A stereo matching algorithm is used to find the best match between a pair of images. To compute the cost of the matching points from the sequence of images, the disparity maps from video streams are estimated. However, the estimated disparity sequences may cause undesirable flickering errors. These errors result in low visibility of the synthesized video and reduce video coding. In order to solve this problem, in this article, the authors propose a spatiotemporal disparity refinement on local stereo matching based on the segmentation strategy. Based on segmentation information, matching point searching, and color similarity, adaptive disparity values to recover the disparity errors in disparity sequences can be obtained. The flickering errors are also effectively removed, and the boundaries of objects are well preserved. The procedures of the proposed approach consist of a segmentation process, matching point searching, and refinement in the temporal and spatial domains. Experimental results verify that the proposed approach can yield a high quantitative evaluation and a high-quality disparity map compared with other methods.