In this article, a corresponding metadata-based stereo object tracking system using disparity-motion estimation (DME) is proposed. In the proposed method, object tracking is performed based on robust characteristics of an object, such as its motion, shape, and color. Elements of metadata and the optimal tracking method are selected at each frame. Because the characteristics of an object are affected by the camera position and by environmental conditions, a tracking method between individual cameras is selected by means of the optimal metadata. Additionally, the time-sequential disparity motion vector (DMV) can be estimated from the feature-based DME, which is extracted from the sequence of the stereo images, and metadata sequences. Using the DMV, the area where the target object is located and its location coordinate are then detected from the stereo images. In an experiment, the proposed system is shown to be robust for nonrigid objects and demonstrates its ability to track a target object adaptively with an average low error ratio of approximately 2.37% between the detected and actual location coordinates of the target object.