This paper presents a new method for tracking moving objects with radical color changes using modified mean shift. The mean shift algorithm seeks the highest density value using a mean shift vector obtained from the density gradient. Density presents various forms of object information such as color and intensity depending on the application. Conventional color-based mean shift methods show good results when tracking nonrigid moving objects. However, they do not provide accurate results when the initial color distribution of the object disappears. In our method, color distribution is used to represent the objects. The mean shift algorithm is first used to derive an object candidate by estimating the maximum increase in density direction from its current position. Next, the color variation of the object is calculated and compared with a specific threshold value. When the color variation of the object exceeds this threshold value, the initial color of the search window is updated. The objective of our method is to provide for robust real-time object tracking with large color variation in the object whose color changes during motion. The implementation of the new algorithm shows effective tracking results with complete object color changing from time to time. Validation of our approach is illustrated by comparison of experimental results obtained using the methods described above.