Object tracking is a hot spot in computer vision and has been developed rapidly in recent years. However, there are still some problems that must be solved in estimation of the state trajectory of the targets (position, orientation, scaling, etc.). In this paper, we focus on the problem of object scale change in tracking and propose scale-adaptive tracking method by combining a kernelized correlation filter with geometric estimation. We combine geometric estimation with low-complexity target scales, so the target scale can be definite during the tracking. Both normal and fault-tolerant target scales are established. Normal target scales are established to seek the optimal solution of the target scale and fault-tolerant target scales are used to correct the results of geometric estimation. Geometric estimation can estimate the target initial size roughly in the early stages of tracking and reduce the scope of scales during the tracking process. Experimental results show that the proposed algorithm is sensitive to the change of target scale and the tracking result is remarkably accurate when target scale changes in practice.