In order to solve tracking failures caused by objects deformation, occlusion and fast motion, a novel algorithm called MS-TLD which under the Tracking-Learning-Detection framework is proposed. The algorithm reconstructs a new tracker with the scale-adaptive mean-shift method. By introducing color histogram features and scale-adaptive, the new tracker can track objects with deformation and fast moving. We establish a new tracking-detection feedback strategy—the inter-correction between tracker and detector. Therefore, the new algorithm has better robustness when objects are occluded. We use TB-50 standard dataset to verify and evaluate our method. The experimental results show that the proposed algorithm can overcome the tracking failures caused by objects with deformation, occlusion, fast motion, as well as background clutters, and has better tracking accuracy and robustness compared with TLD and other 3 classic algorithms.