This paper presents an unsupervised tracking algorithm with a low computational cost using the Temporal Doubly Stochastic Dirichlet Process (TDSDP) mixture model, and we demonstrate it in tracking fish in low quality videos for water quality assurance. The object is captured in the temporal domain with a global dependency prior instead of the Markov assumption, making it particularly suitable for long-term tracking. Furthermore, the TDSDP mixture model can calculate the number of object trajectories automatically. We describe how to construct this mixture model from thinning multiple Dirichlet Process Mixtures (DPMs) with conjugate priors, followed by details of the algorithm for object tracking. Experiments on a fish dataset illustrate that the TDSDP can track multiple fish, and performs well even when they are overlapping in the view. Further experiments also suggest that TDSDP can be applied to other tracking problems.