In this paper, we propose a new approach to tracking the fingertips of guitarists by embedding a CNN-based segmentation module and a temporal grouping-based ROI-association module combined with a particle filter. First, a CNN architecture is trained to segment hand area of each frame of input video. Then, four fingertip candidates (fore, middle, ring and little fingertips) on each frame are located by counting the vote number of template matching (TM) and reversed Hough transform (RHT). Furthermore, temporal grouping-based ROI association is applied to removal noise and group the fingertip candidates on consecutive frames. Finally, particles are distributed between associated fingertip candidates on every two adjacent frames for tracking the fingertips of guitarists. Experiments using videos containing multiple persons’ guitar plays under different conditions demonstrate that the proposed method outperforms the current state-of-the-art tracking algorithm in terms of the hand area segmentation accuracy (98%) and the fingertip tracking mean error (5.16 pixel: 0.22 cm on the guitar neck) as well as computation efficiency.