In order to promote the development of sports and improve the sports literacy of athletes, this paper discusses the video analysis of sports athletes and extends the application of deep learning (DL) algorithm in feature extraction, detection and estimation to improve the analysis effect of sports videos and promote the development of sports. This paper takes basketball players as the main research object to study the sports video analysis. Based on this, first, the Convolutional Neural Network (CNN) and Fully Convolutional Network (FCN) are discussed. Then a DL video detection and segmentation method based on fusion clustering analysis is proposed by combining the advantages of both with clustering analysis. The designed model is F-C-VDS. Second, CNN is applied to the estimation of basketball players’ and individuals’ altitudes by introducing the idea of feature fusion. Finally, the F-C-VDS model is compared with several neural network models. Its effectiveness is tested based on three postures: moving, jumping and standing. The results show that the segmentation accuracy of F-C-VDS is 97.91% and the segmentation time is 1.01 s. Compared with traditional FCN, the segmentation efficiency is improved by 6.49 %, and the whole segmentation process is stable and more significant than CNN. The average pose estimation accuracy of CNN under feature fusion can reach 80.48%, of which 83.32 % corresponds to moving pose, 82.82 % corresponds to jumping pose and 75.31 % corresponds to standing pose. The proposed DL algorithm can effectively detect, segment and estimate athletes’ altitudes from video images and can be applied to various sports videos. This paper not only provides a reference for the development of DL technology, but also contributes to the development of sports.