With the advent of the era of the media, the number of sports videos on the Internet is on the order of the magnitude of geometric growth. However, in the face of mass and form, diversity of sports videos, how users find interesting events and videos influences this growth. Classifying sports videos, games organization and management is required to effectively improve the video retrieval speed, as artificial classification and semantic annotations are highly susceptible to subjective consciousness and the influence of the cultural level of participants, resulting in low classification and labeling efficiency. Therefore, a digital video automatic classification technology has become a research hotspot in the field. Especially for sports videos, accurate automatic classification and semantic tagging can make the coach quickly find relevant video data, targeted guidance and training for athletes and users quickly find and promote interesting sports video program or fragment. Therefore, automatic classification of sports video technology has become an important branch in the field of digital video research. Extensive literature review is presented in this paper with an in-depth discussion and summary of sports videos based on automatic classification. This paper focuses on video classification based on deep learning and puts forward a multi-level multiple granularity based on cascade SRU airspace feature extraction method. First, convolution neural network is used to extract videos of high, medium and low levels of frame characteristics. Second, each layer of the frame uses characteristics such as build time domain pyramid, cascade SRL learning video time dependence and the characteristic of hierarchical structure in time domain. Finally, the three levels of pyramid time domain features are aggregated into multi-level multi-granularity global characteristics of the video. Experiments show that the feature extraction has good representation ability and robustness.