Detection of dim small ground targets in SAR remote sensing images suffers from deficient target information and significant irrelevant background noise. In order to solve this problem, we propose the CD-YOLO (Cross-DRSN YOLO) based on multi-level feature fusion. In CD-YOLO, the convolutional neural network is firstly used to extract features from the input SAR image step by step, thus the spatial pyramid of shallow and deep feature maps is obtained. Cross-level feature fusion is then performed on the spatial pyramid of feature maps, combined with the constructed soft thresholding module which adopts DRSN (Deep Residual Shrinkage Network), to enhance the spatial features of dim small targets and eliminate the noise-related features. Finally, end-to-end target detection is carried out on the two large parallel feature maps generated after soft thresholding. Detection results are output combined with multi-channel information. Due to lack of sufficient image data, a SAR dim small ground target dataset named SGDSTD (SAR Small Ground-based Dim Small Target Dataset) is constructed. Experimental results show that CD-YOLO achieves a real-time performance of 32.3 frames per second, and its AP_0.5 and AP_{0.5: 0.95} on the SGDSTD dataset achieve 91.6% and 51.9%, respectively. Precision and recall of CD-YOLO on the SGDSTD dataset are 86.9% and 81.4%, respectively. The experiment results prove the effectiveness of the proposed method.