This study is aimed at the problems of inadequate accuracy and robustness of defect detection or subtle and insignificant defects located in 110 kV cross-linked polyethylene (XLPE) cable joints. A two-stage CNN-based workmanship defect detection network (WDDCJNet) for XLPE cable joint is proposed. First, an improved attention mechanism called Parallel Convolutional Block Attention Module (PA_CBAM) incorporates parallel connections between the spatial attention and channel attention to enlarge the receptive field of the spatial attention mechanism. PA_CBAM further combined with ResNet50 and Feature Pyramid (FPN) to create an enhanced feature extraction network named PM_Resnet50_FPN, which is specifically designed to emphasize defect features and enhance the capability to extract subtle defects. Then, ROI Align is adopted to solve the problem of the offset of the mapping location area caused by the rounding operation to make anchor frame size more consistent with the defect scale. The K-means algorithm is also used to cluster the cable joint surface defect anchor frame. The proposed method is compared with the other state-of-art CNN algorithms on self-collected cable joints dataset. The experimental results prove that WDDCJNet can greatly improve the detection accuracy of five types of defects and meets the detection requirements of practical applications for cable joint workmanship defects.