Defect detection is an imperative step in ensuring the quality of steel products. To overcome the problem of low accuracy and poor detection of subtle features in current detection methods, a revised neural network model structure has been proposed. VFSN’s main convolutional layer structure is VGG with residuals, where two convolutional blocks are designed. It’s connected to a spatial pyramid pool (SPP) after reducing the aliasing effects and detecting defect features at various scales by constructing multibranch inputs and multilevel feature overlays and fusions. Research on the open-source data set NEU-DET has shown high-accuracy recognition, with the recognition accuracy rate reaching 75.1% and an average accuracy rate improving by 2.7%. The F1 score improved by 1.3%. There’s a significant improvement in detecting small defects using the proposed network structure.