Forest fires wreak havoc on natural ecosystems and represent a grave threat to environmental stability. Establishing a rapid and efficient network for the early detection of forest fires remains a critical challenge and a focal point of research. In response to this problem, this paper proposes Fire & Smoke - You Only Look Once (FS-YOLO) for real-time forest fire detection. FS-YOLO significantly enhances fire detection performance through the integration of three innovative modules: Mixed Attention Cross Stage Partial (MACSP), Cross Stage Feature Pyramid Network (CSFPN), and Scalable Spatial Pyramid Pooling (SSPP). First, the MACSP module targets diverse colors and shapes characteristic of forest fires. By combining channel attention with local spatial attention, it precisely weights the network’s features, achieving greater accuracy in capturing fire characteristics. Second, the CSFPN method merges high-level semantic information with low-level detail via both top-down and bottom-up pathways, creating multi-scale feature maps that boast expanded receptive fields. Lastly, the SSPP method enhances the network’s focus on fire targets across varied scenes through scaling factors, bolstering the model’s robustness. Additionally, this paper organizes and annotates a forest fire dataset. The experimental results show that compared to the baseline model, FS-YOLO achieves an 8% improvement in mean average precision, and the average precision values for flames and smoke increase by 10.1% and 5.7%, respectively, indicating a significant overall performance improvement of the model. Compared to other object detection algorithms, FS-YOLO consistently achieves optimal performance.