In the smart manufacturing process, it is important to closely monitor manufactured parts. To solve the problem of part anomaly detection, this paper proposes a GAM–Boost anomaly detection model using a large-scale dataset (14.3 GB) from the Kaggle competition “Bosch Production Line Performance.” The model first selects the important features using the XGBoost algorithm and then captures the nonlinear relationships between the features using the generalized additive model. To capture the nonlinear relationships between features and at the same time improve the model’s ability to understand the data relationships, feature engineering techniques are applied to transform the nonlinear relationships without ignoring the linear relationship features. Finally the XGBoost model is optimized for anomaly detection using the Bayesian algorithm. The experimental results show that the model achieves lower errors on both training and test sets, the generalization performance of the model is significantly improved, it can better adapt to various data situations, and it achieves better results in terms of flexibility and prediction accuracy.