With the rapid development of artificial intelligence, facial manipulation technology has also evolved remarkably. This has led to the problem of malicious misuse of technology in a society and also developed the research topic of facial forgery detection. In practice, substantial videos are processed by unknown compression and low-quality methods, which makes the detection of facial forgery extremely challenging. We find that deep neural networks trained for specific forged features have better robustness in the presence of unknown compression and low quality. For this reason, we propose an approach to improve facial forgery detection by implementing a divide-and-conquer idea based on the mixture of experts (MoE) framework. (1) Based on the characteristics of the forgery method itself, the facial forgery detection problem is divided into face-swapping detection and expression-swapping detection. (2) Based on the special similarities and differences between expert models, we design an effective lightweight combination method. (3) We propose the focal loss with adaptive decay (FAD) loss function, which effectively alleviates the problem of over-focusing on noisy samples of the focal loss function and further improves the similarity and difference between expert models. Numerous experiments have demonstrated the effectiveness and superiority of our framework on FaceForensics++ (FF++) datasets and benchmarks.