In the recent years, the detection of deepfakes has become a substantial topic in image and video forensics. State-of-the-art blind detection methods can detect deepfakes from synthetic datasets with high accuracies. However, they struggle to classify deepfake material that underwent adversarial post-processing or fail to generalize to unseen video data. In this paper, a refined detection pipeline taking advantage of a semi-blind detection scheme is proposed. It combines background-matching with a state-of-the-art CNN-classifier. When classifying videos from the Deepfake Detection Challenge Dataset the CNN-classifier was previously trained on, the performance did not improve using the new detection scheme. However, the approach was able to achieve superior results on unseen data of the FaceForensics++ Dataset.