4.1. Experimental Setup
4.1.1. Dataset
Currently, most studies have chosen FakeAVCeleb [
39] as the training and testing dataset, and this study also evaluates the performance of the proposed audio–video detector on the FakeAVCeleb dataset.
FakeAVCeleb, a multimodal deepfake dataset containing 500 real videos extracted from the VoxCeleb2 corpus, is used as a base set to generate about 20,000 deepfake videos by various deepfake generation methods. Deepfake video frames are generated using Faceswap and FSGAN, and deepfake audio is synthesized using Real-Time Voice Cloning (RTVC). Then, Wav2Lip is applied to synchronize the video frames with the audio. Therefore, the FakeAVCeleb dataset contains three different types of audio–visual forgery cases, namely FVFA (fake video fake audio), FVRA (fake video real audio) and RVFA (real video fake audio).
In the video preprocessing stage, the duration of the video clip segments is selected to be 1 s, and the segments with less than 1 s at the end of the video are directly discarded, and the final number of video clips obtained is shown in
Table 1. For each category, we divide the training set and validation set according to the ratio of 7:3.
4.1.2. Evaluation Metrics
Accuracy (ACC), Receiver Operating Characteristic Curve (ROC), and Area Under Curve (AUC) are the evaluation metrics used.
ACC is the ratio of the number of samples correctly classified by the classifier to the total number of samples and is calculated as:
where TPrepresents True Positive, TN represents True Negative, FP represents False Positive, and FN represents False Negative. In the dichotomous classification task, the prediction has only two values of 0 and 1 for the classification result, representing both True and False categories. Therefore, TP represents both the label and prediction being true; TN represents both the label and prediction being false; FP represents the label being false and prediction being true; and FN represents the label being true and prediction being false.
AUC is the area under the ROC curve (Receiver Operating Characteristic Curve), which is also often used to evaluate the performance of classification models. The ROC curve is plotted with the True Positive Rate (TPR) as the vertical axis and the False Positive Rate (FPR) as the horizontal axis. FPR, as the horizontal axis, plots the performance of the classifier under different thresholds, where , .
The value of AUC ranges from 0 to 1. The closer the ROC curve gets to the upper left corner, the closer the AUC value is to 1, indicating better classifier performance.
These metrics are calculated using the real/fake labels of the video aspects, obtained by averaging the scores of all segments from the same video. Results from other methods are directly referenced for comparison.
4.1.3. Implementation Details
All experiments were conducted on an Ubuntu 20.04 system using Pytorch and an NVIDIA RTX 3090 with 24GB of VRAM. Each video clip segment is of size . The output dimensions for the video stream are , and for the audio stream, . Each input token for the video prediction model has a length of 2048, with a multi-head self-attention consisting of 8 heads, each head having an embedding dimension of 256.
The output dimension of the visual projection layer is . The proposed model is optimized using the Adam optimizer, with and set to 0.9 and 0.999 respectively, an initial learning rate of 0.001, and a weight decay of 0.0001. The batch size is 2, the number of epochs is 150, and early stopping is used to store the best model weights.
4.2. Comparative Experiment
To evaluate the performance of our method, we compare it with other unimodal and multimodal deepfake detection methods, and the experimental results are shown in
Table 2.
First, compared with the unimodal visual deepfake detection method, the method proposed in this paper significantly outperforms Xception in terms of performance, with both ACC and AUC values improved by more than 15%. Xception, as a classical deepfake detection method, appears to be insufficient in the face of the new type of audio–video forged videos, despite its excellent performance in earlier studies. MAT employs a multiattention network for fine-grained forgery classification, but its performance is still unsatisfactory, which indicates the limitations of unimodal detection methods in dealing with more complex forgery videos. LipForensics detects forgery traces by analyzing high-level semantic irregularities in lip movements, but since it does not incorporate audio features, it cannot distinguish the abnormalities of lip movements in the face of forgery videos in the presence of lip synchronization, and its detection ability is limited.
Second, the methods proposed in this paper also perform well and achieve state-of-the-art performance when compared to multimodal deepfake detection methods. The emotions method identifies authenticity by analyzing the differences in perceived emotions expressed between audio and visual modalities, while AVFakeNet and AvoiD-DF acquire multimodal high-level features by designing different spatio-temporal coding networks and intermodal attentional mechanisms to fuse these features and thus detect discordance in audio–visual modalities. However, these methods mainly focus on intermodal incongruence, while the methods in this chapter not only consider intermodal incongruence but also introduce the incongruence of intra-modal temporal features, which integrates the two kinds of falsification anomalies to achieve the best detection results. By combining audio–visual discordance and timing feature inconsistency, the method proposed in this paper demonstrates higher accuracy in deepfake detection and makes significant progress.
4.3. Ablation Experiment
We conducted ablation experiments on audio–visual modalities, considering three scenarios: only visual, only audio, and bimodal. In these, the models with only a single modality do not consider the contrastive loss during training (i.e.,
). The results are shown in
Table 3, and the corresponding ROC curves are presented in
Figure 6.
Based on the data in
Table 3, it can be seen that the detection model containing only video streams (Only-Visual) has an ACC and AUC of 79.87% and 80.54%, respectively, whereas the detection model containing only audio streams (Only-Audio) achieves an ACC and AUC of 71.35% and 72.26%, respectively. This indicates that the detection performance of the video modality is better than the audio modality in a single modality. This is mainly due to the more complex structure of the 3D convolutional neural network used for visual modality and the use of Transformer structure as a predictor to obtain the inconsistency of the temporal features, whereas the extraction of the audio modality is simpler in comparison. When combining audio and video bimodality for detection (Visual-Audio), the ACC and AUC of the model are 84.33% and 89.91%, respectively, which are significantly better than the detection results of the single modality. This indicates that the combination of audio and video bimodality can better improve the detection performance of depth forgery, while single modality detection is becoming more and more difficult for higher visual quality forgery videos, so capturing and learning the inconsistency between audio and video modalities is an effective way to improve the detection accuracy.
To further investigate the effect of our proposed three loss functions, six different combinations were designed for ablation experiments. The results are shown in
Table 4 and
Figure 7. According to the results of ablation experiments, the impact of using different loss functions on the overall detection performance of the model is analyzed as follows:
(1) When the cross-entropy loss function is used singly, the detection performance of the model is relatively poor, and both ACC and AUC indexes are at a low level. This is because the cross-entropy loss function mainly supervises the classification task and may not be sensitive and fine enough for complex tasks such as deepfake video detection.
(2) When the contrast loss function is used alone, the model performance is improved; in particular, the AUC metrics are improved by nearly two percentage points compared to using only the cross-entropy loss. The contrast loss function can help the model better capture and learn the inconsistency between different modalities, thus improving the accuracy of detection.
(3) When the predictive loss function is used singly, the model performance gains further improvement, with the ACC and AUC metrics reaching 80.39% and 85.47%, respectively, which are significantly better than using the other two loss functions singly. The predictive loss function is able to better capture timing information by predicting the intrinsic relationship between current and future video clips, which is important for detecting video forgery.
(4) When combining the cross-entropy loss with either the comparison loss or the prediction loss, the model performance is improved, and the effect of the prediction loss is more significant. This indicates that the classification supervision effect of cross-entropy loss complements well with the ability of contrast loss to capture similar variability and prediction loss to capture temporal information.
(5) When using the three loss functions simultaneously, the model performance is the best, with an ACC of 84.33% and an AUC as high as 89.91%, which fully verifies that the effective fusion of the three loss functions can maximize their respective advantages and achieve the best results in detecting deeply forged videos.
Overall, the prediction loss function is the most critical for capturing video timing information, the cross-entropy loss provides supervision for the basic classification task, and the contrast loss function learns the similar differences within the video frames also has a certain role, and the use of the three to complement each other and fusion can greatly improve the performance of the detection of deepfake video.