Temporal Feature Prediction in Audio–Visual Deepfake Detection

Abstract

The rapid growth of deepfake technology, generating realistic manipulated media, poses a significant threat due to potential misuse. Therefore, effective detection methods are urgently needed to prevent malicious use, as current approaches often focus on single modalities or the simple fusion of audio–visual signals, limiting their accuracy. To solve this problem, we propose a deepfake detection scheme based on bimodal temporal feature prediction, which innovatively introduces the idea of temporal feature prediction into the audio–video bimodal deepfake detection task, aiming at fully exploiting the temporal laws of audio–visual modalities. First, pairs of adjacent audio–video sequence clips are used to construct input quadruples, and a dual-stream network is employed to extract temporal feature representations from video and audio, respectively. A video prediction module and an audio prediction module are designed to capture the temporal inconsistencies within each single modality by predicting future temporal features and comparing them with reference features. Then, a projection layer network is designed to align the audio–visual features, using contrastive loss functions to perform contrastive learning and maximize the differences between real and fake video modalities. Experiments on the FakeAVCeleb dataset demonstrate superior performance with an accuracy of 84.33% and an AUC of 89.91%, outperforming existing methods and confirming the effectiveness of our approach in deepfake detection.

1. Introduction

Currently, deepfake detection techniques mainly rely on anomalous features of visual modalities in videos. However, as deepfake generative systems continue to advance, visual fraud [1,2,3] and audio tampering [4,5] often coexist in forged videos. These manipulated videos often undergo compression and other disturbances that obscure forensic traces, rendering unimodal detection methods (e.g., hybrid boundaries [6,7], frequency anomalies [8,9,10,11], intra-modal inconsistencies [12,13], and facial or lip motion artifacts [14,15]) less effective. Thus, it is essential to develop robust multimodal detection approaches that leverage both audio and visual information. With the advantage of multimodal learning, the detector can utilize the inconsistency artifacts between different modalities to enhance its detection capability and obtain accurate and reliable results, which provides an effective path to solve the above problems.

Recent research has explored multimodal deepfake detection by integrating audio and visual signals. Muppalla et al. [16] enhanced detection granularity by categorizing samples using combined unimodal labels. However, the full potential of audio–visual correlations remains underexploited. While some works [17,18,19,20,21,22] have innovated in fusing these features, they fall short in addressing the fundamental differences between audio and video signals, leading to gaps in the modal distribution. Furthermore, current methods focusing on specific inconsistencies, like emotional cues [23], risk becoming ineffective if forgers adapt to conceal these anomalies.

A major limitation of current audio–video bimodal forgery detection methods is that the essence of training a network strategy, whether using comparative learning or self-supervision, is to learn the inconsistency between the two modalities. This inherent frame of mind limits the breadth and depth of application of these methods in deepfake detection. Previous research has made significant progress in exploiting intra-frame artifacts and temporal feature inconsistencies, etc., but existing multimodal learning methods tend to ignore these anomalous cues, especially those present in the time series, which leads to subtle traces of forgery being able to evade the detector. Therefore, introducing artifact features, which have been proven to be effective in unimodal detection, to the task of bimodal detection can fill the gap in the current research and better utilize the intra-modal artifacts and inter-modal inconsistency features of videos.

To address the issues raised earlier, this paper presents a deep forgery detection method based on bimodal temporal feature prediction, which makes significant contributions in several key areas:

• A characterization prediction module based on temporal feature prediction is proposed. It segments the input video into several video segments and infers more long-term global feature relationships from the sequence segments. These long-term global features across multiple time steps help to mine visual and auditory regularities and predict future sequence representations.
• A video prediction module and an audio prediction module are designed. By predicting future timing features and comparing them with reference features, the capture of timing inconsistencies within a single modality is realized.
• This paper innovatively proposes the contrast loss function as an important part of the overall objective loss function. The contrast loss function is trained to maximize the distinction between real and fake videos, making real and fake videos distinguishable.
• A projection layer network is designed for the audio and video features, respectively, which is used to merge the feature representations of neighboring audio and video clips and align the bimodal features, thus solving the problem of dimensional inconsistency between the video stream features and the audio stream features.

2. Related Work

Recent advances in deepfake detection have utilized contrast and self-supervised learning methods to identify differences between audio and visual channels. Chugh et al. [24] argued that audio–video dissonance can be a strong basis for detecting deepfake video, as tampered faces have discontinuities in the mouth region that result in a mismatch with the voice. Therefore, the authors proposed to use a dual-stream network to process video features and speech features separately, and to use the modal dissonance score (MDS) to construct a contrast loss function to compute the degree of incongruence between audio–video features. However, as with the frame-based approach, simply training the deep network on the video in a binary classification manner may result in the model overfitting the seen artifacts. To address this problem, Haliassos et al. [25] used a self-supervised approach that exploits known correspondences between visual and auditory modalities in natural videos to capture all the shared information between the two modalities through a cross-modal teacher–student framework. However, this approach requires a two-phase training process and uses audio–video bimodality for cross-modal supervision only in the first phase. In the testing phase, it abandons the audio modality and considers only visual features, which undoubtedly sacrifices the model’s performance and limits its scalability. Yu et al. [26] similarly extracted natural audio–visual correspondences from real videos in a self-supervised manner, and then used the learned real correspondences as targets to guide the extraction of audio–visual inconsistencies in the detection phase. The first stage utilizes two augmented visual views paired with corresponding audio cues to better explore common audio–visual correspondences. The second phase then aligns the audio–visual features of the real video using the network model frozen in the first phase to enable the detection network to better learn the audio–visual inconsistencies of deepfake videos.

Cheng et al. [27] found that voice and facial identity representations in deepfake videos are often mismatched but homogeneous in real videos. Based on this, an effective multimodal matching framework was designed to pretrain models on a generic audio–visual dataset and fine-tune them on downstream deepfake data to detect the plausibility of real and fake videos. Cozzolino et al. [28] created a deepfake detector for persons of interest (POIs) by extracting the audio–visual features that characterize a person’s identity and using these features. They applied comparative learning to extract feature representations of the most unique dynamic face and audio clips for each identity. When a person’s video and audio are manipulated, their representation in the embedding space will become inconsistent with the true identity, allowing the reliable detection of deepfakes. The detector is trained only on real face videos and does not depend on any specific manipulation method, thus providing the highest generalization ability.

In the field of deepfake detection in audio and video, several related works provide valuable references. Umirzakova et al. [29] proposed an improved facial image parsing technique that significantly improves the segmentation accuracy of facial features by combining spatial and channel attention mechanisms. Dam et al. [30] introduced the AYDIV model, which utilizes a contextual vision converter for 3D object detection. This approach may be useful for analyzing spatial and temporal relationships in audio and video. Lee et al. [31] introduced the Watt-EffNet model, which balances accuracy and computational requirements, and is particularly valuable for real-time fake detection, providing insights for resource-efficient model development.

3. Methods

Figure 1 illustrates the detection framework based on bimodal temporal feature prediction, consisting of five parts: preprocessing, audio–visual feature extraction, representation prediction, contrastive learning, and classification. First, the input raw video is subjected to preprocessing operations, including speech extraction and sequence segmentation, to generate the input video stream and audio stream. Second, an audio–visual feature extraction module performs feature learning on the signals of the two modalities to extract the dense representation of the video channel and the coherent features of the audio channel. Then, the representation prediction module performs a prediction task at adjacent video clips based on the learned bimodal features to achieve the detection of timing inconsistencies within a single modality. Meanwhile, after passing through the projection layer, between the two modalities, contrast learning is used to maximize the falsification of the differences between video modalities. Finally, in the classification module, the prediction results are output to identify the video authenticity. The following sections provide a comprehensive account of each of these processes.

3.1. Preprocessing

Assuming a training dataset composed of N videos $D=\{(x^1,y^1),(x^2,y^2),\dots ,(x^N,y^N)\}$, where $x^i$ represents the input video and $y^i$ represents the video label. For any given video $x^i$, it is first segmented into M one-second clip segments $\{x_1^i,x_2^i,\dots ,x_M^i\}$. Subsequently, the S3FD [32] model is used for face detection on the video segment $x_j^i$, and the detected facial regions are cropped to form the video stream input $v_j^i$. Additionally, we use ffmpeg [33] to extract the audio information from each clip segment $a_j^i$. Therefore, for a video $x^i$, the inputs for the audio stream and video stream are $v^i=\{v_1^i,v_2^i,\dots ,v_M^i\}$ and $a^i=\{a_1^i,a_2^i,\dots ,a_M^i\}$, respectively. As shown in Figure 1, we process pairs of adjacent video clip segments at a time, i.e., $\{v_j^i,a_j^i,v_{j+1}^i,a_{j+1}^i\}$, where $v_j^i$ and $a_j^i$ are used to extract the source audio–visual features, and $v_{j+1}^i$ and $a_{j+1}^i$ are used to extract the target audio–visual features.

3.2. Audio–Visual Feature Extraction Process

Audio–visual feature extraction is divided into two parts: video stream processing and audio stream processing. Video stream processing helps the model learn rich spatio-temporal information so as to capture key features of video clips. Audio stream processing, on the other hand, helps the model to more accurately analyze and understand the audio content in order to better identify forged cues in the audio modality.

3.2.1. Video Stream Processing

Channel-Separated Convolutional Networks (CSNs) [34] serve as the feature extraction network $S_v$ for the video stream. Although CSN builds on 3D-ResNet, CSN makes some key improvements over the original 3D-ResNet to better model the spatio-temporal information of the video data. First, CSN uses a channel-separated organizational structure. Within each residual block, the first $1\times 1\times 1$ convolution acts only on the channel dimension, while the second $3\times 3\times 3$ convolution captures both spatial and temporal information. This design reduces the number of parameters and allows for more focused modeling of spatial and time-domain patterns. Second, CSN introduces a tunable time-domain sampling rate. By configuring the time step of different STAGES, CSN can balance the fusion of the space and time domains to achieve the effect of progressively aggregating the timing information. Finally, CSN applies lightweight time domain modeling. Compared to directly stacking a large number of 3D convolutions across the entire network, CSN can learn the timing representation with only a small number of 3D convolution operations, reducing the computational complexity.

We made two improvements to the original CSN network structure. First, the temporal stride is set to 1 at all stages to prevent temporal resampling and maintain complete temporal patterns. Second, at the end of the network, global average pooling is applied only to the spatial dimension to produce a dense feature representation of the video clips rather than a global feature representation of the input video clips, which facilitates the subsequent prediction module.

$S_v$ maps the input source video clip $v_j^i\in R^{T\times C\times H\times W}$ and target video clip $v_{j+1}^i\in R^{T\times C\times H\times W}$ into latent sequential representations $E_j^v=\{e_1^v,e_2^v,\dots ,e_T^v,\}\in R^{T\times C^{\prime }}$ and $E_{j+1}^v=\{e_{T+1}^v,e_{T+2}^v,\dots ,e_{2T}^v,\}\in R^{T\times C^{\prime }}$, where T is the time length (number of frames), C is the number of input channels, and H and W are the spatial resolutions. The number of output feature channels $C^{\prime }$ depends on the network parameter settings, which is set to 2048. Additionally, the visual embedding features maintain the same temporal dimension as the input features, containing rich information on the dynamic patterns.

3.2.2. Audio Stream Processing

Mel Frequency Cepstrum Coefficients (MFCCs) [35] serve as the input for the audio stream. For each audio segment $a_i$ lasting 1 s with a sampling rate of 48,000 Hz, its MFCCs are computed and converted into a heatmap format, with dimensions of $13\times 99\times 1$. Here, the height 13 represents that there are 13 Mel frequency bands in the MFCC, the width 99 indicates that the audio segment is divided into 99 frames, and the channel number is 1. The preprocessing flow of the audio modality is shown in Figure 2. Overall, the audio is encoded as a heatmap image representing the MFCC values for each time step and each Meier band, which is subsequently fed into the audio stream.

Obtaining the MFCC features in an audio clip consists of six main steps as shown in Figure 3.

For the MFCC heatmap representation of the audio signal, ResNet-18 [36] is utilized as the feature extraction network $S_a$, with the number of input channels of the first convolutional layer set to 1. The main feature of ResNet is the use of residual modules and residual connections to build the network, which makes it possible to train deeper networks without the problem of gradient vanishing, and to excel in tasks such as image classification. $S_a$ maps the source audio feature $a_j^i$ and target audio feature $a_{j+1}^i$ to consistent audio representations $E_j^a\in R^{C^{″}\times 1}$ and $E_{j+1}^a\in R^{C^{″}\times 1}$, where $C^{″}$ is 512.

3.3. Representation Prediction Process

With the continuous advancement of deepfake technology, the quality of the generated fake videos is becoming higher and higher, making the obvious specific artifactual features present in low visual quality videos utilized in the past proposed methods severely weakened, as well as facing the challenge of low generalization performance for cross-dataset detection. Therefore, a representation prediction module based on temporal feature prediction is proposed for detecting deepfake videos by predicting temporal feature representations of future time blocks in bimodal detection. In order to predict future feature representations from the present, instead of inferring short-term localized timing information from frame to frame, the input video is segmented into a number of S-second-long video segments, and more long-term global feature relationships are inferred from the sequence segments. These long-term global features across multiple time steps help to mine visual and auditory regularities and predict future sequence representations.

For audio–visual bimodal features, two types of prediction models are designed for visual and audio features, respectively. The visual prediction model $P^v$ uses a 1-block Transformer [37] encoder structure, following the design of ViT [38] as shown in Figure 4. The source video feature $E_j^v$ is processed through the predictor to infer the predicted feature $E_{pre}^v$ as follows:

$$E_{pre}^v=P^v\left(E_j^v\right)=\{p_1^v,p_2^v,\dots ,p_1^v\}$$

(1)

The input to the audio prediction model is the output of the audio stream processing module, which represents a globally consistent feature representation of the audio segment, suitable for a feed-forward neural network structure prediction model. Therefore, the structure of the audio prediction model $P^a$ consists of a $1\times 1$ convolutional layer. The source audio feature $E_j^a$ is processed through the predictor to infer the predicted feature $E_{pre}^a$ as follows:

$$E_{pre}^a=P^a\left(E_j^a\right)$$

(2)

Therefore, in each processing cycle, for the input set $\{v_j^i,a_j^i,v_{j+1}^i,a_{j+1}^i\}$, we obtain the embedded feature representation set $\{E_j^v,E_j^a,E_{j+1}^v,E_{j+1}^a\}$, and the predicted audio–visual feature set $\{E_{pre}^v,E_{pre}^a\}$. The representation prediction module optimizes the model using a correlation-based representation prediction loss. The correlation calculation between the predicted dense visual features and the referenced target video dense features is as follows:

$$cor{r}_v={\displaystyle \frac{{\sum }_{t=1}^T<p_t^v,e_{T+t}^v>}{T}}$$

(3)

The calculation of the correlation between the predicted audio embedding and the referenced target audio embedding is as follows:

$$cor{r}_a=<E_{pre}^a,E_{j+1}^a>$$

(4)

where $<e_1,e_2>$ represents the computation of the correlation coefficient between two vectors.

Thus, the representation prediction loss can be described as follows:

$$L_{pre}=-{\displaystyle \frac{1}{N}}\sum _i(y_i\times ln\left(\sigma (cor{r}_v+cor{r}_a)\right)+(1-y_i)\times ln(1-\left(\sigma (cor{r}_v+cor{r}_a)\right))$$

(5)

where $\sigma \left(x\right)=1/(1+exp(-x\left)\right)$, and $y_i$ represents the label of the i-th video clip segment.

Since real video adjacent segments are overly smooth, their correlation is higher. In contrast, deepfake videos, due to incoherent facial movements, impact predictions, resulting in a lower correlation. This allows distinguishing between genuine and fabricated videos.

3.4. Comparative Learning Process

The contrast loss function amplifies the inconsistency between the different modalities in the fake video while reducing the differences between the modalities of the real video, making the real and fake videos distinguishable.

Due to the inconsistency in dimensions between the video stream features and audio stream features extracted by the audio–visual feature extraction module, it is not feasible to directly use the contrastive loss function for computation. Therefore, projection layer networks have been designed separately for video and audio features to merge and align the audio–visual features of adjacent video segments as shown in Figure 5.

The Euclidean distance is employed to quantify the disparity between the features output by the projection layer for the audio–visual features:

$$d=\parallel E_{pro}^v-E_{pro}^a{\parallel }_F^2$$

(6)

Consequently, the contrast loss function can be expressed as follows:

$$L_{con}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{y}^i{d}^2+(1-y^i)max{(margin-d,0)}^2$$

(7)

where d is the Euclidean distance between $E_{pro}^v$ and $E_{pro}^a$, and the margin is a hyperparameter set to 0.5.

By training with the contrastive loss function, inconsistencies between different modalities in fabricated videos can be amplified while reducing the differences between modalities in real videos, making it possible to distinguish between genuine and fake videos.

3.5. Loss Function

The model architecture designed in this paper uses three loss functions to continue the training, including the representation prediction loss for within-modality, the comparison loss function for between-modality, and the cross-entropy loss function. In theory, two originally similar samples remain similar in the low-dimensional space after dimensionality reduction, while two dissimilar samples cannot be similar in the low-dimensional space. In terms of experimental observation, the faked video will show potential inconsistency relative to the real video. Therefore, the characterization prediction module and the comparison learning module and the corresponding two loss functions are designed.

On the respective sub-networks for the visual and audio modalities, a binary classification model based on MLP is implemented, applying the cross-entropy loss function to enhance anomaly detection capabilities under a single modality as follows:

$$L_{con}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N{y}^{i}log\left(\widehat{y^i}\right)+(1-y^i)log(1-\widehat{y^i})$$

(8)

where $y^i$ represents the predicted label for the i-th video clip segment.

Thus, the total loss function is the weighted sum of the prediction loss function $L_{pre}$, the contrastive loss function $L_{con}$, and the cross-entropy loss function $L_{cro}$, which can be expressed as:

$$L_{total}={\lambda }_1{L}_{pre}+{\lambda }_2{L}_{con}+{\lambda }_3{L}_{cro}$$

(9)

where $L_{total}$ is the final cross-modal loss function used for manipulation detection. ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are the weight parameters used to balance the loss function, set to $1/3$.

The specific algorithmic flow is shown in Algorithm 1:

Algorithm 1 Algorithmicflow of detection framework based on temporal feature prediction.

Input : $x^i\in D\{(x^1,y^1),(x^2,y^2),\dots ,(x^N,y^N)\};0<i<N$ learning rate: $\alpha =0.001$ batch size: $B=2$ epoch: $num\text{\_}iter=150$ Output : Training Models: ${\theta }_{net},{\theta }_{pre},{\theta }_{pro},{\theta }_{cro};S_a,S_v,S_{pre}^v,S_{pre}^a,S_{pro},S_{cls}$   1: for  $i=1⟶num\text{\_}iter$  do   2:    for $v_j^i,a_j^i,v_{j+1}^i,a_{j+1}^i$ range in $x^i$ do   3:        $E_j^v,E_{j+1}^v:=S_v(v_j^i,v_{j+1}^i)$   4:        $E_j^a,E_{j+1}^a:=S_a(a_j^i,a_{j+1}^i)$   5:        $E_{pre}^v\left(j\right):=S_{pre}^v\left(E_j^v\right)$   6:        $E_{pre}^a\left(j\right):=S_{pre}^a\left(E_j^a\right)$   7:        $E_{pro}^v\left(j\right):=S_{pro}^v(E_j^v,E_{j+1}^v)$   8:        $E_{pro}^a\left(j\right):=S_{pro}^a(E_j^a,E_{j+1}^a)$   9:        $\widehat{y_j^i}:=S_{cls}(E_{pro}^v\left(j\right),E_{pro}^v\left(j\right))$ 10:   end for 11:   $g_{{\theta }_{pre}}⟵{▽}_{{\theta }_{pre}}\left[{\displaystyle \frac{1}{B}}{\sum }_{b=1}^{b=B}\left({\sum }_{j=1}^{j=len\left(x^i\right)-1}{L}_{pre}(E_{j+1}^v,E_{j+1}^a,E_{pre}^v\left(j\right),E_{pre}^a\left(j\right))\right)\right]$ 12:   $g_{{\theta }_{pro}}⟵{▽}_{{\theta }_{pro}}\left[{\displaystyle \frac{1}{B}}{\sum }_{b=1}^{b=B}\left({\sum }_{j=1}^{j=len\left(x^i\right)-1}{L}_{pro}(E_{pro}^v\left(j\right),E_{pro}^a\left(j\right))\right)\right]$ 13:   $g_{{\theta }_{cro}}⟵{▽}_{{\theta }_{cro}}\left[{\displaystyle \frac{1}{B}}{\sum }_{b=1}^{b=B}\left({\sum }_{j=1}^{j=len\left(x^i\right)-1}{L}_{cro}(j^v,\widehat{y_j^i})\right)\right]$ 14:   ${\theta }_{net}⟵{\theta }_{net}+\alpha ·Adam({\theta }_{net},sum(g_{{\theta }_{pre}},g_{{\theta }_{pro}},g_{{\theta }_{cro}}))$ 15:   if ${\theta }_{net}$ have not converged then 16:           break 17:   end if 18: end for

4. Experiments

4.1. Experimental Setup

4.1.1. Dataset

Currently, most studies have chosen FakeAVCeleb [39] as the training and testing dataset, and this study also evaluates the performance of the proposed audio–video detector on the FakeAVCeleb dataset.

FakeAVCeleb, a multimodal deepfake dataset containing 500 real videos extracted from the VoxCeleb2 corpus, is used as a base set to generate about 20,000 deepfake videos by various deepfake generation methods. Deepfake video frames are generated using Faceswap and FSGAN, and deepfake audio is synthesized using Real-Time Voice Cloning (RTVC). Then, Wav2Lip is applied to synchronize the video frames with the audio. Therefore, the FakeAVCeleb dataset contains three different types of audio–visual forgery cases, namely FVFA (fake video fake audio), FVRA (fake video real audio) and RVFA (real video fake audio).

In the video preprocessing stage, the duration of the video clip segments is selected to be 1 s, and the segments with less than 1 s at the end of the video are directly discarded, and the final number of video clips obtained is shown in

Table 1. Statistics for the FakeAVCeleb dataset.

Categories	Number of Original Videos	Number of Edited Clips
RVRA	500	3k
RVFA	1000	6k
FVRA	9709	58k
FVFA	10,857	65k

. For each category, we divide the training set and validation set according to the ratio of 7:3.

4.1.2. Evaluation Metrics

Accuracy (ACC), Receiver Operating Characteristic Curve (ROC), and Area Under Curve (AUC) are the evaluation metrics used.

ACC is the ratio of the number of samples correctly classified by the classifier to the total number of samples and is calculated as:

$$ACC={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(10)

where TPrepresents True Positive, TN represents True Negative, FP represents False Positive, and FN represents False Negative. In the dichotomous classification task, the prediction has only two values of 0 and 1 for the classification result, representing both True and False categories. Therefore, TP represents both the label and prediction being true; TN represents both the label and prediction being false; FP represents the label being false and prediction being true; and FN represents the label being true and prediction being false.

AUC is the area under the ROC curve (Receiver Operating Characteristic Curve), which is also often used to evaluate the performance of classification models. The ROC curve is plotted with the True Positive Rate (TPR) as the vertical axis and the False Positive Rate (FPR) as the horizontal axis. FPR, as the horizontal axis, plots the performance of the classifier under different thresholds, where $TPR=TP/(TP+FN)$, $FPR=(FP+TN)$.

The value of AUC ranges from 0 to 1. The closer the ROC curve gets to the upper left corner, the closer the AUC value is to 1, indicating better classifier performance.

These metrics are calculated using the real/fake labels of the video aspects, obtained by averaging the scores of all segments from the same video. Results from other methods are directly referenced for comparison.

4.1.3. Implementation Details

All experiments were conducted on an Ubuntu 20.04 system using Pytorch and an NVIDIA RTX 3090 with 24GB of VRAM. Each video clip segment is of size $30\times 3\times 224\times 224$. The output dimensions for the video stream are $30\times 2048$, and for the audio stream, $512\times 1$. Each input token for the video prediction model has a length of 2048, with a multi-head self-attention consisting of 8 heads, each head having an embedding dimension of 256.

The output dimension of the visual projection layer is $512\times 1$. The proposed model is optimized using the Adam optimizer, with $\beta 1$ and $\beta 2$ set to 0.9 and 0.999 respectively, an initial learning rate of 0.001, and a weight decay of 0.0001. The batch size is 2, the number of epochs is 150, and early stopping is used to store the best model weights.

4.2. Comparative Experiment

To evaluate the performance of our method, we compare it with other unimodal and multimodal deepfake detection methods, and the experimental results are shown in

Table 2. Comparative experimental results on the fakeAVCeleb dataset.

Methods	Modality	ACC (%)	AUC (%)
Xception [40]	visual	67.90	70.50
LipForensics [14]	visual	80.10	82.40
MAT [41]	visual	77.60	79.30
MDS [24]	audio–visual	82.80	86.50
Emotions [23]	audio–visual	78.10	79.80
AVFakeNet [22]	audio–visual	78.40	83.40
AvoiD-DF [18]	audio–visual	83.70	89.20
Our	audio–visual	84.33	89.91

The highest result are highlighted in bold.

.

First, compared with the unimodal visual deepfake detection method, the method proposed in this paper significantly outperforms Xception in terms of performance, with both ACC and AUC values improved by more than 15%. Xception, as a classical deepfake detection method, appears to be insufficient in the face of the new type of audio–video forged videos, despite its excellent performance in earlier studies. MAT employs a multiattention network for fine-grained forgery classification, but its performance is still unsatisfactory, which indicates the limitations of unimodal detection methods in dealing with more complex forgery videos. LipForensics detects forgery traces by analyzing high-level semantic irregularities in lip movements, but since it does not incorporate audio features, it cannot distinguish the abnormalities of lip movements in the face of forgery videos in the presence of lip synchronization, and its detection ability is limited.

Second, the methods proposed in this paper also perform well and achieve state-of-the-art performance when compared to multimodal deepfake detection methods. The emotions method identifies authenticity by analyzing the differences in perceived emotions expressed between audio and visual modalities, while AVFakeNet and AvoiD-DF acquire multimodal high-level features by designing different spatio-temporal coding networks and intermodal attentional mechanisms to fuse these features and thus detect discordance in audio–visual modalities. However, these methods mainly focus on intermodal incongruence, while the methods in this chapter not only consider intermodal incongruence but also introduce the incongruence of intra-modal temporal features, which integrates the two kinds of falsification anomalies to achieve the best detection results. By combining audio–visual discordance and timing feature inconsistency, the method proposed in this paper demonstrates higher accuracy in deepfake detection and makes significant progress.

4.3. Ablation Experiment

We conducted ablation experiments on audio–visual modalities, considering three scenarios: only visual, only audio, and bimodal. In these, the models with only a single modality do not consider the contrastive loss during training (i.e., ${\lambda }_1={\lambda }_3=1/2,{\lambda }_2=0$). The results are shown in

Table 3. Ablation experimental results of different audio–visual modalities.

No.	Ablation Model	ACC (%)	AUC (%)
1	only visual	79.87	80.54
2	only audio	71.35	72.26
3	audio–visual	84.33	89.91

The highest result are highlighted in bold.

, and the corresponding ROC curves are presented in Figure 6.

Based on the data in Table 3, it can be seen that the detection model containing only video streams (Only-Visual) has an ACC and AUC of 79.87% and 80.54%, respectively, whereas the detection model containing only audio streams (Only-Audio) achieves an ACC and AUC of 71.35% and 72.26%, respectively. This indicates that the detection performance of the video modality is better than the audio modality in a single modality. This is mainly due to the more complex structure of the 3D convolutional neural network used for visual modality and the use of Transformer structure as a predictor to obtain the inconsistency of the temporal features, whereas the extraction of the audio modality is simpler in comparison. When combining audio and video bimodality for detection (Visual-Audio), the ACC and AUC of the model are 84.33% and 89.91%, respectively, which are significantly better than the detection results of the single modality. This indicates that the combination of audio and video bimodality can better improve the detection performance of depth forgery, while single modality detection is becoming more and more difficult for higher visual quality forgery videos, so capturing and learning the inconsistency between audio and video modalities is an effective way to improve the detection accuracy.

To further investigate the effect of our proposed three loss functions, six different combinations were designed for ablation experiments. The results are shown in

Table 4. Ablation experimental results with different loss functions.

No.	Loss Function	$\mathit{\lambda }$	ACC (%)	AUC (%)
1	only-$L_{cro}$	$[{\lambda }_1,{\lambda }_2,{\lambda }_3]=[0,0,1]$	67.90	70.50
2	only-$L_{con}$	$[{\lambda }_1,{\lambda }_2,{\lambda }_3]=[0,1,0]$	80.10	82.40
3	only-$L_{pre}$	$[{\lambda }_1,{\lambda }_2,{\lambda }_3]=[1,0,0]$	77.60	79.30
4	$L_{cro}+L_{con}$	$[{\lambda }_1,{\lambda }_2,{\lambda }_3]=[0,{\displaystyle \frac{1}{2}},{\displaystyle \frac{1}{2}}]$	82.80	86.50
5	$L_{cro}+L_{pre}$	$[{\lambda }_1,{\lambda }_2,{\lambda }_3]=[{\displaystyle \frac{1}{2}},0,{\displaystyle \frac{1}{2}}]$	78.10	79.80
6	All	$[{\lambda }_1,{\lambda }_2,{\lambda }_3]=[{\displaystyle \frac{1}{3}},{\displaystyle \frac{1}{3}},{\displaystyle \frac{1}{3}}]$	84.33	89.91

The highest result are highlighted in bold.

and Figure 7. According to the results of ablation experiments, the impact of using different loss functions on the overall detection performance of the model is analyzed as follows:

(1) When the cross-entropy loss function is used singly, the detection performance of the model is relatively poor, and both ACC and AUC indexes are at a low level. This is because the cross-entropy loss function mainly supervises the classification task and may not be sensitive and fine enough for complex tasks such as deepfake video detection.

(2) When the contrast loss function is used alone, the model performance is improved; in particular, the AUC metrics are improved by nearly two percentage points compared to using only the cross-entropy loss. The contrast loss function can help the model better capture and learn the inconsistency between different modalities, thus improving the accuracy of detection.

(3) When the predictive loss function is used singly, the model performance gains further improvement, with the ACC and AUC metrics reaching 80.39% and 85.47%, respectively, which are significantly better than using the other two loss functions singly. The predictive loss function is able to better capture timing information by predicting the intrinsic relationship between current and future video clips, which is important for detecting video forgery.

(4) When combining the cross-entropy loss with either the comparison loss or the prediction loss, the model performance is improved, and the effect of the prediction loss is more significant. This indicates that the classification supervision effect of cross-entropy loss complements well with the ability of contrast loss to capture similar variability and prediction loss to capture temporal information.

(5) When using the three loss functions simultaneously, the model performance is the best, with an ACC of 84.33% and an AUC as high as 89.91%, which fully verifies that the effective fusion of the three loss functions can maximize their respective advantages and achieve the best results in detecting deeply forged videos.

Overall, the prediction loss function is the most critical for capturing video timing information, the cross-entropy loss provides supervision for the basic classification task, and the contrast loss function learns the similar differences within the video frames also has a certain role, and the use of the three to complement each other and fusion can greatly improve the performance of the detection of deepfake video.

5. Conclusions

In conclusion, this paper proposes a deep forgery detection scheme based on bimodal timing feature prediction, which combines two modalities, audio and visual, to achieve the detection of high-quality deep forgery videos by exposing the inter-modal variability and the inconsistency of the timing features within the modalities in the forgery videos. Compared with existing schemes, this scheme solves the challenge that the detection of a single modality is insufficient to cope with a wide variety of forged videos in the network environment. However, this scheme also has shortcomings. The design for the audio stream is relatively simple, and only the MFCC features, which are common in the field of speech detection, are considered as the input to the audio stream, making the subsequent prediction based on the audio information potentially unable to achieve the expected results. Another limitation of the proposed scheme is the failure to achieve real-time detection. Subsequent research, therefore, can consider different audio input forms and try to retain their temporal integrity when designing the network, which can then be used for prediction tasks through the Transformer structure for more effective audio modal detection. In addition, future deepfake detection research can improve efficiency by simplifying models and optimizing algorithms to enhance the timeliness of detection.

Despite some limitations, our approach still has great potential for application in domains that require robust verification of the authenticity of digital media, such as social platforms, news dissemination, and content review and authentication in the judicial domain. Subsequent research can consider different audio input forms and try to preserve their temporal integrity when designing the network, which can then be used for prediction tasks through the Transformer structure to achieve more effective audio modality detection.