Neural Network Ensemble Method for Deepfake Classification Using Golden Frame Selection

Abstract

Deepfake technology poses significant threats in various domains, including politics, cybersecurity, and social media. This study uses the golden frame selection technique to present a neural network ensemble method for deepfake classification. The proposed approach optimizes computational resources by extracting the most informative video frames, improving detection accuracy. We integrate multiple deep learning models, including ResNet50, EfficientNetB0, Xception, InceptionV3, and Facenet, with an XGBoost meta-model for enhanced classification performance. Experimental results demonstrate a 91% accuracy rate, outperforming traditional deepfake detection models. Additionally, feature importance analysis using Grad-CAM highlights how different architectures focus on distinct facial regions, enhancing overall model interpretability. The findings contribute to of robust and efficient deepfake detection techniques, with potential applications in digital forensics, media verification, and cybersecurity.

1. Introduction

The relevance of deepfake detection research in the modern world, especially in the context of hybrid warfare, is exceptionally high. Using deepfakes to manipulate public opinion, political decisions, and international relations has already become a reality, highlighting potential risks to state stability and personal security. Deepfake technologies can be employed to create falsified evidence that distorts real events or the behavior of influential figures, potentially leading to international conflicts or internal misunderstandings within countries [1,2].

At the tactical level, deepfakes can sow confusion among military personnel, undermining morale or even causing unwarranted retreats due to visually verified but falsified enemy incursions. It highlights the serious threats that deepfakes pose to military strategies and the actual security of nations [3]. For instance, Russia employs hybrid warfare tactics, including cyberattacks and disinformation, to destabilize society in Ukraine, demonstrating how technology can be leveraged as a tool of militarized aggression [4,5].

In global politics, deepfakes are used to discredit political leaders, manipulate public opinion, and even interfere in electoral processes, as seen in various cases during elections in the United States and other countries. Raising awareness and advancing detection technologies are crucial for protecting democratic processes and national security from such threats [1,6].

The development and implementation of regulatory measures, such as the European Digital Act, which imposes fines on social media platforms for the inadequate moderation of artificially generated disinformation, highlight the global recognition of the need to control the spread and use of deepfake technologies. These measures underscore the severity of the issue and the necessity of international cooperation to combat emerging forms of cyber threats [1].

To enhance trust in AI-driven deepfake detection in the critical applications, it is crucial to consider its impact on functional safety, e.g., the safety assessment methodologies used in critical systems. The XMECA and EUMECA techniques proposed by [7] offer valuable insights into expert- and tool-based approaches that can be adapted to assess the robustness and trustworthiness of neural network ensembles used in security-sensitive applications.

Given these challenges, this study formulates the following hypotheses:

Hypothesis 1:

The use of an ensemble neural network approach will improve the accuracy of deepfake video classification by combining the strengths of different neural network architectures.

Hypothesis 2:

Selecting “golden frames”, which are the most informative for analysis, will reduce computational costs without compromising classification quality.

Hypothesis 3:

Model activation analysis using Grad-CAM will demonstrate that different architectures focus on various other features, which can be leveraged to enhance deepfake detection.

Thus, this study aims is to develop advanced deepfake detection technologies, specifically through the use of a hybrid neural network ensemble based on the selection of “golden frames”. This approach aims to significantly improve the accuracy of deepfake identification and ensure effective detection amid growing threats related to hybrid warfare and information manipulation. The research focuses on developing methods capable of addressing contemporary challenges, particularly in the context of ongoing international conflicts, such as the war in Ukraine, where deepfake technologies are used for disinformation and societal destabilization. The findings of this study could serve as the foundation for practical solutions that enhance awareness and resilience against information threats across various domains, including media and social networks.

The main contributions of this study are summarized as follows:

• A golden frame selection technique is introducedidentifying the most informative video frames based on grayscale intensity differences between consecutive frames. This approach improves training efficiency while maintaining a good classification performance.
• An ensemble of deep learning models is constructed, combining ResNet50, EfficientNetB0, Xception, InceptionV3, and FaceNet architectures to capture diverse spatial and textural features relevant to deepfake detection.
• A meta-classification model based on XGBoost is applied to aggregate predictions from the base models, enhancing overall accuracy and robustness.
• The application of Grad-CAM enables the interpretation of model behavior by visualizing the regions of interest across different neural networks.

The remainder of this paper is organized as follows. Section 2 provides an overview of related work in the field of deepfake detection. Section 3 presents the proposed methodology, including the golden frame selection technique and ensemble model architecture. Section 4 describes the implementation details and experimental setup, followed by an evaluation of the results, the ablation study, and the comparative analysis. Section 5 discusses findings, limitations, and future research directions. Finally, Section 6 concludes the paper with a summary of key contributions and practical implications.

2. Literature Review

In recent years, deepfake technologies, which enable the manipulation of multimedia content, particularly videos and images, have attracted significant interest from both the scientific community and the general public due to their potential risks to personal security and informational integrity. The development of effective deepfake detection methods has become a key research direction in computer vision and artificial intelligence. High-performance deepfake video detection using CNN focusing on specific target regions and manual feature distillation, proposed by Tran et al. [8], represents one of the latest approaches in this area. This method stands out by utilizing specific video regions to enhance model accuracy while maintaining an optimal model size.

On the other hand, Zhang et al. [9] highlight the need for a heterogeneous ensemble of features to improve the accuracy of deepfake detectors. They proposed a method that integrates various features, such as grayscale gradient, spectral features, and texture features. This approach outperforms several state-of-the-art deepfake detectors, emphasizing the importance of feature ensembles in detecting manipulations.

Methods based on fine-grained classification using subtle, globally relevant features, as demonstrated in the study by Nadimpalli and Rattani [10], achieve high effectiveness across different datasets and manipulation techniques. Their work underscores the advantages of approaches that focus on finer, discriminative features, highlighting the importance of considering local variations in developing deepfake detectors.

Another innovative approach is presented in the work by Guan et al. [11], which uses a multi-functional weighted model based on meta-learning. This model enhances the overall detection capability by optimizing the ability to identify features in different regions.

Finally, it is essential to highlight the work of Khan et al. [12], who conducted a comparative analysis of different deepfake video detection techniques, emphasizing the strengths and weaknesses of each approach and contributing to the improvement of hyperparameter settings to enhance the overall accuracy and efficiency of models.

Chakraborty and Naskar [13] investigate the impact of human physiology and facial biomechanics on the development of reliable deepfake detectors. They analyze how physiological signals can be used to improve the accuracy of detectors, which often face challenges related to demographic and social biases. This approach helps to reduce errors typically encountered in other detection methods that rely on general image or video characteristics.

In their review, Abbas and Taeihagh [14] systematically examine methods for detecting and generating deepfakes using artificial intelligence. The authors explore key algorithms, platforms, and tools, revealing how these technologies can be implemented in various contexts to combat the spread of fake news. They also highlight practical challenges and trends in the implementation of policies aimed at countering deepfake dissemination.

The research by Casu et al. [15] sheds light on a new aspect of the challenges in deepfake detection, focusing on the impact of cognitive biases on decision-making in digital forensics. They introduce the term “Impostor Bias”, which describes the systematic tendency to doubt the authenticity of multimedia content, often assuming it was generated by artificial intelligence. This bias can lead to false judgments and wrongful accusations, undermining the reliability and credibility of forensic evidence. The study emphasizes how the realism of AI-generated multimedia products can amplify the impact of this bias, pointing to the need to develop strategies to prevent and counteract it.

Meanwhile, Firc et al. [16] examine deepfakes as a threat to facial and voice recognition systems, providing a detailed overview of tools and attack vectors in the visual and audio domains. They analyze how deepfakes can affect biometric systems, particularly through spoofing, and categorize deepfakes with corresponding creation tools, datasets, and detection methods. Their primary contribution is the analysis of attack vectors, considering differences between deepfake categories and reports of actual attacks to assess the threats they pose to selected biometric systems.

In their work “ClueCatcher”, Lee et al. [17] use an innovative approach to deepfake detection, focusing on independent features across different domains. They identify and use features such as facial color mismatches, synthesis boundary artifacts, and quality differences between facial and non-facial regions. Using multi-stream convolutional neural networks and inter-patch dissimilarity evaluators allows this model to effectively detect unique deepfake features, significantly enhancing overall detection efficiency and generalizability.

Naitali et al. [18] review a wide range of topics related to deepfakes, including generation methods, detection, available datasets, challenges, and future research directions. Their work provides a foundational understanding of the deepfake problem and highlights its potential risks in areas such as disinformation, political manipulation, reputational damage, and fraud. The authors also focus on current detection methods and identify the need for further research to develop strategies to mitigate the threats posed by deepfakes.

Dincer et al. [19] presented an innovative method for detecting deepfake videos utilizing the golden ratio for frame selection. This approach allows for selecting selecting specific frames, rather than random ones, which can potentially enhance the detection performance by allowing researchers to focus on significant facial regions. The method uses three different feature extraction techniques (VGG19, EfficientNet B0, and EfficientNet B4) and two capsule network models (CapsuleNet and ArCapsNet), demonstrating the ability for deeper and more accurate data analysis. Performance evaluations on two challenging deepfake detection datasets, Celeb-DF and DFDC-P, showed high results, with the fusion of the best models yielding high accuracy and area-under-the-curve (AUC) values, with 93.63% accuracy and 99.14% AUC for Celeb-DF.

Sumanth et al. [20] proposed a hybrid method that combines a Temporal Convolutional Network (TCN) with semantically based content-based frame selection for video summarisation. The method’s novelty lies in integrating the TCN to consider temporal information, which improves the quality of golden frame selection. The advantage is a 6.8% improvement in the F1-score compared to the baseline methods, although the algorithm displayshigh computational complexity for long videos.

In his PhD research, Yousefi [20] ideveloped a method for selecting key frames in egocentric videos with a focus on semantic diversity. The author implemented a network model that minimizes redundancy and highlights the most informative frames. The method ensured the accuracy of frame classification was over 85%, but showed decreased efficiency in videos with low scene contrast.

Gong et al. [21] presented the Diverse Sequential Subset Selection method for the trained generation of video summaries based on human golden annotations. Their model ensures balanced coverage and the diversity of frames in the video, which increases the agreement with human ratings by 12%. However, pre-training is required for each new video, which limits the method’s use in real-time.

Leszczuk and Duplaga [22] proposed a specialized algorithm for the video review of bronchoscopic procedures that automatically identifies golden shots for medical analysis. The uniqueness of the method is its adaptation to the medical domain, with an accuracy of over 92%. However, the algorithm shows lower sensitivity to small details, which is critical for some diagnoses.

Kim and Kim [23] created a conceptual model for the interactive selection of key frames based on the semantics of the object (for example, recognizable symbols such as the Golden Gate Bridge). The authors achieved 80% coverage of significant events involving users, although the method depends on data pre-processing of, which reduces the automaticity of the process.

Alarfaj and Khan [24] conducted a detailed study on fake news classification using machine and deep learning methods. They employed various machine learning models, such as multinomial, Gaussian, and Bernoulli naive Bayes classifiers, logistic regression, and an aggressive classifier without passive learning. In addition to traditional models, the authors explored deep learning models such as LSTM and CNN-LSTM, which showed higher accuracy and more stable classification than traditional methods. These results highlight the potential of deep learning models for effectively combating the spread of fake news.

By examining existing solutions in deepfake detection (

Table 1. Overview of closest analogs.

Author(s)	Year	Method Description	Method Features	Method Novelty
Tran et al. [8]	2021	The use of CNN with a focus on specific target regions.	High performance achieved through manual feature distillation.	Model size optimization for improved performance.
Zhang et al. [9]	2022	Heterogeneous feature ensemble for detection.	Integration of various features (gray gradient, spectral features, and texture features) to improve accuracy.	Improving accuracy through feature ensemble.
Guan et al. [11]	2023	Multifunctional weighted model based on meta-learning.	Utilizing RGB and frequency domains to enhance generalization.	High generalization and adaptation to diverse data.
Lee et al. [17]	2023	“ClueCatcher” with domain-independent feature selection.	Focus on facial color mismatches, synthesis boundaries artifacts, and quality differences between faces and non-facial regions.	Effectiveness in real-world scenarios.

), it can be noted that our method, based on the use of a neural network ensemble with golden frame selection, demonstrates significant advantages compared to other approaches. In particular, a similar method developed by Guan et al. [11], which also employs complex model ensembles to improve accuracy, is worth mentioning. Both methods show high effectiveness under various conditions; however, our approach stands out due to the better optimization of computational resources and adaptability, thanks to targeted frame selection, only focusing on the most informative parts of the video.

Despite the significant progress in developing deepfake detection methods, current approaches still have certain limitations.

• First, most deep learning models, particularly CNN architectures, demonstrate high accuracy on specific datasets but significantly lose effectiveness when the data source or video quality changes.
• Second, some approaches, such as methods based on spectral analysis or manual feature extraction, are sensitive to lighting changes and variations in video streams.
• Third, most methods process videos frame by frame, which results in significant computational costs and limits their practical applicability in real-time scenarios.

These issues justify the need to develop more efficient methods that can maintain high accuracy with lower resource consumption, which is the primary goal of this study.

Therefore, the novelty of our method lies in its ability to effectively utilize golden frames to train models, which significantly enhances processing speed and accuracy compared to traditional methods that analyze each video frame. This approach not only improves efficiency but also provides better generalization when detecting deepfakes across various scenarios, making it ideally suited for applications in media and social networks.

3. Method Description

To improve deepfake recognition accuracy, a neural network ensemble method (Figure 1) is proposed that is based on the extraction of “golden frames”—the most informative frames that reflect significant changes in the scene. This approach allows for a substantial reduction in data volume by focusing on key moments, ensuring more accurate deep learning model training, and enhancing the effectiveness of the final prediction. Figure 1 illustrates the process of extracting golden frames from video and subsequent ensemble training, which enables higher classification accuracy for deepfake content. The following steps outline the proposed method:

Step 1: Data collection. The initial video processing stage involves collecting and organizing data from relevant sources. Video data are stored as files, each containing metadata with class information (e.g., “FAKE” or “REAL”). These metadata are used to label the class tags during the preparation of training and validation datasets. M represents the set of metadata containing information about each video file and its class, while V refers to the directory containing video files.

Step 2: The function for the selection of golden frames from video [19]. This function identifies and selects the most informative frames from the video. This is determined based on significant changes in the scene. The frame selection process is carried out by comparing each frame with the previous one, using the absolute difference, converted into grayscale, to compute the overall intensity of the scene change. Mathematically, this can be represented as calculating the average sum of the intensity difference between two consecutive frames; if it exceeds a given threshold, the frame is added to the set of golden frames. This process continues until the specified number of frames is selected or until the end of the video, ensuring that the most significant scenes are chosen. V represent the video consisting of N frames. The function selects a set of frames F, which significantly differ from each other based on the given threshold of scene change intensity $\theta$.

While the grayscale intensity change between frames may be influenced by factors such as object or camera motion, our goal in using this metric is to identify visually dynamic frames likely to contain facial deformations or artifacts. These transitions often coincide with the manifestation of deepfake inconsistencies, thereby increasing the chances of capturing critical evidence in the selected golden frames.

2.1. Initialization. Initialize the set F with the first frame $f_0$, which we read from V and add to F. Accordingly, we denote $f_0$ as $fprev$.

2.2. Iterative process. We iterate through each $i-\mathrm{t}\mathrm{h}$ frame $V$, where i changes from $\Delta i$ into $N$ with a step of $\Delta i=\max \left(1,⌊{\displaystyle \frac{N}{nu{m}_{frames}}}⌋\right),$ $i-\mathrm{t}\mathrm{h}$ frame $f_i$ from $V$.

Compute the absolute difference D between $fprev$ and $f_i$:

$$D_i=Gray\left(\left|f_i-f_{i-1} \right|\right)$$

(1)

where $f_i$ is the current frame, $f_{i-1}$ is the previous frame, and Gray denotes the grayscale transformation function. The implementation details (such as the use of OpenCV) are described in the main text and/or code comments.

Calculate the difference metric $S$ as the average pixel value in $D$:

$$S={\displaystyle \frac{\sum D}{m\times n}}$$

(2)

where $m\times n$ is the frame size after scaling.

If $S>\theta$, add $f_i$ to $F$ and update $f_{prev}=f_i$.

2.3. Termination condition. The iteration process ends when the number of frames in $F$ reaches $nu{m}_{frames}$ or when all frames in $V$ have been checked.

Step 3: Preprocessing using golden frames involves the function of first checking the presence of each video file listed in the metadata and then extracting a specified number of golden frames that exhibit significant scene changes from each video. Mathematically, the process can be described as iterating through the metadata $M$, which contains pairs of filenames and additional data, followed by calling the function $F\left(f_i,k\right)$ to extract $k$ golden frames from each video.

Let $M$ denote the metadata for video files, where each element $M_i=\left(f_i,d_i\right)$, with $f_i$ is being the filename and $d_i$ as the additional data (such as class labels). The set V denotes the directory where the video files are stored. Let $F\left(f_i,k\right)$ represent the function for selecting golden frames from the video file $f_i$ with a maximum of k frames.

3.1. Frame and label extraction. For each tuple $\left(f_i,d_i\right)\in M$, we perform the following:

• Check whether the file $f_i$ exists in directory $V$.
• If the file exists, apply the golden frame extraction function $F\left(f_i,k\right)$, where $k$ is the maximum number of frames, to extract the following:

  $$G_i=F\left(f_i,k\right)$$

  (3)
• Assign the class label $y_i$ based on the label field in $d_i$:

  $$y_i=\left\{\begin{array}{c}1, if d_i\left[label\right]=FAKE\\ 0, otherwise\end{array}\right.$$

  (4)
• Append all frames in $G_i$ to the list $X$, and append the label $y_i$ to list $Y$ for each frame.

3.2. Returning the results. After processing all videos, the collected data are converted into NumPy arrays to prepare for model training:

$$X=np.array\left(X\right), \mathrm{Y}=np.array\left(Y\right).$$

(5)

Step 4: Loading and preprocessing the training data, as well as splitting them into training and validation sets, can be described using the mathematical form shown below:

4.1. Loading and preprocessing the data. Let M be the metadata (as defined previously) and $V$ be the directory containing the video files. We define a preprocessing function that returns the feature set $X$ and corresponding labels $Y$:

$$(X,Y)=Preprocess(M,V)$$

(6)

where $X\in {\mathbb{R}}^{n\times h\times w\times c}$ represents the set of processed golden frames, nnn is the number of frames, and hhh, www, and ccc are the frame height, width, and number of channels, respectively. $Y\in {\left\{\mathrm{0,1}\right\}}^n$ is the binary label vector corresponding to each frame.

4.2. Splitting the data into training and validation sets. We split the data using an 80/20 ratio to train and validate the model. The split is defined as follows:

$$(X_{train},X_{val},Y_{train},Y_{val})=Split(X,Y,tes{t\_}_{\_}size=0.2,rando{m\_}_{\_}state=42)$$

(7)

where 20% of the data are allocated to the validation set, while 80% are used for training. The parameter random_state = 42 ensures the reproducibility of the split.

Step 5: The creation of base models based on ResNet50 [25], EfficientNetB0 [26], Xception [27], InceptionV3 [28], and Facenet [29], which is based on InceptionResNetV2 [30], can be described as follows: for each model, a pre-trained architecture [31,32] is used to perform initial image processing through convolutional layers for feature extraction (

Table 2. Comparison of features extracted using ResNet50, EfficientNetB0, Xception, InceptionV3, and Facenet.

Feature Type	ResNet50	EfficientNetB0	Xception	InceptionV3	Facenet
Edges and contours of objects	Object edge detection.	Edge detection with computational resource optimization.	Contour detection with deep detailing.	Edge detection through multichannel operations for general objects.	Precise edge detection of faces for feature recognition.
Textural patterns	Surface texture detection.	Textural details at different scales.	Complex texture detection.	Comprehensive texture analysis, adapted for general objects.	Texture extraction tailored for facial features.
Shapes and sizes	Recognition of basic shapes and sizes.	Efficient shape recognition at multiple levels.	Detection of complex shapes and geometric features.	Generalized shape detection for diverse objects.	Detailed face shape recognition for accurate identification.
Lighting and shadows	Lighting variation analysis.	Balancing local and global lighting variations.	High sensitivity to shadows and lighting.	Robustness to lighting changes through multiscaling.	Analysis of subtle lighting variations specific to faces.
Conceptual features	Face, vehicle, and other object recognition.	Object recognition based on simplified features.	High efficiency in recognizing complex objects.	Generalized object recognition across different categories.	Optimized for face recognition and identity identification.
Depth-related features	Complex patterns (shapes, textures, contours).	Complex patterns at different scales.	Structural and textural features.	Generalized features at deep levels, effective for a wide range of objects.	Deep facial traits for differentiating even similar faces.
Spatial features	Geometric details (straight lines, angles)	Local details and global structure.	Complex interactions between elements.	Geometric patterns are adapted for different object scales.	Spatial features for precise facial detail detection.
Color features	-	Color channel correlations.	Separate extraction of spatial and channel features.	Sensitivity to color channels for general objects.	Consideration of subtle color nuances is important for faces.
Contextual features	Interaction of objects in the scene.	Local and global relationships between objects.	Interactions between objects in the scene.	Detection of relationships between objects in complex scenes.	Determining the context of faces in the scene to improve accuracy.
Features across different levels	-	Balance between details and general patterns.	Local details and general characteristics	Different feature levels for different object categories.	Local and global features for face identification.
Detailing features	General characteristics of objects.	High detailing of small objects.	Precise extraction of details and textures.	Different levels of detailing for a wide range of objects.	Super-detailing of facial features for accurate recognition.
Local structure features	Focus on general shapes and textures.	Balance between local and global features.	Focus on local details.	Local and global features for accurate scene analysis.	Focus on local facial features for reliable identification.

). Each model [33] has a structure that includes a processing sequence consisting of pre-trained convolutional layers, fully connected layers with Swish activation, and dropout layers for regularization, and an output using sigmoid activation for binary class prediction. Thus, the following mathematical formulation can be used:

5.1. Definition of input data. Let $X\in R^{N\times H\times W\times C}$ represent the input tensor, where $N$ is the number of samples, $H=224,W=224$, i $C=3$ are the dimensions of the image (height, width, number of channels).

5.2. Using pre-trained base models. For each model architecture, we define a feature extraction [34] function $F_{Model}: {\mathbb{R}}^{H\times W\times C}\to {\mathbb{R}}^d$, which maps a single image to a high-dimensional feature vector. The top classification layers are excluded in all models. The functions are defined as follows:

ResNet50 ₃ LSTM: $F_{ResNet}\left(X\right)=ResNet50\left(X\right), d=2048$

EfficientNetB0: $F_{EffNet}\left(X\right)=EfficientNetB0\left(X\right), d=1280$

Xception: $F_{Xception}\left(X\right)=Xception\left(X\right), d=2048$

InceptionV3: $F_{Inception}\left(X\right)=InceptionV3\left(X\right), d=2048$

Facenet: $F_{Facenet}\left(X\right)=InceptionResNetV2\left(X\right), d=512$

5.3. Flattening the output of the base model. The feature tensor FModel (X) obtained from the base model is converted into a flat vector

$$Z=Flatten\left(F_{Model}\left(X\right)\right)\in {\mathbb{R}}^d$$

(8)

where d—is the dimensionality of the flattened feature vector, which depends on the architecture of the base model.

5.4. Fully connected layer with Swish activation. A fully connected layer with 512 neurons and a Swish activation function is applied to the flattened vector:

$$Z^{\prime }=Swish\left(W1Z+b1\right)$$

(9)

where $W1\in R512$×d is the weight matrix and; $b1\in R512$ is the bias vector.

The Swish function is defined as $Swish\left(x\right)=x\cdot \sigma \left(x\right)$, where $\sigma \left(x\right)$ is the sigmoid function.

5.5. Dropout layer for regularization. To prevent overfitting, a dropout layer with a neuron dropout probability of p = 0.5 is applied:

$$Z^{″}=Dropout\left(Z^{\prime },p\right)$$

(10)

5.6. Additional layers for ResNet50 with LSTM. The following operations are only performed for the ResNet50 model with LSTM:

Dimensionality expansion:

$$Z^{\prime ″}=ExpandDims\left(Z^{″},axis=1\right)$$

(11)

This converts the vector Z′′ into a tensor for compatibility with the LSTM layer.

LSTM layer:

$$ZLSTM=LSTM\left(256\right)\left(Z^{\prime ″}\right)$$

(12)

where the LSTM has 256 memory units.

Distinct properties: the use of LSTM allows the model to account for sequential or temporal dependencies in the input data.

Step 6: Output layer with sigmoid activation

A fully connected output layer with one neuron and sigmoid activation is applied for binary classification.

For models without LSTM, the following holds:

$$\widehat{y}=\sigma \left(W_2{Z}^{″}+b_2\right)$$

(13)

For ResNet50 with LSTM, the following holds:

$$\widehat{y}=\sigma \left(W_2{Z}_{LSTM}+b_2\right)$$

(14)

$W_2\in R^{1\times n},n=512$ for models without LSTM and $n=256$ for the model with LSTM.

$b_2\in R$ is the bias term.

$\sigma \left(x\right)={\displaystyle \frac{1}{1+e^{-x}}}$ is the sigmoid function.

Step 6: The training of the base models based on ResNet50, EfficientNetB0, Xception, InceptionV3, and Facenet can be mathematically represented as shown:

6.1. Model compilation. Let $M_{ResNet}, M_{EfficientNet}, M_{Xception}, M_{InceptionV3}{, M}_{Facenet}$ represent the models based on ResNet50, EfficientNetB0, Xception, InceptionV3, and Facenet, respectively. For each of these models, optimization is performed using the Adam algorithm, and the loss function is defined as binary cross-entropy:

$$L_{binary}\left(y,\widehat{y}\right)=-\left(ylog\left(\widehat{y}\right)+\left(1-y\right)\log \left(1-\widehat{y}\right)\right)$$

(15)

where $y$ the true label, and $\widehat{y}$ is the model’s prediction. The objective is to minimize $L_{binary}$ using the Adam optimizer:

$${\theta }_{new}={\theta }_{old}-\eta {\nabla }_{\theta }{L}_{binary}$$

(16)

where $\theta$ represents the model parameters, and $\eta$ is the learning rate.

6.2. Model training. For each model M with training data $\left(X_{train},Y_{train}\right)$ and validation data $\left(X_{val},Y_{val}\right),$ the model training process can be described as the minimization of the loss function $L_{binary}$ over 10 epochs using a batch size of 8. This can be written as follows:

$$\underset{\theta }{\min }{\displaystyle \frac{1}{N}}\sum _{i=1}^N{L}_{binary}\left(y_i,M\left(X_i;\theta \right)\right)$$

(17)

where $N$ is the number of samples in the training set, $X_i is$ the input data, $y_i$ is the label for the $i-\mathrm{t}\mathrm{h}$ sample, and $\theta$ represents the model’s parameters.

The training process is performed for each model, with the parameters updated after each epoch based on the gradient of the loss function. The accuracy is evaluated based on the validation data after every epoch.

Step 7: The formation of meta-features (predictions of base models) is carried out as follows:

7.1. Predictions for the training dataset. For each base model $M_{ResNet}, M_{EfficientNet}, M_{Xception}$, $M_{InceptionV3}{, M}_{Facenet},$ predictions for the training data $X_{train}$ are computed. Let ${\widehat{y}}_{ResNet}\left(X_{train}\right), {\widehat{y}}_{EfficientNet}\left(X_{train}\right), {\widehat{y}}_{Xception}\left(X_{train}\right), {\widehat{y}}_{InceptionV3}\left(X_{train}\right), {\widehat{y}}_{\mathrm{F}\mathrm{a}\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{t}}\left(X_{train}\right)$ represent the predicted values derived using the base models for the training data:

$${\widehat{y}}_{ResNet}=M_{ResNet}\left(X_{train}\right)$$

(18)

$${\widehat{y}}_{EfficientNet}=M_{EfficientNet}\left(X_{train}\right)$$

(19)

$${\widehat{y}}_{Xception}=M_{Xception}\left(X_{train}\right)$$

(20)

$${\widehat{y}}_{InceptionV3}=M_{InceptionV3}\left(X_{train}\right)$$

(21)

$${\widehat{y}}_{Facenet}=M_{Facenet}\left(X_{train}\right)$$

(22)

7.2. Predictions for the validation dataset. Similarly, predictions are obtained for the validation data $X_{val}$:

$${\widehat{y}}_{ResNet}\left(X_{val}\right), {\widehat{y}}_{EfficientNet}\left(X_{val}\right), {\widehat{y}}_{Xception}\left(X_{val}\right), {\widehat{y}}_{InceptionV3}\left(X_{val}\right), {\widehat{y}}_{Facenet}\left(X_{val}\right)$$

(23)

These are the predicted values derived from the base models for the validation data.

7.3. The production of input data for the meta-model. The input data for the meta-model (stacking) are produced by horizontally concatenating the predictions of the base models for the training and validation datasets:

$$X_{meta}^{train}=hstack\left({\widehat{y}}_{ResNet}\left(X_{train}\right), {\widehat{y}}_{EfficientNet}\left(X_{train}\right), {\widehat{y}}_{Xception}\left(X_{train}\right), {\widehat{y}}_{InceptionV3}\left(X_{train}\right), {\widehat{y}}_{\mathrm{F}\mathrm{a}\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{t}}\left(X_{train}\right)\right)$$

(24)

$$X_{meta}^{val}=hstack\left({\widehat{y}}_{ResNet}\left(X_{val}\right), {\widehat{y}}_{EfficientNet}\left(X_{val}\right), {\widehat{y}}_{Xception}\left(X_{val}\right), {\widehat{y}}_{InceptionV3}\left(X_{val}\right), {\widehat{y}}_{Facenet}\left(X_{val}\right)\right)$$

(25)

The new feature matrices $X_{meta}^{train}$ and $X_{meta}^{val}$ are used as input data for the meta-model, which is trained on the predictions of the base models for final predictions.

Step 8: Training the meta-model involves training it based on the predictions of the base models, using stacked features formed from the predictions for each sample. Mathematically, this process is described below.

8.1. Creating the meta-model. Let $M_{meta}$ be the meta-model that is trained on the features derived from the predictions of the base models. These features are formed by stacking the predictions ${\widehat{y}}_{ResNet}$, ${\widehat{y}}_{EfficientNet}$, ${\widehat{y}}_{Xception}, {\widehat{y}}_{InceptionV3}, {\widehat{y}}_{Facenet}$ for each sample:

$$X_{meta}=hstack({\widehat{y}}_{ResNet}, {\widehat{y}}_{EfficientNet}, {\widehat{y}}_{Xception}, {\widehat{y}}_{InceptionV3}, {\widehat{y}}_{Facenet})$$

(26)

8.2. Training the meta-model. The meta-model is trained on the training meta-features $X_{meta}^{train}$ with the corresponding labels $y_{train}$ minimizing the loss function $L\left(y, \widehat{y}\right)$, where $\widehat{y}$ represents the predictions of the meta-model:

$$\underset{\theta }{\min }{\displaystyle \frac{1}{N}}\sum _{i=1}^{N}L\left(y_i, M_{meta}\left(X_{meta}^{train};{\theta }_{meta}\right)\right)$$

(27)

8.3. Predictions for the validation set. The meta-model makes predictions on the validation set of features $X_{meta}^{val}$, yielding ${\widehat{y}}_{meta}\left(X_{meta}^{val}\right)$.

8.4. Accuracy evaluation of the meta-model. The accuracy of the meta-model is evaluated by comparing ${\widehat{y}}_{meta}$ with the validation labels $y_{val}$:

$$Accuracy={\displaystyle \frac{1}{N_{val}}}\sum _{i=1}^{N_{val}}\mathbb{I}\left(y_i={\widehat{y}}_i\right)$$

(28)

8.5. Confusion matrix formation and visualization. The confusion matrix is defined as follows:

$$C=\left[\begin{array}{cc}TP& FP\\ FN& TN\end{array}\right]$$

(29)

where $TP, FP, FN, TN$ represent the counts of true positives, false positives, false negatives, and true negatives, respectively. This matrix is visualized using a heatmap.

4. Implementation

4.1. Case Study

The implementation of the proposed method, the dataset from the Deepfake Detection Challenge on the Kaggle platform (Kaggle, 2020) [35] was used. This is currently the largest dataset for deepfake detection. It contains over 100,000 video fragments created with the participation of 3426 actors using various deepfake generation methods, including GAN-based techniques. All the videos were generated with the full consent of the participants, making this dataset unique in its kind.

The training dataset consists of 400 videos, of which approximately 209 original videos were used to create these 400 videos. Among these, 323 videos are labeled as “FAKE” and 77 as “REAL”, corresponding to about 80% of videos being fake (Figure 2). This confirms that the dataset contains a large number of modified videos compared to the real ones. Several original videos were used multiple times to generate fake videos, with some originals generating up to six fake videos.

Next, the golden frame selection method is used to reduce the volume of processed video data and highlight the most informative moments in the videos. The method is based on analyzing changes between consecutive frames: frames are extracted from each video at fixed intervals, and then the average brightness difference between the current and previous frames is calculated using the absolute pixel difference in grayscale. A frame is considered significantly different if the value of this difference exceeds a set threshold (30.0), indicating a significant change in the scene. For example (Figure 3), frames from fake videos, such as eekozbeafq.mp4 or dcuiiorugd.mp4, may show more artifacts or lighting changes compared to real videos, like avmjormvsx.mp4.

Data preprocessing includes resizing the selected frames to a standard size of 224 × 224 pixels to meet the input layer requirements of deep learning models, such as ResNet, EfficientNet, and Xception. After that, pixel values are normalized to the range of [0,1], which helps with the stability and efficiency of model training. The frames and their corresponding labels (0 for real videos, 1 for fake ones) are organized into 4D tensors with the shape (num_videos, num_frames, 224, 224, 3) for input data X, and (num_videos) for labels y, ensuring the correct representation of video data for model training. After data preparation, data are split into training and validation sets in an 80% training/20% validation ratio.

A model architecture was developed to analyze video frame sequences, combining pre-trained models ResNet50, EfficientNetB0, and Xception with LSTM to process temporal dependencies. The input consists of 10 frames, each of 224 × 224 pixels, allowing the models to capture both spatial and temporal features. Each frame is processed separately using the TimeDistributed layer, followed by the use of an LSTM with 256 neurons to model dependencies between frames. Training was performed for up to 15 epochs, with early stopping to prevent overfitting, using a batch size of 4 and the binary_crossentropy loss function to classify videos as fake or real. This approach helps the model better understand spatiotemporal patterns to detect manipulations.

The accuracy graph (Figure 4) shows the performance of different models during training and validation. ResNet50 achieved a highest validation accuracy of 68.1%, showing good generalization. FaceNet reached 94.7% training accuracy but only 61.1% validation accuracy, indicating some overfitting. EfficientNet demonstrated stable training progress, finishing with 59.7% validation accuracy. Xception and Inception followed a similar trend, with Xception reaching 56.9% validation accuracy.

The loss graph (Figure 4) shows that the training loss decreased for all models, but validation loss varied. Some models, like FaceNet and Inception, displayed a more significant gap between training and validation loss, suggesting overfitting. However, early stopping was applied to reduce this effect. EfficientNet showed the most stable loss trend, while ResNet50 balanced training and validation loss best, indicating good generalization. Future improvements could include additional regularization techniques or ensemble methods to further enhance performance.

4.2. Feature Impact

To analyze essential features that affect the classification of deepfake videos, the Grad-CAM method was used, allowing the visualization of the activation of neural networks (Figure 5). The resulting heatmaps showed that different neural network architectures focus on different areas of the face (

Table 3. Analysis of Grad-CAM activations in the deepfake classification process.

Model	Main Activation Zones	Main Type of Anomalies
ResNet50	Forehead, cheeks	Textural distortions, uneven lighting
EfficientNetB0	Contours of the mouth, eyes	Facial contour deformations, unnatural light transitions
Xception	Eye area, lips	Motion anomalies, unnatural texture details
InceptionV3	Central part of the face	Color instability, blurring of details
FaceNet	Entire face structure	Global distortions of shape and proportions
ResNet50	Forehead, cheeks	Texture distortions, uneven lighting

). ResNet50 predominantly focuses on the textural features of the skin, particularly the forehead and cheeks, indicating its sensitivity to global distortions. EfficientNetB0 places more emphasis on the contours of the face, especially the lines around the mouth and eyes, making it effective at detecting structural anomalies. The Xception model focuses on the regions around the eyes and lips, which are key areas for deepfake identification, as they often contain unnatural artifacts related to movement and texture. The analysis of the obtained activations suggests that combining different architectures can enhance the effectiveness of deepfake detection by utilizing various approaches to processing visual information.

The obtained results confirm that different neural network architectures employ distinct approaches to identifying features characteristic of deepfakes. The use of Grad-CAM not only facilitates the interpretation of model decisions but also helps to identify key areas influencing classification, such as texture features, facial contours, and motion artifacts. This opens up possibilities for further improving deepfake detection methods by combining models with different sensitivity zones to achieve greater accuracy and reliability.

4.3. Accuracy Evaluation

The analysis of feature importance in the meta-model (Figure 6) demonstrates that the most significant factor in deepfake classification is the average prediction value of the Xception model, which has the highest weight (5.1123). The second most important feature is the median value of the ResNet output predictions (4.8745), confirming the stability and relevance of this architecture in the ensemble approach. Slightly lower but still substantial importance is observed for the mean values of ResNet (1.5623) and the standard deviation of EfficientNet (1.1345), indicating their role in detecting local variations in videos. Features associated with the Inception model (std = 0.2678, median = 0.1452), as well as the mean value of FaceNet (0.0817), had significantly lower weights, suggesting their less significant informativeness in the classification process compared to other factors.

The accuracy evaluation of different meta-models confirms the superiority of XGBoost [36], which achieved a maximum accuracy of 0.91, significantly outperforming Gradient Boosting (0.85) and Random Forest (0.825). This highlights the effectiveness of XGBoost in combining base model predictions, leading to more precise generalization. Interestingly, some features, such as EfficientNet_mean (−0.5352), Xception_std (−0.4789), and ResNet_std (−0.4156), had negative importance values, which may indicate their potential impact in reducing classification efficiency.

The obtained results confirm the effectiveness of ensemble models for deepfake detection and highlight the feasibility of further optimizing the feature weight coefficients in the meta-model.

4.4. Experimental Results

To understand how different models and training settings affect deepfake detection, we tested five models: ResNet50, FaceNet, Xception, Inception, and EfficientNet. We analyzed their accuracy and loss under different conditions:

• With golden frames: using selected keyframes from videos that provide the most important visual features.
• Without golden frames: training without selected keyframes.
• Without batch optimization: turning off batch-level optimizations during training.
• Without dropout: removing dropout layers to see how it affects regularization.
• Without Mish activation: replacing Mish activation functions with ReLU to check their impact.

The results for each setting are summarized in

Table 4. Model performance under different configurations.

Model (Accuracy/Loss)	With Golden Frames	Without Golden Frames	Without Batch Optimization	Without Dropout	Without Mish Activation
ResNet50	0.597/0.673	0.486/0.701	0.431/0.700	0.528/0.686	0.528/0.692
EfficientNetB0	0.486/0.700	0.597/0.679	0.569/0.662	0.528/0.684	0.514/0.725
Xception	0.625/0.654	0.486/0.714	0.528/0.691	0.556/0.689	0.583/0.675
InceptionV3	0.500/0.722	0.486/0.767	0.583/0.704	0.431/0.746	0.514/1.088
Facenet	0.597/0.676	0.569/0.714	0.583/0.678	0.486/0.688	0.528/0.778
Meta-Model XGBoost	0.911/0.224	0.905/0.334	0.903/0.354	0.885/0.386	0.874/0.456

.

The results (Table 4) show that using golden frames improved accuracy for all models, with EfficientNet performing best (66.67%). Removing golden frames reduced accuracy, especially for ResNet50 (59.72%) and Xception (62.50%). Turning off batch optimization lowered performance, while dropout and Mish activation helped maintain stable accuracy. We also tested an XGBoost meta-model using features from ResNet50, EfficientNet, and Xception. With golden frames, XGBoost reached 91.14% accuracy, but accuracy dropped slightly to 90.54% without them. Removing batch optimization and dropout also impacted performance, with accuracy decreasing to 90.27% and 88.54%, respectively. Switching Mish to ReLU led to the lowest accuracy for XGBoost at 87.41%. These results highlight the importance of selecting the proper preprocessing steps and model settings for deepfake detection. Choosing the best frames and using batch optimization, dropout, and Mish activation can significantly improve accuracy and stability.

4.5. Module Description and Operation

The TruScanAI module [37] is designed to automatically analyze video files to detect deepfakes. The user interface allows for video uploads, after which the system processes the content using the proposed ensemble neural network approach, incorporating the golden frame selection method. The module automatically extracts key frames during the analysis and examines textural, contour, and kinematic facial features. Upon completion, the system provides the user with a classification result, labeling the video as REAL or FAKE, supplemented with a visualization of the analysis and key indicators that influenced the model’s decision.

This approach ensures an effective and accessible way to verify the authenticity of video content, catering to both regular users and experts in the field of digital security.

5. Discussion

Analysis of the results: The results confirm the effectiveness of the proposed ensemble approach for DeepFake classification, particularly due to integrating predictions from ResNet50, EfficientNetB0, and Xception into the XGBoost meta-model. The best accuracy among base models was achieved by EfficientNet (66.67%) when using golden frames, while XGBoost with golden frames reached an accuracy of 91.14%, confirming the effectiveness of this technique. In contrast, removing golden frames led to a drop in accuracy for all models, particularly for Xception (48.61%) and ResNet50 (48.61%). Batch optimization and dropout proved essential for stable performance—disabling them caused a noticeable decline in accuracy across all models.

Interpretation of neural network activations: The analysis of model activations using Grad-CAM confirmed that different neural network architectures have specific attention areas when classifying deepfakes. ResNet50 primarily focuses on the textural features of the skin, particularly the forehead and cheeks, which explains its effectiveness in detecting global artifacts. EfficientNetB0 concentrates on the contours of the mouth and eyes, providing more accurate detection of structural anomalies. Xception pays the most attention to the regions of the lips and eyes, which are key areas for detecting anomalies in movement and texture. Combining these models in an ensemble approach helps compensate for their shortcomings and improves deepfake detection accuracy.

Limitations and scalability challenges: Despite the high accuracy rates, the proposed method has certain limitations. One of the challenges is the models’ sensitivity to the quality of input videos: with low resolution or significant compression of video files, classification accuracy may decrease due to the loss of key textural features. Additionally, while the ensemble model demonstrates high overall accuracy, false positives and false negatives remain an issue, particularly in videos that use the latest deepfake generation techniques with enhanced realism. Furthermore, the computational complexity of the ensemble approach may pose a problem when integrating it into real-world applications that require real-time video processing.

Given the parallels between cybersecurity challenges in critical infrastructure and deepfake detection, the GA-and-IMECA-based cybersecurity assessment techniques for FPGA-based I&C systems presented by [38] provide a structured framework that could inform the future risk assessment of AI-based media authentication systems. Applying such rigorous security lifecycle approaches helps align deepfake detection mechanisms with best practices from high-assurance domains.

Opportunities for future research: Future research could focus on improving the model’s resilience to new generative methods, such as videos created using StyleGAN3 or Diffusion Models. A promising direction is the integration of additional features, particularly audio analysis, which would allow for the detection of inconsistencies between the video and audio tracks. Another essential area of development is optimizing computational costs, which could be achieved through the use of transfer learning or model compression without significant loss of accuracy. Additionally, extended testing on independent datasets from real-world sources would help assess the model’s generalizability in practical scenarios, such as fact-checking and digital forensic analysis.

Moreover, extending the current system to support multimodal detection—combining visual and auditory cues—could improve accuracy and robustness, particularly against deepfakes with well-synchronized but artificially generated audio tracks.

Ethical considerations. As deepfake detection systems become more widespread, the ethical balance between detection and censorship must be considered. While the technology aims to protect individuals and institutions from misinformation and fraud, there is a risk of misuse—e.g., automated filtering of legitimate satirical content or overclocking of dissenting voices in political contexts. Transparency, explainability, and human-in-the-loop validation should be prioritized in deploying such systems to mitigate potential misuse and maintain trust.

6. Conclusions

The proposed neural network ensemble method for deepfake classification using the golden frame selection technique demonstrated high efficiency in detecting fake videos. The combination of base models ResNet50, EfficientNetB0, and Xception in an ensemble with the meta-model XGBoost achieved the highest accuracy among the tested methods, reaching 91% accuracy on the validation set, surpassing the results of Random Forest (82.5%) and Gradient Boosting (85%). Feature importance analysis showed that the most significant contributions to classification came from the average predictions of Xception (5.1123) and ResNet (4.8745), while other features had lower significance. Additional activation analysis using Grad-CAM confirmed that different neural network architectures focus on various aspects of the face, making their combination effective at detecting anomalies typical of deepfakes. The golden frame selection method significantly reduced computational costs, which is a critical factor for real-world applications, while maintaining high levels of accuracy and model generalization. The practical application of the developed system is possible in several key areas. First, it can be integrated into fact-checking and media monitoring platforms to verify the authenticity of video materials used in news and social media. This would enable journalists and independent researchers to quickly assess the credibility of content, preventing the spread of misinformation. Furthermore, the system could be applied in digital forensic analysis, helping to detect fake video evidence in criminal cases. In the future, integration of this approach into automatic video stream monitoring systems could allow for real-time manipulation detection, enhancing information security. Despite the results obtained, the research revealed certain challenges, including the model’s sensitivity to the quality of input videos and the difficulty of processing cutting-edge generative methods. Further research should improve model generalization, integrate audio analysis, and optimize computational resources to ensure effective real-time operation. The results of this work may be valuable for developing digital forensic tools, media fact-checking, and ensuring information security in the face of increasing threats related to deepfake technologies.