In this chapter, we present our dataset, training details, qualitative results, quantitative results, and ablation studies.
4.3. Qualitative Results
In
Figure 4, we compare AmazingFT with five other state-of-the-art (SOTA) face-swapping methods (FaceSwap-Gan, DeepFaceLab, SimSwap, Inswapper, BlendSwap). We use FaceForensics++ as the test set, which is a large-scale dataset for facial forgery detection, containing 1000 original video sequences sourced from 977 YouTube videos. These videos feature trackable, primarily frontal faces without occlusions, allowing automatic tampering methods to generate realistic fake videos. In our experiments, we sampled 10 frames from each video for comparison. The results demonstrate that AmazingFT excels in face-swapping performance. The face-swapped images generated by AmazingFT show superior naturalness, consistency, detail preservation, expression conveyance, and lighting matching.
The images generated by AmazingFT are very natural, with no noticeable artifacts or incoherent areas, appearing as if they are part of the original image. The entire face maintains consistency in lighting and skin tone. Features of the source face, such as the eyes, nose, and mouth, are accurately mapped onto the target face and maintain high consistency. The positioning and proportions of facial features blend naturally with the target face, with no noticeable misalignment or distortion. In terms of detail preservation, AmazingFT performs exceptionally well, retaining skin texture, hair, and other fine features. Even at high resolution, the face-swapped images remain delicate and realistic. Furthermore, the images accurately convey the expressions of the source face. Expression changes, whether smiling or frowning, are natural and coherent, with facial expression details well presented. In lighting matching, AmazingFT also performs excellently, seamlessly integrating lighting conditions. The transitions between highlights and shadows are very natural, avoiding noticeable lighting inconsistencies.
In comparison, while BlendSwap performs well, it falls slightly short of AmazingFT in terms of naturalness, detail preservation, and lighting matching. FaceSwap exhibits noticeable artifacts and incoherent areas, DeepFaceLab has deficiencies in expression consistency and lighting matching, SimSwap performs poorly in facial feature consistency and detail handling, and although Inswapper does well in detail preservation, it has shortcomings in expression conveyance and lighting matching, leaving room for significant improvement. These comparisons clearly showcase the outstanding performance of AmazingFT in face-swapping tasks.
We downloaded face videos from YouTube to perform frame-by-frame face-swapping, showcasing five sets of results by extracting the face-swapped results of the 1st frame, 25th frame, 50th frame, 75th frame, and 200th frame. As shown in
Figure 5, AmazingFT demonstrates powerful performance in the field of face-swapping. AmazingFT not only generates high-quality, realistic, and natural face-swapping effects but also effectively handles partially occluded faces. By combining Transformer and GAN technologies, AmazingFT excels in both detail processing and overall visual effects. The frame-by-frame face-swapping results displayed (1st frame, 25th frame, 50th frame, 75th frame, and 100th frame) clearly illustrate its stability and consistency, with facial expressions, lighting conditions, and facial details all handled very naturally.
The Transformer, through its multi-head attention mechanism, captures and maintains the complex semantic relationships between the source face and the target face. Observing the frame-by-frame images, the expression and pose changes in each frame are very natural. For instance, in the first column from the 25th frame to the 100th frame, as the expression changes, the features on the target face also change synchronously, demonstrating the model’s strong capability in capturing dynamic features. Through feature fusion, the Transformer ensures that features at different scales are effectively processed and preserved. In each frame, details such as skin texture, hair, and facial shadows are well preserved, making the face-swapping results more realistic.
From the 1st frame to the 100th frame, the generated face-swapped images maintain high quality at different resolutions. The GAN generator uses progressive upsampling and convolution operations to generate high-resolution images from low resolution, ensuring that details at each layer are effectively processed and preserved. For example, observing the second column at the 50th and 75th frames, details such as skin texture and lighting effects are very clear. The discriminator continually optimizes the generator during training, enabling it to produce more realistic and natural images. The frame-by-frame results show that the target face and the source face maintain consistency and realism in the overall structure and detailed features. For instance, in the fourth column at the 25th and 100th frames, the facial features of the target face are highly consistent with the source face, with no noticeable artifacts or unnatural transitions.
4.4. Quantitative Results
Next, we conducted a quantitative comparison on the FaceForensics++ video dataset using the following metrics: SSIM, FID, Pose Error, Expression Error, and Face Shape Error, to further validate the effectiveness of our AmazingFT. For FaceSwap, DeepFaceLab, and SimSwap, we evenly sampled 10 frames from each video, creating a 10 K test set.
SSIM measures the similarity between the generated image and the reference image by comparing luminance, contrast, and structural information. It aligns more closely with the human visual system’s perception, reflecting image quality better than simple pixel differences. SSIM calculation involves a local window within the image, computing the similarity in terms of luminance, contrast, and structure for each window, and then averaging the similarity across the entire image. In this paper, SSIM is implemented using the scikit-image library in Python. The generated and reference images are converted to grayscale, and the SSIM function computes the SSIM value between the two images. A higher SSIM value indicates better image quality.
Figure 5.
Face-swapping results of AmazingFT with different frame counts in videos.
Figure 5.
Face-swapping results of AmazingFT with different frame counts in videos.
FID evaluates the quality of generated images by calculating the distribution differences between generated images and real images in the feature space of a pre-trained Inception [
24] network. It considers the mean and covariance of the feature distributions; a lower FID value indicates closer distributions and higher image quality. In this paper, FID is implemented using a pre-trained Inception network. The generated and real images are fed into the Inception network to extract features. Then, the mean and covariance of these features are computed, and the FID value is calculated using these statistics. A lower FID value indicates better image quality.
The Pose metric assesses the consistency of facial pose between the generated image and the target pose by comparing facial orientation, angle, and position features. Higher pose consistency indicates better performance in maintaining the target pose. In this paper, pose evaluation is implemented using Dlib [
25] for facial pose estimation. Key facial points of the generated and target images are detected, and their pose parameters (e.g., pitch, yaw, and roll angles) are calculated and compared. A lower Pose value indicates better image quality.
The Expression metric measures the consistency of facial expressions between the generated image and the source face by comparing facial expression features such as mouth corner lifting and eyebrow raising. In this paper, expression evaluation is implemented using OpenFace [
26] for facial expression recognition. Key facial points of the generated and source images are detected, and their expression vectors are calculated and compared. A lower Expression value indicates better image quality.
The Shape metric evaluates the consistency of facial shape between the generated image and the target shape by comparing facial contours and the positions of key points. Higher shape consistency indicates better performance in maintaining facial shape. In this paper, shape evaluation is typically implemented using facial key point detection algorithms, and we use Dlib for this purpose. Key facial points of the generated and target images are detected, and their Euclidean distances and similarity metrics are calculated to assess shape consistency. A lower Shape value indicates better image quality.
As shown in
Table 1, the performance of six different face-swapping methods across multiple evaluation metrics is presented, including FaceSwap, DeepFaceLab, SimSwap, Inswapper, BlendSwap, and AmazingFT (ours). From the table, it is evident that AmazingFT outperforms all other methods in every evaluation metric. It has the highest SSIM value (0.84), indicating the best structural similarity between the generated images and the real images. It also has the lowest FID value (28.45), suggesting the highest quality of generated images with minimal distribution differences from the real images. Additionally, it has the lowest Pose, Expression, and Shape values, which are 2.95, 25.44, and 40.32, respectively, demonstrating superior consistency in pose, expression, and shape of the generated images. In comparison, DeepFaceLab, SimSwap, FaceSwap, Inswapper, and BlendSwap show varying strengths and weaknesses. Overall, AmazingFT leads significantly across all metrics, showcasing its exceptional performance in face-swapping tasks.
In this study, we designed an evaluation system comprising five subjective evaluation metrics to assess the effectiveness of face-swapping algorithms. The five metrics are as follows:
Naturalness: This metric evaluates the degree of naturalness in the face-swapping effect, determining whether the generated face-swapped image appears to be a real face. Images with high naturalness should have no obvious artificial traces, artifacts, or incoherent areas, presenting a smooth and realistic overall visual effect. Consistency: This metric assesses whether the features of the source face are consistent with the target face in the swapped image. This includes the position, proportion, and overall structure of facial features. High consistency in face-swapping should make it difficult for viewers to detect unnatural changes. Detail preservation: This metric evaluates the degree of detail retention in the swapped images, including whether fine features such as skin texture, hair, and wrinkles are preserved. High detail preservation means that the images should appear real and detailed, even at high resolution. Expression conveyance: This metric measures whether the facial expressions of the source face are accurately conveyed in the swapped images. Good expression conveyance should maintain the expression changes in the source face and present the same emotions and expression details on the target face. Lighting consistency: This metric assesses the consistency of lighting in the swapped images, particularly the integration of lighting conditions between the source face and the target face. Good lighting consistency ensures that shadows, highlights, and color tones appear natural and harmonious across different lighting environments.
To ensure the objectivity and reliability of the evaluation results, we invited 60 volunteers to subjectively score the face-swapping outcomes. Each volunteer reviewed the results and rated each image based on the five aforementioned metrics, with scores ranging from 1 to 5. We then conducted statistical analyses on the scores for each metric to assess the performance of the face-swapping algorithms across various aspects. A higher score indicates better performance in the respective metric, while a lower score signifies poorer performance.
Table 2 shows the performance of six different face-swapping methods across five subjective evaluation metrics, including FaceSwap, DeepFaceLab, SimSwap, Inswapper, BlendSwap, and AmazingFT (ours). The scores for each method on Identity, Attribute, Anti-Occlusion, Details, and Fidelity metrics illustrate their performance in various aspects. It is evident from the table that AmazingFT excels in all evaluation metrics, with scores of 4.12 for Identity, 4.08 for Attribute, 4.06 for Anti-Occlusion, 4.33 for Details, and 4.07 for Fidelity. This demonstrates its superior performance in identity recognition, attribute retention, anti-occlusion, detail handling, and fidelity. In contrast, other methods like DeepFaceLab, FaceSwap, SimSwap, Inswapper, and BlendSwap generally score below 4 in these metrics. Overall, AmazingFT leads significantly in comprehensive performance, outperforming other mainstream networks in face-swapping tasks.
4.5. Ablation Study
We conducted three different types of ablation experiments to demonstrate the effectiveness of our designed network architecture.
We evaluated the performance of three different face-swapping model configurations across multiple evaluation metrics: a model without GAN (W/O GAN), a model without Transformer (W/O Transformer), and the complete AmazingFT model, as shown in
Table 3. From the table, it is evident that the complete AmazingFT model performs the best across all metrics. It has the highest SSIM value (0.84), indicating the best structural similarity of the generated images; the lowest FID value (28.45), indicating the highest quality of generated images with minimal distribution differences from the real images; and the lowest Pose, Expression, and Shape values, which are 2.95, 25.44, and 40.32, respectively, demonstrating superior performance in pose, expression, and shape consistency of the generated images.
The model without GAN (W/O GAN) performs close to AmazingFT in SSIM and Pose but shows poorer results in FID, Expression, and Shape, indicating that GAN plays a crucial role in enhancing image quality and feature consistency. The model without Transformer (W/O Transformer) has metrics similar to the W/O GAN model but still falls short of the complete AmazingFT model, indicating that the Transformer plays a significant role in feature transformation. The AmazingFT model exhibits the best overall performance, proving the critical roles of GAN and Transformer in improving face-swapping effects.
Table 4 presents the performance of two different face-swapping model configurations across multiple evaluation metrics: a model without multi-scale (No Multi-Scale) and a model with multi-scale (Multi-Scale). The table shows that the Multi-Scale model performs the best across all metrics, with the highest SSIM value (0.84) and the lowest FID value (28.45). The Pose, Expression, and Shape values are also the lowest, at 2.95, 25.44, and 40.32, respectively. In contrast, the No Multi-Scale model performs slightly less well across all metrics. Although its SSIM value is also high (0.83), close to that of the Multi-Scale model, it falls short in other metrics, particularly in FID (29.14) and Expression (26.11), indicating a slight deficiency in image quality and expression consistency. This suggests that the Transformer plays a crucial role in enhancing feature extraction within the multi-scale model, resulting in superior visual effects and consistency in the generated face-swapped images. Overall, the Multi-Scale model demonstrates better comprehensive performance, proving the significant impact of the Transformer in improving face-swapping effects.
Table 5 shows the performance of different groups in the face-swapping task when using varying numbers of multi-scale models. By comparing the performance of different groups, it is evident that Group C, which uses 4 multi-scale models, performs the best in SSIM, Pose, and Expression metrics, proving the correctness of our choice of 4 multi-scale models. Specifically, Group C has the highest SSIM value, the lowest Pose value, and the lowest Expression value, indicating the best expression retention. In contrast, Groups A and B, which use fewer multi-scale models, improve image quality to some extent but do not achieve the same results as Group C. Groups D and E, while further increasing the number of multi-scale models, do not show performance improvements as significant as those of Group C. In fact, their SSIM and Pose metrics even decrease, possibly due to overfitting or increased computational complexity from too many multi-scale models. This ablation experiment verifies that choosing 4 multi-scale models can maintain high image quality while maximizing pose and expression consistency, demonstrating the superiority and rationality of this configuration.
4.6. Analysis of the Roles of GAN and Transformer
AmazingFT demonstrates exceptional performance in the face-swapping task, particularly in detail handling and feature fusion. From the results of the ablation experiments, it is evident that the face-swapping effect significantly deteriorates when either GAN or Transformer is removed, as shown in
Figure 6.
In the absence of GAN, the generated images exhibit noticeable distortion and blurriness in both details and overall structure. The face-swapping results appear flat and lack detail, with facial features not being sharp enough. The generated images lack texture and realism in detail, and the overall visual effect is inferior to that of the complete AmazingFT model. Without the Transformer, the generated images show issues in feature mapping and fusion, where the features of the source face are not accurately mapped onto the target face, resulting in unnatural and inconsistent facial features. The subtle differences in facial expressions and structures are not well preserved and reproduced, highlighting the critical role of the Transformer in capturing and transforming features.
Using the complete AmazingFT model, the face-swapping results achieve optimal levels in facial detail, feature mapping, and overall visual effect. The details of the source face, such as the eyes and mouth, are accurately preserved and naturally integrated into the target face. The features of the source face are well-presented and fused on the target face, resulting in highly realistic and natural-looking generated images.
4.7. Facial Detail Performance
The synthesized results of the target face and source face appear very natural and exhibit detailed refinement, as shown in
Figure 7. The images demonstrate that the swapped faces maintain the overall facial structure and pose of the target face while successfully incorporating the features of the source face. For instance, in the first set of images, the facial features of the source face, such as the eyes and mouth, are well preserved and presented in the result image, naturally blending with the facial structure of the target face to form a realistic new face.
Close-up details of the eyes and mouth show remarkable refinement. The eye details, such as the eyelids, pupils, and subtle differences in the sclera, are handled very naturally without noticeable edge artifacts. The mouth details are equally well processed, with the shape and texture of the lips being well preserved and seamlessly integrated with the skin tone and texture of the surrounding facial areas. These details highlight the high precision and quality of AmazingFT in handling facial features.
By observing different sets of images, it is evident that regardless of the differences in facial features between the target face and the source face, AmazingFT consistently achieves high-quality face-swapping results while maintaining naturalness and realism. This demonstrates the model’s strong capabilities in feature extraction, feature fusion, and detail processing, enabling it to adapt to various facial features and lighting conditions, thereby generating highly realistic and natural-looking face-swapped images.
4.8. Swapping with the Same Source Faces
By analyzing the face-swapping results of the same source face with different target faces in
Figure 8, it is evident that regardless of facial expressions, lighting conditions, or different target faces, the results consistently maintain the primary features of the source face while naturally blending with the facial structure of the target face. In each image, the main features of the source face (such as the eyes, nose, and mouth) are accurately mapped onto the target face, and the expressions and poses appear very natural, demonstrating the model’s strong capabilities in capturing dynamic features and detail processing.
For example, in the first set of results, the eye and mouth features of the source face are reproduced very realistically on the target face. The second set of results maintains the identity features of the source face while ensuring consistent detail processing across different target faces. This indicates that the model can retain the main features of the source face on various target faces while adapting to different facial structures and expressions.
The third set of results showcases the face-swapping effect of the source face under different lighting conditions. The features of the source face are well-preserved and presented on the target face, particularly with natural handling of lighting and shadows. The fourth set of results demonstrates the adaptability of the source face on different target faces, showing that regardless of the target face’s gender, hairstyle, or facial features, the primary features of the source face naturally integrate with the target face.