Feature Weighted Cycle Generative Adversarial Network with Facial Landmark Recognition and Perceptual Color Distance for Enhanced Face Animation Generation

Abstract

We propose an anime style transfer model to generate anime faces from human face images. We improve the model by modifying the normalization function to obtain more feature information. To make the face feature position of the anime face similar to the human face, we propose facial landmark loss to calculate the error between the generated image and the real human face image. To avoid obvious color deviation in the generated images, we introduced perceptual color loss into the loss function. In addition, due to the lack of reasonable metrics to evaluate the quality of the animated images, we propose the use of Fréchet anime inception distance to calculate the distance between the distribution of the generated animated images and the real animated images in high-dimensional space, so as to understand the quality of the generated animated images. In the user survey, up to 74.46% of users think that the image produced by the proposed method is the best compared with other models. Also, the proposed method reaches a score of 126.05 for Fréchet anime inception distance. Our model performs the best in both user studies and FAID, showing that we have achieved better performance in human visual perception and model distribution. According to the experimental results and user feedback, our proposed method can generate results with better quality compared to existing methods.

1. Introduction

Automatic anime image generation holds several important applications, contributing to both creative and practical domains. Anime images are widely used in various media forms, such as comics, animations, video games, and visual novels. Automatic generation can aid in producing vast amounts of diverse content, reducing production time and costs for these media industries. Anime-style avatars, characters, and illustrations have gained popularity in personal profiles, social media, and online interactions. On YouTube, a new industry called VTuber has emerged, which captures a person’s facial expressions and interacts with the audience through the appearance of an animated character. However, in the production of VTuber characters, it is necessary to manually draw the character first and then use Live2D [1] to set each part of the character, such as the head angle and eye movement position. Then, FaceRig [2] is used to capture the person’s facial features in order to generate corresponding facial animations for the animated character. The purpose of this paper is to use a generative adversarial network in deep learning to achieve anime facial style transfer, thereby reducing the time and cost of creating character models for VTubers. Also, the proposed method allows individuals to customize their digital personas while maintaining a consistent visual style.

In previous methods of image style transfer, Winnemöller et al. [3] used image processing methods to convert images into cartoon styles by detecting edges and setting pixel threshold values manually. Although this method is simple, it can only present simple textures, and it also requires different threshold values to be set manually for different images. Efros et al. [4] proposed using Markov random fields for texture generation. Based on this method, Hertzmann et al. [5] analogized the image style before and after style transfer for other groups to the desired image to obtain new images. Although the style can be specified, it still cannot convert more complex textures well.

With the development of deep learning, related research has begun to apply neural networks to image style transfer. Gatys et al. [6] first proposed the use of minimizing the Gram matrix to make the style of the target image consistent with the style image in the VGG16 [7] feature map by reducing the difference. However, this method requires retraining when inputting new images. Both Johnson et al. [8] and Gatys et al. [6] used VGG16 for feature extraction, but the difference is that Johnson et al. [8] optimized the model instead of the target image, allowing the model to achieve real-time conversion of a single style without the need to retrain with new images.

Isola et al. [9] used generative adversarial networks to achieve better image quality, but this method requires paired style and target images, and data collection needs to be carefully processed. Zhu and Park et al. [10] proposed cycle loss, which requires the original image to be converted back to the original distribution after being converted to the target data distribution and is closer to the input image, allowing image style transfer to learn the transformation of two data distributions without paired image datasets. U-GAT-IT [11], based on CycleGAN [10], added the concept of a class activation map proposed by Zhou et al. [12], weighted the feature map by an auxiliary classifier, and learned the important areas for style transfer. Li et al. [13] proposed adaptive point-wise layer instance normalization to reduce the problem of reducing feature information when combining instance normalization with layer normalization in adaptive instance layer normalization in U-GAT-IT. Zeng et al. [14] used conditional generative adversarial networks to generate high-quality virtual facial animations with facial expressions. The work in [15] also emphasized controlling the expressions of the generated animated faces using their proposed lip-sync landmark animation generation network, emotional landmark animation generation network, and the landmark-to-animation translation network. In [16], the authors translated real-world face images into different artistic styles with reduced computational cost. Yang and Qiu [17] proposed supporting multiple target domains in image conversion and generating images of diverse styles in multiple domains. In [18], cycle consistency was used to facilitate efficient learning and domain adaptation, promoting greater variety and flexibility in expression translation across datasets.

We use U-GAT-IT [11] as the main architecture and enhance its performance in this research. We add adaptive point-wise layer instance normalization in the framework. We propose facial landmark loss based on convolutional pose machines [19], which help the generated images learn the positions of the original facial features. Also, we use perceptual color distance proposed by Zhao et al. [20] to make the generated images more consistent with the original images in color. For the evaluation of anime images, we improved the Fréchet inception distance proposed by Martin et al. [21] and proposed a Fréchet anime inception distance to make the evaluation of generated anime images more reasonable. The main contributions of this work include the following aspects: First, we combine the advantages of different normalization methods through convolution operation to improve the generation quality. Second, we use facial landmark loss to capture and calculate the error between the key point positions of real human faces and generated animated character faces. Third, we use perceptual color loss based on the CIEDE2000 color difference formula to obtain animated images with similar colors as the original image for human eyes. Fourth, we propose a Fréchet anime inception distance to improve the traditional Fréchet inception distance in evaluating animated characters.

2. Methodology

The proposed face animation generation framework is illustrated in Figure 1. The input image goes through an encoder consisting of down-sampling blocks and residual blocks. After the down-sampling blocks encode the input image, it obtains richer feature information and avoids the problem of gradient disappearance through four layers of Resnet blocks. The auxiliary classifier takes the feature map output by the encoder as input and obtains feature vectors through global average pooling and global max pooling. It learns the weight of each channel with a fully connected layer. The output feature maps are multiplied with the output by the original encoder to obtain the attention feature map, which serves as the input of the decoder. The decoder consists of residual blocks and up-sampling blocks. In order to learn the normalization parameters, the attention feature map will obtain γ and β through the fully connected layer. The normalization method used in the residual blocks is adaptive point-wise layer instance normalization (AdaPoLIN) [13]. Finally, the animation image is generated through up-sampling blocks.

2.1. Network Architecture

We use U-GAT-IT [11] as our base network model. In the U-GAT-IT experiment, some generated images had obvious differences in color tone compared to the original images, and others produced blurry results. This paper aims to address these two issues.

To improve the blurry results, we proposed the facial landmark loss, which utilizes a pre-trained convolutional pose machine [19] model to help the generated model learn the facial landmark positions for face transformation. To address the color tone differences in the generated images, we added the perceptual color distance [20] based on CIEDE2000 [22] as a loss function to obtain images that are more consistent with human perception. Finally, we used adaptive point-wise layer instance normalization (AdaPoLIN) [13] to generate images with better quality in terms of color and texture.

We use the PatchGAN discriminator [9] as the discriminator, which consists of five and seven layers of convolutional layers to form two models that output matrices of different sizes to determine the authenticity under different receptive fields. The full loss function can be formulated as Equation (1).

$$\begin{array}{l}{L}_{total}={\lambda }_{LSGAN}{L}_{LSGAN}^{s\to t}+{\lambda }_{Facial\_Landmark\_Loss}{L}_{Facial\_Landmark\_Loss}+{\lambda }_{perc}{L}_{perc}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\lambda }_{cycle}{L}_{cycle}+{\lambda }_{dentity}{L}_{Identity}+{\lambda }_{CAM}{L}_{CAM}\end{array}$$

(1)

In Equation (1), $L_{LSGAN}$, $L_{cycle}$, $L_{Identity}$, and $L_{CAM}$ represent the adversarial loss, cycle loss, identity loss, and class activation map loss, respectively. These losses are described in Equations (2)–(5).

$$L_{LSGAN}^{s\to t}=E_{x~X_t}\left[D_{t-5}{\left(x\right)}^2\right]+ E_{x~X_s}\left[{\left(1-D_{t-5}\left(G_{s\to t}\left(x\right)\right)\right)}^2\right]+E_{x~X_t}\left[D_{t-7}{\left(x\right)}^2\right]+ E_{x~X_s}\left[{\left(1-D_{t-7}\left(G_{s\to t}\left(x\right)\right)\right)}^2\right]$$

(2)

$$L_{cycle}^{s\to t}=E_{x\sim Xs}\left[{|x-G_{t\to s}(G_{s\to t}\left(x\right)\left)\right)|}_1\right]$$

(3)

$$L_{identity}^{s\to t}=E_{x\sim Xt}\left[{|x-G_{s\to t}(x\left)\right|}_1\right]$$

(4)

$$\begin{array}{l}{L}_{CAM}=E_{x~X_t}\left[{\eta }_{D_{t-5}}{\left(x\right)}^2\right]+\mathrm{ }{E}_{\mathrm{x}~X_s}\left[{\left(1-{\eta }_{D_{t-5}}\left(G_{s\to t}\left(x\right)\right)\right)}^2\right]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+E_{x~X_t}\left[{\eta }_{D_{t-7}}{\left(x\right)}^2\right]+\mathrm{ }{E}_{\mathrm{x}~X_s}\left[{\left(1-{\eta }_{D_{t-7}}\left(G_{s\to t}\left(x\right)\right)\right)}^2\right]+{[-(E}_{x~X_s}\left[{\mathrm{l}\mathrm{o}\mathrm{g}(\eta }_s\left(x\right))\right]+\mathrm{ }{E}_{\mathrm{x}~X_t}\left[log\left(1-{\eta }_s\left(x\right)\right)\right]\left)\right]\end{array}$$

(5)

where $X_t$ represents the target domain and $X_s$ represents the source domain. $D_{t-5}$ and $D_{t-7}$ denote the PatchGAN discriminator with five layers and seven layers, respectively. $G_{s\to t}$ denotes the generator from the source to the target domain. In (5), ${\eta }_s$ represents the auxiliary classifier of the generator, while ${\eta }_{D_{t-5}}$ and ${\eta }_{D_{t-7}}$ represent the auxiliary classifiers of discriminators with five layers and seven layers, respectively.

In Equation (1), we set the parameters as ${\lambda }_{LSGAN}=1,{\lambda }_{cycle}=10,{\lambda }_{dentity}=10,\mathrm{a}\mathrm{n}\mathrm{d}{ \lambda }_{CAM}=1000$. These parameters are selected according to [11]. The parameters ${\lambda }_{Facial\_Landmark\_Loss}$ and ${ \lambda }_{perc}$ are set as ${\lambda }_{Facial\_Landmark\_Loss}=5,$ and ${ \lambda }_{perc}=8$ empirically after testing several combinations of weights to balance the influence of facial landmark loss, perceptual color loss, and the other terms in the full loss function.

2.2. Facial Landmark Loss

To make the positions of facial features match before and after conversion, we propose using two pre-trained models to obtain the facial landmarks of the generated and original images. By minimizing the distance between the two, we help the generator learn the position of the facial features.

Real human faces use the facial landmarks detected by the convolutional pose machines pre-trained on the AFLW [23] dataset. For animated faces, we use the convolutional pose machines pre-trained on the animated face dataset. Both models are open-sourced on GitHub [24,25].

Since the facial features of animated faces have significant structural differences from those of real human faces, we only retain four coordinates corresponding to the left eye center, right eye center, nose center, and mouth center in the output of the convolutional pose machines model to avoid generating facial features that do not fit the data distribution, such as eye corners or lips. After obtaining the facial landmarks of the generated and original images, we use Smooth L1 Loss to calculate the coordinates of the corresponding facial features, making the positions of the facial features in the generated animated image close to those in the original image, as shown in Figure 2. The facial landmark loss can be formulated as Equation (6), and the $threshold$ is set as 1. In Equation (6), the term $L\left(G_{s\to t}\left(x\right)\right)$ indicates the key point locations of the generated image, and $L\left(x\right)$ indicates the key point position of the original image.

$$L_{Facial\_Landmark\_Loss}=\left\{\begin{array}{c}{\displaystyle \frac{0.5{\left(L\left(G_{s\to t}\left(x\right)\right)-L\left(x\right)\right)}^2}{threshold}}, \left|L\left(G_{s\to t}\left(x\right)\right)-L\left(x\right)\right|<threshold\\ \left|L\left(G_{s\to t}\left(x\right)\right)-L\left(x\right)\right|-0.5\ast beta, otherwise\end{array}\right.$$

(6)

2.3. Perceptual Color Loss

When calculating the difference between colors, the most commonly used method is the Euclidean distance between two points in the RGB space. However, the perception of different colors by human eyes is not consistent, and some colors are more sensitive to changes, as shown in Figure 3. In Figure 3, the three colors in RGB space are (0, 0, 255), (50, 50, 255), and (50, 0, 205). The distance between (0, 0, 255) and (50, 50, 255) is 100. The distance between (50, 50, 255) and (50, 0, 205) is also 100. Although the two distances in the RGB color space are the same, human eyes would perceive that the color (50, 50, 255) is closer to (0, 0, 255) than (50, 0, 205). The aforementioned method assumes that the impact of the variation of the three colors on human perception is the same, which does not conform to human visual perception. In anime face style transfer, we hope that the generated image can maintain the color distribution of the original image, such as skin tone, hair color, and background color, while changing the style. However, U-GAT-IT generates images that have significant differences in color compared to the original images in Figure 4. To address this issue, we use the CIEDE2000 [22] (G. Sharma, W. Wu, and E. N. Dalal, 2005) color difference formula to obtain generated images that are more in line with human visual perception of color.

CIEDE2000 was proposed by the International Commission on Illumination. It calculates by converting colors from the RGB color space to the CIELAB color space first and then to the CIEL×C×H× color space using Equation (7), as follows:

$$CIEDE2000(\left(L_1^{\mathrm{*}},a_1^{\mathrm{*}},b_1^{\mathrm{*}}\right),(L_2^{\mathrm{*}},a_2^{\mathrm{*}},b_2^{\mathrm{*}}))=\sqrt{{\left({\displaystyle \frac{∆L^{\prime }}{k_L{S}_L}}\right)}^2+{\left({\displaystyle \frac{∆C^{\prime }}{k_C{S}_C}}\right)}^2+{\left({\displaystyle \frac{∆H^{\prime }}{k_H{S}_H}}\right)}^2+R_T{\displaystyle \frac{∆C^{\prime }}{k_C{S}_C}}{\displaystyle \frac{∆H^{\prime }}{k_H{S}_H}}}$$

(7)

where ${\left\{L_1^{\ast },a_1^{\ast },b_1^{\ast }\right\} }_{i=1}^2$ are the representation of two colors in the CIELAB color space and $∆L^{\prime },∆C^{\prime },\mathrm{a}\mathrm{n}\mathrm{d}∆H^{\prime }$ are the differences between two colors in terms of lightness, chroma, and hue, respectively. The rotation term $R_T$ is added to solve the problem of inaccurate calculations in the blue and violet areas. The details of the weighting factors, $S_L, S_C, \mathrm{a}\mathrm{n}\mathrm{d} S_H$, and parametric factors, $k_L,k_C,\mathrm{a}\mathrm{n}\mathrm{d}{k}_H$, can be found in [22].

To improve the color deviation issue of U-GAT-IT, we introduce the differentiable CIEDE2000 formula proposed in [20] as the loss function to calculate the color difference between source images and generated images, making them more similar in human perception. The $L_{perc}$ can be formulated as Equation (8).

$$L_{perc}={\displaystyle \frac{1}{H\ast W\ast 100}}{\sum }_{y=1}^H{\sum }_{x=1}^{W}CIEDE2000(s, G_{s\to t}\left(s\right))$$

(8)

2.4. Adaptive Point-Wise Layer Instance Normalization (AdaPoLIN)

In Figure 5a, batch normalization [26] calculates the mean and variance of the same channel for all feature maps. Although it can obtain the data distribution information in the batch, it loses the features of a single image. In addition, the bias is generated when the sampling size is not large enough or the data distribution in the dataset is not uniform. These two problems will affect the generated image and not meet the goal of style transfer. In Figure 5b, layer normalization [27] calculates the mean and variance of all channels of a single input image, reducing the problem of data distribution in batch sampling in batch normalization. In Figure 5c, instance normalization [28] normalizes a single channel of the feature map and can retain more details of image features compared to batch normalization. Therefore, it is widely used in style transfer-related tasks.

U-GAT-IT proposes to combine instance normalization and layer normalization by training a ρ between zero and one to obtain the advantages of both methods. However, this combination does not consider that each channel represents different content, such as color, shape, and texture, resulting in information loss due to multiplication by ρ.

AniGAN [13] proposes adaptive point-wise layer instance normalization (AdaPoLIN), which concatenates instance normalization and layer normalization by channel before convolution. The parameters $\gamma$ and $\beta$ are learned by the fully connected layer. This method obtains feature maps with the same dimensions without reducing feature information. The AdaPoLIN can be formulated as Equation (9).

$$AdaPoLIN\left(z, \gamma , \beta \right)=\gamma ·Conv\left(\left[{\displaystyle \frac{z-{\mu }_I\left(z\right)}{{\sigma }_I\left(z\right)}} , {\displaystyle \frac{z-{\mu }_L\left(z\right)}{{\sigma }_L\left(z\right)}}\right]\right)+\beta$$

(9)

In Equation (9), $z$ represents the input feature map. The terms ${\mu }_I\left(z\right)$ and ${\sigma }_I\left(z\right)$ are the mean and standard deviation of instance normalization, respectively, while ${\mu }_L\left(z\right)$ and ${\sigma }_L\left(z\right)$ are the mean and standard deviation of layer normalization, respectively.

3. Experimental Results

We trained and evaluated our model on the Selfie2Anime dataset [11]. For better evaluation of anime images, we fine-tuned the InceptionV3 of FID [21] using the Caltech-256 Object Category dataset [29] and subsets of the Danbooru dataset [30] as training images for new categories. The hardware specification and software environment for the experiments are listed in

Table 1. Hardware specification for experiments.

Device	Specification
CPU	Intel® Core™ i9-12900 K @ 3.20 GHz
GPU	NVIDIA Geforce RTX 3090
CPU Memory	64 GB
GPU Memory	24 GB

and

Table 2. Software environment for experiments.

Python	3.8.13
Pytorch	1.13.0
Opencv-python	3.4.11
scikit-learn	0.23.2
OS	Ubuntu 18.04 LTS

, respectively. The optimizer used in the experiments is the Adam optimizer. The initial learning rate was set as 0.0001 for both the generator and discriminator.

In the field of image generation, quality is often measured using the Fréchet inception distance (FID) [21] metric. This metric calculates the distance between the mean and covariance matrix of the feature maps of the generated images and the real images at the intermediate layer of the InceptionV3 network to measure their similarity. However, this metric is based on the feature extraction model (InceptionV3) to obtain high-level features for comparison. The problem is that although Imagenet [31] has thousands of categories, if the input image data do not belong to any of these categories, the model will not be able to determine which category the image belongs to. In Imagenet, there are no data on anime categories, so the original FID cannot be reasonably used as an evaluation metric for our task.

To address this issue, in order to reduce computing resources while increasing animation categories, we fine-tuned the pre-trained inceptionV3 model on the Caltech-256 Object Category dataset, with subsets of Danbooru added. The dataset we used for this purpose contains 258 categories, with 27,726 training data and 3081 testing data. When fine-tuning the model, the training process ran for 16 epochs and achieved 87.8611% accuracy on the testing dataset. We calculated the output feature map of this new model that evaluated anime images. The FID formula was used, and we named it the Fréchet anime inception distance (FAID). To validate its effectiveness, we added real human facial images to a dataset with 100 animated images. Starting from 10, we increased the number by 5 each time. If a metric could distinguish between anime faces and real human faces, its value should have gradually increased to reflect the mixture of different categories in the dataset. As shown in Figure 6, the FAID gradually increased as the number of real human faces increased, while the original FID decreased when 10 or 15 images were added. Also, the FID was smaller at 25 images than at 20 images. This shows that our proposed FAID can more reasonably distinguish between animated and real human faces compared to FID.

For face style transfer, in order to ensure that the color distribution of the transformed image is similar to that of the original image, we used the perceptual color distance (PCD) in Equation (8) as the metric between the original image and the generated image. The smaller the value of this metric, the smaller the color difference between the two images, indicating that the generated image is closer to the original image in terms of human perception.

In order to evaluate whether the generative model simply converts real human faces to animated faces without generating animated faces with facial features in the same position, we used the facial landmark loss in Equation (6) as our evaluation metric, calculating the difference in facial landmarks using L1 loss. The smaller the difference, the closer the facial feature positions are before and after the conversion, indicating that the generated animated faces are more similar to the original real human faces.

Our survey compared the images generated by four models, the model proposed by us, U-GAT-IT [11], ACLGAN [32], and Council-GAN [33], with each model generating 50 images. A total of 13 people participated in the survey, resulting in 650 votes. We calculated the percentage of total votes for each option to evaluate users’ preferences for images generated by different models. This metric served as the main criterion for comparing our models and others.

Figure 7 shows the qualitative comparison results of our model with other models. It can be seen that in terms of generated quality, our model and U-GAT-IT can both generate realistic animated faces, while ACLGAN and Council-GAN have obvious primitive facial lines and features, meaning that our proposed model has better performance in style transfer. As for the color deviation issue, it can be seen that our model generates images that are closer in color to the original image, while U-GAT-IT has obvious color deviation problems. Similarly, it can also be seen that our model is closer to the original image in terms of facial features compared to U-GAT-IT, indicating that our proposed method can effectively improve the generated quality of U-GAT-IT.

Table 3. Quantitative comparison results of the Selfie2Anime testing dataset with U-GAT-IT, Council-GAN, ACLGAN, and our method (↓ lower is better; ↑ higher is better).

Metrics	Methods
	U-GAT-IT	Council-GAN	ACLGAN	Ours
User ↑	11.85%	4.77%	8.92%	74.46%
FAID ↓	150.65	126.55	208.40	126.05
PCD ↓	22.74	22.78	15.82	16.45
Land ↓	6.93	6.57	6.03	6.28

shows the quantitative results of images generated by different models. All four models listed in Table 3 were trained on the Selfie2Anime dataset with an image size of 128. Among these, UGATIT and ACLGAN were trained with the parameters provided in the papers, while Council-GAN used the model weights publicly available in the reference.

Our model performed the best in both the user study and FAID, showing that we had better performance in human visual perception and model distribution. In terms of color distance (PCD) and facial landmark distance (Land), we ranked second in performance metrics, which can be seen in Table 3. However, these two metrics only measure the similarity between the generated anime images and the original input face images. They cannot always accurately reflect the quality of the generated anime images. Due to the obvious facial features of ACLGAN, especially the eye features, the skin around the eye sockets remained the same. On the contrary, the output of our model demonstrated a significant change in eye shape. This led to a lower score on the color difference metric in that area. Nevertheless, it is obvious that the quality of images generated by our method is significantly better, as shown in Figure 7. Therefore, we believe that these two metrics should be used to evaluate the improvement of the model that can already generate images with decent quality, like U-GAT-IT and the proposed method. As Table 3 shows, we have made significant progress on these scores compared to U-GAT-IT.

Table 4. Quantitative results of ablation study (↓ lower is better; ↑ higher is better).

Facial Landmark Loss	Perceptual Color Loss	AdaPoLIN	FAID ↓	PCD ↓	Land ↓
			150.65	22.74	6.93
v			136.72	22.37	6.41
	v		132.85	18.16	6.67
		v	130.73	22.58	6.92
v	v		132.19	16.93	6.36
v	v	v	126.05	16.45	6.28

presents the ablation study that evaluated the impact of each component, including facial landmark loss, perceptual color loss, and AdaPoLIN normalization. Through the experimental results listed in Table 4, we can observe the specific contribution and improvement of each individual component. The original method sometimes failed to correctly generate the facial features of some animated faces. However, this problem was alleviated after adding the facial landmark loss. Our method can indeed help the model learn the positions of facial features in real-person images and generate corresponding facial features in animated images. After adding the perceptual color loss, the color deviation problem of the original method was greatly improved, especially for skin color, hair color, and other areas related to characters. We can see that although adding AdaPoLIN did not help in color preservation and facial feature position, it retained more features than the original method, such as hairstyle and facial expressions. The FAID value was also higher, and the generation quality was better than the original method.

4. Conclusions

We propose a facial image style transfer model based on U-GAT-IT. By combining the advantages of convolutional operation with instance normalization and layer normalization, the model improves the quality of generated images. The facial landmark loss is used to extract the facial landmarks of real human faces and generate animated characters. The error between the two is calculated to help the model generate images with facial features that are more in line with real human faces. In terms of image color, the perceptual color loss based on the CIEDE2000 color difference formula is used to obtain animated images that are visually similar to the original human images. To better evaluate animated characters, we propose the Fréchet anime inception distance to improve the shortcomings of the Fréchet inception distance in evaluating animated characters.

According to the qualitative results, our proposed model can improve the color deviation and inaccurate facial feature positioning issues in U-GAT-IT and generate images that are significantly better than those generated by ACLGAN and Council-GAN. In quantitative experiments, we conducted user surveys to obtain subjective feedback and found that the generated images of our model were more popular among the audience. In addition, the model performs better than our baseline model, U-GAT-IT, in both perceptual color distance and facial landmark distance. However, the proposed method still has some limitations. One of the limitations is that it relies on pre-trained models, such as convolutional pose machines, for facial landmark detection. If the model has biases or inaccuracies in detecting landmarks, the generated image is affected. The second limitation is that there are very few male characters in the dataset for the current system. Therefore, the generation results of male characters are not as good as those of female characters. Adding images with more male characters into the training data could help to resolve this problem.

For future studies, we suggest the following directions: First, since the painting styles of animated characters are very diverse, we believe that after collecting more relevant information, we can improve the ability of the model to identify images of different painting styles to make the value of Fréchet anime inception distance more reasonable. Second, by adding facial semantic segmentation features during model training, more diverse and realistic results can be obtained in the generated images. Third, we believe that by adding datasets with labels, such as glasses, hair color, and other characteristics, and modifying the generation space, we can control the characteristics of the generated characters and achieve the effect of image editing.