Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait

Abstract

With the continuous development of cross-modality generation, audio-driven talking face generation has made substantial advances in terms of speech content and mouth shape, but existing research on talking face emotion generation is still relatively unsophisticated. In this work, we present Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait to synthesize lip-sync and an emotionally controllable high-quality talking face. Specifically, we take a facial reenactment perspective, using facial landmarks as an intermediate representation driving the expression generation of talking faces through the landmark features of an arbitrary emotional portrait. Meanwhile, decoupled design ideas are used to divide the model into three sub-networks to improve emotion control. They are the lip-sync landmark animation generation network, the emotional landmark animation generation network, and the landmark-to-animation translation network. The two landmark animation generation networks are responsible for generating content-related lip area landmarks and facial expression landmarks to correct the landmark sequences of the target portrait. Following this, the corrected landmark sequences and the target portrait are fed into the translation network to generate an emotionally controllable talking face. Our method controls the expressions of talking faces by driving the emotional portrait images while ensuring the generation of animated lip-sync, and can handle new audio and portraits not seen during training. A multi-perspective user study and extensive quantitative and qualitative evaluations demonstrate the superiority of the system in terms of visual emotion representation and video authenticity.

1. Introduction

With the rapid growth in demand for virtual humans [1,2,3], talking face generation has become a hot spot for current research [4,5,6]. The goal of talking face generation is to synthesize realistic, a lip-synced portrait animation from one (or a few) static portrait image(s) and an audio (or motion) sequence [7]. This task has a wide range of applications in multiple fields through simulated visual avatars, such as treatment systems for phantom vision and hearing, virtual anchors, and character customization in games.

Research in talking face animation generation is making good progress as large-scale audiovisual datasets [8,9,10] are becoming more and more available. However, most existing studies have focused on realistic lip syncs [5,11,12,13,14] and head-pose movements [15,16], and research on conveying the emotional aspect, which is a vital ingredient of authenticity in communication, lacks maturity.

Some previous studies have attempted to generate emotionally controlled portrait animations [17,18,19,20]. Sadoughi et al. [18] and Eskimez et al. [20] controlled the expression of the generated animation by means of additional emotion labels, but fixed labels can only roughly express a limited number of expressions. Fang et al. [19] used audio to directly control the emotion of the generated animation, resulting in results with lower visual quality. Building on this, the state-of-the-art method proposed by Ji et al. [17] was used to generate emotional animations while ensuring the high fidelity of the animation. However, due to the complexity of audio emotion itself, determining emotion in animation can lead to ambiguity. Furthermore, for the generation of the emotional animations studied above, the target portrait must be trained by the model, which results in its having a limited ability to render emotions and more limited application scenarios [21].

In this study, we propose Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait, a method that selects portrait images as an emotion-driven source and inputs audio and target identity portraits to generate emotionally diverse facial animations. Meanwhile, facial landmarks are used as intermediate representatives, dividing the model into an audio-to-landmark network and a landmark-to-animation network. Facial landmarks connect audio to pixels and have the following advantages for facial emotion animation generation tasks. On the one hand, facial geometry information is effectively maintained while deleting millions of pixels, resulting in a natural low-dimensional representation for the portrait. This allows the model to capture subtle dynamic landmark displacements of the head, avoiding the mishmash between portrait outlines and background pixels that miss these details. On the other hand, landmark features can convey sufficient emotional information for downstream emotional reenactment tasks and are more structured than intermediate representations such as action units (AU) [22].

In addition, to make the animation more emotionally controllable, we use the design idea of decoupling to divide the audio-to-landmark network into two sub-networks for the prediction of lip-sync landmarks and facial emotion landmarks. Firstly, the prediction of lip-sync landmarks is handled by the lip-sync landmark animation generation network, which is based on MakItTalk [15], by decoupling the audio content in order to improve accuracy. Secondly, for the prediction of facial emotion landmarks, an additional input emotion source portrait is used, which can be derived from an untrained arbitrary portrait. Driven by the co-driving of arbitrary emotional source portraits and audio, our emotional landmark animation generation network effectively translates the emotion to the target portrait frame by frame. Finally, the landmark-to-animation translation network was used to generate an emotionally controllable talking face. Compared to the label and audio-driven emotion methods, our method is more flexible in its ability to control the facial emotions in the animation, and the video covers more expressions while ensuring lip-sync and visual clarity. The advanced performance of our method and its key components is demonstrated through qualitative and quantitative experiments.

In summary, our approach allows the generation of emotionally controlled talking faces through two unseen portraits and one unseen audio sequence, a simplified diagram of which is shown in Figure 1. The contributions of our work can be summarized as follows: (a) We propose an emotionally controllable facial animation generation method based on facial landmarks, using uniform low-dimensional facial landmarks as intermediate variables to avoid the training dependency problem of traditional emotionally controllable talking faces. (b) We adopt a decoupled design idea to divide the facial animation generation task into a lip-sync task and a facial emotion task in order to avoid generating landmarks jumbled between the lip region and facial expressions. (c) We enriched the emotional coverage of the portrait animation with additional emotion sources to reenact the expressions of the target portrait, making it more versatile and flexible for practical applications.

2. Related Work

2.1. Audio-Driven Talking Face Generation

Audio-driven facial animation generation is a popular topic in computer vision and in generative models. Its purpose is to transform a static portrait into a realistic talking face animation through arbitrary audio. Audio-driven facial animation does not require as many video clips of the target individual for training as graphics-based portrait facial animation to maintain the phonemic information of the target [7,23,24]. The methods used by previous researchers for audio-driven facial animation can be divided into two groups based on the presence or absence of an intermediate structural representation.

Direct Image Reconstruction Methods. Raw pixels or dots with lip movements synchronized with the speech signal are directly generated through the audio driver. With the development of audiovisual cross-modals, Chung et al. [11] proposed to generate lip-synced videos by the encoder–decoder CNN model named Speech2Vid. Song et al. [12] used conditional recursive generative networks to combine image and audio features in recursive units to achieve temporal dependence across animation frames. Prajwal et al. [14] developed a new lip-sync discriminator that enhances the robustness of lip synchronization in the wild. Zhou et al. [16] successfully added head motion to the generated animation by defining a 12-dimensional pose encoding in latent space. The above methods focus on matching lip movements to audio, but they were unable to control the video emotion.

More recently, a few methods [10,19,20,21,25] have added an emotional element to the generated talking face. Karras et al. [25] suggested an end-to-end network with a more realistic emotional component using a 3D-based method; however, the learned emotional states are difficult to comprehend and do not model facial features. Fang et al. [19] used a generative adversarial network (GAN) to directly synthesize animations from audio and pixels to express the emotions conveyed by audio; however, their results suffered from fidelity problems (facial distortion, blurring, and other artefacts). Recently, Eskimez et al. [20] developed an end-to-end emotionally controllable conversation face generation system using speech, individual face images, and categorized sentiment labels as inputs. Although the implementation idea is innovative, emotion control is affected by the classification expressions, and it can only cover a limited range of expressions. In addition, some studies have opted to add additional visual input to control emotion. Wang et al. [10] could only achieve a transition from neutral to emotional; Liang et al. [21] controlled head pose and facial expression through multiple visual inputs, but this also resulted in a model with low robustness to complex backgrounds.

Intermediate Representation Method. In the portrait reconstruction pipeline, structural information was employed as an intermediate representation to help the model learn the features of the target. The neural network of Eskimez et al. [26] can generate talking face landmarks from arbitrarily long noisy speeches. Chen et al. [13] divided the process into two stages: an audio-to-landmark network and a landmark-to-animation network based on facial landmarks. This cascade method avoids fitting erroneous audiovisual signal correlations that are unrelated to speech content. Zhou et al. [15] used decoupled learning to separate speech into content and speaker identity components trained separately, whereby spontaneous head posture movement is predicted after successfully completing the lip-sync task. In addition to facial landmarks, previous approaches have also used other intermediate representations to facilitate animated synthesis, such as the approach of Sadoughi et al. [18], who completed emotional talker face synthesis of the lip region via markers from the IEMOCAP corpus, and that of Chen et al. [27] who completed talker face synthesis using action units from the facial action coding system (FACS) [22].

Compared to direct methods for generating talking face, facial emotion animation based on intermediate representatives has been less studied. Recently, a state-of-the-art method was proposed by Ji et al. [17] using the cross-reconstructed emotion disentanglement technique to decompose speech into two decoupled spaces and then using the audio-to-landmark network to generate landmarks for emotional animations. However, this method can only be applied to seen portraits and audio, with weak emotion generalization. In contrast to this approach, our method captures emotion from arbitrary portraits, which can be used for unseen audio and portraits, allowing more flexibility in generating emotions for the talking face.

2.2. Facial Expression Reenactment

Significant progress in face generation and editing [28,29,30] has been made in recent years through advanced image-to-image translation [31,32]. Facial emotional expression reproduction is more challenging than facial attribute editing, which only modifies the appearance of specific facial areas. Facial expression involves the movement of several facial muscles, involving a large amount of geometric information. In the ExprGAN proposed by Ding et al. [33], an expression controller module is designed to learn expressions so that the expression intensity can be controlled during facial expression recognition. Pumarola et al. [34] proposed GANimation using action units (AU) as emotion labels, allowing the intensity of each AU to be controlled individually. This solution can reenact facial emotional expressions in a continuous domain while preserving identity information. Wu et al. [35] used a cascading transformation strategy to perform progressive facial expressions to reenact the local expression focus to better preserve identity-related features and details around the eyes, nose, and mouth, thereby improving the fidelity of the generated image.

On the other hand, there are some studies [36,37,38] that introduce facial landmarks as intermediate representatives to accomplish the task of facial expression reenactment. Qian et al. [36] used contrastive learning to transform geometric manifolds into embedded semantic manifolds of facial expressions to accomplish a cross-identity emotion transfer. Zhang et al. [37] proposed a unified landmark converter to bridge the source and target images via facial landmarks with geometric information. A new triple perceptual loss function was reapplied to enrich the facial details of the image by allowing the model to learn both appearance and geometric information. Liu et al. [38] also used facial landmark transformation, but added a face rotation module and an expression enhancement generator to solve the entanglement problem of expression and pose features. Inspired by the facial landmark-based reenactment of facial expressions, in this study, another portrait is input into the network for emotion generation. The combined operation of the two portraits and audio results in a talking face that is emotionally rich and realistic. Unlike in traditional facial expression reenactment, our method needs to ensure the identity of the target in each frame while warping the geometric information of the video frame by frame to make the video smoother and more natural, which is undoubtedly more difficult for expression reenactment.

3. Method

3.1. Overview

Figure 2 shows a schematic of the general structure of the proposed method, which accepts as input audio and two static portrait images (one representing the identity source and another representing the emotion source) and outputs a lip-sync, spontaneous head movement, and identity source portrait animation with emotion source expression. Facial landmarks are used as intermediary representations in this network to bridge the gap between audio and visual features. The network can be broken down into three main network components: the lip-sync landmark animation generation network; the emotional landmark animation generation network; and the landmark-to-animation translation network. The lip-sync landmark animation generation network is used to generate the speech-content landmark sequence from identity source facial landmarks and audio content (see Section 3.2). The emotional landmark animation generation network reenacts the facial landmarks of the identity source portrait, frame by frame, with respect to an arbitrary emotion source portrait and audio, to create landmark displacements with emotion and head movement (see Section 3.3). The landmark-to-animation translation network is an image-to-image translation network that inputs the target portraits and landmarks predicted by the above network to generate an emotionally controllable talking face (see Section 3.4). In the following sections, the modules of the proposed architecture are described.

3.2. Lip-Sync Landmark Animation Generation Network

The goal of the lip-sync landmark animation generation network is to generate the target portrait lip region landmark animation synchronized with audio content, which requires mapping the audio to the facial landmark locations. This network uses an encoder–decoder network structure containing a multilayer perceptron (MLP) encoder, long short-term memory (LSTM) [39] encoder, and an MLP decoder. Content-independent features in audio reduce the prediction accuracy to a certain degree. Therefore, adopting the idea in [15], disentangled learning in the field of voice conversion using AutoVC [40] was used to design a state-of-the-art approach to disentangle audio into content embedding and style embedding. The speaker-independent content embedding $Ec\in R^{T\times C}$ was separated from audio $V$, where $T$ is the total number of input audio frames ($t$ denotes any frame), and $C$ is the content dimension. In addition to content embedding, a landmark embedding vector is also extremely important, providing facial shape and identity information for this network. To this end, a 3D landmark detector [41] is first used to create a landmark $L_a\in R^{68\times 3}$, followed by an MLP encoder to extract the landmark feature from $L_a$. The deconstructed content embedding with landmark feature is then fed into an LSTM encoder that captures the audio content and continuous lip landmark dependencies. Finally, the MLP decoder accepts the output of the LSTM encoder and predicts the lip landmark displacement $\mathsf{\Delta }{P}_t$, which is summed with the original landmarks to obtain the lip-sync landmark sequence, $P_t$. Mathematically, this network can be expressed as follows:

$$\mathsf{\Delta }{P}_t=ML{P}_c\left(LST{M}_c\left(E{c}_{t\to t+\lambda },ML{P}_L\left(L_a;W_{mlp,l}\right);W_{lstm}\right);W_{mlp,c}\right)$$

(1)

$$P_t=L_a+\mathsf{\Delta }{P}_t$$

(2)

where {$W_{mlp,l},W_{lstm},W_{mlp,c}$} are the learnable parameters of the MLP encoder, LSTM encoder, and MLP decoder, respectively; the MLP encoder $ML{P}_L$ consists of two layers with internal hidden state vectors of size 256 and 128; the LSTM encoder $LST{M}_c$ is made up of three layers of units, each with a hidden state vector of 256; and the MLP decoder $ML{P}_c$ comprises three layers with internal hidden state vectors of size 512, 256, and 204 (68 × 3); {$Ec,L_a,\mathsf{\Delta }{P}_t,P_t$} is consistent with the above descriptions; for content embedding $Ec$, the window size for each frame $t$ is $\lambda =$ 18.

To improve the realism of the lip landmark displacement, we used the loss function ${ℒ}_C$ to minimize the distance between the reference landmark ${\widehat{P}}_t$ and the predicted landmark $P_t$(where $i$ is the index of each individual landmark):

$${ℒ}_L={\displaystyle \sum _{t=1}^T{\displaystyle \sum _{i=1}^N}}{\Vert P_{i,t}-{\widehat{P}}_{i,t}\Vert }_2^2$$

(3)

3.3. Emotional Landmark Animation Generation Network

A realistic talking face must include lips that match the audio content as well as facial expressions and the head movements. To make the generated animations richer in emotion, we propose to extract emotional features from the expressions of arbitrary portrait landmarks instead of speech signals, as depicted in Figure 2.

With the input of the emotion source portrait, the audio and emotional conditions can be separated, and the model can efficiently transform emotions from any portrait to the target portrait, allowing the emotions to be controlled when synthesizing the animation. The visual modality inputs to the emotional landmark animation generation network consist of the same identity source landmark $L_a\in R^{68\times 3}$ as the lip-sync landmark animation generation network and an arbitrary emotion source landmark $L_b\in R^{68\times 3}$. The input of the network auditory modality then uses the content embedding $Ec$ and the style embedding $Es$ isolated from AutoVC [40], where style embedding is a feature of the speaker. The reason for using a speech conversion neural network to decouple audio is that there is a geometric relationship between the discourse content and the speaker, notably because the mouth shape for the same sentence varies somewhat from person to person, not to mention features such as eyebrows and eyes. Our network consisted of an audio encoder (${\varphi }_s$), two face landmark encoders (${\varphi }_{e1}$ and ${\varphi }_{e2}$), and a face landmark decoder (${\psi }_{\mathrm{e}}$).

To capture the effect of audio on facial expressions, the audio encoder ${\varphi }_s$ was modeled after the approach of the MakeItTalk [15] model component. Unlike MakeItTalk, MLP was not used to conduct direct landmark prediction on the acquired audio features, which were instead passed on to a downstream task for multimodal fusion. Consequently, the visual and auditory modal data complement and corroborate each other, resulting in high-quality portrait animation with emotion. First, the content embedding $Ec$ is fed into an LSTM with the same network structure and time window as the lip-sync network, except for the input size. Second, the style embedding $Es$ goes through the model of [42] to maximize the similarity of embeddings between different discourses of the same speaker and to minimize the similarity between different speakers. The internal hidden state vector sizes were 256, 128, and 128. Finally, the above output is assembled in a self-attention layer [43] to capture the longer structured dependencies because the head action lasts much longer than tens of milliseconds of phonemes. The extraction of audio features can be formulated as

$$S_t=Attn\left(LST{M}_{c^{\prime }}\left(E{c}_{t\to t+\lambda };{W^{\prime }}_{lstm}\right),ML{P}_s\left(Es;W_{mlp,s}\right);W_{attn}\right)$$

(4)

where {${W^{\prime }}_{lstm},W_{mlp,s},W_{attn}$} are the learnable parameters of the network; and {$LST{M}_{c^{\prime }},ML{P}_s,Attn$} are the models described above; {$Ec,Es$} are consistent with those described above. The output is the feature value $S_t$; the length is the total number of input audio frames; and the width is 64.

Encoders ${\varphi }_{e1}$ and ${\varphi }_{e2}$ are assigned the tasks of extracting landmark features of the target portrait landmark $L_a$ and emotional portrait landmark $L_b$. Both have similar structures and are lightweight neural networks consisting of seven-layer MLP, but their functions are not the same. The encoder ${\varphi }_{e1}$ primarily extracts geographic information $F_a$ from the identity and position of $L_a$, whereas encoder ${\varphi }_{e2}$ primarily extracts geographic information $F_b$ from the facial expressions of $L_b$. Subsequently, the three extracted features are linearly fused to obtain the fused feature $F_t$. In addition, the decoder ${\psi }_{\mathrm{e}}$ consisting of three-layer MLP accepts the fused feature $F_t$ and reconstructs the emotional landmark displacement $\mathsf{\Delta }{Q}_t$ for each frame of the identity source. Lastly, $\mathsf{\Delta }{Q}_t$ is used to revise the lip-sync landmark sequence $P_t$ to obtain the target portrait facial landmark sequence with emotional source expressions, $Q_t$. To summarize, the module uses the following modifications to describe the output landmarks in sequential order:

$$F_a={\varphi }_{e1}\left(L_a;W_{e1}\right)$$

(5)

$$F_b={\varphi }_{e2}\left(L_b;W_{e2}\right)$$

(6)

$$F_t=concat\left(F_a,F_b,S_t\right)$$

(7)

$$\mathsf{\Delta }{Q}_t={\psi }_{\mathrm{e}}\left(F_t;W_e\right)$$

(8)

$$Q_t=P_t+\mathsf{\Delta }{Q}_t$$

(9)

where {$W_{e1},W_{e2},W_e$} are the learnable parameters of the network; the encoder {${\varphi }_{e1},{\varphi }_{e2}$} internal hidden state vectors are all 256 in size; the decoder ${\psi }_{\mathrm{e}}$ for the three layers is 512, 256, and 204 (68 × 3) in size; {$L_a,L_b,F_a,F_b,S_t,F_t$, $\mathsf{\Delta }{Q}_t,P_t,Q_t$} are consistent with those described above; and $concat$ denotes the horizontal connectivity.

In the training phase, the overall loss is given by the sum of the three loss terms, ${ℒ}_L,{ℒ}_{D_L}$ and ${ℒ}_{D_T}$:

$${ℒ}_s={\lambda }_1{ℒ}_L+{\lambda }_2{ℒ}_{D_L}+{\lambda }_2{ℒ}_{D_T}$$

(10)

where ${\lambda }_1=$ 1 and ${\lambda }_2=$ 0.001 are set through hold-out validation.

Landmark Loss. ${ℒ}_L$ is defined as the distance between the minimized reference landmark ${\widehat{Q}}_t$ and the predicted landmark $Q_t$, with the aim of promoting the correct placement of the emotional landmarks while preserving the target identity.

$${ℒ}_L={\displaystyle \sum _{t=1}^T{\displaystyle \sum _{i=1}^N}}{\Vert Q_{i,t}-{\widehat{Q}}_{i,t}\Vert }_2^2$$

(11)

Adversarial Loss. Generates facial landmarks with stable frame rates and rich facial expressions during the training process. To this end, we created two discriminator networks following GAN design ideas and inspired by [37]. The goal of discriminator $D_L$ is to determine the authenticity of the predicted landmarks. To ensure the fluidity of the landmark sequence and to prevent the interval frame landmarks from shifting too much, we used the discriminator $D_T$ to determine the similarity of the interval frame identities. ${ℒ}_{D_L}$ and ${ℒ}_{D_T}$ are then their loss functions:

$${ℒ}_{D_L}=\log D_L\left({\widehat{Q}}_t\right)+\log \left(1-D_L\left(Q_t\right)\right)$$

(12)

$${ℒ}_{D_T}=\log D_T\left({\widehat{Q}}_{t-1},{\widehat{Q}}_t\right)+\log \left(1-D_T\left({\widehat{Q}}_{t-1},Q_t\right)\right)$$

(13)

where ${\widehat{Q}}_{t-1}$ denotes the reference landmark of the previous frame

3.4. Landmark-To-Animation Translation Network

This network was used to generate portrait animation along with the input identity source portrait $L_a$ after obtaining the generated landmark sequence $Q$. Our network follows the basic idea of Zakharov et al. [44]. The U-Net architecture of the encoder–decoder network performs the landmark-to-animation translation while preserving image identity information and facial texture to act as a face generator. A frame-by-frame 256 × 256 three-channel facial sketch is drawn by connecting the expected landmark with predefined colorful lines and then adding it to the original image channel to generate a 256 × 256 6-channel image. The preprocessed image is passed to the network with 12 layers. The first six layers are encoders with each layer consisting of a 2-step convolution with a core of 3 × 3 followed by two residual blocks, and the last six decoder layers are symmetric upsampling structures. Skip connections also exist between the encoding and decoding layers with the same spatial resolution. The output facial image sequences are combined in frames to obtain the result, a lip-sync facial animation of the target portrait with the emotion source expression.

4. Experiment

4.1. Experimental Settings

Datasets. Two audiovisual datasets: MEAD [10] and the Obama Weekly Address database [8] were employed. MEAD is a high-resolution emotional audiovisual dataset consisting of 60 actors from different continents with eight emotions, three intensities, and seven poses. A subset of this database was used for the pre-trained model of the emotional landmark animation generation network and the landmark-to-animation translation network. Videos of the frontal pose were selected from the MEAD database to train our network. The dataset was divided into three sections: 60%, 20%, and 20% for training, hold-out validation, and testing, respectively. For the emotional landmark animation generation network, the reference was a video of a target portrait with emotional source expression. The Obama Weekly Address database contains high-quality video clips of weekly presidential speeches from a single speaker, former President Barack Obama. Its consistent frontal pose with little emotion is ideal for training the lip-sync landmark animation generation network.

Implementation Details. All audio waveforms in the dataset were sampled at 16KHz and all video was converted to 30 fps. All images of the network model were 256 × 256 pixels in size. There are 68 key points of landmarks provided in [41]. The models were trained using PyTorch, including a 16 GB Nvidia Tesla V100 GPU from Santa Clara, CA, USA.

Comparing Methods. Our approach was compared with two state-of-the-art representative algorithms for image-based face video generation, AVTG and MakeItTalk. The cascading network structure of AVTG based on face landmarks was proposed by Chen et al. [13]. In addition, MakeItTalk by Zhou et al. [15] uses decoupled learning to generate spontaneous head movements. These two methods were used extensively for comparison.

4.2. Quantitative Evaluation

To evaluate the quality of the images generated by different methods, the metrics SSIM [45], PSNR and FID [46] were selected. To evaluate the semantic level of lip synchronization and facial expressions, the landmark distance (LD) and landmark velocity difference (LVD) were selected as metrics [13,15,17]. LD denotes the average Euclidean distance between all predicted landmark locations and the reference location. The LVD is the average Euclidean distance that represents the landmark velocity between two sequences, and velocity refers to the difference in landmark positions between consecutive frames. Prior to evaluation, the resultant sequences were aligned with the ground truth sequences and the facial landmarks were extracted from them. The results are presented in

Table 1. Quantitative comparison with state-of-the-art methods. Calculation of landmark accuracy and video quality by comparing the results of different methods with real data. L- represents the lip area and F-represents the face area.

Method	SSIM↑	PSNR↑	FID↓	L-LD↓	L-LVD↓	F-LD↓	F-LVD↓
Chen et al. [13]	0.60	28.55	67.60	3.27	2.09	3.82	1.71
Zhou et al. [15]	0.62	28.94	17.33	2.49	1.66	3.55	1.64
Wang et al. [10]	0.68	28.61	22.52	2.52	2.28	3.16	2.01
Ji et al. [17]	0.71	29.53	7.99	2.45	1.78	3.01	1.56
Ours	0.62	28.92	16.80	2.52	1.68	2.98	1.48

, with metrics including video quality (SSIM, PSNR, FID), lip sync (L-LD, L-LVD), and facial expression (F-LD, F-LVD), where the facial expression metric includes all landmarks in the facial region. As can be seen, our generated animation adds additional emotion, and the data outperformed Chen et al. [13] and Zhou et al. [15]. Furthermore, our results are comparable to the existing state-of-the-art algorithms of Ji et al. [17] and Wang et al. [10] Notably, emotion can be controlled in our method using arbitrary portraits, avoiding the training dependency problem, whereby the method is individual specific and a model must be trained for a particular person.

4.3. Qualitative Evaluation

Subject evaluation is equally crucial for the visual content generation and emotion generation tasks. As shown in Figure 3, two advanced techniques (listed in Section 4.1) were compared with the proposed approach. Our system requires the additional input of an emotional source portrait (the portrait in the lower left corner) in addition to audio and target portrait inputs. The proposed approach allows the target portrait to produce expressions corresponding to the seen emotional source portrait while ensuring the visual quality of the video and the identity of the target. Specifically, in Chen et al. [13] and Zhou et al. [15], lip movements are acceptable, and Zhou et al. [15] additionally considers head posture. However, the expressions of both characters are constrained by the input portrait and less attention is accorded to the emotional aspect. In our case, lip movement and head posture are guaranteed, and the input of the emotional source image allows control of the emotional content in the output video. It has been shown that multimodal inputs may provide rich emotional qualities, allowing the creation of emotionally richer videos via the fusion of diverse input types.

In addition to this, we used our proposed method to perform a comparison between the same target portrait with audio and different emotion-driven sources as input. As shown in Figure 4, we generated seven emotions for the target portrait from top to bottom: anger, contempt, disgust, fear, happiness, sadness, and surprise. Our approach allows for diverse expressions to be generated based on the emotion source. Expressions with more evident traits, such as anger, happiness, and melancholy, provide better outcomes, but the more muddled disdain and disgust, surprise, and sadness produce fewer effective results. In addition to the basic categories of facial emotions, there are also compound emotions, meaning those that can be constructed by combining the basic component categories to create new ones [47]. Theoretically, the proposed method can generate expressions of arbitrary emotional origin, including compound emotions. The experimental results show that this system ensures the visual quality and target identity to a certain extent and enriches the emotional representation ability of the talking face animation. However, it is limited by the identity source portrait and the landmark-to-animation translation network, leading to some confusion in the reenactment of certain expressions.

4.4. Ablation Study

To evaluate the impact of each component on our approach, an ablation study was carried out. We created two variants, the first being the Lip-Sync Landmark Animation Generation Network and the Landmark-to-Animation Translation Network, referred to as the content variant. Meanwhile, the second was an emotional variant, which consisted of the Emotional Landmark Animation Generation Network and the Landmark-to-Animation Translation Network. As shown in Figure 5, this content variant has a well-moving lip effect and has the same expression profile as the identity source portrait. The emotion variant is driven by the audio and the emotion source portrait to generate a video that matches the expression features of the emotion source portrait. Unlike the previous variant, our method generates lip-sync while also migrating expression features from another portrait. Furthermore, to further demonstrate the versatility and flexibility of our approach, we chose unseen portraits for this portrait input (the identity source is the corresponding author of this paper, and the emotion source is the first author of this paper).

4.5. User Study

The user study was conducted in three phases, each with 126 participants. The first phase focused on assessing image quality and lip syncing, while the second and third phases focused on assessing the emotional validity of the videos. The input video and audio were obtained from MEAD, and the input image was the first frame of the video in this database.

The first stage was a cross-sectional comparison, with the four groups included in the study consisting of real data, our proposed method, Chen et al. [13], and Zhou et al. [15]. The selected target identity pictures had neutral emotion (no emotion on the face) in three frontal poses and three voices were randomly selected for each emotion, except for the neutral emotion (seven in total), generating a total of sixty-three animations per group. The emotion source input to our method was the first frame of the video of the target identity for the remaining seven emotions, except for the neutral emotion. Participants were required to evaluate the videos on a scale of 5 out of 5 on the authenticity of the video and the degree of lip-sync. The comparison results of the four groups are shown in Figure 6, where error bars indicate standard deviation. Asterisks represent a significant difference, ** indicates p < 0.01. Pink, Orange, Blue and Green histograms denote Chen et al. [13] and Zhou et al. [15], with similar scores, our method and real data, respectively.

The second phase investigated whether the emotional rendering of our generated videos was authentic and effective. Participants at this stage were asked to classify the sentiments of the videos generated on the basis of real data, Wang et al. [10], and our proposed method. It was decided that the video samples for this stage would all be taken from the same portrait and the same audio, but with different emotions, for a total of 64 videos for each method. Additionally, the videos would have no background sound to allow participants to classify their feelings using visual cues. The confusion matrix for the subjective emotion classification in the user survey is shown in Figure 7. Our videos were more similar to ground-truth video patterns and produced more diagonal confusion matrices than those of Wang et al. The overall classification accuracies of 64.2% (ground-truth), 43.5% (Wang), and 50.9% (our) validate the effectiveness of the proposed model in generating emotionally controlled talking face videos, showing the power of the emotional landmark animation generation network.

In the last stage, an emotionally perceptive user survey was used regarding emotionally mismatched videos as in Eskimez et al. [20], to ask participants to identify the major and secondary emotions (if any) generated by each video. The input consisted of identity source portraits of neutral emotions as well as audio and emotion source portraits covering eight emotions. Sixty-four audiovisual emotion videos were generated by our system through permutation, of which eight were matched with each other. The evaluations of the participants on the emotions displayed in these videos can be used to some extent as a reference for determining the emotion recognition mechanism people rely on the most in recognizing the emotions depicted in these videos. As can be seen from the confusion matrix in Figure 8, the matching value of the visual emotion as the primary emotion is greater than the matching value of the auditory emotion as the primary emotion, with ratios of 41.4% and 34.9%, respectively. This indicates that, for the emotional perception of audiovisual content, the subjects relied much more on the visual modality than on the audio modality. Perceiving these eight emotions, such as happiness, sadness, and anger through the visual modality is relatively simple. The pairs of disgust/contempt and fear/surprise are more easily confused. The audio modality has three pairs of confusing emotions: disgust/contempt, happy/surprised, and fearful/sad.

5. Conclusions

In this study, we proposed a generative network of talking faces based on facial landmarks from the perspective of decoupled design and expression reenactment to enrich the emotion of the generated animation. Our method allows the extraction of expression features from the additional input of unseen emotional portraits and processes them together with the audio to control the generation of animated facial expressions. This method generates videos that satisfy authenticity and high fidelity while enhancing the emotion of the existing animation, enriching the video material, and providing flexible control over the expression of the animation. Qualitative and quantitative experiments have validated the efficiency and flexibility of our approach. In the future, we will use more efficient networks to capture long-term temporal and spatial correlations between pixels to reduce the artifacts of expression reenactment. The emotional element of audio will also be further explored to generate emotionally rich audio-visual animations.