Multi-Representation Joint Dynamic Domain Adaptation Network for Cross-Database Facial Expression Recognition

Abstract

In order to obtain more fine-grained information from multiple sub-feature spaces for domain adaptation, this paper proposes a novel multi-representation joint dynamic domain adaptation network (MJDDAN) and applies it to achieve cross-database facial expression recognition. The MJDDAN uses a hybrid structure to extract multi-representation features and maps the original facial expression features into multiple sub-feature spaces, aligning the expression features of the source domain and target domain in multiple sub-feature spaces from different angles to extract features more comprehensively. Moreover, the MJDDAN proposes the Joint Dynamic Maximum Mean Difference (JD-MMD) model to reduce the difference in feature distribution between different subdomains by simultaneously minimizing the maximum mean difference and local maximum mean difference in each substructure. Three databases, including eNTERFACE, FABO, and RAVDESS, are used to design a large number of cross-database transfer learning facial expression recognition experiments. The accuracy of emotion recognition experiments with eNTERFACE, FABO, and RAVDESS as target domains reach 53.64%, 43.66%, and 35.87%, respectively. Compared to the best comparison method chosen in this article, the accuracy rates were improved by 1.79%, 0.85%, and 1.02%, respectively.

1. Introduction

As one of the research directions in the field of human–computer interaction, emotion recognition has attracted much attention. Emotion is a key factor in understanding human behavior. In emotion recognition, facial expression recognition develops a major role in the domain of human–computer interaction and computer vision. With the progress of computer hardware and the vigorous development of the era of big data, human emotion poses a new challenge to computers and human–computer interaction. Although many research advances have been made in facial expression and emotion recognition, there is still room to improve the accuracy and efficiency of facial emotion judgments. This will open up many new opportunities for the technology-driven future society, especially in the fields of artificial intelligence-driven medical care, robotics, security monitoring, and corporate marketing, which have broad application prospects [1,2,3,4]. The process of facial expression recognition usually involves the use of computer technology to extract face information from images. Firstly, the face detection algorithm is used to recognize and locate the face in the image. Subsequently, the image is preprocessed to improve its quality and prepare it for further analysis. Then, through the feature extraction method, important facial features are identified, and finally, emotion classification is carried out. Facial expression recognition can aid healthcare institutions in identifying individuals’ mental stress issues, thereby facilitating their ability to cope with the challenges pertaining to their psychological well-being. Moreover, this technology can also offer support to businesses, sales, and other service industries by analyzing consumers’ facial expressions for the development of corresponding marketing strategies [5,6,7,8,9,10,11,12].

When conducting experiments, existing facial emotion datasets have some limitations, including insufficient data volume, low image quality, and insufficient diversity of expression labels. In addition, accessing some datasets has become more difficult due to copyright and image right restrictions. Even if an individual wants to build a dataset, it takes a lot of time, effort, and cost to take photos and annotate information [13,14,15,16]. Some researchers in this field have carried out related research to solve these problems. In Section 2, we present relevant research on cross-database facial expression recognition.

Cross-database expression recognition can reduce the difference in sample distribution between different databases, but conducting cross-database expression experiments requires hardware support and a large amount of computing resources. At the same time, it also faces technical challenges such as inconsistent labels between different databases [14,17]. In the field of deep learning, cross-database transfer learning has emerged as a prominent research direction, with numerous effective methods being developed. Drawing inspiration from the Multi-Representation Adaptation Network (MRAN) approach [18], this paper proposes a hybrid structure composed of multiple sub-structure feature extraction units for multi-representation feature extraction. It maps the original facial expression images into multiple distinct sub-feature spaces, enabling the alignment of image features from different perspectives in both source and target domains. Previous methods in each sub-feature space solely focus on aligning the global distributions of data between source and target domains while neglecting the unique details within each category that are present in both domains. To extract more fine-grained information from expressions, this paper proposes a multi-representation joint dynamic domain adaptation network (MJDDAN) that performs joint dynamic domain adaptation on source and target domain data within each sub-feature space for cross-database facial expression recognition research.

The main contributions of this paper are as follows:

(1)

A novel Joint Dynamic Maximum Mean Difference (JD-MMD) model integrating a subdomain and global domain is proposed, not only enabling adaptation between different domains but also the incorporation of fine-grained information within the same class.

(2)

The acquisition of image features from multiple angles enables a more comprehensive description of the features. By incorporating these features into the joint dynamic domain adaptive model, we propose a multi-representation joint dynamic domain adaptive network (MJDDAN) that effectively addresses feature distribution discrepancies across different domains, thereby enhancing model performance and adaptability.

(3)

The experimental results demonstrate that the domain adaptive method based on MJDDAN exhibits significant advantages in the task of facial expression recognition. This is validated through experiments conducted on the RAVDESS [19], FABO [20], and eNTERFACE [21] datasets. In comparison to other methods, MJDDAN achieves superior implementation outcomes.

The subsequent sections are organized as follows: The Section 2 provides an overview of the relevant studies conducted in the field of cross-database identification research. The Section 3 presents the multi-representation joint dynamic domain adaptive network (MJDDAN) and outlines the network’s overall structure and principles of each module. The Section 4 provides a detailed description of the experiments, including information on the experimental environment, dataset selection, experimental details, procedures, and evaluation metrics. Subsequently, an analysis and demonstration of the experimental results is presented to validate the effectiveness of our proposed method through comparisons with other approaches. Finally, in the Section 5, we summarize our proposed methodology by reviewing its core ideas and objectives while summarizing the key steps and techniques employed. Furthermore, we highlight the advantages and innovations offered by our approach.

2. Related Work

The TMMLDL migration model, proposed by Ni and his colleagues [22], integrates the techniques of dictionary learning and metric learning. This study also investigates the utilization of pairwise constraints and global structural information to ensure effective transfer learning across diverse domains. The proposed method by Zhang et al. [23], known as Joint Local-Global Discriminative Subspace Transfer Learning (LGDSTL), employs joint local-global graphs to quantify the dissimilarity between different domains, with the objective of achieving a balanced knowledge transfer from global graphs in each database and local discriminative geometric structures. Transfer Component Analysis (TCA) [24] is one of the classical transfer learning methods. It is a method based on subspace projection, which maps the data of the source domain and target domain into a shared feature space to realize knowledge transfer between the domains. It has achieved good results in many practical applications, which proves its practicability and effectiveness in solving the problem of domain adaptation. GFK (Geodesic Flow Kernel) [25] is a transfer learning method utilizing manifold learning. The GFK method uses the manifold distance between the covariance matrices of the source and target domains to construct a new kernel that can be used for classification. Specifically, the GFK method learns a new kernel function by finding the minimum manifold distance between the source domain and the target domain, which can be classified over the target domain. Subspace Alignment (SA) [26] is a transfer learning method utilizing kernel normalization, which aims to minimize the difference between the source domain and target domain through subspace alignment, thereby improving the performance of transfer learning.

In addition to traditional transfer learning methods, many transfer learning methods based on deep learning have also been proposed in recent years. A DANN (Domain Adversarial Neural Network) [27] uses the idea of domain classifier and adversarial training to achieve feature transfer and self-adaptation among different domains by minimizing the difference between the source domain and target domain. This method can help the feature representation learned by the model to have strong generalization performance to achieve better performance in the target domain. A Deep Subdomain Adaption Network (DSAN) [28] proposes an idea of self-adaptation between local subdomains to improve the effect of transfer learning. This approach recognizes that the differences between the source domain and the target domain are not global but exist in local subdomains. Therefore, a DSAN introduces the local maximum mean difference (LMMD) to measure the degree of difference between local subdomains. Ren et al. [29] developed a representational learning model that is transferable and can improve recognition performance. The method models bionic face representations as approximately stable and structured representations to identify commonalities between source and target domains. This method significantly improves the recognition performance by improving the grouping effect of the feature output and introducing a reasonable method to emphasize and share the discriminant scale and direction. Long et al. [30] verify that when inputting data with different attributes, they can complement each other and explore their role in making the learning model robust to domain differences. As a result, they propose a universal framework called Graph Co-Regularized Transfer Learning (GTL), which integrates various matrix factorization models. Specifically, the main objective of GTL is to extract latent common factors by preserving the statistical properties of different domains and further optimizing them using the geometric structure of each domain to mitigate negative transfer effects. The CDNet [31] presents a novel approach for recognizing complex facial expressions across databases with only a limited number of camera angles. By employing a cascaded decomposition network, the CDNet effectively handles various scenarios of facial expressions and achieves more accurate recognition performance in cross-domain learning tasks. On the other hand, Zhu et al. [18] introduced a method called multi-representation adaptation that focuses on aligning the distributions of multiple representations. On the basis of this approach, the Multi-Representation Adaptation Network (MRAN) can capture different aspects of image information to successfully accomplish cross-database facial expression recognition.

3. Multiple-Representation Joint Dynamic Domain Adaptive Network

3.1. Network Overview

The multi-representation joint dynamic domain adaptive network (MJDDAN) introduces global adaptive maximum mean difference (MMD) [24,32,33] and local subdomain adaptive LMMD [28,34,35] into the multi-representation feature extraction module, and it carries out joint dynamic domain adaptive in each substructure, respectively, to obtain fine-grained information about different feature spaces. The MJDDAN model framework is shown in Figure 1. The whole network structure can be divided into three parts. The first part is the ResNet network [36] model, represented by G, which is used to extract low-dimensional features of facial expression images. The second part is a multi-feature extraction module embedded with Joint Dynamic Maximum Mean Difference (JD-MMD), represented by H, which incorporates diverse substructures for extracting feature information from multiple perspectives. Within each substructure, JD-MMD combines a globally domain adaptive MMD module [24,32,33] and locally subdomain adaptive LMMD module [28,34,35], facilitating domain adaptation. The third part is classifier S.

3.2. Multi-Representation Feature Extraction Module

In domain adaptive image classification, we usually use convolutional neural networks to extract image feature representations. However, due to the limitation of the network structure, the extracted features often contain only part of the information of the picture, which may lead to an unsatisfactory migration effect [18]. In Figure 2, (a) represents the original facial expression picture, while (b)–(d) represent the facial expression features after feature extraction. It can be observed that the facial expression features extracted only contain part of the picture’s information [18]. These features may contain either useful information that is adaptive to subsequent domains or information that is not adaptive to the domain. In order to solve this problem, it is necessary to comprehensively consider the feature information of the original facial expression images from multiple perspectives when feature extraction is carried out. Through a comprehensive consideration of multiple perspectives, we can obtain a more comprehensive description of image features so as to carry out a more comprehensive analysis and gain a better understanding of the original face expression image object. This holistic approach helps improve the effectiveness and accuracy of domain adaptation and reduces the possibility of extracting irrelevant or redundant information. To avoid the time consumption of training multiple convolutional neural networks, one option is to extract different features by introducing different components into the neural network structure [18].

The whole MJDDAN model can be divided into three modules: $g(·)$ is the ResNet50 [36] network module used to extract features, $h\left(·\right)$ is used to replace the average pooling layer with a multi-feature extraction module, and JD-MMD, $s(·)$ is the classification module. The final complete network model is [16]

$$y=f\left(x\right)=\left(s·h·g\right)\left(x\right)\left(h·g\right)\left(x\right)=h\left(g\left(x\right)\right)$$

(1)

The multi-representation feature extraction module has n distinct substructures, $h_1\left(·\right)\cdots h_n(·)$, each of which consists of two convolution operations or pooling operations. The objective optimization function of the multi-representation feature extraction module is as follows:

$$\underset{}{\mathit{min}}\sum _i^n\widehat{d}(\left(h_i·g\right)\left(X_s\right),\left(h_i·g\right)\left(X_t\right))$$

(2)

$X_s$ indicates the source domain database, $X_t$ indicates the target domain database, and $\widehat{d}(·,·)$ indicates the distance between the source domain and the target domain [18].

After domain adaptive is completed in multiple subspaces, it is necessary to carry out the whole classification task. Multiple sub-feature extraction structures are connected through a full-join layer to obtain a join vector $[\left(h_1·g\right)\left(X\right),\cdots \left(h_n·g\right)\left(X\right)]$, which is then input into the classifier $s(·)$. Finally, the entire network model $y=f\left(x\right)$ embedded with the multi-representation feature extraction module can be expressed in the following form [18]

$$y=f\left(x\right)=s\left(\right[\left(h_1·g\right)\left(X\right),\cdots \left(h_n·g\right)\left(X\right)\left]\right)$$

(3)

3.3. Joint Dynamic Maximum Mean Difference

When unsupervised domain adaptation is carried out, the labeled samples are stored in the source domain as $D_S={\{x_i^s,y_i^s\}}_{i=0}^{n_s}$, and the unlabeled samples are stored as $D_T={\left\{x_i^t\right\}}_{i=0}^{n_t}$. $n_s$ and $n_t$ represent the number of samples in the source domain and the target domain, respectively, and $y_i$ is used to indicate the label information of the I−th sample in the source domain. $D_S$ and $D_T$ are sampled from different data distributions, p and q, where p ≠ q. The purpose of transfer learning is to reduce the distribution difference between the source domain and the target domain by designing a network $f\left(x\right)$ [27,37,38,39,40].

In past attempts of unsupervised domain adaptation, the alignment between the source domain and target domain was in the global domain, but the alignment between the two domains was ignored, which not only means that the fine-grained information of the class was ignored, but also the discriminant structure became chaotic. Therefore, this paper proposes the Joint Dynamic Maximum Mean Difference (JD-MMD) to deal with local subdomain adaptive as well as global adaptive. While minimizing the MMD, the LMMD is simultaneously used to introduce the target domain sample labels predicted by the network model into the feature alignment process so as to realize a joint dynamic domain adaptive method that focuses more on fine-grained information. The global MMD expression is [24,28,32,33]

$${MMD}_H(P,Q)\triangleq {\left|\right|E_p\left[\phi \left(x^s\right)\right]-E_q\left[\phi \left(x^t\right)\right]\left|\right|}_H^2$$

(4)

The globally adaptive loss function $L_M$ is denoted by ${MMD}_H\left(P,Q\right)$.

The local LMMD expression is [28,34,35]

$${LMMD}_H(P,Q)\triangleq {E_c\left|\right|E_{p^{\left(c\right)}}\left[\phi \left(x^s\right)\right]-E_{q^{\left(c\right)}}\left[\phi \left(x^t\right)\right]\left|\right|}_H^2$$

(5)

By minimizing the above formula, we can narrow the distance between subfields of the same class. In order to carry out domain adaptation more finely, an unbiased estimator is made according to the weight $w^c$ of each class, which can evaluate the distance of the degree quantum field more accurately.

$$\widehat{{LMMD}_H}(P,Q)=\frac{1}{C}\sum _{c=1}^C||\sum _{x_i^s\in D_s}{w}_i^{sc}\phi \left(x_i^s\right)-\sum _{x_j^t\in D_t}{w}_j^{tc}\phi \left(x_j^t\right)|{|}_H^2$$

(6)

In the given equation, the weights corresponding to $x_i^s$ and $x_j^t$ in category C are denoted as $w_i^{sc}$ and $w_j^{tc}$, respectively. $\sum _{x_i^s\in D_s}{w}_i^{sc}=\sum _{x_j^t\in D_t}{w}_j^{tc}=1$. The weighted sum of category C is represented as $\sum _{x_i\in D}{w}_i^c\phi \left(x_i\right)$. The calculation formula for $w_i^c$ is as follows: [28]

$$w_i^c=\frac{y_{ic}}{{\sum }_{(x_i,y_j)\in D}{y}_{jc}}$$

(7)

For the source domain, we can easily obtain the label $y_{ic}$ but during unsupervised domain adaptation, it is difficult to obtain the label information in the target domain. In order to deal with the lack of labels in the target domain, the probability distribution output by the deep neural network can be used as the soft label of the target domain sample, and these soft labels can be used to calculate the difference between the distribution of related subdomains to achieve domain adaptation. So, the formula is [28]

$$\begin{array}{cc}\widehat{{\mathrm{L}\mathrm{M}\mathrm{M}\mathrm{D}}_L}\left(P,Q\right)=& \frac{1}{C}{\sum }_{c=1}^C\left[{\displaystyle \sum _{i=1}^{n_s}}{\displaystyle \sum _{j=1}^{n_t}}{w}_i^{sc}{w}_j^{sc}k\left(z_i^{sl},z_j^{sl}\right)\right.\hfill \\ & \left.+{\displaystyle \sum _{i=1}^{n_s}}{\displaystyle \sum _{j=1}^{n_t}}{w}_i^{tc}{w}_j^{tc}k\left(z_i^{tl},z_j^{tl}\right)-2{\displaystyle \sum _{i=1}^{n_s}}{\displaystyle \sum _{j=1}^{n_t}}{w}_i^{sc}{w}_j^{tc}k\left(z_i^{sl},z_j^{tl}\right)\right]\end{array}$$

(8)

where $z^l$ is the L-layer activation l ∈ {1,2, …, L}.

The local subdomain adaptation loss function $L_L$ is denoted by $\widehat{{\mathrm{L}\mathrm{M}\mathrm{M}\mathrm{D}}_L}\left(P,Q\right)$, so joint dynamic domain adaptive can be expressed as

$$L_J={\lambda }_M\times L_M+{\lambda }_L\times L_L$$

(9)

${\lambda }_M$ and ${\lambda }_L$ are the weight parameters of the two domain adaptive methods, which are used to balance the importance of the two domain adaptive loss terms. $L_J$ represents the loss function of joint dynamic domain adaptation, which realizes domain adaptation by dynamically adjusting the MMD and LMMD.

In order to prevent the network from falling into the optimal solution during training and to balance the optimization progress of two loss functions, $L_M$ and $L_L$, we use a set of dynamic weighting factors for training. These dynamic weighting factors can be adjusted according to the performance of the model during training to ensure a balance between the training speed of different loss functions. The dynamic weighting factor is expressed as follows:

$${\lambda }_i=\frac{{\alpha }_i\times L_i}{{\sum }_{i\in \{M,L\}}{L}_i}$$

(10)

When training the network model, we use a fixed initial hyperparameter ${\alpha }_i$ to enhance the contribution of different loss functions. In order to achieve the global optimal solution, we introduce the dynamic weight factor ${\alpha }_i$, which adjusts as the model iteratively updates the loss function value $L_i$. This dynamic adjustment can balance the contribution of different loss functions in the model training and ensure that the model does not fall into the local optimal solution to achieve the best migration effect.

$$\underset{f}{\mathit{min}}\frac{1}{n_s}\sum _{i=1}^{n_s}J(f\left(x_i^s\right),y_i^s)+\lambda \sum _i^n{\widehat{d}}_J(\left(h_1·g\right)\left(X\right),····\left(h_n·g\right)\left(X\right))$$

(11)

After feature alignment, the data from the source domain $D_S$ and target domain $D_T$ enter the classifier module S, but only the data from the source domain $D_S$ undergoes classification loss training because the classification performance of the network model needs to be controlled by the labeled source domain $D_S$ data, and then the parameters of the entire network model are adjusted during training. Therefore, the objective function of MJDDAN mainly includes cross entropy loss function and joint dynamic domain adaptive loss function. In the loss formula of MJDDAN, $J(·,·)$ is the cross-entropy loss function, that is, the classification loss; ${\widehat{d}}_J(·,·)$ is the joint dynamic domain adaptive loss; $L_J$ in the substructure is calculated using Equation (11); and $\lambda >0$ is the tradeoff parameter.

4. Experimental Results and Analysis

The experimental environment and database employed in this study are detailed in this section. Within this experimental framework, we conduct a comparative analysis as well as an ablation experiment of the MJDDAN model for cross-database facial expression recognition.

4.1. Experimental Environment

The experimental setup of this study involves two distinct operating system platforms. On the personal host, we employ the Windows operating system for video processing in the database, to extract required images, and to conduct comparative experiments on traditional migration methods. On the server side, we conduct deep domain adaptive experiments using the Ubuntu operating system (graphics card: NVIDIA GTX1080Ti).

4.2. Facial Expression Database

The experiments in this paper encompass three distinct databases, providing a comprehensive validation of the proposed methodology. These databases include the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) [19], the Bimodal Face and Body Gesture Database (FABO) [20], and the eNTERFACE ‘05 Audio-Visual Emotion Database (eNTERFACE) [21].

(1)

The RAVDESS database is a multi-modal emotion database that includes two parts (voice emotions and song emotions). The performances of 12 male and 12 female professional actors were recorded, with each actor performing in a neutral American accent and ensuring correct content matching with different vocabulary. In the song mode, there are five types of emotions (tranquility, joy, grief, fear, and anger), while in the voice mode, there are expressions of serenity, joy, grief, anger, fear, surprise, and disgust. Each emotion has a neutral expression as well as two expressions of varying intensity [19,41,42,43,44]. Figure 3 is a sample of partial expressions in the RAVDESS database.

(2)

The FABO database was created by Hatice Gunes and others at the University of Technology Sydney in 2005, and it is the first publicly available bimodal emotion database that contains data on facial expressions and body language. Out of the 23 participants in the database, 11 are men, and 12 are women. During the process of collecting data, each subject was presented with different illustrations and situations, and their generated facial expressions and body language were recorded. Each video contains 2–4 fully displayed emotional expressions of the same type. The database recorded nine emotional expressions (surprise, anger, disgust, fear, boredom, anxiety, and uncertainty) [20,45,46,47,48]. Figure 4 is a sample of partial expressions in the FABO database.

(3)

Martin and his team at KU Leuven developed the eNTERFACE database, which is a two-modal emotion database containing speech emotion and video emotion. The database was created with the selection of 46 subjects, 81% of whom were male, with the remaining being female, from 14 different countries. The participants expressed genuine feelings of happiness, fear, anger, surprise, disgust, and sadness in six different situations. Finally, the database was reviewed by a human expert who deleted the video data with unclear emotional expressions. In the end, data from 42 participants were retained [21,49,50,51,52]. Figure 5 is a sample of partial expressions in the eNTERFACE database.

4.3. Experimental Details

As the three databases all store video information, it is necessary to extract image data containing emotional information from these videos. Initially, the OpenCV library is employed to process the videos into sequential frames, and subsequently, eight frames are selected at regular intervals for each video. Furthermore, considering that each database is collected in distinct environments, it becomes imperative to customize them by eliminating irrelevant data while preserving segments that have potential for generating emotional information. The objective of this customization process is to optimize the dataset which focuses on emotionally relevant content. Specifically, the facial region of interest spans from the left ear to the right ear and includes areas above (such as the forehead) and below it (including the chin), as depicted in Figure 6. This cropping technique aids in retaining crucial facial features for enhanced sentiment analysis and recognition.

To ensure the consistency of label information between the source and target domains, we selected anger, disgust, fear, happiness, sadness, and surprise as common emotion classification labels based on the criteria of the eNTERFACE database, which offers a minimal number of emotion classifications. By refining these three databases and standardizing their label information to encompass these six emotions, we can guarantee coherence and comparability in tasks related to emotion recognition.

Finally, in the RAVDESS database, we selected 192 video samples for each of the six emotions, resulting in a total of 1152 samples. Similarly, in the eNTERFACE database, we chose 215 video samples for each emotion, resulting in a total of 1290 video samples. In terms of the FABO dataset, varying quantities were selected for each of the six emotions (anger had 230 samples, disgust had 90 samples, fear had 86 samples, happiness had 118 samples, sadness had 68 samples, and surprise had 64 samples), resulting in a total of 656 samples. By employing such diverse sample selections across multiple databases, we can acquire an ample number of examples to effectively conduct experiments and evaluations on facial expression classification algorithms.

4.4. Cross-Database Facial Expression Recognition Experiment and Result Analysis

The experiment conducts nine transfer learning tasks on three databases, including $F\to R$, $e\to R$, $F\&e\to R$, $R\to F$, $e\to F$, $R\&e\to F$, $R\to e$, $F\to e$, and $R\&F\to e$. By comparing experiments and conducting ablation tests, the effectiveness of MJDDAN as well as the impact of each module on the overall network performance are demonstrated.

The existing transfer learning methods for cross-database facial expression recognition primarily encompass TCA [24], GFK [25], SA [26], DANN [27], DSAN [28], CDNet [31], and MRAN [18]. In this paper, we propose the MJDDAN model, which is compared with the above methods under identical experimental conditions. The evaluation criteria include the F1-Score, confusion matrix, and accuracy rate. The accuracy rates of the experimental results are presented in

Table 1. The experimental accuracy of the proposed MJDDAN and other compared methods.

	TCA	GFK	SA	DANN	DSAN	CDNet	MRAN	MJDDAN
F→R	31.47%	34.83%	35.65%	38.94%	40.75%	41.06%	43.27%	43.92%
e→R	39.28%	42.39%	42.76%	44.27%	46.89%	48.66%	50.52%	52.53%
$F\&e\to R$	39.75%	43.27%	43.94%	46.63%	48.37%	51.85%	50.95%	53.64%
$R\to F$	28.35%	31.62%	31.82%	34.28%	35.89%	38.63%	38.10%	39.84%
$e\to F$	33.91%	37.83%	37.42%	39.16%	40.23%	42.72%	40.79%	42.54%
$R\&e\to F$	33.33%	38.42%	37.33%	40.49%	40.98%	42.81%	41.36%	43.66%
$R\to e$	27.83%	29.73%	30.03%	29.06%	31.67%	31.56%	30.92%	32.29%
$F\to e$	27.79%	30.92%	31.84%	32.23%	32.78%	33.37%	32.68%	33.96%
$R\&F\to e$	28.67%	31.85%	32.29%	32.03%	34.34%	34.85%	33.77%	35.87%

, while the F1-Scores are displayed in

Table 2. The experimental F1-Scores of the proposed MJDDAN and other compared methods.

	TCA	GFK	SA	DANN	DSAN	CDNet	MRAN	MJDDAN
F→R	0.3168	0.3462	0.3527	0.3940	0.4088	0.4127	0.4478	0.4672
e→R	0.3953	0.4271	0.4288	0.4436	0.4657	0.4835	0.5004	0.5191
$F\&e\to R$	0.3992	0.4357	0.4425	0.4683	0.4851	0.5127	0.5063	0.5295
$R\to F$	0.2965	0.3133	0.3171	0.3491	0.3588	0.3759	0.3905	0.4028
$e\to F$	0.3477	0.3791	0.3769	0.3966	0.4016	0.4236	0.4153	0.4314
$R\&e\to F$	0.3448	0.3869	0.3745	0.4027	0.4077	0.4257	0.4285	0.4652
$R\to e$	0.2864	0.3016	0.3020	0.2973	0.3181	0.3282	0.3127	0.3250
$F\to e$	0.2839	0.3077	0.3154	0.3244	0.3292	0.3384	0.3239	0.3436
$R\&F\to e$	0.2894	0.3150	0.3251	0.3252	0.3450	0.3404	0.3356	0.3612

.

According to the experimental data presented in Table 1, the following conclusions can be drawn: The proposed MJDDAN outperforms traditional transfer learning methods (TCA, GFK, and SA) and deep learning methods (DANN, DSAN, CDNet, and MADTN) in cross-database facial expression recognition tasks. In Table 1, the bold text highlights the maximum value for each row’s result. Generally speaking, deep learning methods exhibit superior performance compared to traditional approaches in this task. Traditional methods, such as TCA and SA, rely on linear transformations that are inadequate for nonlinear data mapping. Despite GFK being a nonlinear method that utilizes the locally linear embedding technique, it still fails to achieve satisfactory results. Therefore, traditional methods are not ideal for cross-database facial expression recognition tasks. Among the mentioned deep learning methods, DANN and DSAN only consider global or local domain adaptation, respectively, and both employ a single network structure for the domain adaptive deep learning approach. The proposed MJDDAN performs domain adaptation by simultaneously using four different substructures across four feature subspaces. This significantly enhances the accuracy of cross-database expression recognition. Compared with the latest CDNet method, only the experimental accuracy of the $e\to F$ cross-database expression recognition task is slightly lower, and the accuracy of the other recognition tasks is higher than that of the current excellent methods. Compared with the MRAN method, the proposed MJDDAN applies the local subdomain adaptive method in multiple subspaces to make use of more fine-grained information in different expression categories. According to the experimental results, the proposed MJDDAN in this paper achieves higher accuracy than the MRAN. For cross-database facial expression recognition tasks, the accuracy of the proposed MJDDAN is improved by up to 2.23% for the RAVDESS target domain. For the FABO target domain, the accuracy is improved by up to 1.54%. For the eNTERFACE target domain, the accuracy is improved by up to 1.76%. Among the nine groups of cross-database facial expression recognition experiments, the confusion matrix of the proposed MJDDAN is shown in Figure 7. An, Di, Fe, Ha, Sa, and Su represent anger, disgust, fear, happiness, sadness, and surprise, respectively.

4.5. Ablation Experiment

Ablation experiments are conducted in this section to evaluate the contribution of the multi-representation feature extraction module and the JD-MMD module to the entire MJDDAN model. To ensure the effectiveness and fairness of the experiment, we use the same experimental environment and strategy as those used in the comparative experiment in the previous section. The RAVDESS database is used as the target domain data, while the FABO database and eNTERFACE data are used as the source domain data. We designed several variants of the network model to prove the effectiveness of each module. The first variant only adds the multi-representation feature extraction module to the ResNet network (MFER), without embedding the JD-MMD, in order to verify its contribution. The second method embeds the JD-MMD into the ResNet network (JDDAN) and performs joint dynamic domain adaptation for verification purposes. The third variant simultaneously adds both modules mentioned above to the ResNet network, creating the MJDDAN model. Moreover, a global adaptive network based on multi-representation feature extraction (MGDAN) and a local subdomain adaptive network based on multi-representation feature extraction (MLDAN) are constructed to verify their effectiveness, respectively.

Table 3. Ablation experiment accuracy rates and F1-Scores of cross-database facial expression recognition task with RAVDESS database as target domain.

	$\mathit{F}\to \mathit{R}$		$\mathit{e}\to \mathit{R}$		$\mathit{F}\&\mathit{e}\to \mathit{R}$
	ACC	F1-Score	ACC	F1-Score	ACC	F1-Score
ResNet	26.74%	0.2693	34.66%	0.3417	35.62%	0.3451
MFER	28.59%	0.2945	36.43%	0.3629	37.92%	0.3751
JDDAN	40.75%	0.4088	46.89%	0.4657	48.37%	0.4851
MGDAN	41.66%	0.4281	50.47%	0.5017	51.03%	0.5152
MLDAN	43.74%	0.4625	52.25%	0.5173	53.27%	0.5228
MJDDAN	43.92%	0.4672	52.53%	0.5191	53.64%	0.5295

shows the ablation experiment accuracy rates and F1-Scores of the cross-database facial expression recognition task with the RAVDESS database as the target domain.

The results of the ablation experiment presented in Table 3 demonstrate that the two modules contribute differently to cross-database facial expression recognition (the bold text indicates the maximum value for each line). Compared to the ResNet network based on multi-representation feature extraction, a multi-representation feature extraction module replaces the last fully connected layer. The experimental results show that all three cross-database facial expression recognition tasks achieve higher accuracy than the ResNet network, with improvements of 1.85%, 1.77%, and 2.3%, respectively, proving that the multi-representation feature extraction module can enhance the experimental accuracy without requiring domain adaptation between the source and target domains for cross-database facial expression recognition purposes. Furthermore, incorporating the JD-MMD into the last fully connected layer significantly reduces facial expression feature distribution across different databases compared to the ResNet network based on joint dynamic domain adaptation. As observed in the experimental results, all three cross-database facial expression recognition tasks yield substantially higher accuracy rates compared to both the ResNet network and multi-representation feature extraction network model, achieving increases of 14.01%, 12.23%, and 17.65%, respectively, compared to the ResNet network. In comparison to the ResNet network that relies on multi-representation feature extraction, our proposed MJDDAN demonstrates significant improvements of 12.16%, 10.46%, and 10.45%, respectively, in the cross-database facial expression recognition experiment, highlighting the importance of domain adaptation between different databases. By incorporating the MJDDAN into various subspaces of the ResNet network for multi-representation feature extraction, we effectively align feature distributions from diverse facial expression spaces to capture more detailed information. The experimental accuracy of all three cross-database facial expression recognition tasks increased by 3.17%, 5.64%, and 5.27%, respectively, when compared to the accuracy obtained when using a single network structure like ResNet with a joint dynamic domain adaptive network alone. These results indicate that the migration effect achieved by our multi-subspace joint dynamic domain adaptive method is significantly superior to that obtained with a single network structure. The proposed MJDDAN in this paper can enhance accuracy compared to both the global domain adaptive network and the local subdomain adaptive network based on multi-representation feature extraction. This finding proves that the proposed MJDDAN exhibits superior migration performance over both the global domain adaptive and local subdomain adaptive approaches.

The primary objective of this study is to address the issue of distribution discrepancy in cross-database facial expression recognition. The proposed approach in this paper involves extracting features from images through multiple sub-feature spaces at different angles and aligning the source and target domain features through the Joint Dynamic Maximum Mean Difference (JD-MMD) within the sub-feature space. The comparative experiments revealed that, although the accuracy for the $e\to F$ experiment and the F1-Score for the $R\to e$ experiment are slightly lower than those of the CDNet method, our method outperforms other approaches in other respective experiments; therefore, the effectiveness of our proposed approach is partially validated.

5. Conclusions

This paper proposes a novel multi-representation joint dynamic domain adaptation network (MJDDAN) model for cross-database facial expression recognition. The model proposes a Joint Dynamic Maximum Mean Difference (JD-MMD) module and integrates it into each sub-feature space to effectively reduce the feature distribution differences between different domains. The computational cost of the JD-MMD module is slightly higher compared to that of the LMMD and MMD. Finally, extensive experiments are conducted using the MJDDAN model for cross-database facial expression recognition on the RAVDESS database, FABO database, and eNTERFACE database. Compared to the best comparison method chosen in this article, the accuracy of the MJDDAN model was improved by 1.79%, 0.85%, and 1.02%, respectively. In the future, we will apply the MJDDAN model to achieve large-scale cross-database facial expression recognition. Although the JD-MMD is an effective domain adaptation method, it may not necessarily be the best choice. In future research, we can design a network structure that can autonomously select domain adaptation methods that are suitable for each subspace and select the best measurement method through network training.