Multimodal Emotion Recognition Based on Facial Expressions, Speech, and Body Gestures

Abstract

The research of multimodal emotion recognition based on facial expressions, speech, and body gestures is crucial for oncoming intelligent human–computer interfaces. However, it is a very difficult task and has seldom been researched in this combination in the past years. Based on the GEMEP and Polish databases, this contribution focuses on trimodal emotion recognition from facial expressions, speech, and body gestures, including feature extraction, feature fusion, and multimodal classification of the three modalities. In particular, for feature fusion, two novel algorithms including supervised least squares multiset kernel canonical correlation analysis (SLSMKCCA) and sparse supervised least squares multiset kernel canonical correlation analysis (SSLSMKCCA) are presented, respectively, to carry out efficient facial expression, speech, and body gesture feature fusion. Different from the traditional multiset kernel canonical correlation analysis (MKCCA) algorithms, our SLSKMCCA algorithm is a supervised version and is based on the least squares form. The SSLSKMCCA algorithm is implemented by the combination of SLSMKCCA and a sparse item (L1 Norm). Moreover, two effective solving algorithms for SLSMKCCA and SSLSMKCCA are presented in addition, which use the alternated least squares and augmented Lagrangian multiplier methods, respectively. The extensive experimental results on the popular public GEMEP and Polish databases show that the recognition rate of multimodal emotion recognition is superior to bimodal and monomodal emotion recognition on average, and our presented SLSMKCCA and SSLSMKCCA fusion methods both obtain very high recognition rates, especially for the SSLSMKCCA fusion method.

1. Introduction

Intelligent human–computer interfaces have been developed for several decades, approximately. Nowadays, the application of artificial intelligence in such interfaces has become more and more extensive, and intelligent human–computer interfaces are also increasingly demanded to act the same as or approximate human-to-human interfaces when it comes to emotional intelligence in interactions. This includes all kinds of emotions such as happiness, sadness, disgust, and many more [1,2,3]. Consequently, this comes at a higher requirement for the research of intelligent human–computer interfaces; after all, the computer still has a wide gap in emotional intelligence when compared to human beings. If tomorrow’s intelligent human–computer interfaces aim to achieve parity with human-to-human emotional interaction, accurate emotion recognition ability is the key necessary prerequisite [4].

Human-to-human emotional interaction is very common in daily life for human beings. The form of such human-to-human emotional interaction is, however, very sophisticated, variable, and dynamic. For human beings with different cultural backgrounds, races, genders, ages, personalities, and, furthermore, differences in traits, the form of human-to-human emotional interaction is often significantly different [5,6]. Even for the same person, it may highly vary in various environments. Among those complexities and variabilities, however, one can also find a certain generality of human-to-human emotional interaction. This generality is given by the fact that most human beings interact emotionally with other people using a multimodal combination form most of the time [7,8,9,10]. This multimodal combination form most often includes the modalities of facial expressions, speech, and body gestures [11,12,13,14,15,16]. Therefore, if the ability of intelligent human–computer interfaces achieves or approximates the level of human-to-human emotional interface, it is essential to research such a multimodal emotion recognition for computers. Specifically, the research of multimodal emotion recognition based on facial expressions, speech, and body gestures is, accordingly, of crucial relevance for the next generation of intelligent human–computer interfaces [3,17,18]. However, it is a considerably challenging task and has seldom been researched in the past years.

In general, emotion recognition can be divided into two types according to the literature: monomodal emotion recognition and multimodal emotion recognition [9,12,19,20,21,22]. Monomodal emotion recognition is based on only one modality, while multimodal emotion recognition is based on two or more than two modalities [3,10,23,24,25]. The typical monomodal emotion recognition covers facial expression recognition, speech emotion recognition, body gesture emotion recognition, physiological signal emotion recognition, and likes [13,14,25,26,27]. Among those monomodal emotion recognition types, most of the emotion recognition researchers focus on facial expression recognition and speech emotion recognition. Body gesture emotion recognition, physiological signal emotion recognition, and other monomodal emotion recognition types have been less researched in the past years [25,26,27].

Aside from monomodal emotion recognition, multimodal emotion recognition also covers bimodal emotion recognition based on facial expressions and speech, bimodal emotion recognition based on facial expressions and physiological signals, bimodal emotion recognition based on facial expressions and body gestures, bimodal emotion recognition based on speech and physiological signals, multimodal emotion recognition based on facial expressions, speech, and body gestures, multimodal emotion recognition based on facial expressions, physiological signal, body gestures, and more [3,9,10,11,17,22,24,25,28,29,30,31,32,33,34]. Among the multimodal emotion recognition types, bimodal emotion recognition based on two modalities is relatively present in the body of literature, but multimodal emotion recognition based on three or more than three modalities is very rare in research, as adding further modalities obviously increases the difficulty and complexity for building the multimodal emotion database and multimodal emotion recognition engine [22,25]. However, as described in the first two paragraphs in this article, if the ability of an emotionally intelligent human–computer interface can reach or get close to the level of human-to-human emotional interaction, it is essential to study multimodal emotion recognition based on facial expressions, speech, and body gestures [29,30,34].

Although the research of multimodal emotion recognition based on facial expressions, speech, and body gestures is a very difficult task, there are a few researchers that have preliminarily carried out this task [12,21]. Caridakis and Castellano et al. [17,18] used two fusion levels (feature level and decision level) and a Bayesian classifier to reach multimodal emotion recognition based on facial expressions, speech, and body gestures, and their experimental results show that multimodal emotion recognition based on three modalities significantly improves the recognition performance over using less modalities. Metallinou et al. [35] present a multimodal emotion recognition from a facial expression, speech, and head movement approach, which is implemented by decision-level fusion and a Bayesian model. Similar to the work of [17,18], Kessous et al. [36] also use two fusion levels (feature level and decision level) and a Bayesian model to implement multimodal emotion recognition based on facial expressions, speech, and body gestures, and also obtain similar multimodal emotion recognition experimental results in their established speech-centered interaction multimodal emotion corpus. Different from the above recognition-based works, Nicolaou et al. [37] researched multimodal dimensional emotion prediction based on facial expressions, speech, and body gestures by using two regression methods including bidirectional long short-term memory neural networks and support vector machines for regression. Ozkul et al. [38] used a hidden Markov model and a mutual information measure to realize multimodal emotion recognition based on facial expressions, speech, and upper-body gestures on their spontaneous multimodal corpora. Ghayoumi et al. [39] recently set up a real-time multimodal emotion recognition engine based on facial expressions, speech, and body gestures, which was implemented as a formal model based on decision-level fusion. Moreover, other related works on this task are described in references [40,41].

From the above discussion, we can see that multimodal emotion recognition based on facial expressions, speech, and body gestures has not been well studied in the past years, and there are still many steps to be taken. In this article, we focus on the study of multimodal emotion recognition from facial expressions, speech, and body gestures using the GEMEP [42,43] and Polish databases [41]. Our work includes the feature extraction, feature fusion, and multimodal classification of three modalities.

Specifically for feature fusion, two novel algorithms including supervised least squares multiset kernel canonical correlation analysis (SLSMKCCA) and sparse supervised least squares multiset kernel canonical correlation analysis (SSLSMKCCA) are presented in this article to carry out facial expression, speech, and body gesture feature fusion. Different from the traditional multiset kernel canonical correlation analysis (MKCCA) algorithms [44,45], our SLSKMCCA algorithm is a supervised version and is based on the least squares form. The SLSMKCCA algorithm is implemented by the combination of SLSMKCCA and a sparse item (L1 Norm) [3,46,47,48,49], and it can be used to select the useful features of three modalities by joining the sparse item (L1 Norm). Moreover, two effective solution algorithms for SLSMKCCA and SSLSMKCCA are also presented, which use the alternated least squares [15,49,50,51] and augmented Lagrangian multiplier method [22,52,53,54,55,56], respectively.

The main novelty of this article stems from the following three contributions. The first contribution is that two novel algorithms including SLSMKCCA and SSLSMKCCA are presented to carry out facial expression, speech, and body gesture feature fusion. The second contribution is that two effective solution algorithms for SLSMKCCA and SSLSMKCCA are presented. The last contribution is that we implemented the multimodal emotion recognition based on facial expressions, speech, and body gestures using the SLSMKCCA and SSLSMKCCA algorithm.

The rest of the paper is organized as follows. Section 2 and Section 3 introduce the traditional multiset canonical correlation analysis (MCCA) and multiset kernel canonical correlation analysis (MKCCA) algorithms, respectively. Section 4 and Section 5 introduce our proposed supervised least squares multiset kernel canonical correlation analysis (SLSMKCCA) and sparse least squares multiset canonical correlation analysis (SSLSMKCCA) algorithms, respectively. Section 6 and Section 7 introduce the experimental result on the GEMEP and Polish multimodal emotion databases and conclusion, respectively.

2. Multiset Canonical Correlation Analysis

Different from the canonical correlation analysis (CCA) method, the multiset canonical correlation analysis (MCCA) method [15,57,58,59,60,61,62,63] was proposed to implement a linear correlation analysis between three or more sets of data. So far, it has been applied to implement feature fusion in many applications such as handwritten digit recognition, remote sensing image classification, face recognition, blind source separation, and more [64,65].

As this paper focuses on multimodal emotion recognition from three modalities (facial expressions, speech, and body gestures), in the following, we will introduce the MCCA in the form of three sets of data to be handled, which correspond to the above three modalities, respectively. We suppose that $\mathbf{X}\in {\mathbf{R}}^{p\times N}$, $\mathbf{Y}\in {\mathbf{R}}^{q\times N}$, and $\mathbf{Z}\in {\mathbf{R}}^{s\times N}$ are the feature matrices of facial expression, speech, and body gesture modalities, respectively, p, q, and r are the corresponding dimension of the three modalities, and N is the number of the three modalities’ samples. $\mathbf{A}$, $\mathbf{B}$, and $\mathbf{C}$ are three projection matrices of the MCCA method, which correspond to facial expression, speech, and body gesture modalities, respectively. Then, the MCCA method can be shown as [57,58,59,60]

$$\begin{array}{ccc}\hfill arg\underset{\mathbf{A},\mathbf{B},\mathbf{C}}{max}& & {\displaystyle \frac{{\mathbf{A}}^T\mathbf{X}{\mathbf{Y}}^T\mathbf{B}}{\sqrt{{\mathbf{A}}^T\mathbf{X}{\mathbf{X}}^T\mathbf{A}}\sqrt{{\mathbf{B}}^T\mathbf{Y}{\mathbf{Y}}^T\mathbf{B}}}}\hfill \\ \hfill & & +{\displaystyle \frac{{\mathbf{A}}^T\mathbf{X}{\mathbf{Z}}^T\mathbf{C}}{\sqrt{{\mathbf{A}}^T\mathbf{X}{\mathbf{X}}^T\mathbf{A}}\sqrt{{\mathbf{C}}^T\mathbf{Z}{\mathbf{Z}}^T\mathbf{C}}}}\hfill \\ & & +{\displaystyle \frac{{\mathbf{B}}^T\mathbf{Y}{\mathbf{Z}}^T\mathbf{C}}{\sqrt{{\mathbf{B}}^T\mathbf{Y}{\mathbf{Y}}^T\mathbf{B}}\sqrt{{\mathbf{C}}^T\mathbf{Z}{\mathbf{Z}}^T\mathbf{C}}}}.\hfill \end{array}$$

(1)

The formulation (1) of the MCCA method can be transformed to optimize the following equation:

$$\begin{array}{ccc}\hfill arg\underset{\mathbf{A},\mathbf{B},\mathbf{C}}{max}& & {\mathbf{A}}^T\mathbf{X}{\mathbf{Y}}^T\mathbf{B}+{\mathbf{A}}^T\mathbf{X}{\mathbf{Z}}^T\mathbf{C}+{\mathbf{B}}^T\mathbf{Y}{\mathbf{Z}}^T\mathbf{C}\hfill \\ \hfill s.t.& & {\mathbf{A}}^T\mathbf{X}{\mathbf{X}}^T\mathbf{A}=1,\hfill \\ \hfill & & {\mathbf{B}}^T\mathbf{Y}{\mathbf{Y}}^T\mathbf{B}=1,\hfill \\ & & {\mathbf{C}}^T\mathbf{Z}{\mathbf{Z}}^T\mathbf{C}=1.\hfill \end{array}$$

(2)

Lastly, the optimization formulation of (2) can be solved by the traditional Lagrangian multiplier method.

3. Multiset Kernel Canonical Correlation Analysis

Based on the linear MCCA method, the MKCCA method [44,45,66] was proposed next to implement a non-linear correlation analysis between three or more sets of data by non-linear mapping. So far, the MKCCA method has also been applied to implement feature fusion in many applications such as remote sensing image classification, genomic data analysis, blind source separation, and many more [44,45,67,68].

We suppose that $\varphi (\mathbf{X})\in {\mathbf{R}}^{p_{\varphi }\times N}$, $\psi (\mathbf{Y})\in {\mathbf{R}}^{q_{\psi }\times N}$, and $\phi (\mathbf{Y})\in {\mathbf{R}}^{s_{\phi }\times N}$ are the feature matrices of the facial expression, speech, and body gesture modalities, respectively, which are mapped into a higher space by three corresponding mapping functions $\varphi$, $\psi$, and $\phi$. $p_{\varphi }$, $q_{\psi }$, and $s_{\phi }$ are the corresponding dimensions of the three modalities in higher space. Then, the MKCCA method can be shown as [44,45,67,68]

$$\begin{array}{ccc}\hfill arg\underset{\mathbf{A},\mathbf{B},\mathbf{C}}{max}& & {\displaystyle \frac{{\mathbf{A}}^T\varphi (\mathbf{X})\psi {(\mathbf{Y})}^T\mathbf{B}}{\sqrt{{\mathbf{A}}^T\varphi (\mathbf{X})\varphi {(\mathbf{X})}^T\mathbf{A}}\sqrt{{\mathbf{B}}^T\psi (\mathbf{Y})\psi {(\mathbf{Y})}^T\mathbf{B}}}}+\hfill \\ \hfill & & {\displaystyle \frac{{\mathbf{A}}^T\varphi (\mathbf{X})\phi {(\mathbf{Y})}^T\mathbf{C}}{\sqrt{{\mathbf{A}}^T\varphi (\mathbf{X})\varphi {(\mathbf{X})}^T\mathbf{A}}\sqrt{{\mathbf{C}}^T\phi (\mathbf{Y})\phi {(\mathbf{Y})}^T\mathbf{C}}}}+\hfill \\ & & {\displaystyle \frac{{\mathbf{B}}^T\psi (\mathbf{Y})\phi {(\mathbf{Y})}^T\mathbf{C}}{\sqrt{{\mathbf{B}}^T\psi (\mathbf{Y})\psi {(\mathbf{Y})}^T\mathbf{B}}\sqrt{{\mathbf{C}}^T\phi (\mathbf{Y})\phi {(\mathbf{Y})}^T\mathbf{C}}}}.\hfill \end{array}$$

(3)

It is noted that the three projection matrices of the MKCCA method $\mathbf{A}$, $\mathbf{B}$, and $\mathbf{C}$ can be represented as

$$\begin{array}{ccc}\hfill \mathbf{A}& =& \varphi (\mathbf{X})\mathbf{D},\hfill \end{array}$$

(4)

$$\begin{array}{ccc}\hfill \mathbf{B}& =& \psi (\mathbf{Y})\mathbf{E},\hfill \end{array}$$

(5)

$$\begin{array}{ccc}\hfill \mathbf{C}& =& \phi (\mathbf{Y})\mathbf{G},\hfill \end{array}$$

(6)

where $\mathbf{D}$, $\mathbf{E}$, and $\mathbf{G}$ are three other matrices which, respectively, correspond to the three projection matrices of the MKCCA method $\mathbf{A}$, $\mathbf{B}$, and $\mathbf{C}$.

Then, the formulation (3) of the MKCCA method can be represented as

$$\begin{array}{ccc}\hfill arg& & \underset{\mathbf{D},\mathbf{E},\mathbf{G}}{max}{\displaystyle \frac{{\mathbf{D}}^T\varphi {(\mathbf{X})}^T\varphi (\mathbf{X})\psi {(\mathbf{Y})}^T\psi (\mathbf{Y})\mathbf{E}}{\sqrt{{\mathbf{D}}^T[\varphi {(\mathbf{X})}^T\varphi (\mathbf{X}){]}^2\mathbf{D}}\sqrt{{\mathbf{E}}^T[\psi {(\mathbf{Y})}^T\psi (\mathbf{Y}){]}^2\mathbf{E}}}}\hfill \\ \hfill & & +{\displaystyle \frac{{\mathbf{D}}^T\varphi {(\mathbf{X})}^T\varphi (\mathbf{X})\phi {(\mathbf{Y})}^T\phi (\mathbf{Y})\mathbf{G}}{\sqrt{{\mathbf{D}}^T[\varphi {(\mathbf{X})}^T\varphi (\mathbf{X}){]}^2\mathbf{D}}\sqrt{{\mathbf{G}}^T[\phi {(\mathbf{Y})}^T\phi (\mathbf{Y}){]}^2\mathbf{G}}}}\hfill \\ & & +{\displaystyle \frac{{\mathbf{E}}^T\psi {(\mathbf{Y})}^T\psi (\mathbf{Y})\phi {(\mathbf{Y})}^T\phi (\mathbf{Y})\mathbf{G}}{\sqrt{{\mathbf{E}}^T[\psi {(\mathbf{Y})}^T\psi (\mathbf{Y}){]}^2\mathbf{E}}\sqrt{{\mathbf{G}}^T[\phi {(\mathbf{Y})}^T\phi (\mathbf{Y}){]}^2\mathbf{G}}}}.\hfill \end{array}$$

(7)

Now, let us define three kernel matrices of the MKCCA method ${\mathbf{K}}_X=\varphi {(\mathbf{X})}^T\varphi (\mathbf{X})$, ${\mathbf{K}}_Y=\psi {(\mathbf{Y})}^T\psi (\mathbf{Y})$, and ${\mathbf{K}}_Z=\phi {(\mathbf{Y})}^T\phi (\mathbf{Y})$ for facial expression, speech, and body gesture modalities respectively. Then, the formulation (7) of the MKCCA method can be represented as

$$\begin{array}{ccc}\hfill arg\underset{\mathbf{D},\mathbf{E},\mathbf{G}}{max}& & {\displaystyle \frac{{\mathbf{D}}^T{\mathbf{K}}_X{\mathbf{K}}_Y\mathbf{E}}{\sqrt{{\mathbf{D}}^T{({\mathbf{K}}_X)}^2\mathbf{D}}\sqrt{{\mathbf{E}}^T{({\mathbf{K}}_Y)}^2\mathbf{E}}}}\hfill \\ \hfill & & +{\displaystyle \frac{{\mathbf{D}}^T{\mathbf{K}}_X{\mathbf{K}}_Z\mathbf{G}}{\sqrt{{\mathbf{D}}^T{({\mathbf{K}}_X)}^2\mathbf{D}}\sqrt{{\mathbf{G}}^T{({\mathbf{K}}_Z)}^2\mathbf{G}}}}\hfill \\ & & +{\displaystyle \frac{{\mathbf{E}}^T{\mathbf{K}}_Y{\mathbf{K}}_Z\mathbf{G}}{\sqrt{{\mathbf{E}}^T{({\mathbf{K}}_Y)}^2\mathbf{E}}\sqrt{{\mathbf{G}}^T{({\mathbf{K}}_Z)}^2\mathbf{G}}}}.\hfill \end{array}$$

(8)

Similar to the above MCCA method, the formulation (8) of the MKCCA method can also be transformed to optimize the following equation:

$$\begin{array}{ccc}\hfill arg\underset{\mathbf{D},\mathbf{E},\mathbf{G}}{max}& & {\mathbf{D}}^T{\mathbf{K}}_X{\mathbf{K}}_Y\mathbf{E}+{\mathbf{D}}^T{\mathbf{K}}_X{\mathbf{K}}_Z\mathbf{G}+{\mathbf{E}}^T{\mathbf{K}}_Y{\mathbf{K}}_Z\mathbf{G}\hfill \\ \hfill s.t.& & {\mathbf{D}}^T{({\mathbf{K}}_X)}^2\mathbf{D}=1,\hfill \\ \hfill & & {\mathbf{E}}^T{({\mathbf{K}}_Y)}^2\mathbf{E}=1,\hfill \\ & & {\mathbf{G}}^T{({\mathbf{K}}_Z)}^2\mathbf{G}=1.\hfill \end{array}$$

(9)

Lastly, the optimization formulation of (9) can also be solved by the traditional Lagrangian multiplier method.

4. Supervised Least Squares Multiset Kernel Canonical Correlation Analysis

In this section, we introduce the optimization formulation of the MCCA method. For the CCA method, the formulation (1) can be changed as (we suppose that we only consider the modality of facial expressions and speech in this formulation) [57,58]

$$\begin{array}{ccc}\hfill arg\underset{\mathbf{A},\mathbf{B}}{max}& & {\displaystyle \frac{{\mathbf{A}}^T\mathbf{X}{\mathbf{Y}}^T\mathbf{B}}{\sqrt{{\mathbf{A}}^T\mathbf{X}{\mathbf{X}}^T\mathbf{A}}\sqrt{{\mathbf{B}}^T\mathbf{Y}{\mathbf{Y}}^T\mathbf{B}}}}.\hfill \end{array}$$

(10)

Unlike the traditional form (10) of the CCA method, it also can be shown as another least squares form [3,15,50,51]:

$$\begin{array}{ccc}\hfill & & arg\underset{\mathbf{A},\mathbf{B}}{min}{\mathbf{W}}_{CCA}\hfill \\ \hfill =& & arg\underset{\mathbf{A},\mathbf{B}}{min}\parallel {(\mathbf{X}{\mathbf{X}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{X}-\mathbf{A}{\mathbf{B}}^T\mathbf{Y}){\parallel }_F^2.\hfill \end{array}$$

(11)

The formulations (10) and (11) of the CCA method are equivalent, which has been proven in the literature [50]. Based on the least squares form (11) of the CCA method, we present the supervised least squares multiset kernel canonical correlation analysis (SLSMKCCA) algorithm as follows:

$$\begin{array}{ccc}\hfill & & arg\underset{\mathbf{A},\mathbf{B},\mathbf{C},\mathbf{H}}{min}{\mathbf{W}}_{SLSMKCCA}\hfill \\ \hfill =& & arg\underset{\mathbf{A},\mathbf{B},\mathbf{C},\mathbf{H}}{min}\parallel {(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}[\mathbf{J}-\mathbf{H}{\mathbf{A}}^T\varphi (\mathbf{X})]{\parallel }_F^2\hfill \\ \hfill & & +\parallel {(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}[\mathbf{J}-\mathbf{H}{\mathbf{B}}^T\psi (\mathbf{Y})]{\parallel }_F^2\hfill \\ & & +\parallel {(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}[\mathbf{J}-\mathbf{H}{\mathbf{C}}^T\phi (\mathbf{Y})]{\parallel }_F^2,\hfill \end{array}$$

(12)

where $\mathbf{J}\in {\mathbf{R}}^{m\times N}$ is the label matrix [15,46,51,56] for $\mathbf{X}$, $\mathbf{Y}$, and $\mathbf{Z}$, $\mathbf{H}$ is the projection matrix of the label matrix $\mathbf{J}$, and m is the dimension of the label matrix $\mathbf{J}$.

Substituting $\mathbf{A}=\varphi (\mathbf{X})\mathbf{D}$, $\mathbf{B}=\psi (\mathbf{Y})\mathbf{E}$ and $\mathbf{C}=\phi (\mathbf{Y})\mathbf{G}$ in Section 3 into the formulation (12), the SLSMKCCA algorithm can be shown as

$$\begin{array}{ccc}\hfill & & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\mathbf{H}}{min}{\mathbf{W}}_{SLSMKCCA}\hfill \\ \hfill =& & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\mathbf{H}}{min}\parallel {(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}[\mathbf{J}-\mathbf{H}{\mathbf{D}}^T\varphi {(\mathbf{X})}^T\varphi (\mathbf{X})]{\parallel }_F^2\hfill \\ \hfill & & +\parallel {(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}[\mathbf{J}-\mathbf{H}{\mathbf{E}}^T\psi {(\mathbf{Y})}^T\psi (\mathbf{Y})]{\parallel }_F^2\hfill \\ & & +\parallel {(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}[\mathbf{J}-\mathbf{H}{\mathbf{G}}^T\phi {(\mathbf{Y})}^T\phi (\mathbf{Y})]{\parallel }_F^2,\hfill \end{array}$$

(13)

Substituting ${\mathbf{K}}_X=\varphi {(\mathbf{X})}^T\varphi (\mathbf{X})$, ${\mathbf{K}}_Y=\psi {(\mathbf{Y})}^T\psi (\mathbf{Y})$, and ${\mathbf{K}}_Z=\phi {(\mathbf{Y})}^T\phi (\mathbf{Y})$ in Section 3 into the formulation (13), the SLSMKCCA algorithm can be shown as

$$\begin{array}{ccc}\hfill & & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\mathbf{H}}{min}{\mathbf{W}}_{SLSMKCCA}\hfill \\ \hfill =& & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\mathbf{H}}{min}\parallel {(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{J}-\mathbf{H}{\mathbf{D}}^T{\mathbf{K}}_X){\parallel }_F^2\hfill \\ \hfill & & +\parallel {(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{J}-\mathbf{H}{\mathbf{E}}^T{\mathbf{K}}_Y){\parallel }_F^2\hfill \\ & & +\parallel {(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{J}-\mathbf{H}{\mathbf{G}}^T{\mathbf{K}}_Z){\parallel }_F^2;\hfill \end{array}$$

(14)

then, it can be reformulated as

$$\begin{array}{ccc}\hfill & & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\mathbf{H}}{min}{\mathbf{W}}_{SLSMKCCA}=\hfill \\ \hfill & & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\mathbf{H}}{min}tr\{[{(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{J}-\mathbf{H}{\mathbf{D}}^T{\mathbf{K}}_X)]\hfill \\ \hfill & & [{(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{J}-\mathbf{H}{\mathbf{D}}^T{\mathbf{K}}_X){]}^T\}+\hfill \\ \hfill & & tr\{[{(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{J}-\mathbf{H}{\mathbf{E}}^T{\mathbf{K}}_Y)]\hfill \\ \hfill & & [{(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{J}-\mathbf{H}{\mathbf{E}}^T{\mathbf{K}}_Y){]}^T\}+\hfill \\ \hfill & & tr\{[{(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{J}-\mathbf{H}{\mathbf{G}}^T{\mathbf{K}}_Z)]\hfill \\ & & [{(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{J}-\mathbf{H}{\mathbf{G}}^T{\mathbf{K}}_Z){]}^T\};\hfill \end{array}$$

(15)

then, it can be shown as

$$\begin{array}{ccc}\hfill & & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\mathbf{H}}{min}{\mathbf{W}}_{SLSMKCCA}=\hfill \\ \hfill & & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\mathbf{H}}{min}tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\mathbf{D}}^T{\mathbf{K}}_X{\mathbf{K}}_X\mathbf{D}{\mathbf{H}}^T\}+\hfill \\ \hfill & & tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\mathbf{E}}^T{\mathbf{K}}_Y{\mathbf{K}}_Y\mathbf{E}{\mathbf{H}}^T\}+\hfill \\ \hfill & & tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\mathbf{G}}^T{\mathbf{K}}_Z{\mathbf{K}}_Z\mathbf{G}{\mathbf{H}}^T\}-\hfill \\ \hfill & & 2tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_X\mathbf{D}{\mathbf{H}}^T\}-2tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_Y\mathbf{E}{\mathbf{H}}^T\}\hfill \\ & & -2tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_Z\mathbf{G}{\mathbf{H}}^T\},\hfill \end{array}$$

(16)

then,

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\partial {\mathbf{W}}_{SLSMKCCA}}{\partial \mathbf{D}}}& =& 2{\mathbf{K}}_X{\mathbf{K}}_X\mathbf{D}{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}-\hfill \\ & & 2{\mathbf{K}}_X{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}=0,\hfill \end{array}$$

(17)

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\partial {\mathbf{W}}_{SLSMKCCA}}{\partial \mathbf{E}}}& =& 2{\mathbf{K}}_Y{\mathbf{K}}_Y\mathbf{E}{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}-\hfill \\ & & 2{\mathbf{K}}_Y{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}=0,\hfill \end{array}$$

(18)

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\partial {\mathbf{W}}_{SLSMKCCA}}{\partial \mathbf{G}}}& =& 2{\mathbf{K}}_Z{\mathbf{K}}_Z\mathbf{G}{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}-\hfill \\ & & 2{\mathbf{K}}_Z{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}=0,\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\partial {\mathbf{W}}_{SLSMKCCA}}{\partial \mathbf{H}}}& =& 2{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\mathbf{D}}^T{\mathbf{K}}_X{\mathbf{K}}_X\mathbf{D}\hfill \\ \hfill & & +2{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\mathbf{E}}^T{\mathbf{K}}_Y{\mathbf{K}}_Y\mathbf{E}\hfill \\ \hfill & & +2{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\mathbf{G}}^T{\mathbf{K}}_Z{\mathbf{K}}_Z\mathbf{G}\hfill \\ \hfill & & -2{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_X\mathbf{D}\hfill \\ \hfill & & -2{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_Y\mathbf{E}\hfill \\ & & -2{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_Z\mathbf{G}=0;\hfill \end{array}$$

(20)

then, $\mathbf{D}$, $\mathbf{E}$, $\mathbf{G}$, and $\mathbf{H}$ can be written as

$$\begin{array}{c}\hfill \mathbf{D}={\mathbf{K}}_X^{-1}{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{[{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}]}^{-1},\end{array}$$

(21)

$$\begin{array}{c}\hfill \mathbf{E}={\mathbf{K}}_Y^{-1}{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{[{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}]}^{-1},\end{array}$$

(22)

$$\begin{array}{c}\hfill \mathbf{G}={\mathbf{K}}_Z^{-1}{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{[{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}]}^{-1},\end{array}$$

(23)

$$\begin{array}{ccc}\hfill \mathbf{H}=& & (\mathbf{J}{\mathbf{K}}_X\mathbf{D}+\mathbf{J}{\mathbf{K}}_Y\mathbf{E}+\mathbf{J}{\mathbf{K}}_Z\mathbf{G})\hfill \\ \hfill & & ({\mathbf{D}}^T{\mathbf{K}}_X{\mathbf{K}}_X\mathbf{D}+{\mathbf{E}}^T{\mathbf{K}}_Y{\mathbf{K}}_Y\mathbf{E}+\hfill \\ & & {\mathbf{G}}^T{\mathbf{K}}_Z{\mathbf{K}}_Z{\mathbf{G})}^{-1}.\hfill \end{array}$$

(24)

According to the above derivation, the iterative solution algorithm of the SLSMKCCA method using the alternated least squares [3,15,22,49,50,51,54] is given in Algorithm 1.

Algorithm 1: SLSMKCCA

Input: Label matrix $\mathbf{J}\in {\mathbf{R}}^{m\times N}$, three feature matrices $\mathbf{X}\in {\mathbf{R}}^{p\times N}$, $\mathbf{Y}\in {\mathbf{R}}^{q\times N}$, and $\mathbf{Z}\in {\mathbf{R}}^{s\times N}$;  1. Calculate ${\mathbf{K}}_X=\varphi {(\mathbf{X})}^T\varphi (\mathbf{X})$, ${\mathbf{K}}_Y=\psi {(\mathbf{Y})}^T\psi (\mathbf{Y})$, ${\mathbf{K}}_Z=\phi {(\mathbf{Y})}^T\phi (\mathbf{Y})$; Repeating four steps until convergence:  1. Update $\mathbf{D}$ when $\mathbf{H}$ is fixed: $\mathbf{D}={\mathbf{K}}_X^{-1}{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{[{\mathbf{H}}^T{(\mathbf{H}{\mathbf{H}}^T)}^{-1}\mathbf{H}]}^{-1}$;  2. Update $\mathbf{E}$ when $\mathbf{H}$ is fixed: $\mathbf{E}={\mathbf{K}}_Y^{-1}{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{[{\mathbf{H}}^T{(\mathbf{H}{\mathbf{H}}^T)}^{-1}\mathbf{H}]}^{-1}$;  3. Update $\mathbf{G}$ when $\mathbf{H}$ is fixed: $\mathbf{G}={\mathbf{K}}_Z^{-1}{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{[{\mathbf{H}}^T{(\mathbf{H}{\mathbf{H}}^T)}^{-1}\mathbf{H}]}^{-1}$;  4. Update $\mathbf{H}$ when $\mathbf{D}$, $\mathbf{E}$, and $\mathbf{G}$ are all fixed: $\mathbf{H}=(\mathbf{J}{\mathbf{K}}_X\mathbf{D}+\mathbf{J}{\mathbf{K}}_Y\mathbf{E}+\mathbf{J}{\mathbf{K}}_Z\mathbf{G})$ ${({\mathbf{D}}^T{\mathbf{K}}_X{\mathbf{K}}_X\mathbf{D}+{\mathbf{E}}^T{\mathbf{K}}_Y{\mathbf{K}}_Y\mathbf{E}+{\mathbf{G}}^T{\mathbf{K}}_Z{\mathbf{K}}_Z\mathbf{G})}^{-1}$; Output: $\mathbf{D}$, $\mathbf{E}$, $\mathbf{G}$.

Lastly, the SLSMKCCA method carries out facial expression, speech, and body gesture feature fusion by [3,8,11,15,19,22,31]

$$\begin{array}{c}\hfill \left(\begin{array}{c}{\mathbf{D}}^T{\mathbf{K}}_X\hfill \\ {\mathbf{E}}^T{\mathbf{K}}_Y\hfill \\ {\mathbf{G}}^T{\mathbf{K}}_Z\hfill \end{array}\right).\end{array}$$

(25)

5. Sparse Supervised Least Squares Multiset Kernel Canonical Correlation Analysis

In recent years, based on the MCCA method, the sparse multiset canonical correlation analysis (SMCCA) [69,70,71] method has also been proposed and applied to implement feature fusion in applications such as genomic data analysis, neoadjuvant breast cancer prediction, imaging, and (epi)genomics [69,70,72].

Different from the above SMCCA algorithm, we present the sparse supervised least squares multiset kernel canonical correlation analysis (SSLSMKCCA) algorithm as follows based on our SLSMKCCA algorithm in Section 4:

$$\begin{array}{ccc}\hfill & & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\mathbf{H}}{min}{\mathbf{W}}_{SSLSMKCCA}\hfill \\ \hfill =& & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\mathbf{H}}{min}\parallel {(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{J}-\mathbf{H}{\mathbf{D}}^T{\mathbf{K}}_X){\parallel }_F^2\hfill \\ \hfill & & +\parallel {(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{J}-\mathbf{H}{\mathbf{E}}^T{\mathbf{K}}_Y){\parallel }_F^2\hfill \\ \hfill & & +\parallel {(\mathbf{J}{\mathbf{J}}^T)}^{-{\displaystyle \frac{1}{2}}}(\mathbf{J}-\mathbf{H}{\mathbf{G}}^T{\mathbf{K}}_Z){\parallel }_F^2\hfill \\ & & +{\lambda }_D{\parallel \mathbf{D}\parallel }_1+{\lambda }_E{\parallel \mathbf{E}\parallel }_1+{\lambda }_G{\parallel \mathbf{G}\parallel }_1,\hfill \end{array}$$

(26)

where ${\lambda }_D$, ${\lambda }_E$, and ${\lambda }_G$ denote the L1 Norm parameter for $\mathbf{D}$, $\mathbf{E}$, and $\mathbf{G}$, respectively.

Then, the formulation (26) of the SSLSMKCCA algorithm can also be written as

$$\begin{array}{ccc}\hfill & & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\mathbf{H}}{min}{\mathbf{W}}_{SSLSMKCCA}=\hfill \\ \hfill & & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\mathbf{H}}{min}tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\mathbf{D}}^T{\mathbf{K}}_X{\mathbf{K}}_X\mathbf{D}{\mathbf{H}}^T\}+\hfill \\ \hfill & & tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\mathbf{E}}^T{\mathbf{K}}_Y{\mathbf{K}}_Y\mathbf{E}{\mathbf{H}}^T\}+\hfill \\ \hfill & & tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\mathbf{G}}^T{\mathbf{K}}_Z{\mathbf{K}}_Z\mathbf{G}{\mathbf{H}}^T\}-\hfill \\ \hfill & & 2tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_X\mathbf{D}{\mathbf{H}}^T\}-2tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_Y\mathbf{E}{\mathbf{H}}^T\}\hfill \\ \hfill & & -2tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_Z\mathbf{G}{\mathbf{H}}^T\}\hfill \\ & & +{\lambda }_D{\parallel \mathbf{D}\parallel }_1+{\lambda }_E{\parallel \mathbf{E}\parallel }_1+{\lambda }_G{\parallel \mathbf{G}\parallel }_1,\hfill \end{array}$$

(27)

Based on the augmented Lagrangian multiplier approach [3,15,22,49,51,52,53,54,55,56], we can add three equality constraints $\tilde{\mathbf{D}}=\mathbf{D}$, $\tilde{\mathbf{E}}=\mathbf{E}$, and $\tilde{\mathbf{G}}=\mathbf{G}$ for formulation (27); then, the formulation (27) can be noted as

$$\begin{array}{ccc}\hfill & & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\tilde{\mathbf{D}},\tilde{\mathbf{E}},\tilde{\mathbf{G}},\mathbf{H}}{min}{\mathbf{W}}_{SSLSMKCCA}=\hfill \\ \hfill & & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G},\tilde{\mathbf{D}},\tilde{\mathbf{E}},\tilde{\mathbf{G}},\mathbf{H}}{min}tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\tilde{\mathbf{D}}}^T{\mathbf{K}}_X{\mathbf{K}}_X\tilde{\mathbf{D}}{\mathbf{H}}^T\}\hfill \\ \hfill & & +tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\tilde{\mathbf{E}}}^T{\mathbf{K}}_Y{\mathbf{K}}_Y\tilde{\mathbf{E}}{\mathbf{H}}^T\}+\hfill \\ \hfill & & tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\tilde{\mathbf{G}}}^T{\mathbf{K}}_Z{\mathbf{K}}_Z\tilde{\mathbf{G}}{\mathbf{H}}^T\}-\hfill \\ \hfill & & 2tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_X\tilde{\mathbf{D}}{\mathbf{H}}^T\}-2tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_Y\tilde{\mathbf{E}}{\mathbf{H}}^T\}\hfill \\ \hfill & & -2tr\{{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_Z\tilde{\mathbf{G}}{\mathbf{H}}^T\}\hfill \\ \hfill & & +{\displaystyle \frac{{\beta }_D}{2}}\parallel \tilde{\mathbf{D}}-\mathbf{D}{)\parallel }_F^2+tr\{{\Gamma }_D^T(\tilde{\mathbf{D}}-\mathbf{D})\}\hfill \\ \hfill & & +{\displaystyle \frac{{\beta }_E}{2}}\parallel \tilde{\mathbf{E}}-\mathbf{E}{)\parallel }_F^2+tr\{{\Gamma }_E^T(\tilde{\mathbf{E}}-\mathbf{E})\}\hfill \\ \hfill & & +{\displaystyle \frac{{\beta }_G}{2}}\parallel \tilde{\mathbf{G}}-\mathbf{G}{)\parallel }_F^2+tr\{{\Gamma }_G^T(\tilde{\mathbf{G}}-\mathbf{G})\}\hfill \\ & & +{\lambda }_D{\parallel \mathbf{D}\parallel }_1+{\lambda }_E{\parallel \mathbf{E}\parallel }_1+{\lambda }_G{\parallel \mathbf{G}\parallel }_1,\hfill \end{array}$$

(28)

where the set of parameters ${\beta }_D$, ${\beta }_E$, and ${\beta }_G$ denote three constants, and the set of parameters ${\Gamma }_D$, ${\Gamma }_E$, and ${\Gamma }_G$ denote three Lagrangian multiplication matrices.

Then,

$$\begin{array}{ccc}\hfill & & {\displaystyle \frac{\partial {\mathbf{W}}_{SSLSMKCCA}}{\partial \tilde{\mathbf{D}}}}=2{\mathbf{K}}_X{\mathbf{K}}_X\tilde{\mathbf{D}}{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}-\hfill \\ & & 2{\mathbf{K}}_X{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}+{\Gamma }_D+{\beta }_D\tilde{\mathbf{D}}-{\beta }_D\mathbf{D}=0,\hfill \end{array}$$

(29)

$$\begin{array}{ccc}\hfill & & {\displaystyle \frac{\partial {\mathbf{W}}_{SSLSMKCCA}}{\partial \tilde{\mathbf{E}}}}=2{\mathbf{K}}_Y{\mathbf{K}}_Y\tilde{\mathbf{E}}{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}-\hfill \\ & & 2{\mathbf{K}}_Y{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}+{\Gamma }_E+{\beta }_E\tilde{\mathbf{E}}-{\beta }_E\mathbf{E}=0,\hfill \end{array}$$

(30)

$$\begin{array}{ccc}\hfill & & {\displaystyle \frac{\partial {\mathbf{W}}_{SSLSMKCCA}}{\partial \tilde{\mathbf{G}}}}=2{\mathbf{K}}_Z{\mathbf{K}}_Z\tilde{\mathbf{G}}{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}-\hfill \\ & & 2{\mathbf{K}}_Z{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}+{\Gamma }_G+{\beta }_G\tilde{\mathbf{G}}-{\beta }_G\mathbf{G}=0,\hfill \end{array}$$

(31)

$$\begin{array}{ccc}\hfill & & {\displaystyle \frac{\partial {\mathbf{W}}_{SSLSMKCCA}}{\partial \mathbf{H}}}=2{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\tilde{\mathbf{D}}}^T{\mathbf{K}}_X{\mathbf{K}}_X\tilde{\mathbf{D}}+\hfill \\ \hfill & & 2{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\tilde{\mathbf{E}}}^T{\mathbf{K}}_Y{\mathbf{K}}_Y\tilde{\mathbf{E}}+2{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}{\tilde{\mathbf{G}}}^T{\mathbf{K}}_Z{\mathbf{K}}_Z\tilde{\mathbf{G}}\hfill \\ \hfill & & -2{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_X\tilde{\mathbf{D}}-2{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_Y\tilde{\mathbf{E}}\hfill \\ & & -2{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{J}{\mathbf{K}}_Z\tilde{\mathbf{G}}=0,\hfill \end{array}$$

(32)

then, $\tilde{\mathbf{D}}$, $\tilde{\mathbf{E}}$, $\tilde{\mathbf{G}}$, and $\mathbf{H}$ can be shown as

$$\begin{array}{c}\hfill \tilde{\mathbf{D}}=\mathbf{D}+{\displaystyle \frac{2{\mathbf{K}}_X{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}}{{\beta }_D}}-{\displaystyle \frac{{\Gamma }_D}{{\beta }_D}}-\\ \hfill {\displaystyle \frac{2{\mathbf{K}}_X{\mathbf{K}}_X\tilde{\mathbf{D}}{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}}{{\beta }_D}},\end{array}$$

(33)

$$\begin{array}{c}\hfill \tilde{\mathbf{E}}=\mathbf{E}+{\displaystyle \frac{2{\mathbf{K}}_Y{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}}{{\beta }_E}}-{\displaystyle \frac{{\Gamma }_E}{{\beta }_E}}-\\ \hfill {\displaystyle \frac{2{\mathbf{K}}_Y{\mathbf{K}}_y\tilde{\mathbf{E}}{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}}{{\beta }_E}},\end{array}$$

(34)

$$\begin{array}{c}\hfill \tilde{\mathbf{G}}=\mathbf{G}+{\displaystyle \frac{2{\mathbf{K}}_Z{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}}{{\beta }_G}}-{\displaystyle \frac{{\Gamma }_G}{{\beta }_G}}-\\ \hfill {\displaystyle \frac{2{\mathbf{K}}_Z{\mathbf{K}}_Z\tilde{\mathbf{G}}{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}}{{\beta }_G}},\end{array}$$

(35)

$$\begin{array}{ccc}\hfill \mathbf{H}=& & (\mathbf{J}{\mathbf{K}}_X\tilde{\mathbf{D}}+\mathbf{J}{\mathbf{K}}_Y\tilde{\mathbf{E}}+\mathbf{J}{\mathbf{K}}_Z\tilde{\mathbf{G}})\hfill \\ \hfill & & ({\tilde{\mathbf{D}}}^T{\mathbf{K}}_X{\mathbf{K}}_X\tilde{\mathbf{D}}+{\tilde{\mathbf{E}}}^T{\mathbf{K}}_Y{\mathbf{K}}_Y\tilde{\mathbf{E}}+\hfill \\ & & {\tilde{\mathbf{G}}}^T{\mathbf{K}}_Z{\mathbf{K}}_Z\tilde{\mathbf{G}}{)}^{-1}.\hfill \end{array}$$

(36)

When $\tilde{\mathbf{D}}$, $\tilde{\mathbf{E}}$, $\tilde{\mathbf{G}}$, and $\mathbf{H}$ are fixed, we only need to solve $\mathbf{D}$, $\mathbf{E}$, and $\mathbf{G}$, and formulation (28) of the SSLSMKCCA algorithm can be reformulated as

$$\begin{array}{ccc}\hfill & & arg\underset{\mathbf{D},\mathbf{E},\mathbf{G}}{min}{\mathbf{W}}_{SSLSMKCCA}=\hfill \\ \hfill & & +{\displaystyle \frac{{\beta }_D}{2}}\parallel (\tilde{\mathbf{D}}-\mathbf{D}{)\parallel }_F^2+tr\{{\Gamma }_D^T(\tilde{\mathbf{D}}-\mathbf{D})\}\hfill \\ \hfill & & +{\displaystyle \frac{{\beta }_E}{2}}\parallel (\tilde{\mathbf{E}}-\mathbf{E}{)\parallel }_F^2+tr\{{\Gamma }_E^T(\tilde{\mathbf{E}}-\mathbf{E})\}\hfill \\ \hfill & & +{\displaystyle \frac{{\beta }_G}{2}}\parallel (\tilde{\mathbf{G}}-\mathbf{G}{)\parallel }_F^2+tr\{{\Gamma }_G^T(\tilde{\mathbf{G}}-\mathbf{G})\}\hfill \\ & & +{\lambda }_D{\parallel \mathbf{D}\parallel }_1+{\lambda }_E{\parallel \mathbf{E}\parallel }_1+{\lambda }_G{\parallel \mathbf{G}\parallel }_1,\hfill \end{array}$$

(37)

which can be resolved into

$$\begin{array}{c}\hfill arg\underset{\mathbf{D}}{min}{\displaystyle \frac{{\lambda }_D}{{\beta }_D}}{\parallel \mathbf{D}\parallel }_1+{\displaystyle \frac{1}{2}}\parallel \mathbf{D}-(\tilde{\mathbf{D}}+{\displaystyle \frac{{\Gamma }_D}{{\beta }_D}}){\parallel }_F^2,\end{array}$$

(38)

$$\begin{array}{c}\hfill arg\underset{\mathbf{E}}{min}{\displaystyle \frac{{\lambda }_E}{{\beta }_E}}{\parallel \mathbf{E}\parallel }_1+{\displaystyle \frac{1}{2}}\parallel \mathbf{E}-(\tilde{\mathbf{E}}+{\displaystyle \frac{{\Gamma }_E}{{\beta }_E}}){\parallel }_F^2,\end{array}$$

(39)

$$\begin{array}{c}\hfill arg\underset{\mathbf{G}}{min}{\displaystyle \frac{{\lambda }_G}{{\beta }_G}}{\parallel \mathbf{G}\parallel }_1+{\displaystyle \frac{1}{2}}\parallel \mathbf{G}-(\tilde{\mathbf{G}}+{\displaystyle \frac{{\Gamma }_G}{{\beta }_G}}){\parallel }_F^2;\end{array}$$

(40)

then, based on the Lemma in [3,22,49,52,53,54,56], $\mathbf{D}$, $\mathbf{E}$, and $\mathbf{G}$ can be shown as

$$\begin{array}{ccc}\hfill & & \mathbf{D}=U_{{\displaystyle \frac{{\lambda }_D}{{\beta }_D}}}[\tilde{\mathbf{D}}+{\displaystyle \frac{{\Gamma }_D}{{\beta }_D}}]=\hfill \\ & & \left\{\begin{array}{cc}{(\tilde{\mathbf{D}}+{\displaystyle \frac{{\Gamma }_D}{{\beta }_D}})}_{ij}-{\displaystyle \frac{{\lambda }_D}{{\beta }_D}},\hfill & if{(\tilde{\mathbf{D}}+{\displaystyle \frac{{\Gamma }_D}{{\beta }_D}})}_{ij}>{\displaystyle \frac{{\lambda }_D}{{\beta }_D}}\hfill \\ {(\tilde{\mathbf{D}}+{\displaystyle \frac{{\Gamma }_D}{{\beta }_D}})}_{ij}+{\displaystyle \frac{{\lambda }_D}{{\beta }_D}},\hfill & if{(\tilde{\mathbf{D}}+{\displaystyle \frac{{\Gamma }_D}{{\beta }_D}})}_{ij}<-{\displaystyle \frac{{\lambda }_D}{{\beta }_D}},\hfill \\ 0,\hfill & otherwise\hfill \end{array}\right.\hfill \end{array}$$

(41)

$$\begin{array}{ccc}\hfill & & \mathbf{E}=U_{{\displaystyle \frac{{\lambda }_E}{{\beta }_E}}}[\tilde{\mathbf{E}}+{\displaystyle \frac{{\Gamma }_E}{{\beta }_E}}]=\hfill \\ & & \left\{\begin{array}{cc}{(\tilde{\mathbf{E}}+{\displaystyle \frac{{\Gamma }_E}{{\beta }_E}})}_{ij}-{\displaystyle \frac{{\lambda }_E}{{\beta }_E}},\hfill & if{(\tilde{\mathbf{E}}+{\displaystyle \frac{{\Gamma }_E}{{\beta }_E}})}_{ij}>{\displaystyle \frac{{\lambda }_E}{{\beta }_E}}\hfill \\ {(\tilde{\mathbf{E}}+{\displaystyle \frac{{\Gamma }_E}{{\beta }_E}})}_{ij}+{\displaystyle \frac{{\lambda }_E}{{\beta }_E}},\hfill & if{(\tilde{\mathbf{E}}+{\displaystyle \frac{{\Gamma }_E}{{\beta }_E}})}_{ij}<-{\displaystyle \frac{{\lambda }_E}{{\beta }_E}},\hfill \\ 0,\hfill & otherwise\hfill \end{array}\right.\hfill \end{array}$$

(42)

$$\begin{array}{ccc}\hfill & & \mathbf{G}=U_{{\displaystyle \frac{{\lambda }_G}{{\beta }_G}}}[\tilde{\mathbf{G}}+{\displaystyle \frac{{\Gamma }_G}{{\beta }_G}}]=\hfill \\ & & \left\{\begin{array}{cc}{(\tilde{\mathbf{G}}+{\displaystyle \frac{{\Gamma }_G}{{\beta }_G}})}_{ij}-{\displaystyle \frac{{\lambda }_G}{{\beta }_G}},\hfill & if{(\tilde{\mathbf{G}}+{\displaystyle \frac{{\Gamma }_G}{{\beta }_G}})}_{ij}>{\displaystyle \frac{{\lambda }_G}{{\beta }_G}}\hfill \\ {(\tilde{\mathbf{G}}+{\displaystyle \frac{{\Gamma }_G}{{\beta }_G}})}_{ij}+{\displaystyle \frac{{\lambda }_G}{{\beta }_G}},\hfill & if{(\tilde{\mathbf{G}}+{\displaystyle \frac{{\Gamma }_G}{{\beta }_G}})}_{ij}<-{\displaystyle \frac{{\lambda }_G}{{\beta }_G}},\hfill \\ 0,\hfill & otherwise\hfill \end{array}\right.\hfill \end{array}$$

(43)

According to the above derivation, the iterative solution algorithm of the SSLSMKCCA method using the alternated least squares [3,15,22,49,50,51,54] is shown in Algorithm 2. In our experiment, the initial matrices of ${\Gamma }_D$, ${\Gamma }_E$, and ${\Gamma }_G$ are all set to 0 matrix, and ${\rho }_D$, ${\rho }_E$, and ${\rho }_G$ are all set to $1.1$. ${\lambda }_D$, ${\lambda }_E$, and ${\lambda }_G$ are all set to $0.88$, and ${\beta }_D$, ${\beta }_E$, and ${\beta }_G$ are all set to $2\times {10}^5$.

Algorithm 2: SSLSMKCCA

Input: Label matrix $\mathbf{J}\in {\mathbf{R}}^{m\times N}$, three feature matrices $\mathbf{X}\in {\mathbf{R}}^{p\times N}$, $\mathbf{Y}\in {\mathbf{R}}^{q\times N}$, and $\mathbf{Z}\in {\mathbf{R}}^{s\times N}$, the set of constant parameters ${\beta }_D$, ${\beta }_E$, and ${\beta }_G$, the set of Lagrangian multiplication matrices ${\Gamma }_D$, ${\Gamma }_E$, and ${\Gamma }_G$, the set of L1 Norm parameters ${\lambda }_D$, ${\lambda }_E$, and ${\lambda }_G$, the set of constant parameters ${\rho }_D$, ${\rho }_E$, and ${\rho }_G$;  1. Calculate ${\mathbf{K}}_X=\varphi {(\mathbf{X})}^T\varphi (\mathbf{X})$, ${\mathbf{K}}_Y=\psi {(\mathbf{Y})}^T\psi (\mathbf{Y})$, ${\mathbf{K}}_Z=\phi {(\mathbf{Y})}^T\phi (\mathbf{Y})$; Repeating nine steps until convergence:  1. Update $\mathbf{D}$ when $\tilde{\mathbf{D}}$, ${\beta }_D$, and ${\Gamma }_D$ are fixed: $\mathbf{D}=U_{{\displaystyle \frac{{\lambda }_D}{{\beta }_D}}}[\tilde{\mathbf{D}}+{\displaystyle \frac{{\Gamma }_D}{{\beta }_D}}]$  2. Update $\mathbf{E}$ when $\tilde{\mathbf{E}}$, ${\beta }_E$, and ${\Gamma }_E$ are fixed: $\mathbf{E}=U_{{\displaystyle \frac{{\lambda }_E}{{\beta }_E}}}[\tilde{\mathbf{E}}+{\displaystyle \frac{{\Gamma }_E}{{\beta }_E}}]$  3. Update $\mathbf{G}$ when $\tilde{\mathbf{G}}$, ${\beta }_G$, and ${\Gamma }_G$ are fixed: $\mathbf{G}=U_{{\displaystyle \frac{{\lambda }_G}{{\beta }_G}}}[\tilde{\mathbf{G}}+{\displaystyle \frac{{\Gamma }_G}{{\beta }_G}}]$  4. Update $\tilde{\mathbf{D}}$ when ${\tilde{\mathbf{D}}}^{k-1}$ ($k-1$ denotes the previous iteration value), $\mathbf{D}$, ${\beta }_D$, and ${\Gamma }_D$ are fixed: $\tilde{\mathbf{D}}=\mathbf{D}+{\displaystyle \frac{2{\mathbf{K}}_X{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}}{{\beta }_D}}-{\displaystyle \frac{{\Gamma }_D}{{\beta }_D}}-$ ${\displaystyle \frac{2{\mathbf{K}}_X{\mathbf{K}}_X{\tilde{\mathbf{D}}}^{k-1}{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}}{{\beta }_D}}$;  5. Update $\tilde{\mathbf{E}}$ when ${\tilde{\mathbf{E}}}^{k-1}$, $\mathbf{E}$, ${\beta }_E$, and ${\Gamma }_E$ are fixed: $\tilde{\mathbf{E}}=\mathbf{E}+{\displaystyle \frac{2{\mathbf{K}}_Y{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}}{{\beta }_E}}-{\displaystyle \frac{{\Gamma }_E}{{\beta }_E}}-$ ${\displaystyle \frac{2{\mathbf{K}}_Y{\mathbf{K}}_Y{\tilde{\mathbf{E}}}^{k-1}{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}}{{\beta }_E}}$;  6. Update $\tilde{\mathbf{G}}$ when ${\tilde{\mathbf{G}}}^{k-1}$, $\mathbf{G}$, ${\beta }_G$ and ${\Gamma }_G$ are fixed: $\tilde{\mathbf{G}}=\mathbf{G}+{\displaystyle \frac{2{\mathbf{K}}_Z{\mathbf{J}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}}{{\beta }_G}}-{\displaystyle \frac{{\Gamma }_G}{{\beta }_G}}-$ ${\displaystyle \frac{2{\mathbf{K}}_Z{\mathbf{K}}_Z{\tilde{\mathbf{G}}}^{k-1}{\mathbf{H}}^T{(\mathbf{J}{\mathbf{J}}^T)}^{-1}\mathbf{H}}{{\beta }_G}}$;  7. Update $\mathbf{H}$ when $\tilde{\mathbf{D}}$, $\tilde{\mathbf{E}}$, and $\tilde{\mathbf{G}}$ are all fixed: $\mathbf{H}=(\mathbf{J}{\mathbf{K}}_X\tilde{\mathbf{D}}+\mathbf{J}{\mathbf{K}}_Y\tilde{\mathbf{E}}+\mathbf{J}{\mathbf{K}}_Z\tilde{\mathbf{G}})$ $({\tilde{\mathbf{D}}}^T{\mathbf{K}}_X{\mathbf{K}}_X\tilde{\mathbf{D}}+{\tilde{\mathbf{E}}}^T{\mathbf{K}}_Y{\mathbf{K}}_Y\tilde{\mathbf{E}}+$ ${\tilde{\mathbf{G}}}^T{\mathbf{K}}_Z{\mathbf{K}}_Z\tilde{\mathbf{G}}{)}^{-1}$;  8. Update ${\Gamma }_D$, ${\Gamma }_E$, and ${\Gamma }_G$: ${\Gamma }_D={\Gamma }_D^{k-1}+{\beta }_D(\tilde{\mathbf{D}}-{\tilde{\mathbf{D}}}^{k-1})$, ${\Gamma }_E={\Gamma }_E^{k-1}+{\beta }_E(\tilde{\mathbf{E}}-{\tilde{\mathbf{E}}}^{k-1})$, ${\Gamma }_G={\Gamma }_G^{k-1}+{\beta }_G(\tilde{\mathbf{G}}-{\tilde{\mathbf{G}}}^{k-1})$,  9. Update ${\beta }_D$, ${\beta }_E$, and ${\alpha }_G$: ${\beta }_D={\rho }_D{\beta }_D^{k-1}$, ${\beta }_E={\rho }_E{\beta }_E^{k-1}$, ${\beta }_G={\rho }_G{\beta }_G^{k-1}$; Output: $\mathbf{D}$, $\mathbf{E}$, $\mathbf{G}$.

Lastly, the SSLSMKCCA method also carries out facial expression, speech, and body gesture feature fusion by [3,8,11,15,19,22,31]

$$\begin{array}{c}\hfill \left(\begin{array}{c}{\mathbf{D}}^T{\mathbf{K}}_X\hfill \\ {\mathbf{E}}^T{\mathbf{K}}_Y\hfill \\ {\mathbf{G}}^T{\mathbf{K}}_Z\hfill \end{array}\right).\end{array}$$

(44)

6. Experiments

In this section, we conducted single-modal and multimodal emotion recognition experiments using the GEMEP and Polish databases to validate the effectiveness of our proposed SLSMKCCA and SSLSMKCCA algorithms, and compared them with other methods.

6.1. The GEMEP Multimodal Emotion Database

Among all of the existing emotion databases, monomodal emotion database and bimodal emotion database are relatively common [9,10,12,21,25]. However, multimodal emotion databases (three modalities or more than three modalities) are relatively rare, especially for the case of a multimodal emotion database consisting of facial expressions, speech, and body gestures. The Geneva Multimodal Emotion Portrayals (GEMEP) [42,43] is a public multimodal emotion database that contains this combination of modalities, built by Banziger, Mortillaro, and Scherer at the University of Geneva in recent years.

The original GEMEP database, in total, collected 1260 samples from 10 professional actors. That is to say, each modality contains 1260 samples. Banziger et al. evaluated the validity of the 1260 samples by rating, and, lastly, only kept 145 reliable samples for public usage and abandoned most of the samples [42,43]. The reserved parts of the GEMEP database cover 17 emotion categories including tenderness, cold anger (irritation), amusement, anxiety, contempt, despair, pride, hot anger (rage), interest, fear (panic), joy (elation), surprise, pleasure (sensory), relief, sadness, admiration, and disgust [42,43].

Although the GEMEP database has 17 emotion categories, some categories are similar to each other. For example, the emotion categories hot anger (rage) and cold anger (irritation) are actually similar in terms of arousal and, particularly, valence (it refers to the degree of pleasure or displeasure associated with an emotional experience), essentially measuring the positive or negative characteristics of an emotion. Moreover, the sample number of each emotion category is very small. Therefore, we clustered the similar emotion categories into the same emotion category in the following. In our experiment, we specifically merged the emotion categories cold anger (irritation) and hot anger (rage) as the ‘anger’ emotion category, the emotion categories contempt and disgust as the ‘disgust’ emotion category, the emotion categories anxiety and fear (panic) as the ‘fear’ emotion category, the emotion categories amusement, joy (elation), pleasure (sensory), and pride as the ‘happy’ emotion category, the emotion categories of despair and sad as the ‘sad’ emotion category, the emotion categories surprise and admiration as the ‘surprise’ emotion category, and the emotion categories of interest, relief, and tenderness as the ‘calm’ emotion category. Thus, we obtained seven emotion categories from the GEMEP database after merging.

Table 1. The information about the seven emotion categories and the number of samples in our experiment.

Category	Included Categories	Number of Samples
Anger	Cold anger, Hot anger	20
Disgust	Contempt, Disgust	10
Fear	Anxiety, Fear	20
Happy	Amusement, Joy, Pleasure, Pride	40
Sad	Despair, Sad	20
Surprise	Surprise, Admiration	10
Calm	Interest, Relief, Tenderness	25
All		145

provides the information on the seen emotion categories and the number of samples in our experiments.

Due to the sample size of the GEMEP database being too small (each modality only provides 145 samples), we utilized traditional methods rather than a deep learning method for feature extraction of facial expressions, body gesture analysis, and speech analysis. The feature extraction methods for the three modalities used in our experiment are summarized as follows. Moreover, the classifier, experimental strategy, comparison fusion method, kernel function, and other information used in our experiment are given together in Table 2.

• Facial expression modality: Local Binary Patterns (LBPs) [73] were used to extract features from facial expression images from the GEMEP database. Those facial expression images were selected from facial expression videos from the GEMEP database by manual selection, and every facial expression video corresponds to one facial expression image. The resolution of those facial expression images is $128\times 128$. The facial expression feature dimension of LBP was 3776 in our experiment.
• Body gesture modality: spatial–temporal features [8,11,15,31,74] were used to extract body gesture features from body gesture videos from the GEMEP database. The dimension of the spatial-temporal features used in our experiment was 200.
• Speech modality: the openSMILE tool [3,19,22,75] was used to extract speech features (“emobase” set) from the audio data of the GEMEP database. The speech feature dimension used in our experiment was 1582.

Table 2. Information on the experiments run on the GEMEP database.

GEMEP database	10 subjects; 145 samples; 3 modalities; 7 categories
Feature extraction	facial expressions: LBP; body gestures: spatial–temporal features; speech: openSMILE
Fusion method	PCA fusion [31]; CCA [58]; kernel CCA (KCCA) [58]; MCCA [61]; MKCCA [44,45]; SMCCA [69]; SLSMKCCA; SSLSMKCCA; decision-level fusion [76]
Classifier	support vector machines (SVMs)
Strategy	leave one subject out
Kernel function	Gaussian kernel

Table 2. Information on the experiments run on the GEMEP database.

GEMEP database	10 subjects; 145 samples; 3 modalities; 7 categories
Feature extraction	facial expressions: LBP; body gestures: spatial–temporal features; speech: openSMILE
Fusion method	PCA fusion [31]; CCA [58]; kernel CCA (KCCA) [58]; MCCA [61]; MKCCA [44,45]; SMCCA [69]; SLSMKCCA; SSLSMKCCA; decision-level fusion [76]
Classifier	support vector machines (SVMs)
Strategy	leave one subject out
Kernel function	Gaussian kernel

(1) Monomodal emotion recognition experimental results: Firstly, we conducted three sets of monomodal emotion recognition experiments based on facial expressions, body gestures, and speech modality. The monomodal emotion recognition experimental results on the GEMEP database are shown in

Table 3. Three sets of monomodal emotion recognition experimental results on the GEMEP database (ARR = average recognition rate).

Modality	ARR (%)	Recall (%)	F1-Score (%)
Facial expressions	55.2	48.5	51.2
Body gestures	29.7	23.5	23.6
Speech	44.8	36.9	35.1

.

From Table 3, we can see that the recognition rates of three sets of monomodal emotion recognition experiments are not high, especially for body gestures as a modality. The recognition rates of facial expression and speech analysis are higher than for body gesture analysis, obviously, especially in comparison to the facial expression modality.

(2) Bimodal emotion recognition experimental results: Secondly, we conducted three sets of bimodal emotion recognition experiments between facial expressions, body gestures, and speech using PCA fusion, CCA, and KCCA. Three sets of bimodal emotion recognition experiments included bimodal emotion recognition based on facial expressions and body gestures, bimodal emotion recognition based on facial expressions and speech, and, finally, bimodal emotion recognition based on speech and body gestures. The corresponding experimental results on the GEMEP database are shown in

Table 4. Three sets of bimodal emotion recognition experimental results on the GEMEP database (ARR = average recognition rate).

Method (Modality)	ARR (%)	Recall (%)	F1-Score (%)
PCA (facial and gesture)	49.0	42.0	43.5
CCA (facial and gesture)	55.9	48.9	51.6
KCCA (facial and gesture)	55.2	48.5	51.3
PCA (facial and speech)	52.4	42.5	42.4
CCA (facial and speech)	58.0	48.8	49.9
KCCA (facial and speech)	58.0	48.8	49.9
PCA (body and speech)	42.8	34.3	33.6
CCA (body and speech)	42.8	34.4	32.1
KCCA (body and speech)	42.1	33.7	31.5

.

From Table 4, we can see that the recognition rates of bimodal emotion recognition based on facial expressions and speech are higher than bimodal emotion recognition based on facial expressions and body gestures and bimodal emotion recognition based on speech and body gestures for all fusion methods. The recognition rates of bimodal emotion recognition based on speech and body gestures are visibly low.

(3) Multimodal emotion recognition experimental results: Lastly, we conducted the experiment aiming for ‘full’ multimodal emotion recognition based on facial expressions, body gestures, and speech as modalities using PCA fusion, MCCA, MKCCA, SMCCA, Convolutional Neural Networks (CNNs) [77], decision-level fusion, and our presented SLSMKCCA and SSLSMKCCA fusion methods. The corresponding multimodal emotion recognition experimental results on the GEMEP database are provided in

Table 5. Multimodal emotion recognition experimental results on the GEMEP database (ARR = Average recognition rate).

Method (Facial, Gesture, Speech)	ARR (%)	Recall (%)	F1-Score (%)
PCA	52.4	43.0	43.1
MCCA	57.9	47.7	48.6
MKCCA	56.6	50.9	52.3
SMCCA	58.6	48.3	49.2
CNN	48.3	48.5	48.8
Feature level fusion + Bayesian [17,18,36]	57.9	47.7	48.6
Decision level fusion + Bayesian [17,18,35,36]	58.5	50.0	51.6
SLSMKCCA	57.2	44.9	45.8
SSLSMKCCA	61.4	52.3	54.3

. Moreover, the confusion matrices of the SLSMKCCA and SSLSMKCCA fusion methods on the GEMEP database are shown in Figure 1 and Figure 2, respectively.

From Table 3, Table 4 and Table 5 and Figure 1 and Figure 2, we can see that the recognition rate of multimodal emotion recognition is higher than the ones of bimodal and monomodal emotion recognition, on average, and especially higher than for monomodal emotion recognition. Our presented SLSMKCCA and SSLSMKCCA fusion methods both obtain high recognition rates, especially for SSLSMKCCA as a fusion method, and our SSLSMKCCA method obtains the best recognition rate amongst all monomodal, bimodal, and multimodal emotion recognition experimental results. In addition, the multimodal emotion recognition results of CNNs are poor due to the limited sample number of the GEMEP database.

6.2. The Polish Multimodal Emotion Database

The Polish emotional database [41] is another public multimodal emotion database that contains the modality of facial expressions, speech, and body gestures. It collected 560 samples from 16 professional actors for each modality and covers 7 emotion categories such as surprise, sadness, happiness, neutral, fear, anger, and disgust.

The feature extraction method for facial expression and speech modality on the Polish database [41] is the same as the aforementioned GEMEP database (for details, refer to the feature extraction part of the GEMEP database in Section A). However, the feature extraction method for the body gesture modality is different from the GEMEP database. As the Polish database only provides the point cloud data (each frame contains 25 key points and each key point contains three location data points and three rotation data points) instead of video data for body gesture modality. We selected 10 frames from each body gesture sample at a uniform interval and concatenated the point cloud data of 10 frames as the feature vector. Finally, the body gesture feature dimension of the Polish database was 1500 ($10\times 25\times (3+3)=1500$) in our experiment.

Moreover, the classifier, experimental strategy, comparison fusion method, and kernel function used in the experiment of the Polish database were the same as the aforementioned GEMEP database, which are given together in Table 2.

(1) Monomodal emotion recognition experimental results: Similar to the GEMEP database, we also conducted three sets of monomodal emotion recognition experiments on the Polish database first. The monomodal emotion recognition experimental results are shown in

Table 6. Three sets of monomodal emotion recognition experimental results on the Polish database (ARR = average recognition rate).

Modality	ARR (%)	Recall (%)	F1-Score (%)
Facial expressions	51.4	51.4	50.9
Body gestures	52.5	52.3	52.3
Speech	53.9	53.5	53.0

.

From Table 6, we can see that the recognition rates of three sets of monomodal emotion recognition experiments on the Polish database are higher than the GEMEP database in general, especially for body gestures as a modality. The main reason is that the Polish database covers more samples than the GEMEP database.

(2) Bimodal emotion recognition experimental results: Secondly, we also conducted three sets of bimodal emotion recognition experiments between facial expressions, body gestures, and speech using PCA fusion, CCA, and KCCA on the Polish database. Similar to the GEMEP database, the corresponding three sets of bimodal emotion recognition experiment results on the Polish database are shown in

Table 7. Three sets of bimodal emotion recognition experimental results on the Polish database (ARR = average recognition rate).

Method (Modality)	ARR (%)	Recall (%)	F1-Score (%)
PCA (facial and gesture)	65.1	64.7	64.2
CCA (facial and gesture)	63.3	62.8	62.2
KCCA (facial and gesture)	63.3	63.1	62.5
PCA (facial and speech)	60.7	60.2	60.1
CCA (facial and speech)	61.8	61.6	61.2
KCCA (facial and speech)	60.7	60.2	59.4
PCA (body and speech)	61.1	60.8	60.5
CCA (body and speech)	63.1	62.9	62.7
KCCA (body and speech)	62.2	62.0	61.5

.

From Table 7, we can see that the recognition rates of the three sets of bimodal emotion recognition experiments are all better than the three sets of monomodal emotion recognition for all fusion methods, which demonstrates the effectiveness of the fusion strategy.

(3) Multimodal emotion recognition experimental results: Similar to the GEMEP database, we also conducted the experiment aiming for ‘full’ multimodal emotion recognition on the Polish database. The corresponding multimodal emotion recognition experimental results on the Polish database are provided in

Table 8. Multimodal emotion recognition experimental results on the Polish database (ARR = average recognition rate).

Method (Facial, Gesture, Speech)	ARR (%)	Recall (%)	F1-Score (%)
PCA	66.8	66.5	66.3
MCCA	67.3	66.9	67.0
MKCCA	64.2	63.9	63.6
SMCCA	67.7	67.4	67.3
CNN	57.5	57.4	57.2
Decision-level fusion + SVM	64.8	64.5	64.1
Feature level fusion + Bayesian [17,18,36]	62.0	61.7	61.4
Decision level fusion + Bayesian [17,18,35,36]	51.2	51.5	50.3
SLSMKCCA	63.3	62.9	61.6
SSLSMKCCA	68.1	67.7	67.6

. Moreover, the confusion matrices of the SLSMKCCA and SSLSMKCCA fusion methods are shown in Figure 3 and Figure 4 respectively.

From Table 6, Table 7 and Table 8 and Figure 3 and Figure 4, we also can see that the recognition rate of multimodal emotion recognition is higher than the ones of bimodal and monomodal emotion recognition, on average, which is similar to the GEMEP database. Moreover, our presented SLSMKCCA and SSLSMKCCA fusion methods both obtain high recognition rates, and the recognition rate of our SSLSMKCCA method is 68.1% and it obtains the best recognition rate amongst all monomodal, bimodal, and multimodal emotion recognition experimental results. In addition, the multimodal emotion recognition results of CNNs are poor due to the limited sample number of the Polish database.

From the experimental results on the GEMEP and Polish databases, our presented SLSMKCCA and SSLSMKCCA fusion methods both obtain high recognition rates, especially for the SSLSMKCCA fusion method, and the possible reason is that the SLSMKCCA method effectively establishes the correlation between three modal features and category labels. The SSLSMKCCA method adds a sparse item (L1 Norm) on the basis of the SLSMKCCA method, which can more effectively select useful features from the three modalities.

7. Conclusions

In this article, we focused on the rare case of three-fold multimodal emotion recognition from facial expressions, speech, and body gestures based on the GEMEP and Polish databases, including the feature extraction, feature fusion, and multimodal classification of the three modalities. In particular, for feature fusion, two novel algorithms, namely SLSMKCCA and SSLSMKCCA, were presented in this article to carry out facial expression, speech, and body gesture feature fusion. The experimental results on the GEMEP and Polish databases showed that the recognition rates of multimodal emotion recognition were considerably higher than those of bimodal and monomodal emotion recognition on average, and our presented SLSMKCCA and SSLSMKCCA fusion methods both obtained very high recognition rates, especially for the SSLSMKCCA fusion method. Due to the limited sample number of the GEMEP and Polish databases, the recognition rate of various methods is not high. Therefore, we aimed to build a larger multimodal emotion database that covers facial expressions, speech, and body gestures as modalities and test our presented SLSMKCCA and SSLSMKCCA fusion methods on a larger database in the future. Other future work of interest includes transfer learning and generative adversarial network topologies to better cope with the data sparseness faced. In addition, we also consider applying the SLSMKCCA and SSLSMKCCA algorithms to practical multimodal scenarios. In real-world settings, the proposed SLSMKCCA and SSLSMKCCA algorithms can accommodate additional modalities such as four or more modalities, demonstrating a certain degree of robustness. Moreover, our SSLSMKCCA algorithm incorporates a sparse term (L1 Norm) to select features and eliminate irrelevant noise.