Learning Modality Consistency and Difference Information with Multitask Learning for Multimodal Sentiment Analysis

Abstract

The primary challenge in Multimodal sentiment analysis (MSA) lies in developing robust joint representations that can effectively learn mutual information from diverse modalities. Previous research in this field tends to rely on feature concatenation to obtain joint representations. However, these approaches fail to fully exploit interactive patterns to ensure consistency and differentiation across different modalities. To address this limitation, we propose a novel framework for multimodal sentiment analysis, named CDML (Consistency and Difference using a Multitask Learning network). Specifically, CDML uses an attention mechanism to assign the attention weights of each modality efficiently. Adversarial training is used to obtain consistent information between modalities. Finally, the difference among the modalities is acquired by the multitask learning framework. Experiments on two benchmark MSA datasets, CMU-MOSI and CMU-MOSEI, showcase that our proposed method outperforms the seven existing approaches by at least 1.3% for Acc-2 and 1.7% for F1.

1. Introduction

With the recent surge in social media platforms, there has been an increasing focus on multimodal learning [1,2]. Multimodal data, consisting of Acoustic (Ac), Visual (Vi), and Language (La) modalities, have become prevalent in various applications. Multimodal sentiment analysis (MSA) [3,4,5,6] is an approach that leverages information from multiple modalities to enhance the prediction accuracy of emotional states. Researchers in MSA have primarily concentrated on studying intricate fusion models [7,8,9], such as tensor-based fusion [10] and attention-based fusion. However, despite the advancements made in previous studies using benchmark datasets, the fusion mechanisms employed in MSA still encounter difficulties in effectively addressing the existing modality gaps between diverse modalities. These modality gaps refer to the variances that exist among the different modalities, posing a challenge in integrating their information seamlessly for accurate sentiment prediction.

In the field of multimodal sentiment analysis, researchers have proposed various approaches to address the challenges associated with heterogeneous modalities. One such approach is the Multimodal Transformer (MulT) provided by Tsai et al. [11]. The model leverages a crossmodal attention mechanism to adapt information between different modalities. However, MulT ignores the interaction between modalities, and MulT can not obtain comprehensive joint representations. Another important aspect emphasized by Lv et al. [12] is the consideration of information exchange among all modalities for effective multimodal fusion. According to their argument, the capture of joint representations is a difficult task because of inherent differences in heterogeneity across modalities. Simply concatenating modality features to create a common representation does not fully explore the interrelationships between different modalities. On the other hand, projecting different modalities into the same representation space can achieve consistency but overlooks complementarity. In conclusion, the main challenges for multimodal sentiment analysis are capturing the joint representation and exploiting effective interactions across modalities. This involves the integration of consistency and complementarity to achieve enhanced accuracy in sentiment analysis.

To address the aforementioned challenges, we propose a novel framework for multimodal sentiment analysis called Consistency and Difference using a Multitask Learning network (CDML). CDML can integrate unimodal diversity and bimodal consistency in a unified framework. For bimodality, we put forward a Joint Representation Translator (JRTran) to integrate joint representations of Vi-La, Ac-La, and Vi-Ac. The JRTran employs cycle translation. Generators and discriminators are designed in the cycle translation for adversarial training, which enables the translated features to better approximate the source features and further improve the performance of sentiment analysis. The process of translating between a source modality and a target modality yields an intermediate representation that acquires consistent information. Additionally, the attention mechanism is introduced to effectively assign the attention weights of each modality. Based on attention weights, the features of each modality can be efficiently fused. In the meantime, the generation of redundant information and interference of nonimportant modalities can be effectively reduced. At the end of CDML, a multitask learning framework is introduced to enhance the classification performance of the model. The model can pay more attention to the differentiated representations of each unimodal itself by jointly training multimodal tasks and unimodal subtasks. Multitask learning acts as a model regularizer with the help of unimodal auxiliary tasks.

To sum up, the main novelties of this paper can be characterized as follows:

• A multimodal sentiment analysis model that operates on joint representations is introduced. This model employs an attention mechanism to effectively allocate attention weights to each modality, thereby enabling the features of each modality to be more effectively integrated.
• Adversarial training is used to ensure the consistency of the distribution between the source and translated modalities, which acquire consistent information between modalities.
• The generalization ability of the model is improved by a multitask learning framework. A single-modal label generation module is adopted to obtain single-modal sentiment labels and thus learn the differences among the modalities.

The remainder of this paper is organized as follows. Section 2 discusses the related work on multimodal sentiment analysis and multitask learning. Section 3 describes the proposed CDML method. Section 4 presents the relevant experiment and analysis. Section 5 shows our conclusions.

2. Related Work

2.1. Multimodal Sentiment Analysis

Multimodal fusion networks provide a distinct advantage in various applications, showcasing their superiority over unimodal approaches. They excel in tasks such as sentiment analysis [6,10,13], action recognition [14,15], and semantic segmentation [16,17,18], benefiting from the integration of multiple modalities to achieve improved performance and accuracy. Multimodal fusion is a core problem for multimodal sentiment analysis. According to the different stages in the algorithm model, existing methods can be categorized into early fusion (feature-level fusion) and late fusion (decision-level fusion) [19]. Early fusion methods are the fusion on the feature level by simple concatenation. Williams et al. [8] proposed Early Fusion LSTM (EF-LSTM), which connects feature vectors from different modalities as input and captures the long-range dependencies with LSTM. Zadeh et al. [9] proposed a memory fusion network, which stores the interaction information between modalities through gated memory units. To effectively address the complexities of multimodal relationships, Zadeh et al. [10] introduced a Tensor Fusion Network (TFN), which utilizes tensor fusion to model the intermodal dynamics, while the intramodal dynamics are captured by three modal embedding subnetworks. Furthermore, Liu et al. [20] introduced Low-Rank Multimodal Fusion (LMF), leveraging modality-specific low-rank factors to improve efficiency in the fusion process. Wang et al. [21] presented an advanced framework that utilizes an attention-based model to dynamically adapt word embeddings based on auxiliary nonverbal features such as video and audio features. Sun et al. [22] devised the Interaction Canonical Correlation Network (ICCN), which uses the outer product of feature pairs and canonical correlation to establish associations between language-based video and language-based audio.

Our work is fundamentally different from these available works. We do not use contextual information and neither focus on complex tensor fusion mechanisms. Instead, we emphasize the importance of joint representation and multitask learning.

2.2. Multitask Learning

Multitask learning [23,24,25] endeavors to enhance the performance of each task by integrating multiple tasks into one model. In multitask learning, the model can learn and perform multiple related tasks simultaneously, thus better exploring shared information and complementary features among different tasks. Compared with single-task learning, multitask learning has the following advantages: multitask learning can make full use of the relationships among tasks to obtain more information than a single task, and different and similar tasks added to the model training are similar to a model regularization to improve the robustness of the model. Furthermore, multitask learning can enhance the generalization ability and reduce the risk of model overfitting. Jiang et al. [26] introduced a multitask learning framework that exhibits the ability to dynamically adjust the weight of loss for each subtask. Given the intricate interconnection between sentiment analysis and emotion recognition tasks, Akhtar et al. [27] posited a multitask framework rooted in contextual cues. Zheng et al. [28] posited a discriminative joint multitask framework (DJMF) that concurrently achieves sentiment prediction and emotion recognition. Yu et al. [29] adopted self-supervised learning and introduced adaptive weights to balance the variability between modalities.

While previous work has focused more on adjusting the weights between multitasks, our work is designed to capture joint representations between modalities in a multitasking framework. We jointly learn unimodal and multimodal tasks with the self-supervised strategy. Our method learns similarity information from multimodal tasks and learns differentiated information from unimodal tasks.

3. Proposed Method

In this section, we explain the proposed CDML in detail. The overall architecture of the CDML is shown in Figure 1. In the following subsections, Section 3.1 presents the overall architecture of CDML. Section 3.2 and Section 3.3 describe important components of the CDML. Section 3.4 introduces the loss function.

3.1. Overall Architecture

The input of the CDML contains acoustic, visual, and language modalities. To handle diverse input sources, the architecture is structured into three branches with each branch dedicating to a specific input source. Firstly, after the data processing, we obtain three distinct sequences of processed modality features, which are represented as $X_i\in {\mathbb{R}}^{T_i\times d_i}$, $i\in \{Ac,Vi,La\}$. Subsequently, we adopt a Joint Representation Translator (JRTran) to capture joint representations of bimodality. Next, a Cross-Modality Transformer (CMTran) is adopted to investigate the synergistic relationship among modalities. Afterward, the CMTran used to process different branches generates modality reinforcement representations $S_{Ac}$, $S_{Vi}$, and $S_{La}$. The attention network distinguishes the weights of different modalities. Finally, we make the prediction with the fusion feature $X_M^{\ast }$ through a fully connected layer. In the model training stage, to focus on the differences across modalities, CDML produces three additional unimodal outputs, denoted as ${\widehat{y}}_s\in \mathbb{R}$, where $s\in \{Ac,Vi,La\}$. In the model reasoning phase, we solely utilize ${\widehat{y}}_M$ as the ultimate predictive output. The primary focus of this paper is the setting of word alignment to ensure that $T_{Ac}=T_{Vi}=T_{La}=T$. Considering these sequences $U_{Ac}$, $U_{Vi}$, and $U_{La}$, the primary task is to predict the sentimental intensity result ${\widehat{y}}_M\in \mathbb{R}$.

3.2. Data Processing

This section presents the methods for extracting modality features for different modalities of the multimodal sentiment analysis task. For acoustic, we employ COVAREP [30] to extract low-level acoustic features. The dimension of the feature is 74. For visual, we employ Facet [31] to encode each video frame to signify 35 facial action units for representing basic and advanced emotions on a per-frame basis. For language, we employ Glove [32] to transform visual transcripts into language embeddings. The embedding is a 300-dimensional vector. Thus, we obtain the feature sequence $U_i\in {\mathbb{R}}^{T_i\times d_i}$, $i\in \{Ac,Vi,La\}$, where $T_i$ represents the length of the sequence, $d_i$ represents the dimension of each modality. Feature sequences $U_i\in {\mathbb{R}}^{T_i\times d_i}$, $i\in \{Ac,Vi,La\}$ are passed into a one-dimensional convolutional neural network to ensure that they are of the same dimension:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill U_i^{\prime }=\mathrm{Conv}\mathrm{lD}\left(U_i,k_i\right)\in {\mathbb{R}}^{T_i\times d}\end{array}\end{array}$$

(1)

For the modal sequence to incorporate temporal information, we employ positional embedding (PE) to the output of the convolutional layer. As a result, we obtain the processed feature sequences, which are denoted as $X_i\in {\mathbb{R}}^{T_i\times d_i}$, $i\in \{Ac,Vi,La\}$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill X_i=U_i^{\prime }+PE(T_i,d)\end{array}\end{array}$$

(2)

3.3. Modality Fusion

3.3.1. Joint Representation Translator

To effectively capture the joint representation of different modalities, we adopt a Joint Representation Translator (JRTran). This approach involves sequentially applying cyclic translations on each pairwise modality. By taking this approach, the JRTran enables the modeling of intermodal relationships and enhances the understanding of multimodal data. Figure 2 explains the process of obtaining the joint representation between modality Vi and Ac. The whole process is denoted as Joint Representation Translator (Vi, Ac). Other paired Joint Representation Translators are similar, i.e., Joint Representation Translator (La, Vi) and Joint Representation Translator (La, Ac).

In the JRTran, the presence of two modalities is denoted by $\alpha \in {\mathbb{R}}^{T_{\alpha }\times d_{\alpha }}$ and $\beta \in {\mathbb{R}}^{T_{\beta }\times d_{\beta }}$, where $\alpha \ne \beta$ and $T_{\alpha }=T_{\beta }=T$. We adopt the Seq2Seq model [33], which is implemented to design the encoder-decoder that serves the purpose of the JRTran $\left(\begin{array}{c}\alpha ,\beta \end{array}\right)$. During the forward translation process, the encoder E processes the input in modality $\alpha$ and generates the joint representation Z. Next, the decoder D translates the joint representation into the target modality $\tilde{\beta }$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Z=E\left(\alpha \right),\tilde{\beta }=D\left(Z\right)\end{array}\end{array}$$

(3)

During the backward translation process, the modality $\tilde{\beta }$, which has been translated, is fed into the identical encoder E. Subsequently, the decoder takes the translated modality $\tilde{\beta }$ and performs a backward translation to convert it back into modality $\tilde{\alpha }$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Z=E\left(\tilde{\beta }\right),\tilde{\alpha }=D\left(Z\right)\end{array}\end{array}$$

(4)

Especially, the encoder E employs the GRU architecture to process modality $\alpha$. The representation of the hidden layer’s output of the encoder can be summarized as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill z_i=GRU(z_{i-1},{\alpha }_i),i=1,\cdots ,T\end{array}\end{array}$$

(5)

The forward joint representation $Z=\left[z_1,\cdots ,z_T\right]$ is derived by amalgamating the previous outputs of the hidden layers. The decoder D utilizes this forward joint representation Z along with the modality $\beta$ as a prior condition to perform the decoding process:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill p\left(\beta \right)=\prod _{i=1}^{T}p\left({\beta }_i\right|Z,{\beta }_1,\cdots ,{\beta }_{i-1})\end{array}\end{array}$$

(6)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \tilde{\beta }=arg\underset{\beta }{max}p\left(\beta \right|\alpha )\end{array}\end{array}$$

(7)

We employ backward translation to ensure that the joint representation is learned to retain the maximum information of all modalities. Backward translation allows the encoder to capture the target modality. The information obtained by backward translation is combined with the source modality information. Thus, we acquire significant joint representation.

To facilitate the handling of acoustic, visual, and language modalities, there are three Joint Representation Translators:

(1) To achieve modality fusion between language and visual, we design a Joint Representation Translator $(\mathrm{La},\mathrm{Vi})$ composed of the encoder $E_{LaVi}$ and the decoder $D_{LaVi}$. The forward translation (FT) process is shown in Equation (8) as follows, and the backward translation (BT) process is shown in Equation (9) as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \begin{array}{c}\hfill FT:{\tilde{X}}_{Vi}=D_{LaVi}\left(E_{LaVi}\left(X_{La}\right)\right),Z_{LaVi}=E_{LaVi}\left(X_{La}\right)\end{array}\end{array}\end{array}$$

(8)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill BT:{\tilde{X}}_{La}=D_{LaVi}\left(E_{LaVi}\left(X_{Vi}\right)\right),Z_{ViLa}=E_{LaVi}\left(X_{Vi}\right)\end{array}\end{array}$$

(9)

(2) To achieve modality fusion between language and acoustic, we design a Joint Representation Translator $(\mathrm{La},\mathrm{Ac})$ composed of the encoder $E_{LaAc}$ and the decoder $D_{LaAc}$, as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill FT:{\tilde{X}}_{Ac}=D_{LaAc}\left(E_{LaAc}\left(X_{La}\right)\right),Z_{LaAc}=E_{LaAc}\left(X_{La}\right)\end{array}\end{array}$$

(10)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill BT:{\tilde{X}}_{La}=D_{LaAc}\left(E_{LaAc}\left(X_{Ac}\right)\right),Z_{AcLa}=E_{LaAc}\left(X_{Ac}\right)\end{array}\end{array}$$

(11)

(3) To achieve modality fusion between acoustic and visual, we design a Joint Representation Translator $(\mathrm{Vi},\mathrm{Ac})$ composed of the encoder $E_{ViAc}$ and the decoder $D_{ViAc}$, as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill FT:{\tilde{X}}_{Ac}=D_{ViAc}\left(E_{ViAc}\left(X_{Vi}\right)\right),Z_{ViAc}=E_{ViAc}\left(X_{Vi}\right)\end{array}\end{array}$$

(12)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill BT:{\tilde{X}}_{Vi}=D_{ViAc}\left(E_{ViAc}\left(X_{Ac}\right)\right),Z_{AcVi}=E_{ViAc}\left(X_{Ac}\right)\end{array}\end{array}$$

(13)

Upon fusing $Z_{ViAc}$ and $Z_{AcVi}$, we project the resultant representation to ${\overline{Z}}_{ViAc}$ for further interaction. ${\overline{Z}}_{ViAc}$ is utilized as the joint representation of bimodality. Eventually, we obtain three distinct joint representations: ${\overline{Z}}_{LaVi}$, ${\overline{Z}}_{LaAc}$, ${\overline{Z}}_{ViAc}$.

Cycle consistency [34] requires that the translated modality be close to the original modality. Inspired by GAN [35], we design a generator G: Z→X mapping the joint representation $z\in Z$ to the translated modality $x\in X$, and a discriminator D trying to identify whether the modality is translated or original. The discriminator D is treated as a binary classification network, where the translated modality is set with label 0 while the original modality is set with label 1. For training, the adversarial loss L is defined as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L\left(G,D,Z,X\right)={\mathbf{E}}_x\left[logD\left(x\right)\right]+{\mathbf{E}}_z\left[log\left(1-D\left(G\left(z\right)\right)\right)\right]\end{array}\end{array}$$

(14)

Specifically, we design 3 discriminators, $D_{Vi}$, $D_{La}$, $D_{Ac}$, for visual, language, and acoustic modalities, respectively, which encourage the translated modality ${\tilde{X}}_{\{Ac,Vi,La\}}$ to be similar to the original modality $X_{\{Ac,Vi,La\}}$. As a result, the distribution of the translated modality is indistinguishable from the distribution of the original modality. Combining adversarial loss in each JRT(La, Ac), JRT(Vi, Ac), and JRT(La, Vi), we obtain the overall adversarial loss $L_{adv}$.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \begin{array}{cc}\hfill L_{adv}=& L\left(G_{LaVi},D_{Vi},Z_{LaVi},X_{Vi}\right)+L\left(G_{LaVi},D_{La},Z_{ViLa},X_{La}\right)+L\left(G_{LaAc},D_{Ac},Z_{LaAc},X_{Ac}\right)+\hfill \\ & L\left(G_{LaAc},D_{La},Z_{AcLa},X_{La}\right)+L\left(G_{ViAc},D_{Ac},Z_{ViAc},X_{Ac}\right)+L\left(G_{ViAc},D_{Vi},Z_{AcVi},X_{Vi}\right)\hfill \end{array}\end{array}\end{array}$$

(15)

3.3.2. Cross-Modality Transformer

Following the acquisition of the joint representations Z between bimodalities, we employ a Cross-Modality Transformer (CMTran) to exploit interaction among modalities. The CMTran is designed to preserve unimodal information while integrating shared information across different modalities. Additionally, the CMTran leverages joint representations to reinforce a specific modality, further improving its performance.

For the sake of brevity, we denote bimodal information as $BI\in {\mathbb{R}}^{T_b\times d_b}$ and unimodal information as $UI\in {\mathbb{R}}^{T_u\times d_u}$. Subsequently, we explain the procedure by which the joint representation Z is transmitted to the language modality, denoted as “Cross-Modality Transformer (Vi-Ac)→La”. The diagram illustrating the architecture of CMTran (Vi-Ac)→La can be observed in Figure 3. The CMTran architecture of other branches is similar.

During modality interaction, we employ a multihead attention mechanism [36] with $N_h$ heads to promote interactions. The joint representations ${\overline{Z}}_{\{LaVi,LaAc,ViAc\}}$ are converted to key/value pairs to interact with the unimodality $X_{\{Ac,Vi,La\}}$. For the j-th head, we denote query, key, and value as $q_j=UI{W}_j^q,k_j=BI{W}_j^k,v_j=BI{W}_j^{\nu }$, respectively, where $W_j^q\in {\mathbb{R}}^{d_u\times d/N_h},W_j^{\dot{\kappa }},W_j^{\nu }\in {\mathbb{R}}^{d_b\times d/N_h}$, d represents the hidden dimension. Modality Interaction (MI) from bimodality to unimodality is computed as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill hea{d}_j=softmax\left(\frac{UI{W}_j^q{\left(BI{W}_j^k\right)}^T}{\sqrt{d/N_h}}\right)BI{W}_j^{\nu }\end{array}\end{array}$$

(16)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill MI(BI,UI)=Concat\left(hea{d}_1,\cdots ,hea{d}_{N_h}\right)W_O\end{array}\end{array}$$

(17)

where $MI(BI,UI)\in {\mathbb{R}}^{T_u\times d},W_o\in {\mathbb{R}}^{d\times d},hea{d}_j\in {\mathbb{R}}^{T_u\times d/N_h}$ and Concat represents concatenation.

In Figure 3, each CMTran is built with N layers of blocks. Each layer computes in a feed-forward manner, gradually transforming bimodal information into an unimodal representation. The layers use Layer Normalization (LN) and a Position Feedforward (PF) sublayer with a skip connection to enhance the transformation process. Throughout the computation, the output of each layer is denoted as $S^{\left[n\right]}$. By stacking N layers together, the final output is represented as $S^{\left[N\right]}$. The computation is computed as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill S^{\left[0\right]}=UI\end{array}\end{array}$$

(18)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\widehat{S}}^{\left[n\right]}=MI\left(LN\left(BI\right),LN\left(S^{[n-1]}\right)\right)+LN\left(S^{[n-1]}\right)\end{array}\end{array}$$

(19)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill S^{\left[n\right]}=PF\left(LN\left({\widehat{S}}^{\left[n\right]}\right)\right)+LN\left({\widehat{S}}^{\left[n\right]}\right),n=1,\cdots ,N\end{array}\end{array}$$

(20)

where N represents the number of layers in the CMTran.

Eventually, the acquired modality-reinforced representation provides a comprehensive integration of both the unique features from each modality and the common features shared among different modalities. Specifically, we obtain modality-reinforced representations $S_{Ac}$, $S_{Vi}$, and $S_{La}$, which are computed as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill S_{Ac}=CMTran\left(JRT(X_{La},X_{Vi}),X_{Ac}\right)\end{array}\end{array}$$

(21)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill S_{Vi}=CMTran\left(JRT(X_{La},X_{Ac}),X_{Vi}\right)\end{array}\end{array}$$

(22)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill S_{La}=CMTran\left(JRT(X_{Vi},X_{Ac}),X_{La}\right)\end{array}\end{array}$$

(23)

3.3.3. Attention Network

The features output from the Cross-Modality Transformer $S_{\{Ac,Vi,La\}}$ are input to the attention network, and the attention network assigns different attention weights ${\lambda }_{\{Ac,Vi,La\}}$ according to different modalities. The fusion feature $X_M^{\ast }$ for sentiment task prediction is obtained by the weighted accumulation of each modality feature $S_{\{Ac,Vi,La\}}$, which is calculated as shown below.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill C_{\{Ac,Vi,La\}}=tanh\left(W_{attn}{S}_{\{Ac,Vi,La\}}+b_{attn}\right)\end{array}\end{array}$$

(24)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\lambda }_{\{Ac,Vi,La\}}=softmax\left(C_{\{Ac,Vi,La\}}\right)\end{array}\end{array}$$

(25)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill X_M^{\ast }=\sum _{\left\{Ac,Vi,La\right\}}{S}_{\left\{·\right\}}{\lambda }_{\left\{·\right\}}^T\end{array}\end{array}$$

(26)

where $C_{\left\{Ac,Vi,La\right\}}$ is the hidden layer, and $W_{attn}$ and $b_{attn}$ are the weights and biases of the linear transformation matrix, respectively. ${\lambda }_{\{Ac,Vi,La\}}$ are the normalized weight vectors. Then, the fused features $X_M^{\ast }$ are used as inputs to a fully connected layer to predict sentiment, which is calculated as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\widehat{y}}_M=f\left(X_M^{\ast }\right)\end{array}\end{array}$$

(27)

3.3.4. Multitask Learning

The three individual unimodal tasks share modal representations with the multimodal task. To mitigate the dimensional disparity between modalities, we project them into a novel feature space and use linear regression to derive unimodal results, which are computed as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill X_s^{\ast }=ReLU\left(W_{l1}^{sT}{X}_s+b_{l1}^s\right)\end{array}\end{array}$$

(28)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\widehat{y}}_s=W_{l2}^{sT}{X}_s^{\ast }+b_{l2}^s\end{array}\end{array}$$

(29)

where $s\in \{Ac,Vi,La\}$.

To facilitate the single-modal training process, we adopt a Single-modal Label Generation Module (SLGM) to acquire single-modal labels. The details of the SLGM are elaborated as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill y_s=SLGM\left(y_M,X_M^{\ast },X_s^{\ast }\right)\end{array}\end{array}$$

(30)

The SLGM aims to obtain three single-modal sentiment labels. The SLGM shares the same input with the multimodal task. The SLGM calculates the offset based on the relative distance between modality representations and class centers, as depicted in Figure 4. Each subtask has a corresponding positive sentiment center $\left(C_k^p\right)$ and negative sentiment center $\left(C_k^n\right)$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill C_k^p=\frac{{\sum }_{j=1}^{N}I\left(y_k\left(j\right)>0\right)·X_{kj}^g}{{\sum }_{j=1}^{N}I\left(y_k\left(j\right)>0\right)}\end{array}\end{array}$$

(31)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill C_k^n=\frac{{\sum }_{j=1}^{N}I\left(y_k\left(j\right)<0\right)·X_{kj}^g}{{\sum }_{j=1}^{N}I\left(y_k\left(j\right)<0\right)}\end{array}\end{array}$$

(32)

where $k\in \{M,Ac,Vi,La\}$, M denotes the multimodal task; $Ac$, $Vi$, $La$ denotes the acoustic, visual, and language modality, respectively, N represents the number of samples involved in training, $I\left(·\right)$ indicates the characteristic function, and $X_{kj}^g$ refers to the global representation of the $j_{th}$ sample in the modality k.

We use L2 regularization to calculate the distance between the modal representation and the positive and negative class centers:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill D_k^p=\frac{{∥X_k^{\ast }-C_k^p∥}_2^2}{\sqrt{d_k}}\end{array}\end{array}$$

(33)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill D_k^n=\frac{{∥X_k^{\ast }-C_k^n∥}_2^2}{\sqrt{d_k}}\end{array}\end{array}$$

(34)

where $d_k$ is the dimension of the modal representation.

Since the feature representations of different modalities are distributed in their respective feature spaces, the use of absolute distances between modal representations and positive and negative class centers lacks accuracy, so this paper uses relative distances to assess the proximity of the modal representation to the class centers of the positive and negative, which are calculated as shown below:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\alpha }_k=\frac{D_k^n-D_k^p}{D_k^p+\epsilon }\end{array}\end{array}$$

(35)

where $\epsilon$ is the amount of deviation to prevent the denominator from being zero.

Based on the fact that the ratio of unimodal to multimodal labels is proportional to the ratio of the corresponding unimodal to multimodal predicted values, the following equation can be derived:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \begin{array}{c}\hfill \frac{y_s}{y_M}\propto \frac{{\widehat{y}}_s}{{\widehat{y}}_M}\propto \frac{{\alpha }_s}{{\alpha }_M}\rightrightarrows y_s=\frac{{\alpha }_s\ast y_M}{{\alpha }_M}\end{array}\end{array}\end{array}$$

(36)

We can obtain the sentiment labels of related subtasks:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill y_s=y_M+\frac{{\alpha }_s-{\alpha }_M}{2}\ast \frac{y_M+{\alpha }_M}{{\alpha }_M}=y_M+{\delta }_M\end{array}\end{array}$$

(37)

where ${\delta }_M$ indicates the offset of the generated unimodal supervisory value to the multimodal label value.

Eventually, we jointly learn multimodal and unimodal tasks. Notably, these unimodal tasks are solely present during the training stage. Hence, we utilize ${\widehat{y}}_M$ as the ultimate output. For the unimodal auxiliary task, this paper uses the difference between the single-modal labels and the multimodal sentiment labels as the weight of the loss function. This allows the model to focus more on samples with large differences between multimodal labels and single-modal labels, and the loss function for the unimodal auxiliary task is shown below:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{aux}=\sum _s^{\{Ac,Vi,La\}}tanh\left(\left|y_s-y_M\right|\right)\ast \left|{\widehat{y}}_s-y_s\right|\end{array}\end{array}$$

(38)

3.4. Loss Function

For the training phase, we employ the Mean Absolute Error (MAE) to measure the disparity between the ground truth and predicted output. Finally, the loss function $L_{task}$ for task prediction is defined as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{task}=E\left({∥\widehat{y}-y∥}_1\right)\end{array}\end{array}$$

(39)

where ${∥·∥}_1$ represents the MAE, $\widehat{y}$ refers to the prediction output and y represents ground truth. The ultimate loss function $L_{all}$ is formulated by integrating task prediction loss $L_{task}$, adversarial loss $L_{adv}$, and auxiliary task loss $L_{aux}$.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{all}=L_{task}+\mu L_{adv}+\eta L_{aux}\end{array}\end{array}$$

(40)

where $\mu ,\eta$ are the trade-off hyperparameters. All parameters are optimized by minimizing $L_{all}$. For the hyperparameters, we conduct a grid search to select the model with the optimal loss.

4. Experiment and Analysis

In this section, we assess the effectiveness of the CDML on datasets CMU-MOSI [37] and CMU-MOSEI [38]. We illustrate our experiments based on the subsequent aspects. Firstly, we furnish details regarding the datasets and baselines employed in our experiment. Then, we introduce the basic settings including experimental details and evaluation metrics. Finally, we proceed to present and engage in a comprehensive discourse on our experimental results.

4.1. Dataset

The effectiveness of the proposed model is assessed using CMU-MOSI and CMU-MOSEI datasets, recognized benchmarks in the domain of multimodal sentiment analysis. These two public datasets are also more conducive to comparative analysis with models proposed by other researchers in order to better evaluate the performance of the proposed models.

Table 1. Statistics of CMU-MOSI and CMU-MOSEI.

Dataset	Train	Valid	Test	All
CMU-MOSI	1284	229	686	2199
CMU-MOSEI	16,326	1871	4659	22,856

presents important statistics about these datasets. Here, we provide a succinct overview of the aforementioned datasets.

CMU-MOSI: The dataset is a multimodal corpus derived from 93 opinion videos. The videos are segmented into 2199 segments, and each example is endowed with a sentiment intensity rating that ranges from −3 (indicating extremely negative emotions) to 3 (representing extremely positive emotions).

CMU-MOSEI: This dataset is based on the above dataset with improvements, which offers a larger and more diverse collection, consisting of 23,453 annotated video clips. It involves 1000 distinct speakers discussing a wide range of 250 topics. Like its predecessor, the dataset assigns sentiment scores within the range of [−3, +3] to each video clip, enabling in-depth sentiment analysis.

4.2. Baselines

To determine the effectiveness of CDML, a comprehensive comparative study was conducted. This study evaluates the CDML along with several excellent baseline models. These models can be categorized as follows:

(i) Neural Network based methods: RAEVN, MulT, MISA, MoNIG;

(ii) Translation based methods: MCTN, CMJRT;

(iii) Multitask-learning-based methods: Self-MM.

RAVEN [21]: The recurrent attended variation embedding network utilizes information in the nonverbal modality to supplement the semantic information represented by the words.

MCTN [39]: The Multimodal Cyclic Translation Network is based on Seg2seq, which requires only unimodal input data and robust multimodal joint representation learning methods in the testing phase.

MulT [11]: The Multimodal Transformer provides a mechanism for fusing cross-modal information by directly focusing on the lower-order features of other modalities, fusing multimodal information.

MISA [40]: Modality-Invariant and -Specific Representations decompose each modality into modality-invariant and -specific representations and then fuse them to predict affective states.

Self-MM [29]: The Self-Supervised Multitask Multimodal sentiment analysis network generates corresponding modal labels for each modality by self-supervision.

MoNIG [41]: The mixture of Normal-Inverse Gamma distributions algorithm ensures the reliability of the final output by capturing both modality-specific and fused uncertainties.

CMJRT [42]: The Cross-Modal Joint Representation Transformer enables effective information integration and processing across modalities by leveraging hierarchical interactions. Additionally, the passing of joint representations allows for a comprehensive understanding of the interplay between different modalities.

4.3. Basic Settings

To enable a fair comparison, the hyperparameters used in each benchmark are provided in

Table 2. The hyperparameter settings adopted in each benchmark.

Setting	CMU-MOSEI	CMU-MOSI
Batch Size	16	32
Initial Learning Rate	1 × ${10}^{-3}$	1 × ${10}^{-3}$
Optimizer	Adam	Adam
Cross-modal Layers	4	4
Multihead Attention Heads	8	8
Dropout	0.1	0.1
Epochs	50	100

. The performance of CDML and the compared models is evaluated based on five runs, and the average performance was reported. All experiments were carried out on an RTX-2080Ti GPU.

To evaluate the efficacy of the models, a range of metrics including 7-class accuracy (Acc7), binary accuracy (Acc2), F1 score, Mean Absolute Error (MAE), and correlation (Corr) are employed. Higher values for Acc7, Acc2, F1 score, and Corr signify better performance, indicating that the models have achieved more accurate and reliable results. In contrast, for MAE, lower values showcase better performance as they indicate smaller errors in predictions.

4.4. Experimental Results

Firstly, we assess the effectiveness of the CDML on the CMU-MOSI benchmark.

Table 3. Comparison with baselines on CMU-MOSI benchmark. ↑ means higher is better and ↓ means lower is better.

Model	Acc-7 (↑)	Acc-2 (↑)	F1 (↑)	Corr (↑)	MAE (↓)
RAVEN	33.2	78.0	76.6	0.691	0.915
MCTN	35.6	79.3	79.1	0.676	0.909
MulT	35.1	80.2	80.1	0.679	0.972
MISA	35.6	80.4	79.8	0.689	0.969
Self-MM	35.8	80.7	80.6	0.693	0.954
MoNIG	34.1	80.6	80.6	0.680	0.951
CMJRT	37.9	82.4	82.8	0.704	0.887
CDML	39.1	83.7	84.3	0.725	0.864

Bold is to show more clearly the optimal result under the same rubric. The arrows (↑ & ↓) and bold meanings of the other tables in the paper are consistent with this table.

: Perspective analysis results from multimodal data with different models on the CMU-MOSI dataset. By comparison, the CDML model demonstrates notable enhancements across all evaluation metrics when juxtaposed with the baselines. Regarding the 2 classification accuracy, Acc-2, in the sentiment task classification metric, CDML achieved 83.7%, showcasing an enhancement ranging from 1.3% to 5.7% in comparison to the baseline models. Regarding the classification metric F1, CDML improved by 1.5–7.7% compared with the baseline models. CDML reached 39.1% on the 7 classification accuracy Acc-7, an improvement of about 1.2–5.9% compared with the baseline models. Regarding the sentiment task regression metric Pearson’s coefficient Corr, the model improved by about 0.021–0.049. All of the above metrics reached the optimal level in the baseline model.

Subsequently, we assess the performance of the CDML model on the CMU-MOSEI benchmark.

Table 4. Comparison with baselines on the CMU-MOSEI benchmark.

Model	Acc-7 (↑)	Acc-2 (↑)	F1 (↑)	Corr (↑)	MAE (↓)
RAVEN	50.0	79.1	79.5	0.662	0.614
MCTN	49.6	79.8	80.6	0.670	0.609
MulT	51.1	80.1	80.3	0.628	0.610
MISA	51.2	80.5	80.8	0.662	0.629
Self-MM	51.0	80.8	81.2	0.674	0.609
MoNIG	50.0	81.0	81.5	0.600	0.688
CMJRT	51.6	82.9	82.6	0.742	0.581
CDML	52.9	84.3	84.3	0.761	0.553

showcases the empirical findings of diverse baselines, alongside CDML, within the CMU-MOSEI dataset. Evidently, the CDML model outperforms the alternative baselines notably. The primary reasons for this superiority are as follows: The proposed attention network can focus on filtering the modal information that is more important for the sentiment prediction task, and the generative adversarial constraint is added to the loss function, which helps the features of the translation modality to better approximate the source modality features during the translation process. In addition, the multitask learning strategy is used to collaboratively train the multimodal task and each single-modal auxiliary task, which helps the model pay more attention to the differences between different modalities, and then more comprehensively handle the sentiment prediction task and improve the prediction accuracy of the model.

4.5. Ablation Study

4.5.1. Modality Ablation Experiment

To show the degree of influence of different modal combinations on the final sentiment task prediction, modality ablation experiments are conducted in this section on the dataset CMU-MOSEI.

Table 5. Modality combination comparison experiment on CMU-MOSEI. ✓ means information containing this modal.

Acoustic Modality	Visual Modality	Textual Modality	Acc-7 (↑)	Acc-2 (↑)	F1 (↑)	Corr (↑)	MAE (↓)
✓			47.6	79.9	80.2	0.598	0.704
	✓		48.5	80.4	80.9	0.625	0.693
		✓	48.9	80.4	80.6	0.614	0.698
✓	✓		50.7	81.2	81.6	0.648	0.659
✓		✓	50.3	81.3	81.8	0.632	0.662
	✓	✓	51.2	81.7	82.0	0.702	0.624
✓	✓	✓	52.9	84.3	84.3	0.761	0.553

illustrates the outcomes of the experiments with separate combinations of unimodal, bimodal, and trimodal modes. It should be noted that the acoustic, visual, and language unimodality in Table 5 refer to a branch in the left half of Figure 1 (multimodal task) and the corresponding unimodal branch in the right half of Figure 1 (unimodal task). For example, the check mark “Acoustic modal” in the second row of the table means that it contains branch 1 in the left half of Figure 1 and branch 1 in the right half of Figure 1 (acoustic unimodal subtask).

From Table 5, it is obtained that when the model contains all the branches (i.e., when acoustic, visual, and language modality tasks are used simultaneously), the model has the closest sentiment prediction performance to the labeled values and the best fusion performance. When fusing bimodal information, the performance of the experimental evaluation metrics is the next best. The performance is worse when the model contains only unimodal branches. It can be concluded that multimodal information fusion helps to capture the consistency and difference information between modalities; providing global feature information helps the accuracy of model sentiment task prediction.

4.5.2. Algorithm Ablation Experiment

Combined with the CDML proposed, this subsection compares the effects of the proposed enhancement algorithms on the model performance. The experiments are conducted in the datasets CMU-MOSI and CMU-MOSEI, respectively.

Table 6. Comparison of enhancement algorithm ablation experiments on CMU-MOSI. In each row, the corresponding component is progressively removed from the CDML.

Model Design	Acc-7 (↑)	Acc-2 (↑)	F1 (↑)	Corr (↑)	MAE (↓)
CDML (full model)	39.1	83.7	84.3	0.725	0.864
(-) multitask learning	38.7	83.0	83.7	0.717	0.871
(-) adversarial training	38.4	82.9	83.6	0.714	0.875
(-) attention mechanism	37.9	82.4	82.8	0.704	0.887

and

Table 7. Comparison of enhancement algorithm ablation experiments on CMU-MOSEI. In each row, the corresponding component is progressively removed from the CDML.

Model Design	Acc-7 (↑)	Acc-2 (↑)	F1 (↑)	Corr (↑)	MAE (↓)
CDML (full model)	52.9	84.3	84.3	0.761	0.553
(-) multitask learning	52.6	83.7	83.6	0.751	0.573
(-) adversarial training	52.2	83.5	83.4	0.747	0.578
(-) attention mechanism	51.6	82.9	82.6	0.742	0.581

show the results of the algorithmic ablation experiments for CMU-MOSI and CMU-MOSEI. Comparing Table 6 and Table 7, it can be concluded that when either enhancement algorithm is missing, all performance evaluation indexes of the model are reduced to some extent. That is, the proposed enhancement algorithm can improve the performance of sentiment prediction within the model and validate the effectiveness of the enhancement algorithm.

The experimental results of the attention-based mechanism, adversarial training, and multitask learning augmentation algorithm show significant improvement compared with CMJRT. Regarding the evaluation metrics Acc-7, Acc-2, and F1, the amalgamation of all augmentation algorithms (CDML) improved by 1.2%, 1.3%, and 1.5%, respectively, over CMJRT on the CMU-MOSI, and by 1.3%, 1.4%, and 1.7%, respectively, on the CMU-MOSEI. The experiments illustrate that the attention mechanism can focus on filtering features that are more important to the effective task. The unimodal subtask in multitask learning shares feature information with the main sentiment prediction task, which makes the model pay more attention to samples with greater variability and helps provide more comprehensive information for the model sentiment prediction task.

4.6. Case Study

The case study is to show the performance of our proposed method on the dataset CMU-MOSI in a visual form. We aim to illustrate in an intuitive way that the single-modal label generation module makes the model more attentive to the differences between unimodal and multimodal labels. In this section, a case study of the unimodal label generation module is performed, and three typical samples from the CMU-MOSI are selected for visualization, and the samples are positive, neutral, and negative. Section 4.4 describes in detail the comparison of our proposed model with the baselines. The case study of the label generation module is shown in Figure 5. The acoustic, visual, and language label values in the figure are the calculated sentiment values from the unimodal label generation module, and the raw labels are the sentiment annotated values from the multimodal dataset.

The sentiment state in the first case tends to be consistent with that of the label generation values for each unimodal state. In the third case, the sentiment prediction of the audio modal label generation values is shifted concerning the multimodal labels. The predictions obtained from both the textual and visual modalities are negative, but the differences in the audio modality may give a small adjustment to the overall prediction, which is more comprehensive and accurate. That is, the label generation module can make the model pay more attention to the difference between the unimodal labels and the multimodal labels so that the CDML can obtain not only consistency between the modalities during the learning process but also pay more attention to the difference in the sentiment representations.

5. Conclusions

In this article, we introduced CDML, which is a framework for multimodal sentiment analysis. While current researches focus more on fusion mechanisms, our work aims to capture information about consistency and differentiation among modalities in a multitask framework. It facilitates the transfer of joint representations from bimodality to unimodality by using hierarchical interactions among modalities. First, the attention mechanism is used to efficiently assign the attention weights of each modality. Then, adversarial training is introduced to constrain the training process to optimize the ability to extract consistent information between modalities. Moreover, the difference among the modalities is acquired by the multitask learning framework. Finally, the proposed CDML outperforms the comparison model in metrics and achieves an improvement in overall performance.

In real-world situations, we sometimes encounter the problem of missing modalities, where it is unclear which modalities are missing. In the future, we plan to propose an efficient model that addresses the uncertain missing modality problem to improve robustness.