Speech Emotion Recognition Using Multi-Scale Global–Local Representation Learning with Feature Pyramid Network

Abstract

Speech emotion recognition (SER) is important in facilitating natural human–computer interactions. In speech sequence modeling, a vital challenge is to learn context-aware sentence expression and temporal dynamics of paralinguistic features to achieve unambiguous emotional semantic understanding. In previous studies, the SER method based on the single-scale cascade feature extraction module could not effectively preserve the temporal structure of speech signals in the deep layer, downgrading the sequence modeling performance. To address these challenges, this paper proposes a novel multi-scale feature pyramid network. The enhanced multi-scale convolutional neural networks (MSCNNs) significantly improve the ability to extract multi-granular emotional features. Experimental results on the IEMOCAP corpus demonstrate the effectiveness of the proposed approach, achieving a weighted accuracy (WA) of 71.79% and an unweighted accuracy (UA) of 73.39%. Furthermore, on the RAVDESS dataset, the model achieves an unweighted accuracy (UA) of 86.5%. These results validate the system’s performance and highlight its competitive advantage.

1. Introduction

Speech emotion recognition (SER) refers to extracting and analyzing emotion-related features from speech signals, allowing the computer to understand the speaker’s emotional expression [1]. As an essential part of affective computing, SER is the key to facilitating natural human–computer interactions (HCIs) [2]. The HCI system with an affective computing module has been applied to many tasks, such as psychological assessment [3], mobile services [4], and safe driving [5]. Decades of research in SER have been devoted to modeling emotional representations from linguistic (e.g., lexical, syntactic, discourse and rhetorical features) and paralinguistic (e.g., supra-segmental phoneme and prosodic features) characteristics and to developing appropriate algorithms for implementing robust and effective emotion recognition [6,7,8,9,10].

Recently, significant progress has been made in the field of SER using multi-scale convolutional neural networks (MSCNNs). MSCNNs can extract multi-scale temporal and spatial features. Compared to single-scale network methods, MSCNNs [11,12,13,14,15] effectively capture emotion-related features of variable lengths from speech inputs and are expected to further enhance semantic understanding and emotion recognition.

However, the MSCNN module in the previous research used a one-way transmission design. The temporal dynamic range that deep emotional features can retain depends on the size of the convolutional kernel. This unidirectional design can lead to the loss of temporal structure of speech in progressive resolution reduction. To enhance SER performance, it is crucial to learn both global–local semantic representation and temporal dynamics in speech. The main limitations comprise the following: (1) The high-level semantic features learned by the hierarchical MSCNN layers suffer from the loss of temporal structure of speech. (2) In context-aware semantic understanding, the MSCNN independently produces features that ignore the long-term relation between multi-scale sound units, and the correlations between multi-scale features should be explored.

Based on the limitations of the existing feature extraction networks in emotional feature extraction, we propose a novel framework learning context-aware representation using a multi-scale feature pyramid network (MSFPN) for SER. We explore the bi-directional fusion of multi-scale semantic features using a feature pyramid network and preserve the resolution for temporal dynamics learning and global–local semantic understanding. More specifically, we adopt parallel MSCNN groups for multi-scale feature learning, supplemented with a forward fusion mechanism to fuse these features into high-level semantic features. Moreover, to learn the local-global correlations in MSCNN, this paper improves and uses the convolutional self-attention (CSA) [16] layer to focus on the emotion-related periods in the local region effectively. The MSFPN enhances the connections between adjacent elements and captures the interactions between features extracted by different attention heads. In the top-down pathway, features are rejoined into low-level acoustic features by backward fusion, and bi-directional long short-term memory (BiLSTM) is adopted to generate an utterance-level representation for emotion classification.

Our research contributions can be summarized as follows:

1. 

We have conducted a detailed exploration of the application of feature pyramids in speech emotion recognition for the first time. We enhanced the MSCNN by integrating the CSA module to better capture local emotional correlations.

2. 

We have improved the CSA by using a multi-scale convolutional module, avoiding the degradation problem of the convolutional attention network.

3. 

We designed a backward fusion approach that effectively captures features across different levels of detail, successfully preserving the importance of local dynamics and deep semantics in emotional representation.

The rest of this paper is organized as follows: We summarize related work in Section 2. In Section 3, we propose our framework in detail. Experiments are reported in Section 4. We summarize this paper in Section 5.

2. Related Work

2.1. Background of Speech Emotion Recognition

Emotions are psychological states triggered by neurophysiological changes, and they are related differently to thoughts, feelings, behavioral responses, and varying degrees of happiness or unhappiness. Currently, there is no scientific consensus on the definition of emotions, which often intertwine with feelings, temperament, personality, character, and creativity. Emotions consist primarily of subjective experiences, physiological responses, and behavioral reactions, playing a crucial role in building interpersonal relationships. They can be recognized through speech, facial expressions, and body language. Basic emotions include surprise, joy, disgust, sadness, anger, and fear, while complex emotions, such as contempt, amusement, and embarrassment, are blends of multiple feelings and are more challenging to identify.

Speech emotion recognition (SER) technology aims to identify human emotions through voice. Typically, people are not very accurate in recognizing others’ emotions, making emotion recognition a burgeoning field of study where appropriate technology can enhance accuracy. The core of this process lies in identifying emotions from speech without considering cognitive content. SER mainly involves two processes, feature extraction and emotion classification, and it holds significant potential for applications in security, healthcare, entertainment, and education. The task of emotion recognition is complex due to the highly subjective nature of emotions, and there is yet to be a unified standard for classifying or measuring them.

As illustrated in Figure 1, an SER system consists of modules for speech signal input, pre-processing, feature extraction and selection, classification, and emotion recognition. The ability to extract robust emotional features from speech is crucial to the success of an SER system. Studies have shown that extracting features across multiple modal [17,18,19,20,21,22,23] and multiple scales can significantly improve the accuracy of emotion recognition.

2.2. Multi-Scale Network Model

A multi-scale network model is a deep learning architecture designed to capture and process features across different scales simultaneously, thereby enhancing overall model performance. Typically, the model consists of multiple branches, each handling input data at various scales. These features are then fused at specific stages, enabling the extraction of more robust and comprehensive representations. In speech emotion recognition (SER), where emotions are subjective and vary among individuals, many researchers focus on extracting emotion features from speech at multiple scales to enhance SER performance. Zhu et al. [13] utilized a global perceptual fusion method to extract features across various scales in speech emotion recognition. They created a special neural network that learns and combines these multi-scale features using a global perceptual fusion module. Peng et al. [11] proposed a framework using an MSCNN with statistical pooling units to obtain both the audio and text hidden representations and employed an attention mechanism to improve performance further. Xie et al. [24] utilized multi-head attention to extract emotional features from both the temporal and frequency domains. They employed additive attention in the frequency domain, enhancing the capability to extract nonlinear features. In modeling temporal dependence, Chen et al. [15] employed an MSCNN with a bi-directional long short-term memory (BiLSTM) network to model local-aware temporal dynamics for SER. Gan et al. [25,26,27,28] focused on temporal features and proposed a Transformer-based model for sequential feature extraction, incorporating multi-scale temporal feature operators and attention modules. This approach effectively extracts emotion features across different time scales, thereby improving the accuracy of emotion recognition.

Nowadays, multi-scale feature extraction for emotion has become widely adopted. However, existing multi-scale networks often employ one-way transfers, which can compromise the temporal structure of speech. To tackle this, we have incorporated a bi-directional transfer design in our model, which better maintains the emotional features’ temporal structure compared to other models.

2.3. Feature Pyramid Network Model

The feature pyramid network [29] was initially designed to address multi-scale challenges in object detection. Due to its capability to effectively capture features at various granularities, it has become widely adopted in object detection tasks. Lately, researchers have been investigating how feature pyramid networks can be applied to sequential tasks like speech recognition and classification. Liu et al. [30] introduced a contextual pyramid generative adversarial network for speech enhancement. This design effectively captures speech details at various levels and removes audio noise in a structured way. Luo et al. [31] explored sound event detection based on feature pyramid networks, and their experiments demonstrated that the model utilizing feature pyramid networks surpassed long short-term memory (LSTM) in performance. Furthermore, researchers [32,33,34] have utilized feature pyramid networks to enhance speech recognition capabilities by combining multi-level features.

Previous studies have shown that feature pyramid networks can effectively capture multi-scale features in sequential tasks. Yet, there has been limited research on how these networks benefit extracting emotion features from speech. So, we delved deeper into using feature pyramid networks for speech emotion recognition. We enhanced the original feature pyramid network by introducing a more powerful convolutional attention network to extract stronger emotion features.

3. Methodology

In this section, we propose our framework in detail. The proposed framework, as illustrated in Figure 2, comprises two primary modules: the multi-scale feature pyramid model and the global–local representation learning model. The multi-scale feature pyramid model consists of three sets of convolutional blocks (Conv Blocks), each characterized by distinct kernel sizes. Specifically, the first set employs 3 × 3 kernels, the second set utilizes 5 × 5 kernels, and the third set incorporates 7 × 7 kernels. This varied kernel configuration facilitates the effective extraction of emotional features across different scales. Furthermore, the groups interact and fuse features through both forward and backward fusion mechanisms, enhancing the robustness of the extracted multi-scale emotional characteristics. The global–local representation learning model is composed of three bi-directional long short-term memory (BiLSTM) networks, which are specifically designed to capture global features from the outputs of the multi-scale feature pyramid model.

The process commences with the input speech signal being processed through a pre-trained Hidden-Unit BERT (HuBERT) model [35], which is a self-supervised learning model designed for speech representation. HuBERT captures high-level acoustic features by learning from large amounts of unlabeled speech data, providing robust embeddings for downstream tasks. These embeddings are subsequently input into both the multi-scale feature pyramid model and the global–local representation learning model to extract multi-scale discourse-level features. Ultimately, the framework generates the final emotional output through two fully connected layers.

3.1. Multi-Scale Feature Pyramid Network

The overview architecture of the MSFPN is proposed as shown in Figure 2. The overall structure consists of two paths: one for extracting deep emotional features from bottom to top and another for merging multi-scale emotional features from top to bottom.

In the bottom-up pathway, as shown in Figure 3, the MSFPN includes a series of ConvBlock layers to learn and protrude the fine-grained emotion-related features in the different levels. Each ConvBlock contains two CNN layers for feature learning and is mixed with an improved convolutional self-attention (detailed in Section 3.2) layer for local correlation learning.

Specifically, there are three groups $(X,Y,Z)$ of bottom-up pathways with different kernel widths ($kw$ = 3, 5, 7), each extracting three levels of semantic features at different time scales. Given a speech dataset with n utterances, the characteristics of Group1 $(x_1,x_2,x_3)$ are calculated using hierarchical ConvBlock layers shown in Figure 3. After each layer, the number of output channels is reduced by half. To be specific, the output channels for $x_1$ are 1024, for $x_2$ are 512, and for $x_3$ are 256. The deep features of Group2 ($y_1$, $y_2$, $y_3$) and Group3 ($z_1$, $z_z$, $z_3$) are computed in the same way. The difference is that $y_i$ and $z_i$ can receive the previous fine-grained acoustic emotional features from the above groups by a forward fusion to enhance the local semantic understanding of phrases. Typically, the forward fusion here is adding function.

In the top-down pathway, a backward fusion mechanism is introduced to combine semantic feature maps with high-resolution acoustic features. As shown in Figure 4, for deep features $F_i=[x_i,y_i,z_i]$, firstly, calculate the attention score ${\alpha }_i$ for the i-th layer using Equation (1). Then, multiply ${\alpha }_i$ with features from various depth levels to extract the most pertinent emotion features for the i-th layer. Lastly, sum these products to derive the feature $G_i$ for the i-th layer.

$${\alpha }_i=softmax(W_2\times Tanh(W_1{F}_i^T))$$

(1)

$$G_i=\sum _{i=1}^3{\alpha }_i\times F_i$$

(2)

where $G_i$ is the multi-scale deep features of the i-th level in the MSFPN, and $W_1$ and $W_2$ are trainable parameters.

Then, the multi-scale deep feature G is upsampled by a transposed convolution function, and the low-level features add high-level semantic information. We acquire the deep features that contain adequate semantic information for understanding the emotional expression of each utterance and a high resolution with the sequential structure for temporal dependence learning.

3.2. Convolutional Self-Attention

A vanilla convolutional self-attention mechanism is proposed in [16] for learning short-range dependencies. Self-attention [36] is a mechanism that allows a model to focus on different parts of an input sequence when processing each element, enabling it to capture long-range dependencies by assigning attention weights across all elements. However, the self-attention mechanism distributes attention across all elements, which can sometimes lead to the neglect of relationships between adjacent elements and phase-level patterns. To address this, convolutional self-attention (CSA) [16] introduces a convolutional layer within the self-attention framework. CSA applies localized filters that focus on neighboring elements, enabling the model to capture short-range dependencies and fine-grained features between adjacent utterances. This makes CSA particularly effective for extracting local patterns in speech data. As illustrated in Figure 5a, the multi-head convolutional self-attention mechanism operates with each color representing a different attention head. The darker regions indicate the areas that convolutional self-attention focuses on. From the figure, it is evident that CSA restricts its attention to local regions, concentrating on the contextual relationships between adjacent elements. This localized attention allows CSA to capture finer details and better extract local features, making it especially suited for modeling dependencies between neighboring elements.

However, CSA with a fixed kernel size has limitations in fusing multi-scale features. Specifically, when CSA projects multi-head spaces of the same dimension, the restricted regions of interest and the high number of heads can cause the regions projected by different heads to converge into similar feature spaces, leading to a lack of diversity in the feature representations learned by the different heads.

To overcome this limitation, we propose a multi-scale feature extraction approach to enhance the feature space projection. This allows self-attention to compute temporal dependencies across different regions of information while mitigating the overfitting problem associated with the multi-head mechanism in CSA. In this paper, we introduce an improved convolutional self-attention layer that incorporates multi-scale feature extraction. As shown in Figure 5b, we use different kernel sizes across the attention heads, enabling the extraction of fine-grained features at multiple scales. Given an input sequence X, CSA focuses on local regions for each query $q_i$ and restricts its attention region to a local scope of fixed size $m+1$ ($m\in 3,7,11,15$), centered at position i. To maintain the same output dimension as the input, we reduce the output dimension for each kernel width by a factor of four.

$$\widehat{K_m}=\{k_{i-{\displaystyle \frac{m}{2}}},\cdots ,k_i,\cdots ,k_{i+{\displaystyle \frac{m}{2}}}\}.$$

(3)

$$\widehat{V_m}=\{v_{i-{\displaystyle \frac{m}{2}}},\cdots ,v_i,\cdots ,v_{i+{\displaystyle \frac{m}{2}}}\}.$$

(4)

Each head’s linear mapping query, key, and value with dimensions of $dk$, $dk$, and $dv$. In practice, self-attention computes the attention function on a set of queries simultaneously. The query, key, and value are packed together into a matrix Q, $\widehat{K_m}$, and $\widehat{V_m}$. The attention output is calculated as

$$Attention(Q,\widehat{K_m},\widehat{V_m})=softmax({\displaystyle \frac{Q{\widehat{K_m}}^T}{\sqrt{d_k}}})\widehat{V_m}.$$

(5)

$$Hea{d}_{i,m}=Attention(Q{W}_i^Q,\widehat{K_m}{W}_i^{\widehat{K_m}},\widehat{V_m}{W}_i^{\widehat{V_m}})$$

(6)

$$MultiHead(Q,\widehat{K_m},\widehat{V_m})=Concat(hea{d}_{1,3},\cdots ,hea{d}_{l,m})$$

(7)

where $W_i^Q$, $W_i^{\widehat{K_m}}$, $W_i^{\widehat{V_m}}$ are the weight matrices in multi-head attention with dimensions $d_k/l$, $d_k/l$, $d_v/l$, respectively.

3.3. Global–Local Representation Learning Module

In previous studies, the BiLSTM network has been proven effective in capturing the long temporal dynamics of deep features to aggregate global–local representations [15], but the progressive resolution reduction limits the sequence modeling performance [10]. In this paper, we explore the relative interaction between each emotional state in the progressive acoustic feature extraction. We model the multi-scale temporal dependence to generate the global–local representations for SER. In practice, the representation $\mathcal{R}$ is aggregated by BiLSTM $H=(h_1,h_2,\cdots ,h_t)$ for the multi-scale deep features $\mathcal{G}=(g_1,g_2\cdots ,g_t)$.

$$\begin{array}{ll}\hfill & f_t=\sigma (W_f[g_t,h_{t-1}]+b_f)\\ \hfill & i_t=\sigma (W_i[g_t,h_{t-1}]+b_i)\\ \hfill & o_t=\sigma (W_o[g_t,h_{t-1}]+b_o)\\ \hfill & C_t=f_t\ast C_{t-1}+i_t\ast tanh(W_c[g_t,h_{t-1}]+b_c)\\ \hfill & h_t=o_t\ast tanh(C_t)\\ \hfill & \mathcal{R}=h_t^1\oplus h_t^2\oplus h_t^3\end{array}$$

(8)

Here, $\sigma$ denotes the sigmoid activation function, while f, i, o, and C represent the vectors for the input gate, forget gate, output gate, and memory cell activation, respectively. The weight matrices and bias vectors for each gate are indicated by W and b. The last hidden output, $h_t^i$, serves as the utterance-level representation $\mathcal{R}$, which is subsequently input into fully connected layers for emotion inference.

4. Experiments

In this section, we evaluate the proposed framework in the IEMOCAP corpus. This paper compares the results with related state-of-the-art methods and deploys ablation studies to measure each component’s contribution.

4.1. Corpora Description

The Interactive Emotional Dyadic Motion Capture (IEMOCAP) corpus [37] is commonly utilized for evaluations. IEMOCAP is an interactive emotional binary motion capture database developed by the SAIL Lab at the University of Southern California, containing a total of 12 h of recordings. The data were recorded by ten professional actors (five males and five females) in a studio setting. Each recording is accompanied by discrete emotional labels and annotations for emotional dimensions. The IEMOCAP database consists of five sessions, each featuring dialogues between a female and a male actor, divided into two parts: improvised performances and scripted performances. The former involves spontaneous dialogues without predetermined content, while the latter follows a predefined script. This database encompasses multimodal information, including audio and text, making it suitable for various unimodal emotion recognition studies. To ensure a balanced representation of audio samples across categories, this study combines the excitement emotion into the happiness category. Ultimately, the dataset comprises 5531 audio samples, distributed as follows: 1103 instances of anger, 1084 instances of sadness, 1708 instances of calmness, and 1636 instances of happiness. Figure 6 illustrates the number of audio samples corresponding to each emotional label. As there is no predefined data split in IEMOCAP, we perform 10-fold cross-validation with a leave-one-person-out strategy to achieve comparable results.

In addition to IEMOCAP, we also evaluate our model on the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) dataset. RAVDESS consists of 24 actors (12 males and 12 females) and includes both emotional speech and song samples, covering a range of emotions such as happiness, sadness, anger, fear, surprise, and neutral. For this dataset, we adopt a random 8:2 split for the training and test sets. Figure 7 illustrates the number of audio samples corresponding to each emotional label. This split strategy aligns with the approach used in other studies on the RAVDESS dataset, facilitating comparisons of our results with those reported in the literature. The differing data split strategies between IEMOCAP and RAVDESS are intended to ensure consistency with prior research on the RAVDESS dataset.

In this study, we used the HuBERT pre-trained model to extract speech embeddings, which served as the input to our pyramid network model. The HuBERT model, pre-trained on a large corpus of speech data, provides high-quality representations that capture both phonetic and prosodic features. These embeddings were crucial in facilitating accurate emotion recognition, as they allowed the model to focus on high-level speech characteristics rather than low-level acoustic details. By incorporating these pre-trained embeddings, the pyramid network could more effectively discern emotional cues in speech, leading to improved model performance across different emotion categories. This approach highlights the importance of leveraging pre-trained models for feature extraction in speech-based emotion recognition tasks.

4.2. Implementation Details

The proposed framework was implemented in PyTorch. For training, we used the Adam optimizer with a learning rate of $1\times {10}^{-5}$, a batch size of 32, and applied early stopping [38]. To handle data imbalance, we evaluated performance using both weighted accuracy (WA) and unweighted accuracy (UA). WA measures the overall classification accuracy by dividing the total number of correctly predicted samples by the total number of samples, while UA calculates the average accuracy for each emotion category, providing insight into the model’s performance across different emotions. The experiments were conducted on a local environment with an NVIDIA GeForce RTX 3080 GPU, sourced from NVIDIA, Santa Clara, CA, USA.

$$\begin{array}{cc}\hfill & \hfill WA={\displaystyle \frac{{\sum }_{i=1}^K{N}_i\ast Accurac{y}_i}{{\sum }_{i=1}^K{N}_i}}.\hfill \\ \hfill & \hfill UA={\displaystyle \frac{1}{K}}\sum _{i=1}^{K}Accurac{y}_i.\hfill \end{array}$$

(9)

where K denotes the number of emotion categories and i represents the i-th emotion category. $N_i$ denotes the data quantity of the i-th emotion category.

4.3. Experiment Results and Discussion

To fairly compare the performance of the proposed framework, we implement the end-to-end method (E2ESA without multi-task learning), resolution maintained method (DRN), and multi-scale feature representation method (GLAM without data augmentation) for evaluation. The experimental results are shown in

Table 1. Comparisons of UA and WA with state-of-the-art methods on IEMOCAP. The best results are highlighted in bold.

Model	UA	WA
GLAM [13]	69.70%	68.75%
E2ESA [39]	70.86%	69.25%
Xie [24]	70.0%	68.8%
DRN [10]	71.59%	70.23%
MSFPN	73.39%	71.79%

. In the MSCNN-based method, GLAM’s multiple feature representations are conducted on the high-level semantic features, which are limited by the fine-grained temporal dynamics and local representation learning. Compared to the end-to-end method, E2ESA uses single-scale correlation modeling and limits global–local representation learning for context-aware emotion classification. The DRN method appropriately preserves the resolution of deep features, which avoids the loss of temporal structure of speech in hierarchical CNN. However, directly applying the MSCNN module in the DRN leads to a gridding effect when learning high-level temporal dynamics upon the mediate hidden features. In this paper, the MSFPN mainly benefits from the high semantic and full resolution feature map for global–local representation learning and achieves the highest results with 3.69% UA (GLAM), 3.39% UA (Xie [24]), 2.53%UA (E2ESA), and 1.8%UA (DRN) improvements, respectively. Additionally, the model inference time on the IEMOCAP dataset is 0.5 s, ensuring efficient real-time processing.

Furthermore, we also evaluate the performance of MSFPN on the RAVDESS dataset. Since most existing studies report results on the RAVDESS dataset using unweighted accuracy (UA) as the evaluation metric, we adopt UA as the evaluation metric for MSFPN on this dataset as well. The experimental results are shown in

Table 2. Comparisons of UA with state-of-the-art methods on RAVDESS. The best results are highlighted in bold.

Model	UA
Chakhtouna [40]	82.6%
Ullah [41]	82.3%
MSFPN	86.5%

. Compared to the methods proposed by Chakhtouna [40] and Ullah [41], MSFPN better extracts multi-granularity emotional features across different scales. As shown in Table 2, the ability to capture multi-granularity features significantly improves emotion recognition accuracy on the RAVDESS dataset as well. The comparison results demonstrate that the proposed method effectively captures multi-scale contextual information in speech, achieving superior performance on the RAVDESS dataset as well. The model inference time on the RAVDESS dataset is 0.2 s, further emphasizing the efficiency of the proposed method.

Moreover, the feature space of MSFPN is well compacted and assembled. The t-SNE visualization is depicted in Figure 8. This figure shows that our method can produce differentiated emotional features. Unlike DRN, neutral emotional features are more condensed and refined and less mixed in other emotional spaces.

4.4. Ablation Study

To further measure the contributions of each component of the proposed model, the results of ablation experiments depicted in the IEMOCAP dataset, which is shown in

Table 3. Performance of ablation studies on IEMOCAP dataset. ‘w/o’ denotes the vanilla MSFPN framework without certain components.

Method	UA	WA
w/o CSA	66.89%	65.35%
w/o MSCNN	70.95%	69.05%
w/o forward fusion	71.85%	70.09%
w/o backward fusion	72.72%	71.26%
MSFPN	73.39%	71.79%

.

From the results in Table 3 and

Table 4. Performance of ablation studies on RAVDESS dataset. ‘w/o’ denotes the vanilla MSFPN framework without certain components.

Method	UA
w/o CSA	68.8%
w/o MSCNN	80.6%
w/o forward fusion	80.9%
w/o backward fusion	83.0%
MSFPN	86.5%

, there is a clear measurement of the contributions of each component. The forward fusion provides the fusion of multi-scale deep features for robust semantic feature leaning in progressive downsampling. The backward fusion mainly focuses on the salient emotional periods in multi-scale deep features and aggregates them into a high-resolution feature map. The CSA in MSCNN is important for local correlation learning, which effectively enhances the expression of emotion in semantically strong features. The core component in our proposed framework is the MSCNN, which extracts multiple deep features and enables the fusion mechanism to appropriately capture emotional-related features.

5. Conclusions

In this paper, we propose a multi-scale feature pyramid network for context-aware speech emotion recognition. The MSCNN-based feature extraction module with an improved CSA layer is useful for capturing global–local correlations in the speech sequence. With bottom-up and top-down connections, semantically strong features are effectively aggregated with high-resolution acoustic features, which contain adequate emotional characteristics and retain temporal structure for global–local representation learning. The experimental results on IEMOCAP demonstrate our framework’s effectiveness and are significantly better than the state-of-the-art approaches.

6. Future Work

While the proposed multi-scale feature pyramid network has proven effective in capturing multi-scale features from speech, its potential in multimodal emotion recognition was explored only to a limited extent. Previous studies have shown that integrating multiple modalities can further improve recognition performance [42,43,44]. We plan to extend this framework to a multimodal setting by incorporating complementary information from diverse modalities, such as text, visual cues, and physiological signals. By harnessing the strengths of multi-scale feature learning across modalities, we aim to achieve a deeper understanding of emotional expressions and further enhance the system’s robustness and performance in complex real-world scenarios.