Speech Emotion Recognition Using RA-Gmlp Model on Time–Frequency Domain Features Extracted by TFCM

Abstract

Speech emotion recognition (SER) is a key branch in the field of artificial intelligence, focusing on the analysis and understanding of emotional content in human speech. It involves a multidisciplinary knowledge of acoustics, phonetics, linguistics, pattern recognition, and neurobiology, aiming to establish a connection between human speech and emotional expression. This technology has shown broad application prospects in the medical, educational, and customer service fields. With the evolution of deep learning and neural network technologies, SER research has shifted from relying on manually designed low-level descriptors (LLDs) to utilizing complex neural network models for extracting high-dimensional features. A perennial challenge for researchers has been how to comprehensively capture the rich emotional features. Given that emotional information is present in both time and frequency domains, our study introduces a novel time–frequency domain convolution module (TFCM) based on Mel-frequency cepstral coefficient (MFCC) features to deeply mine the time–frequency information of MFCCs. In the deep feature extraction phase, for the first time, we have introduced hybrid dilated convolution (HDC) into the SER field, significantly expanding the receptive field of neurons, thereby enhancing feature richness and diversity. Furthermore, we innovatively propose the residual attention-gated multilayer perceptron (RA-GMLP) structure, which combines the global feature recognition ability of GMLP with the concentrated weighting function of the multihead attention mechanism, effectively focusing on the key emotional information within the speech sequence. Through extensive experimental validation, we have demonstrated that TFCM, HDC, and RA-GMLP surpass existing advanced technologies in enhancing the accuracy of SER tasks, fully showcasing the powerful advantages of the modules we proposed.

1. Introduction

Speech, as the most instinctive and natural form of human communication, has always been a focal point for scientists and researchers. Based on this, the study of SER has emerged, aiming to build a bridge between human speech information and emotional expression using computer science and technology. SER is a key component of artificial intelligence (AI), involving multiple disciplines, such as acoustics, phonetics, linguistics, information theory, pattern recognition, and neurobiology. In human–computer interaction, SER plays a crucial role, aiming to more accurately understand and recognize the emotional state conveyed by the speaker through the voice signal. This technology has been widely practiced and applied in various intelligent applications. For example, in intelligent interaction systems, voice customer service has become the main means to achieve effective interaction [1]. Moreover, in the medical field [2,3], SER can assist doctors and medical workers in more accurately understanding the emotional state of mental patients, thereby providing more personalized and precise medical services. In the field of autonomous driving, by analyzing the emotional state of drivers through SER technology, the system can more accurately determine whether to take corresponding safety measures to avoid potential dangers. In the field of education, teachers and educators can use SER to more comprehensively understand the emotional state of students, to provide more personalized and effective teaching methods and psychological support. In the field of criminal investigation, by analyzing the voice of suspects, one can more accurately understand their emotional state, thus providing important auxiliary information for the investigation and trial of cases. These application examples demonstrate the potential value and practical application of SER in different fields. With further research and development, SER is expected to play a larger role in enhancing human–machine interaction, medical services, autonomous driving, education, and criminal investigation.

Traditional SER methods mostly rely on manually designed low-level descriptors (LLDs) for classification. Lampropoulos et al. [4] evaluated the application of MPEG-7 audio descriptors in emotion category modeling, especially in the emotion classification task based on an RBF-SVM classifier, showing good performance. However, with the continuous development and maturity of deep learning and neural network technologies, there has been a gradual shift towards using neural networks to extract more advanced and comprehensive features. Although these methods have achieved certain success in enhancing the diversity and richness of features, there are still many challenges and problems in practical applications. Among them, the main challenge is how to extract the most representative and informative features from the vast and complex speech signals, which remains a highly challenging issue. To address these issues, researchers have proposed various methods and techniques. For instance, through the multi-attribute decision-making (MADM) method, M. Virvou [5] innovatively integrates the image and speech mode in the dual-mode user interface, realizing the dynamic recognition of the six basic emotions of computer users. M. R. Makiuchi et al. [6] suggest combining acoustic information with textual information, utilizing high-level contextual information to aid in emotion prediction. Zhang et al. [7] use features of three modalities, speech, text, and image, to achieve a more comprehensive and accurate emotion recognition. Moreover, Heqing Zou et al. [8] have predicted tasks by integrating spectrogram features, MFCC features, and the time domain and frequency domain information of Wav2Vec2. However, these methods have not yet sufficiently considered the impact of time domain and frequency domain features on the SER tasks.

Without increasing parameters, by enlarging the receptive field of neurons, we can extract more diverse and rich emotional features, thereby more accurately reflecting the speaker’s emotional state. In the field of SER, we are the first to apply high-dimensional convolution (HDC) to speech feature extraction. In recent studies, Mirsamadi et al. have proposed a recurrent neural network (RNN) [9] with a local attention architecture to learn features in SER. F. Tao et al. [10] have introduced a new variant of long short-term memory (LSTM), namely, advanced LSTM (A-LSTM), to better model the temporal context for SER. Chen et al. [11] have employed an attention-based convolutional RNN (ACRNN) to extract high-level emotional feature representations from log Mel spectrograms. In [12], the recognition of the emotion was achieved using vision transformers. With the introduction of MLP-Mixer [13], which does not use convolution or self-attention mechanisms, it has effectively replaced vision transformers [14]. This has led to a renewed interest in MLP research. Xiaohan Ding [15] proposed RepMLP, combining the global feature extraction capability of MLPs with the local feature extraction ability of CNNs, thereby improving performance in translation-invariant tasks, such as semantic segmentation. Zongzhao Qiu [16] achieved a 33% reduction in mean squared error (MSE) by stacking GMLP and ReZero to learn sequential drug-target affinity information. Jiale Yan [17] introduced TT-MLP using tensor decomposition to compress deep MLPs, thus reducing the amount of training parameters. Wenjing Zhu [18] utilized GMLP components and the global attention module (Glam) to achieve good results. However, speech emotional information is not uniformly distributed across speech features, and the aforementioned MLP structures did not consider how to focus the model on the required information. Therefore, we propose a new type of MLP structure called RA-GMLP, which uses multihead attention to weight the importance information of emotion. This allows for a more focused approach to the key information in the frame, comprehensively understanding and representing the emotional information in speech, thereby significantly improving the accuracy of SER tasks.

In this paper, we will detail the design and implementation specifics of the model we propose and validate its performance and effectiveness through extensive experiments and analysis. We will also compare it with other mainstream SER methods to demonstrate the superiority of our model. In summary, the main contributions of this paper in the field of SER research are as follows:

(1)

We propose a residual attention-based gated multilayer perceptron (RA-GMLP) structure, where the GMLP helps the model determine which information is reserved, and the attention mechanism further weights this important emotion information. This structure fills the gap of the GMLP mechanism in terms of attention, enhancing the accuracy of SER.

(2)

This study proposes TFCM and uses a new method to achieve a time–frequency domain fusion of MFCC features. The experimental results show that this fusion method is actually superior to traditional feature extraction methods.

(3)

To increase the receptive field of neurons and the diversity of speech emotion features, hybrid dilated convolution is introduced to the SER task for the first time. Experimental results show that the model achieves outstanding performance in speech emotion classification tasks.

(4)

This paper proposes a time–frequency SER model based on the RA-GMLP module, which has higher classification accuracy.

2. Related Work

2.1. Traditional Features

In the classification of speech emotions, traditional methods involve using low-level descriptors (LLDs) that are manually designed for classification. Different high-level statistical functions (HSFs), such as the mean, maximum value, and variance, are applied to each LLD, and then the results are concatenated into a long feature vector. With the development of artificial intelligence, features have become more diverse. Currently, the main features in SER include prosodic and spectral features [19]. Prosodic features mainly consist of fundamental frequency-related features and energy-related features [20], which can extract both global and local features, such as maximum, minimum, mean, and variance, forming a high-dimensional feature set. Spectral features reflect the differences in the frequency domain of the signal, including linear spectral features and cepstral features. Common linear spectral features include linear prediction coefficients (LPCs) and log frequency power coefficients (LFCCs), while common cepstral features include Mel-frequency cepstral coefficients (MFCCs) and linear predictive cepstral coefficients (LPCCs). Studies have shown that cepstral features have better discriminative power for speech emotions than linear spectral features, as demonstrated by the research of Bou-Ghazale et al. [21]. In feature extraction, Han et al. [22] used a 16-dimensional feature set, where the first 9 dimensions were prosodic features and the remaining 7 were spectral features, including energy, formant, harmonics-to-noise ratio, etc., and utilized a nonlinear proximal support vector machine based on the Gaussian kernel for classification. Hsiao et al. [23] extracted features such as fundamental frequency, frequency perturbation, zero-crossing rate, energy, harmonics-to-noise ratio, and MFCC, and used a deep RNN model for classification, increasing the unweighted average recall (UAR) from 37.00% to 46.30% in the FAU-Aibo task. Meanwhile, Hudesheng et al. [24] extracted features including mean zero-crossing rate, energy, fundamental frequency, formant, and MFCC, and used a main auxiliary network feature fusion method to achieve 72.50% unweighted accuracy on the IEMOCAP dataset.

2.2. Popular SER Neural Network

Recurrent neural networks (RNNs) have been widely applied to SER tasks. RNNs are capable of modeling temporal dependencies within audio sequences. In contrast, convolutional neural networks (CNNs) [25] focus on extracting local information within sequences, while long short-term memory networks (LSTMs) [26] are used to handle long-term dependencies. Therefore, some studies have adopted a CNN+LSTM [27] structure, which has achieved good results. With the advancement of deep learning, the Transformer [28] was proposed and has been successfully applied to fields like machine translation, text classification, and text generation. In the realm of SER, there has been research applying the Transformer [29] to emotion recognition tasks, and a key-sparse Transformer [30] was proposed to better focus on emotional information. Recently, Google’s ViT team introduced a novel visual framework, MLP-Mixer, which uses multilayer perceptrons (MLPs) instead of the convolution operations in traditional CNNs and the self-attention mechanism in Transformers, achieving comparable results to Transformers. Additionally, by combining spatial projections and gated units [31], a method for capturing the most important emotional information within a global range was also proposed. Finally, Zhu [18] achieved good accuracy using GMLP as a classifier, proving the effectiveness of GMLP in the field of speech.

2.3. Hybrid Dilated Convolution

Dilated convolution is a convolution operation widely applied in the fields of computer vision and image processing. Compared with traditional convolution operations, dilated convolution introduces a dilation rate parameter to increase the receptive field of the convolution kernel, thereby capturing a larger range of contextual information while maintaining computational efficiency. The origin of dilated convolution can be traced back to 2015, first proposed by Fisher Yu and Vladlen Koltun in their research [32]. In this study, they introduced a new convolution operation, the dilated convolution, for image semantic segmentation tasks. Dilated convolution incorporates dilations (or holes) in the convolution kernel, sliding the kernel with a fixed sampling interval, indirectly enlarging its receptive field. The dilation rate parameter determines the size of the sampling interval; when the dilation rate is 1, the dilated convolution is equivalent to the traditional convolution operation. When the dilation rate is greater than 1, the dilated convolution can expand the receptive field. Dilated convolution has achieved remarkable results in fields such as image segmentation, object detection, and image semantic analysis. For example, Chen et al. [33] extensively applied dilated convolution in the DeepLab series of methods proposed in 2017. By incorporating multiscale dilated convolutions, DeepLab can effectively capture context information at different scales, achieving excellent performance in image segmentation tasks. In addition, dilated convolution has also been applied to tasks such as image super-resolution and image generation. For instance, Lin et al. [34] applied dilated convolution to the task of image super-resolution reconstruction in the dilated residual networks proposed in 2017, achieving remarkable results. Although extended convolution has a wide range of applications and developments in the field of images, it has not yet been applied in the field of SER, which is a unique aspect of our work. Our research will fill this gap, exploring the potential of dilated convolution in SER.

3. Architecture Design

In the following sections, we will elaborate in detail on the architecture of the model we propose, as illustrated in Figure 1. The time domain features, frequency domain features, and time–frequency domain features, derived from the same audio segment, are represented as $X_t^{B_t\times C_t\times T_t{\times }_t}$, $X_f^{B_f\times C_f\times T_f\times F_f}$, and $X_{tf}^{B_{tf}\times C_{tf}\times T_{tf}\times F_{tf}}$, respectively. The model consists of three main parts: the MFCC feature extraction module for the TF-feature, the HDC module for the deep feature, the RA-GMLP blocks for the classification block.

The MFCC feature extraction module uses three parallel convolution blocks to extract time domain information: Xt, frequency domain information: Xf, and time–frequency domain interaction information: Xtf, which, after a map, Xt ⊕ Xf ⊕ Xtf->X1, reach $X_1^{B\times C\times F\times T}$. We then introduce a multiscale dilated convolution structure to enrich the features and capture a time–frequency domain deep feature in the speech signal. Subsequently, the concatenated deep features $X_2^{B\times C\times F\times T}$ are input into the RA-GMLP for further extraction and understanding of more comprehensive feature information. Finally, the classification is performed through a fully connected layer, realizing the recognition and classification of different speech signals. This architecture effectively leverages the individual strengths of the MFCC time domain features, MFCC frequency domain features, and MFCC time–frequency domain interaction features, enhancing the model’s performance in complex speech signal processing tasks.

3.1. Feature Extraction Block

As Figure 2 shows, this method extracts time domain and frequency domain information separately, then concatenates the original time–frequency features to achieve a more effective feature fusion. Figure 1 shows the size variation of our features; we used a kernel size of 1 × 3 for temporal convolution. To better capture frequency domain information, we used a kernel size of 3 × 1 for frequency domain convolution. To maintain the connectivity between the time and frequency domains, we performed convolution on the overall features using a kernel size of 3 × 3, with the number of output channels set to 16.

In addition, batch normalization is performed after tconv fconv and tfconv extraction. We concatenated the outputs of the temporal convolution, frequency domain convolution, and time–frequency domain convolution. After passing through the concatenation layer, we obtained time–frequency domain features $X_1^{B\times C\times F\times T}$(B is the batch size of 32, C is the number of channels of 16, F is the frequency dimension of 78, and T is the time domain dimension of 57). Through such processing, we are able to better capture the temporal and frequency domain information in speech signals, enhancing the model’s understanding of speech features.

3.2. Dilated Convolution

In the field of imaging, deep convolutional neural networks have been proven to effectively extract feature representations of images. However, we aim to achieve a larger receptive field with fewer parameters to capture richer features. To address this issue, dilated convolution is introduced since it allows us to achieve our goals, including the reduction of parameter quantity. However, dilated convolution may lead to the gridding effect by inserting zeros between pixels in the convolution kernel, potentially losing important information. This paper employs a hybrid dilated convolution block for further extraction of the time–frequency mixed features $X_1^{B\times C\times F\times T}$. This hybrid dilated convolution block effectively avoids the gridding effect and can capture spectral feature maps while preserving more spatial information. As illustrated in Figure 1, the deep feature extraction module includes three dilated convolutional layers. After each convolution, we perform downsampling and add a max pooling layer after each convolutional layer. As shown in Figure 3, the choice of dilation rates follows a heuristic method similar to a sawtooth wave, sequentially choosing dilation rates of 1, 2, and 3. Using three consecutive dilated convolution blocks, the receptive field is 13 × 13, which is much larger than that of ordinary convolution.The bluer the color, the more times the feature points are counted. According to the calculation of Formula (1), where ‘a’ represents the dilation rate,’k1’ represents the Convolution kernel,’k’ represents the Dilated Convolution kernel. ’p’ represents padding of Convolution,’w’ represents the dimension of feature, we ultimately obtained $X_2^{B\times C\times F\times T}$ (B is the batch size of 32, C is the number of channels of 16, T is the temporal dimension of 18, and F is the frequency domain dimension of 12). By combining dilated convolutions with different dilation rates, we expanded the receptive field, making it more adaptable to the complexity of time–frequency domain features. Therefore, we can capture the emotional features of speech more richly, thereby improving the performance of the model in emotion classification tasks. Finally, the features $X_3^{B\times C\times D}$ were obtained through reshape (Change of dimension).

$$\begin{array}{c}k=k1+(k1-1)·(a-1)\\ w=\frac{w-k+2p}{s}+1\end{array}$$

(1)

3.3. Residual Attention Perceptron Block

Figure 4 illustrates the overall structure of RA-GMLP; to compensate for the shortcomings of GMLP, we supplemented GMLP with an attention mechanism, which we introduce in detail in this section as residual attention-gated multilayer perceptron (RA-GMLP). The input to RA-GMLP is $X_3^{B\times C\times D}$ (where B represents the batch size of 32, C represents the channel size of 128, and D represents the speech feature dimension of 216). The self-attention mechanism represents content stored in memory by key-value pairs, where each element consists of an address (key) and a value (value). A query can match a keyword, and depending on the relevance between the query and the keyword, the corresponding value is retrieved from the memory. As shown in Formula (2), the query, key, and value are typically multiplied by parameter matrices W to obtain Q, K, and V. By calculating attention scores, the model can capture complex contextual relationships between speech frames to enhance the capability of feature expression.

$$\begin{array}{c}Q=W_Q\times \mathrm{query},K=W_K\times \mathrm{key},V=W_V\times \mathrm{value}\\ \mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^T}{\sqrt{D_k}}\right)V\end{array}$$

(2)

After the multihead attention layer, the dimensions of the output features remain unchanged. Then, we perform a linear projection along the D dimension, expanding the D dimension of the feature $X_3$ to 4D, and introduce nonlinearity with the GELU activation function. Next, we split the features along the D dimension into D1 and D2, where D = D1 + D2. In the gating mechanism, we use fw, b(D2) as the gate to judge the importance of the information in D1. Specifically, each position in D1 is multiplied by a learnable gating vector D2, resulting in a new position representation. This position representation retains some features of the original position while also diminishing unimportant emotional features. Thus, through the gating mechanism, we can control the flow of information in feature D1, thereby improving the model’s expressiveness and generalization ability.

$$\begin{array}{cc}\hfill f_{w,b}\left(D\right)& =wD+b\hfill \\ \hfill S\left(Z\right)& =D1\times f_{w,b}\left(D2\right)\hfill \\ \hfill X4& =S\left(X3\right)\hfill \end{array}$$

(3)

As illustrated by Formula (3), we perform multiplication along the D dimension, resulting in new features $X_4$ that have the same dimensions as the input features $X^{B\times C\times D}$. The synergistic operation of the attention and gating mechanisms enhances the model’s capability to capture information: the former reveals the complex contextual relationships between different features at the current moment, while the gating mechanism controls the retention of information related to emotions, accurately reflecting the dynamic variations in the speech signal. In this way, we can more accurately express the emotional information of the input features, thereby improving the model’s classification accuracy.

4. Detailed Analysis of the Dataset and Precise Configuration of Experimental Parameters

4.1. Analysis of the Dataset

In this section, we will provide a detailed introduction to the performance of our model and comprehensively evaluate the impact of different input variables and model components on the final performance through a series of meticulously designed ablation studies. To validate the effectiveness of our proposed model, we conducted experiments using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. This dataset will be introduced in detail below.

The IEMOCAP dataset was recorded by the University of Southern California and contains about 12 h of multimodal audiovisual data, recorded in five sessions. There are two methods for organizing the IEMOCAP dataset, one based on scripts and the other improvised. The dataset includes performances by male and female professional actors in both scripted and improvisational scenarios. The scripted part consists of the actors performing predetermined emotional scenes, while the improvisational part is closer to natural speech and performance. These samples are annotated in terms of size and classification. The dataset includes a total of 5531 samples, with happiness accounting for 29.5%, sadness accounting for 19.6%, anger accounting for 19.9%, and neutrality accounting for 30.8%.

4.2. Configuration of Model Parameters

In our research, the implementation of the model is based on the PyTorch Python library version 3.6. We tested these models on a TITAN RTX (GPU).

Table 1. Experimental parameter settings.

	Baseline		HDC		HDC+RA-MLP
	(W/O TF)	(W/TF)	(W/O TF)	(W/TF)	(W/O TF)	(W/TF)
Segments length	2 s	2 s	2 s	2 s	2 s	2 s
Overlap	1.6 s	1.6 s	1.6 s	1.6 s	1.6 s	1.6 s
Batch size	32	32	32	32	32	32
Epoch	50	50	50	50	50	50
Learning rate	${10}^{-3}$	${10}^{-3}$	${10}^{-3}$	${10}^{-3}$	${10}^{-3}$	${10}^{-3}$
Feature input	(26, 57)	(78, 57)	(26, 57)	(78, 57)	(26, 57)	(78, 57)
Optimizer	Adam	Adam	Adam	Adam	Adam	Adam
Feature output	(32, 4)	(32, 4)	(32, 4)	(32, 4)	(32, 4)	(32, 4)

summarizes the comparison parameters of our proposed model and the baseline models. For the experiments, we randomly divided the IEMOCAP dataset into 80% training set and 20% test set. Additionally, we meticulously processed each speech segment in the dataset, splitting it into 2 s clips with a 1.6 s overlap time set. In terms of feature extraction, we developed a time–frequency domain feature extraction method based on Mel-frequency cepstral coefficient (MFCC) features. Further, we introduced the hybrid dilated convolution technique to extract deeper features and proposed the RA-GMLP module to enhance performance. The experiments were divided into two main parts: the first three sets of experiments did not use the time–frequency domain feature extraction method, and their input size was 26 × 57; the latter three sets of experiments applied time–frequency domain feature extraction, resulting in an input size of 78 × 57. The training configurations for all experiments were consistent, using a learning rate of ${10}^{-3}$, a batch size of 32, and a total of 50 epochs of training. As the loss function, we chose cross-entropy and selected the Adam optimizer. The output size for all experiments was uniformly set to 32 × 4. Finally, we chose weighted average accuracy (WA), unweighted average accuracy (UA),training time, t-sne scatter diagram, and loss function as the evaluation criteria for our model.

5. Detailed Analysis and Discussion of Experimental Results

5.1. Our Model Was Compared and Analyzed against Current Advanced Models

When evaluating the performance of deep learning models, accuracy and training efficiency are two crucial metrics. As seen in

Table 2. Model comparisons with other methods on IEMOCAP (scripted + improvised) in terms of WA UA and time.

Model	Wa	Ua	(Time/Epoch)
SPSL-MFCC - Mel-spec [35]	60.00	58.00	1.2 s
SEQCAP-Spectrogram [36]	72.73	59.71	2.7 s
APCNN-MFCC [18]	69.00	67.00	1.5 s
MHcnn-mfcc [18]	69.80	70.09	3.5 s
AAcnn-mfcc [18]	70.94	71.04	1.3 s
OUR	75.31	75.09	1.5 s

, our model demonstrates a moderate level of time consumption per training epoch, specifically 1.5 s. This duration is neither as rapid as the SPSL-MFCC model, which clocks the best training speed at 1.2 s per epoch, nor as slow as the SEQCAP-Spectrogram model, which marks the slowest in this comparison at 2.7 s per epoch. In terms of accuracy, our model, measured by Wa and Ua, achieves scores of 75.31% and 75.09%, respectively, surpassing all other competing models. While the SEQCAP-Spectrogram model comes close to our model in terms of accuracy, its training time per epoch is 2.7 s, significantly slower than our model, indicating that our model maintains a high level of training efficiency while achieving high accuracy.

Figure 5 illustrates the trend of loss values for various deep learning models over training epochs. In the initial phase of the training, the spsl-mfcc and apcnn-mfcc models demonstrated lower loss values. Apart from the SEQCAP-Spectrogram model, the loss values for the other models showed a pattern of rapid decline. However, as the training progressed, particularly by the 7th epoch, our model (indicated by the red line) began to show a significant advantage in the reduction of loss values. This trend was maintained and intensified in the subsequent training. By the 30th epoch, our model had reached the lowest point in loss values compared with the other models, demonstrating optimal performance on the training set. This result clearly signifies the superior performance and effectiveness of our model.

Based on

Table 3. The distribution of the IEMOCAP dataset.

Emotion Type	Sample Size (Total)	Sample Size (Male)	Sample Size (Female)
neu	929	466	463
hap	855	430	425
sad	653	251	402
ang	1071	531	540

, we can observe the sample distribution of the IEMOCAP dataset that we have utilized. We have utilized four emotion categories from this dataset: neutral (neu), happy (hap), sad (sad), and angry (ang), with each emotion type having varying sample sizes. Notably, the angry category has the highest number of samples, while the sad category has the least. The sample size is also approximately balanced between genders, which is beneficial in avoiding gender bias impacting the model’s performance.

Figure 6 displays the dimensionality-reduced visualization of test data after model training. Blue represents ‘neu’, red represents ‘hap’, yellow represents ‘sad’, and green represents ‘ang’. From the visualization, it is evident that our model can distinctly differentiate between various emotional categories, with clear boundaries between each category, indicating robust classification capabilities. The balanced and diverse distribution of samples aids the model in learning more generalized features, thus enhancing the model’s performance on unseen data. Moreover, the substantial volume of samples provides the model with ample information to learn complex decision boundaries, which is particularly crucial for fine-grained tasks such as emotion classification. In Figure 6, it is apparent that our model has utilized these balanced data more effectively during training, leading to a better distinction of categories in the dimensionality reduction visualization. In summary, the balanced distribution of training samples is critical for the overall performance of the model, and our model excels in this regard, which also explains its superior classification results.

5.2. Investigating the Impact of Varying Dilation Rates in Hybrid Dilated Convolution on Model Performance

Figure 7 demonstrates the changes in the receptive field when using different dilation factors. Meanwhile, Figure 8 shows the accuracy of three different combinations of HDC we attempted while extracting deep features using dilated convolution. Among these methods, when the dilation factors for all three layers of dilated convolution are set to 2, the model’s performance is lower than most of the models we compared. However, when all dilation factors are set to 3, the model’s performance is on par with current methods. This phenomenon is mainly due to the gridding effect caused by dilated convolutions. Compared with existing SER models, when the dilation factors are set to 1, 2, and 3, respectively, there is a significant improvement in accuracy. Therefore, to avoid the gridding effect and fully leverage the advantages of dilated convolutions, we set the dilation factors to 1, 2, and 3, respectively, in the three layers of dilated convolutions. Such parameter settings not only prevent the gridding effect but also effectively extract features. Based on these considerations, we chose the last parameter setting method, the third one, to ensure the correct application of dilated convolutions and optimize performance.

5.3. Exploring the Specific Impact of Different Attention Heads on Model Performance

Multihead attention is one of the core components of the RA-GMLP model, which allows the model to capture emotional information across different representational subspaces by computing multiple sets of attention weights in parallel. Vectors from different spaces are then aggregated (concatenated) together and passed through a linear transformation to produce the final output. For the processing of deep speech emotion features, we have conducted experimental comparisons between multihead attention mechanism GMLP with 1, 2, 4, and 8 heads and GMLP without attention. As shown in Figure 9, apart from RA-GMLP-Head2, which performs comparably to GMLP, the other multihead attention mechanisms significantly outperform GMLP, thanks to the attention mechanism’s ability to focus on key information.

5.4. A Detailed Comparative Analysis of the GMLP Mechanism and Its Enhanced Version, the RA-GMLP Mechanism

As shown in Figure 10, after incorporating the attention module, the accuracy rates for emotions other than “Anger” improved, while the accuracy for “Anger” became slightly lower. Moreover, it substantially eliminated the confusion set “Ang-Sad”. Based on the experimental results, we adopted the most outstanding 8-head attention mechanism in RA-GMLP to ensure the best model performance.

5.5. Conducted Ablation Studies and Provided an In-Depth Analysis of the Results

Figure 11 and Figure 12 clearly depict the trajectory of accuracy changes for our proposed modules during the first 50 epochs. To measure performance, we used two key indicators: WA and UA. If the sum of WA and UA does not show an increase in any epoch, the values of WA and UA are no longer updated for that epoch.

From Figure 11 and Figure 12, it can be seen that from the 1st to the 10th epoch, there was a significant improvement in both WA and UA for all models. Then, from the 10th to the 30th epoch, the growth of accuracy began to plateau. In the subsequent 20 epochs, there was a very slight change in accuracy. Initially, the “HDC (w/TF)” model held the lead in accuracy, but as the training progressed, the “HDC+RA-GMLP (w/TF)” model gradually surpassed the other models in accuracy. TFCM showed the best performance, both in the early stages and towards the end of the training. After completing 50 epochs of training, the “HDC+RA-GMLP (w/TF)” model achieved the highest level of accuracy.

As seen in Figure 13, under the condition of using the same model architecture, we found that the method using TFCM significantly outperforms the traditional conv method in both WA and UA. For instance, without TFCM, the baseline model (Baseline w/o TF) achieved a WA accuracy of 70.10%, while with TFCM (w/TF), the accuracy increased to 71.03%. Similarly, when the HDC module and RA-GMLP module were introduced, the “HDC (w/TF)” and “HDC+RA-GMLP (w/TF)” models achieved WA accuracies of 74.19% and 75.31%, respectively. This is a respective increase of 4.29% and 1.16% over “HDC (w/o TF)” and “HDC+RA-GMLP (w/o TF)” without TFCM. Models utilizing TFCM also showed an increase in UA accuracy, proving the effectiveness of our proposed feature extraction method. As the first three bar graphs indicate, even without TFCM, “HDC (w/o TF)” showed a modest increase in WA and UA over the baseline model, but with the addition of the RA-GMLP module, WA and UA were respectively improved by 4.05% and 3.96%, indicating an overall increase in accuracy. In the last three bar graphs, we observed that the addition of the HDC module and RA-GMLP module, when using TFCM, brought respective improvements of 3.36% and 0.92% in WA, with a steady increase in UA accuracy as well. Therefore, our proposed modules demonstrated good performance in both types of feature extraction methods involved in this study.

As shown in Figure 14, without the use of TFCM, the HDC module improved the recognition accuracy for the “Anger (Ang)” and “Sadness (Sad)” emotions, but at the same time, it reduced the recognition accuracy for the “Happiness (Hap)” and “Neutral (Neu)” emotions. Nonetheless, the overall accuracy rate was still improved. The RA-GMLP module effectively reduced the confusion between “Neutral-Anger (Neu-Ang)” and “Sadness-Anger (Sad-Ang)”, resulting in an increase of 27 and 13 correctly identified samples for the “Happiness” and “Neutral” emotions, respectively. As depicted in Figure 15, with the use of TFCM, the model eliminated the confusion between “Happiness-Neutral (Hap-Neu)” and “Neutral-Sadness (Neu-Sad)”. The number of correctly recognized samples for the “Neutral (Neu)”, “Happiness (Hap)”, and “Anger (Ang)” emotions increased by 8, 9, and 4, respectively. Moreover, with the basis of time–frequency domain feature module extraction, the HDC module eliminated the confusion in “Neutral-Anger (Neu-Ang)”, significantly improving the recognition accuracy for the “Neutral” emotion, with an increase of 14 correct samples.

6. Conclusions

This study introduces an innovative SER architecture based on time–frequency domain features of speech. The architecture utilizes the time–frequency signal fusion model (TFCM) to extract time–frequency domain features from Mel-frequency cepstral coefficients (MFCCs). Experiments indicate that, compared with the baseline model, this feature extraction method can improve recognition performance by 1% over traditional methods. To broaden the receptive field of neurons and enrich feature representation, this paper is the first to introduce high-dimensional convolution (HDC) into the task of emotion recognition. Through experimental analysis, it was found that the newly proposed residual attention-gated multilayer perceptron (RA-GMLP) can more accurately capture key emotional information in sequences, thereby achieving a higher recognition accuracy. While the presented HDC and RA-GMLP demonstrate significant improvements in emotion recognition, their adaptability to other domains and larger, more diverse datasets has not been fully assessed. This constitutes an avenue for future research. To comprehensively evaluate the performance of the model, tests were conducted on the IEMOCAP dataset. The results show that the proposed model achieved a WA of 75.31% and a UA of 75.09%, demonstrating superior recognition performance compared with current mainstream models. These promising results pave the way for future applications in more interactive and emotionally aware computing systems, such as adaptive e-learning platforms and empathetic customer service chatbots. Further research will focus on refining the model to cater to these applications while exploring the integration of real-time emotional feedback mechanisms.