Enhancing Embedded Space with Low–Level Features for Speech Emotion Recognition

Abstract

This work proposes an approach that uses a feature space by combining the representation obtained in the unsupervised learning process and manually selected features defining the prosody of the utterances. In the experiments, we used two time-frequency representations (Mel and CQT spectrograms) and EmoDB and RAVDESS databases. As the results show, the proposed system improved the classification accuracy of both representations: 1.29% for CQT and 3.75% for Mel spectrogram compared to the typical CNN architecture for the EmoDB dataset and 3.02% for CQT and 0.63% for Mel spectrogram in the case of RAVDESS. Additionally, the results present a significant increase of around 14% in classification performance in the case of happiness and disgust emotions using Mel spectrograms and around 20% in happiness and disgust emotions for CQT in the case of best models trained on EmoDB. On the other hand, in the case of models that achieved the highest result for the RAVDESS database, the most significant improvement was observed in the classification of a neutral state, around 16%, using the Mel spectrogram. For CQT representation, the most significant improvement occurred for fear and surprise, around 9%. Additionally, the average results for all prepared models showed the positive impact of the method used on the quality of classification of most emotional states. For the EmoDB database, the highest average improvement was observed for happiness—14.6%. For other emotions, it ranged from 1.2% to 8.7%. The only exception was the emotion of sadness, for which the classification quality was average decreased by 1% when using the Mel spectrogram. In turn, for the RAVDESS database, the most significant improvement also occurred for happiness—7.5%, while for other emotions ranged from 0.2% to 7.1%, except disgust and calm, the classification of which deteriorated for the Mel spectrogram and the CQT representation, respectively.

1. Introduction

Identifying the emotional state from the speech signal is an integral part of the human–computer interaction systems. Knowing the speaker’s emotional state in such systems allows additional context to be added to the human–computer interaction process. Such information may also be helpful in speech recognition tasks while taking into account the emotional condition in the speech synthesis process, which can significantly improve the realism of the produced speech signal. The task of effectively recognizing the speaker’s emotional state in the speech signal is complex because, besides low–level attributes estimated from the signal, prosodic information, the speaker’s characteristics, the language he uses and the content of his speech significantly impact effectiveness. These factors make the search for convenient representation effective determining the emotional state from a speech signal is still ongoing by many research groups around the world. Despite the dominance of unsupervised learning systems in classification applications emotional states in the speech signal, there are solutions based on modifying the feature space by introducing additional attributes to increase the effectiveness of the classification process. For example, in work [1] the authors proposed a method of combining features, which implements the fusion of representations from the convolutional neural network (CNN) with heuristic-based discriminative features and utilize extreme learning machine in the learning process. Experiments with the EmoDB database show a 40% relative error reduction in F1-score compared to a solution based on a CNN network trained on spectrograms and the learning mechanism using bidirectional long short term memory. An interesting approach to feature fusion for emotional speech can be found in [2]. Authors proposed a new feature fusion model called Dual-TBNet where two 1D convolutional layers for features calculation, two Transformers and two bidirectional long short-term memory (BiLSTM) modules were used. The attention mechanism was used to capture the connections between features and BiLSTM was used to enhance the contextual information of the fusion. Model was tested on five popular emotional speech datasets and in the results Authors achieved decent results. Cao, et al. proposed in [3] a hierarchical network for integration of static and dynamic features for speech emotion recognition. The system uses encoder to encode static and dynamic features, then gated multi-features unit is used to determine intermediate representations and attention module performs the final prediction. The experimental stage shows that proposed approach outperforms state-of-the-art baselines. The combination of convolutional neural network-based multiperspective awareness module with a frame-level fine-grained fusion strategy for speech emotion recognition was presented in [4]. Authors showed, that fusion of a multiperspective–aware module to obtain emotional information from speech, an attention mechanism to focus on the salient attributes, and finally a fine–grained fusion strategy leads to classification accuracy over 72% on the IEMOCAP dataset. Another system based on transformer for speech emotion recognition in presented in [5]. Authors proposed cross–attention transformer with the fuse of raw speech signal, spectrogram, and mel–frequency cepstral coefficients (MFCC) features. In the result they reported classification accuracy over 73% which outperforms existing approaches. A hybrid representation of speech including temporal and spatial information is proposed in [6]. The system utilizes CNN with spectrogram and deep neural network (DNN) for features and long short–term memory (LSTM) for extracting temporal information using MFCC feature. Using obtained representations and applaying two different fusion strategies, the final classification accuracy was equal 91.25% for EmoDB dataset and 72.02% for IEMOCAP dataset. Static and dynamic time–frequency network for speech emotion recognition was proposed by Liu and others [7]. In their work, to generate static and dynamic emotional features, the time–frequency convolutional neural network and time–frequency transformer modules were used. To integrate both feature sets and generate meaningful representations a dedicated SD–Cross module was introduced. In the last step, to reduce the risk of overfitting, an augmentation stage was added. In the result of performed experiments, the proposed system outperforms state–of–the–art methods. Another transformer-based system for speech emotion recognition is presented in [8]. Authors proposed multimodal embedding module using pretrained models which provide a priori knowledge of multimodal information to the model. To learn multimodal emotional features and improve contextual emotional semantic understanding, a mutual transformer was introduced. The experimental results show that proposed architecture achieved over 82% classification accuracy. In [9], authors examined various ASR (Automatic Speech Recognition) outputs and fusion methods for speech emotion recognition. The results showed that using ASR hidden and text output with a hierarchical co-attention fusion improves speech emotion recognition accuracy in joint automatic speech and speech emotion recognition training. A self-attention mechanism and multi-scale fusion were evaluated in [10] for speech emotion recognition. The presented approach uses a self-attentional multi-channel convolutional neural network for learning dedicated features from text and a multi-scale fusion. The results show an improvement of weighted accuracy by 1.48% and of unweighted accuracy by 3% on the IEMOCAP dataset. A supervised approach based on a fusion of low-level feature sets for speech emotion classification is presented in [11], where the authors proposed a technique which uses two-stage feature selection and two fusion strategies. In the experimental stage, three sets of features and four popular datasets were used. Applying the proposed scheme, the obtained classification for all datasets exceeded 85% and was highest for EmoDB at 95.29%. Yao et al. [12] propose a framework that integrates three types of neural networks in the classification stage: DNN, CNN, and RNN. The proposed scheme for speech emotion recognition for four emotions is based on speech descriptors at frame, segment and utterance levels fed up separately to classifiers. Then, a confidence-based fusion strategy was designed to integrate the results of classification by separate classifiers. Authors reported achievement of 51.1% weighted and 58.3% unweighted accuracy on the IEMOCAP corpus. Speech emotion recognition approach with co-attention-based fusion was proposed in [13]. Multi-level acoustic information extracted from low-level features and embedded high-level information were treated as multimodal inputs to the dedicated co-attention module working as a fusion stage. The performed experiments show that the proposed technique achieves competitive performance with different cross-validation methods. Another different approach is presented by the authors in the paper [14], where they use separate Multitask Transformers architectures fusion of audio and text data. The described study focuses on three emotional speech datasets: IEMOCAP, MSP-IMPROV, and EmoDB, in a multimodal cross-corpus classification. The obtained results show an improvement in classification compared to using only audio or text data. Another fusion mechanism is presented in the paper [15]. This technique uses a CNN to extract multi-scale features, which are then fused. The authors present results obtained for six SER datasets: CASIA, IEMOCAP, EmoDB, RAVDESS, EMOVO, and SAVEE, for which an improvement in classification was achieved. On the other hand, the paper [16], presents the fusion of features generated using Wav2Vec 2.0 and Pitch Encoder, which are based on raw audio and pitch sequence, respectively. The authors conducted studies on tonal languages, for which they achieved significant improvement.

In this study, we have shown how low–level features describing fundamental frequency contour and the vocal tract’s resonances can improve the convolutional neural network’s embedded space and the final classification accuracy.

The main contribution of our work include:

• Improvement of classification accuracy by using modified embedded space in ResNet architecture with prosody features.
• Confirmation that the Constant-Q transform (CQT) spectrograms better describe emotional speech than Mel spectrograms, thus more suitable for emotional speech classification tasks.
• An analysis of negative and positive influences of the injection of low–level prosody features of speech signals to classify emotional states.

Our motivation to perform such experiments was to determine how low-level speech features can influence the embedded space obtained in the learning process in an unsupervised manner and how it can improve discriminative power in the classification of emotional speech.

The paper is organized as follows: the methodology, audio representations and brief description of deep neural network architecture are presented in the next section. Section 3 describes the dataset used, experimental setup, evaluation approach, and the result. Finally, the last section concludes the work.

2. Materials and Methods

2.1. Methodology

The proposed method uses low–level features of the speech signal, which are combined with the features generated by the selected neural network model at the final stage of classification.

The diagram shown in Figure 1 and the Algorithm 1 shows a general outline of the proposed solution. First, for each recording $s_i$ of emotional speech from the set two representations are determined, base features: $r_1^i={\gamma }_1\left(s_i\right)$ and injected features: $r_2^i={\gamma }_2\left(s_i\right)$ resulting in two sets, $R_1={\left\{r_1^i\right\}}_{i=1}^N$ and $R_2={\left\{r_2^i\right\}}_{i=1}^N$. In other words, each $s_i$ recording is represented by a pair $\{r_1^i,r_2^i\}$. The functions ${\gamma }_1$ and ${\gamma }_2$ return the time–frequency representation of the signal and selected low–level features where N is the number of recordings in the considered set. In the next step, each pair of features represents subsequent recordings processed as follows, the representation of $r_1$ is processed by neural network and then projected to a given size in the dense layer ${FC}_1$. As a result, a vector $x_1$ of size $Z=M-K$ is created, where K is the number of injected features and M is the size of the final layer classification ${FC}_2$. In both cases, $FC$ represents a single fully connected layer. The $r_2$ representation is normalized in the $NL$ layer [17], which creates the vector $x_2$. Then, the vectors $x_1$ and $x_2$ are combined in the concatenation process, giving final feature vector $X=[x_1\mid x_2]$ with length Z, which is processed by the final classification layer ${FC}_2$ and then $softmax$ operation is applied.

2.2. Audio Representations

In the performed experiments, as base features (${\gamma }_1$), we have used two time–frequency representations – CQT (Constant–Q Transform) and Mel spectrograms. This choice depends on previous research [18], where these representations obtained significantly higher results than the regular spectrogram in classifying emotional states using convolutional networks. Moreover, these representations present two different frequency scales. The frequency distribution is geometric for the constant–Q spectrogram and the ratio of the band’s center frequency to its width is constant. As a result, the resulting frequency scale has a different accuracy reproduction in the low and medium frequency range compared to Mel scale [19].

Algorithm 1 Injection mechanism

Input: Audio representations ${\{R_1^i,R_2^i\}}_{i=1}^N$ Output: Predicted labels ${\left\{Y^i\right\}}_{i=1}^N$ $Y=\left\{\right\}$ fori = 1 to N do Input $r_1^i$ into neural network and next to ${FC}_1$ for mapping to specific size and get feature map $x_1^i$. Send $r_2^i$ to $norm$ layer to get vector $x_2^i$. Concatenate $x_1^i$ and $x_2^i$ to get final feature vector $X^i$. Send features $X^i$ to final classification layer ${FC}_2$, execute $softmax$ operation and get predicted label $y^i$. $Y^i\leftarrow y^i$ end for

Mel spectrogram is calculated based on the spectrogram, which is obtained using the STFT (Short–Time Fourier Transform) algorithm, and then, the frequencies are converted to the Mel scale according to Formula (1):

$$f_{mel}\left(f_{hz}\right)=2595{log}_{10}\left(1+{\displaystyle \frac{f_{hz}}{700}}\right),$$

(1)

where $f_{mel}$ is the Mel frequency and $f_{hz}$ is the frequency in Hertz. In turn, in CQT, compared to a standard spectrogram, each signal frame is transformed using the Constant–Q Transform [20] method. This method uses a bank of filters corresponding to subsequent octaves. The central frequencies of successive filters are determined according to Formula (2).

$$f_m={\left(2^{\frac{1}{b}}\right)}^m·f_{min},$$

(2)

where $f_m$ is the center frequency of the m–th filter, b is the number of filters per octave and $f_{min}$ is the minimum frequency. The constant Q is then calculated according to the Equation (3).

$$Q={\displaystyle \frac{f_m}{\Delta f_m}}={\displaystyle \frac{f_m}{f_{m+1}-f_m}}={\left(2^{{\displaystyle \frac{1}{b}}}-1\right)}^{-1}.$$

(3)

In the final step, the frequency domain representation is computed for subsequent signal frames according to the Formula (4).

$$X_p\left[m\right]={\displaystyle \frac{1}{N\left[m\right]}}·\sum _{n=0}^{N\left[m\right]-1}W[m,n]·x_p\left[n\right]·e^{{\displaystyle \frac{-j2\pi Qn}{N\left[m\right]}}},$$

(4)

where W is the window function, p is the signal frame number, and $N\left[m\right]$ is the frame length for bin m, which is determined according to the following equation:

$$N\left[m\right]=Q·{\displaystyle \frac{f_s}{f_m}},$$

(5)

where $f_s$ denotes sampling rate. This representation was initially proposed for reproducing the Western musical scale [21], but it can also be used to classify emotional states based on speech signals [18,22].

As a set of injected low–level features (${\gamma }_2$), we have used a set of statistical features calculated based on the fundamental frequency signal $F_0$, three consecutive formants: $F_1$, $F_2$, $F_3$ and derivatives of these values, resulting in 99 features in total. Our motivation for selecting the fundamental frequency and its harmonics is based on numerous studies [16,23,24,25], that confirm the crucial role of $F_0$ in emotion recognition from speech signals. The fundamental frequency and its harmonics are strongly linked to speech intonation, which is an important element in expressing emotions. Moreover, in our work, we also considered previous studies focused on the interpretation of convolutional networks in the context of emotion recognition [26]. These studies demonstrated a significant relationship between the fundamental frequency and how CNNs learn to recognize emotions in time-frequency representations, further supporting the rationale for choosing this feature in our approach. A general list of the feature space and the number of individual groups is presented in Figure 2. The letters U, V and G represent accordingly: unvoiced part, voiced segment and gradient. The symbols $LR$, $\Delta$, J, and $SH$ represent linear regression, derivative, jitter and shimmer. Whereas the letter S represents a set of statistical descriptors which includes mean, minimum and maximum value, standard deviation, range, and interquartile range. The numbers in the brackets on the bottom represent the number of low–level features in each group.

Additionally, in the Figure 3 examples of individual representations of the speech signal are depicted.

2.3. Deep Neural Network Architectures

Due to the use of time–frequency representation in this work, we decided to use the ResNet [27] convolutional architecture as the base model of the neural network. This architecture is prevalent in image processing, the classification of emotions in speech signal [28,29], and it is also used in connection with time–frequency representations [30]. This architecture was designed to solve the problem of vanishing gradient that appears in models composed of many layers. By using residual connections in residual blocks, from which it is built, it is possible to make deep models and train their layers without significant risk of getting stuck in a local minimum. Moreover, this type of mechanism makes it highly efficient, even when using many layers. The ResNet152 variant was used in this work.

3. Experiments and Results

We have tested the proposed mechanism in various proportions $\beta =K/M$, where K is the number of injected features and M specifies the total number of attributes in the final classification stage. Then, we compared the results with those obtained by base architectures with the same dimensionality of the final feature space but without an injection mechanism. The size of the final feature space M took values from two different sets. For the base models, these were the values 32, 64, 128, 256, 512, 1024 and 2048. These models with an injection mechanism were the same, omitting the first two, 32 and 64. It was necessary due to the number of injected features $K=99$, which would be greater than the target feature space. Additionally, to obtain more reliable results for each variant, we have prepared five models with different starting seeds, then a part of results we have presented as an average. As a result, jointly for both base representations and different values of M, we have prepared 120 models.

We used the following parameters to calculate the CQT representation: number of bins equal to 84, 12 bins per octave and minimum frequency equal to 32.70 Hz. In the case of Mel spectrograms, we have used 128 filters. In training individual models, we used the following parameters: 100 epochs, a learning rate equal to 0.001, an Adam optimizer and cross entropy as a loss function. The data was randomly divided into a train, validation and test subsets as $60\%$, $20\%$ and $20\%$, respectively, for all recordings.

3.1. Datasets

The experiments were conducted on the Berlin Database of Emotional Speech (EmoDB) [31] and RAVDESS dataset [32].

The EmoDB database contains 535 sentences in German language that are recorded as monophonic with a sample rate equal to 16 kHz. The utterances are spoken by ten actors, including five men and five women. These actors speak ten different sentences in the following emotional states: anger (ang), disgust (dis), fear (fea), happiness (hap), sadness (sad), boredom (bor) and neutral (neu). Figure 4 shows distributions of recordings for each emotion. Additionally,

Table 1. Distribution of recordings for individual speakers and the emotional states.

Speaker ID	ang	bor	fea	hap	neu	sad	dis	Sum
03	14	5	4	7	11	7	1	49
08	12	10	6	11	10	9	0	58
09	13	4	1	4	9	4	8	43
10	10	8	8	4	4	3	1	38
11	11	8	10	8	9	7	2	55
12	12	5	6	2	4	4	2	35
13	12	10	7	10	9	5	8	61
14	16	8	12	8	7	10	8	69
15	13	9	8	6	11	4	5	56
16	14	14	7	11	5	9	11	71
Total	127	81	69	71	79	62	46	535

shows the distribution of recordings for individual speakers and emotional states.

The RAVDESS database contains 1440 monophonic sentences in English language with a 48kHz sampling rate. The utterances are spoken by 24 actors, including 12 men and women, two sentences per actor in eight emotional states: neutral, calm (cal), happiness, sadness, anger, fear, disgust, and surprised (sur). Recordings representing particular emotions in the RAVDESS database have two levels of intensity, unlike the EmoDB database. For a more accurate comparison of the results obtained between databases, we used only recordings with the same emotional intensity equal to 2 from the RAVDESS database, except for the neutral state, where all recordings had the same intensity level. The final set contained 96 recordings for every emotional state, giving 768 examples.

3.2. Evaluation Metric

Due to an imbalance problem in the EmoDB database (Table 1 and Figure 4), we adopt two different evaluation metrics to compare speech emotion recognition results: unweighted accuracy (UA) and weighted accuracy (WA) [8,33,34].

In the case of the RAVDESS database, where the distribution of recordings for individual emotions is equal, we presented the results only in weighted accuracy. The weighted accuracy is calculated as:

$$WA={\displaystyle \frac{{\sum }_{c=1}^C{TP}_c}{{\sum }_{c=1}^C({TP}_c+{FN}_c)}},$$

(6)

whereas unweighted accuracy is calculated using the following equation:

$$UA={\displaystyle \frac{1}{C}}\sum _{c=1}^C{\displaystyle \frac{{TP}_c}{{TP}_c+{FN}_c}},$$

(7)

where $TP$ (True Positives) represents the number of samples correctly predicted as positive, $FP$ (False Positives) is the number of samples wrongly predicted as positive, $FN$ (False Negatives) is the number of samples wrongly predicted as negative, and C is the number of emotion classes.

3.3. Results

Figure 5 shows the distribution of classification results for the test set using all prepared models without injection mechanism, divided into the individual sizes of the target feature space M. The graph on the left side (Figure 5a) shows the results in the form of $WA$ obtained by the models trained on the EmoDB database, and the chart on the right side (Figure 5b) presents results for the models trained on the RAVDESS database. On the other hand, Figure 6 also illustrates the distribution of classification results in the same manner but obtained for models with injection mechanisms. Like Figure 5, the left graph (Figure 6a) presents results for the EmoDB database and the right (Figure 6b) for the RAVDESS database. As you can observe, in the general classification for both databases, models trained on CQT representations achieved significantly better classification accuracy for both variants, with and without injection mechanism, than models trained on Mel spectrograms. Additionally, in both cases, EmoDB and RAVDESS, these graphs illustrate the noticeable improvement in classification with the feature injection mechanisms for both CQT and Mel spectrogram. Moreover, in the case of EmoDB and models trained on CQT representation, the injection mechanism significantly reduced the range of results obtained by individual models within a given variant, which is especially visible for $M=128$ and $M=2048$. Furthermore, by comparing the highest accuracy obtained by the baseline models with the best result using the injection mechanism, an improvement of 3.9% for CQT and 6.25% for the Mel spectrogram was achieved for EmoDB. In the case of RAVDESS, it was 1.06% for the Mel spectrogram and 4.12% for CQT representation.

Table 2. Mean values for $UA$, $WA$, and their changes ${WA}_{\Delta }$, ${UA}_{\Delta }$ for each M and $\beta$ values (CQT spectrogram)—EmoDB.

M	$\mathit{WA}$	$\mathit{UA}$	$\mathit{\beta }$	${\mathit{WA}}_{\mathit{I}}$	${\mathit{UA}}_{\mathit{I}}$	${\mathit{WA}}_{\mathbf{\Delta }}$	${\mathit{UA}}_{\mathbf{\Delta }}$
32	74.01	71.61	-	-	-	-	-
64	68.59	66.07	-	-	-	-	-
128	70.62	68.41	0.77	75.31	72.48	4.69	4.07
256	72.03	69.81	0.39	75.16	72.31	3.13	2.5
512	69.84	67.47	0.19	72.97	70.36	3.13	2.89
1024	67.5	63.40	0.10	72.66	69.67	5.16	6.27
2048	65.78	62.03	0.05	74.69	72.98	8.91	10.95

,

Table 3. Mean values for $UA$, $WA$, and their changes ${WA}_{\Delta }$, ${UA}_{\Delta }$ for each M and $\beta$ (Mel spectrogram)—EmoDB.

M	$\mathit{WA}$	$\mathit{UA}$	$\mathit{\beta }$	${\mathit{WA}}_{\mathit{I}}$	${\mathit{UA}}_{\mathit{I}}$	${\mathit{WA}}_{\mathbf{\Delta }}$	${\mathit{UA}}_{\mathbf{\Delta }}$
32	60.78	59.52	-	-	-	-	-
64	62.19	61.95	-	-	-	-	-
128	58.44	56.91	0.77	64.69	62.69	6.24	5.78
256	58.44	56.48	0.39	65.02	62.89	6.58	6.41
512	62.5	61.15	0.19	63.75	62.02	1.24	0.87
1024	58.28	56.73	0.10	66.25	65.48	7.82	8.75
2048	59.22	56.39	0.05	63.28	61.79	4.06	5.4

and

Table 4. Mean values for $WA$, and their changes ${WA}_{\Delta }$, for each M and $\beta$ values (CQT and Mel spectrogram)—RAVDESS.

	CQT Spectrogram				Mel Spectrogram
$\mathit{M}$	$\mathit{WA}$	${\mathit{WA}}_{\mathit{I}}$	${\mathit{WA}}_{\mathbf{\Delta }}$		$\mathit{WA}$	${\mathit{WA}}_{\mathit{I}}$	${\mathit{WA}}_{\mathbf{\Delta }}$	$\mathit{\beta }$
32	66.04	-	-		56.25	-	-	-
64	63.85	-	-		54.58	-	-	-
128	64.06	66.49	2.43		53.96	55.10	1.14	0.77
256	64.79	69.06	4.27		54.27	56.35	2.08	0.39
512	63.44	65.21	1.77		56.77	56.67	−0.1	0.19
1024	62.71	67.19	4.48		53.85	57.40	3.55	0.10
2048	64.90	68.65	3.75		54.37	56.98	2.61	0.05

show averaged results for individual base representations and sizes of the target feature space M, as well as the subsequent values of $\beta$. The results are presented as $WA$ for the base models and ${WA}_I$ for the injection mechanism models. In the case of the EmoDB database, the results are additionally presented in the form of $UA$ for base models and ${UA}_I$ for models with injection mechanisms. Additionally, for each measure, the value (${WA}_{\Delta }$, ${UA}_{\Delta }$) is shown by what improvement occurred after applying the feature injection mechanism for individual sizes M. In the case of EmoDB, the highest average scores, 74.01% (WA) for the base models, were obtained for CQT representation and $M=32$, while for the variant using the injection mechanism, was equal to 75.31% (WA) for $M=128$, which gives an improvement of 1.3% (WA). For the same value $M=128$, the improvement was 4.69% (WA).

The highest average improvement was observed for $M=2048$, equal to 8.91% (WA), $\beta$ was 0.05, so the injected features represented only 5% of the final feature space. The lowest improvement of 2.5% (WA) was obtained for $M=256$, where the injected features constituted about 40% of the feature space. On the other hand, for the Mel spectrogram, the highest average accuracy was 62.5% (WA), for the models base values were obtained for $M=512$, and the variant with the injection mechanism was equal to 66.25% (WA) for $M=1024$, which gives an improvement of 3.75% (WA). However, for $M=1024$, for which the overall highest result was obtained for the Mel spectrogram when injecting features, the average improvement was 8.75% (WA), where the injected features constituted 10% of the total. The lowest average difference was observed for $M=512$, which was 0.87% (WA) for $\beta =0.19$.

For the RAVDES database, the highest average accuracy, 66.04% (WA), for the base models, was also achieved for CQT representation and $M=32$. The variant with injection mechanism was 69.06% (WA) for $M=256$, which gives an improvement equal to 3.02% (WA). For the same value $M=256$, the improvement was 4.27% (WA). The best average improvement, 4.48% (WA), was achieved for $M-1024$ where $\beta$ was equal to 0.10. On the other hand, the lowest improvement 1.77% was observed for $M=512$ and $\beta =0.19$. In the case of base models trained on Mel spectrograms, the highest average result was 56.77% (WA), which, compared to the best score of 57.4% (WA) for the variant with injection mechanism, gives improvements equal to 0.63% (WA).

The graphs in Figure 7 show the relationship of classification quality ($WA$) with the size of the target feature space. Figure 7a and Figure 7c present information for models without injection mechanism for EmoDB and RAVDESS databases, respectively. Figure 7b,d illustrate results for models with injection mechanism also for EmoDB and RAVDESS. Moreover, comparing graphs contained results for EmoDB, the use of the injection mechanism not only had a positive impact on the average classification result but also reduced the average spread of results obtained for different values of M parameter. Additionally, the Figure 7b shows the $\beta$ coefficient changes depending on the M value.

Moreover, changes in the $\beta$ coefficient do not significantly impact the average level of classification, which remains at a more or less similar level throughout the entire range of $\beta$ changes. Such a situation and the average improvement in results after using the injection mechanism may prove the importance of the injected features, which partially reduces the complexity of the neural network architecture while maintaining its level of prediction.

Figure 8 and Figure 9 show the confusion matrices of models trained on EmoDB, without and with the mechanism of injection that received the highest scores for CQT and Mel spectrograms, respectively. Similarly, Figure 10 and Figure 11 lustrate confusion matrices for RAVDESS database.

Table 5. Differences between classification accuracy $WA$ of individual emotions with and without injection mechanisms for the best models trained on the EmoDB database.

	neu	ang	fea	hap	sad	dis	bor
Mel	21	0	38.9	−16.6	0	0	0
CQT	−5.3	6.7	16.7	0	0	30	−11.1

and

Table 6. Differences between classification accuracy $WA$ of individual emotions with and without injection mechanisms for the best models trained on the RAVDESS database.

	neu	ang	fea	hap	sad	dis	cal	sur
Mel	0	0	4.2	−8.3	0	12.5	−4.2	4.2
CQT	−12.5	−8.3	8.4	16.7	−4.1	16.7	12.5	−4.1

present a classification accuracy improvement in individual emotional states for the models depicted on confusion matrices. In the case of EmoDB, injection of additional features ($\beta =0.05$) for CQT representation significantly improved classification accuracy for dis emotional state, 30% (WA). A slight improvement was also observed for the fea and ang states, which were equal to 16.7% (WA) and 6.7% (WA), respectively. The classification accuracy of emotions such as neu and bor was decreased. However, the situation for hap and sad did not change. A different result occurred in the case of the Mel spectrogram ($\beta =0.1$), where only hap got worse by 16.6% (WA). However, significant improvement was observed for neu equal to 21% (WA) and for fea equal to 38.9% (WA). No changes for ang, sad, dis and bor. A different situation can be observed in the results for the RAVDESS database. The most significant improvement occurred for dis—12.5% (WA) in the case of Mel spectrogram and for hap, dis equal to 16.7% (WA) in the case of CQT representation. On the other hand, the accuracy of classification was decreased in the model trained on Mel spectrograms for hap and cal. For the CQT representations, the classification of neu, ang, sad, and sur has deteriorated. In the case of EmoDB, for both representations the classification was noticeably improved fea emotional state. On the other hand, in both cases, accuracy for sad remained unchanged. For RAVDESS, the highest, positive change was occurred in dis for both representations. In both case, EmoDB and RAVDESS, for Mel spectrogram, accuracy for ang remained unchanged but accuracy for hap was decreased. For the models trained on CQT representation, for both databases, classification of neu was also decreased.

Moreover, Figure 12 and Figure 13 shows the classification distribution of individual emotional states of the models with the size of the final space feature M, for which the highest average improvement was observed when using the injection mechanism. In the case of EmoDB, for CQT representation and Mel spectrogram, it was $M=2048$ and $M=1024$, respectively. For RAVDESS, it was $M=1024$ for both representations. The ranges of some results have changed significantly in reduction. For CQT, these were hap and fea, and neu for Mel spectrogram. In both cases, the range of results for the bor state has changed extension.

Additionally, in

Table 7. Average accuracy of all models without $WA$ and with ${WA}_I$ injection mechanism and their changes ${WA}_{\Delta }$ for individual emotional states—EmoDB.

	CQT Spectrogram				Mel Spectrogram
	$\mathit{WA}$	${\mathit{WA}}_{\mathit{I}}$	${\mathit{WA}}_{\mathbf{\Delta }}$		$\mathit{WA}$	${\mathit{WA}}_{\mathit{I}}$	${\mathit{WA}}_{\mathbf{\Delta }}$
neu	64.0	70.5	6.5		48.6	57.2	8.7
ang	91.2	92.4	1.2		78.3	81.7	3.5
fea	47.1	49.3	2.2		35.4	41.3	5.9
hap	49.9	64.5	14.6		41.3	46.9	5.6
sad	88.6	90.7	2.1		88.8	87.7	−1.0
dis	50.0	54.0	4.0		51.1	55.2	4.1
bor	77.9	79.6	1.6		65.7	70.7	5.0

and

Table 8. Average accuracy of all models without $WA$ and with ${WA}_I$ injection mechanism and their changes ${WA}_{\Delta }$ for individual emotional states—RAVDESS.

	CQT Spectrogram				Mel Spectrogram
	$\mathit{WA}$	${\mathit{WA}}_{\mathit{I}}$	${\mathit{WA}}_{\mathbf{\Delta }}$		$\mathit{WA}$	${\mathit{WA}}_{\mathit{I}}$	${\mathit{WA}}_{\mathbf{\Delta }}$
neu	76.8	77.3	0.5		77.0	81.8	4.8
ang	77.3	78.7	1.4		62.1	65.2	3.0
fea	61.8	63.8	2.1		46.7	46.8	0.2
hap	37.9	45.3	7.5		25.2	26.3	1.1
sad	49.8	56.8	7.1		40.6	46.8	6.2
dis	57.7	57.7	−0.1		41.4	43.2	1.8
cal	71.7	75.2	3.5		76.1	69.5	−6.6
sur	81.2	83.7	2.5		69.8	72.3	2.6

average values from all models, without injection mechanism ($WA$) and with injection mechanism $\left({WA}_I\right)$, are presented for individual emotional states. Furthermore, for every emotion, the change in the classification accuracy after using the ${WA}_{\Delta }$ injection mechanism was shown. Table 7 contains results for the EmoDB database. How can be noticed, in the case of CQT representation, the highest average improvement was recorded for emotions hap—14.6% and neu—6.5%. Classification of other emotional states also improved slightly in the range from 1.2% to 4%. In turn, for Mel spectrograms, the classification of neu emotional state improved significantly 8.7% compared to other emotions, for which there was also an improvement, but to a lesser extent range 4.1–5.9%. The exception was sad emotion, where the injection mechanism worsened the average classification level by 1%. The situation is different in the case of the RAVDESS database, which is shown in Table 8. The results for the CQT representation show significant improvement for emotions like hap—7.5% and sad—7.1%. However, a minimal deterioration of 0.1% dis classification accuracy has been observed. For the remaining emotional states, a slight improvement occurred from 0.5% to 3.5%. For models trained on Mel spectrograms, significant deterioration was observed for cal—6.6%, the highest improvement occurred for sad emotional state equal to 6.2%. For the rest of the emotions, accuracy slightly improved from 0.2% to 4.8%.

4. Conclusions

We have proposed a simple approach to fuse the embedded space of convolutional neural networks and low–level features to obtain feature space with higher discriminative power for emotion recognition in speech signals. The experiments showed that source representation in CQT spectrograms achieves better classification results than popular Mel spectrograms for emotional speech. The results show that the models that used additional information from low–level features achieved better performance in the overall classification. The average improvement for the models trained on CQT representations was equal to 1.29% and 3.75% (weighted accuracy) for models trained on Mel spectrograms in the case of the EmoDB dataset. However, for RAVDESS, it was 3.02% and 0.63% for CQT representation and Mel spectrogram, respectively. Furthermore, for variants with the highest change in classification score, the results demonstrate that the average classification performance was improved for almost all emotions compared to variants without an injection mechanism for both databases. In the case of EmoDB, for best models where Mel spectrograms were base representations, the accuracy of all emotions was improved. The highest improvement, around 14%, was obtained for sad and dis states, and the lowest, around 5%, for ang, hap, and bor emotional states. Using models trained on CQT representations, the most significant improvement in classification occurs for hap and dis (more than 20%) emotional states. On the other hand, the classification accuracy of anger and boredom decreases states by around 2%. For models that achieved the best accuracy on the RAVDESS database, the classification of neu was improved by around 16% using the Mel spectrogram, and in the case of CQT representations, the classification of emotional states like sur and fea was increased by around 9%. In most cases, the resulting average classification accuracy was improved for individual emotional states. The most positive impact after applying the injection mechanism was observed for hap—14.6% and neu—8.7% emotions for the EmoDB database. However, for the RAVDESS dataset, the most significant improvement equal to 7.5% and 7.1% occurred for hap and sad emotional states. Additionally, the classification of sad emotion for EmoDB combined with Mel spectrogram has deteriorated by 1%. A similar situation occurred in the case of the RAVDESS database, where the Mel spectrogram had the highest average accuracy decline by 6.6%, but this time for cal emotional state.

In future work, we plan to use other low–level features, including other attributes that combine the properties of a signal in different domains Additionally, we plan to apply our method to recurrent neural networks and transformer architectures.