High-Accuracy Classification of Multiple Distinct Human Emotions Using EEG Differential Entropy Features and ResNet18

Abstract

The high-accuracy detection of multiple distinct human emotions is crucial for advancing affective computing, mental health diagnostics, and human–computer interaction. The integration of deep learning networks with entropy measures holds significant potential in neuroscience and medicine, especially for analyzing EEG-based emotion states. This study proposes a method combining ResNet18 with differential entropy to identify five types of human emotions (happiness, sadness, fear, disgust, and neutral) from EEG signals. Our approach first calculates the differential entropy of EEG signals to capture the complexity and variability of the emotional states. Then, the ResNet18 network is employed to learn feature representations from the differential entropy measures, which effectively captures the intricate spatiotemporal dynamics inherent in emotional EEG patterns using residual connections. To validate the efficacy of our method, we conducted experiments on the SEED-V dataset, achieving an average accuracy of 95.61%. Our findings demonstrate that the combination of ResNet18 with differential entropy is highly effective in classifying multiple distinct human emotions from EEG signals. This method shows robust generalization and broad applicability, indicating its potential for extension to various pattern recognition tasks across different domains.

1. Introduction

Recognizing human emotional states through EEG signals is of great significance, and has wide applications in neuroscience, psychology, and brain–computer interface fields [1]. These signals contribute to a deeper comprehension of emotional mechanisms and cognitive processes within the brain [2]. By analyzing EEG signal changes, researchers can uncover patterns and characteristics of brain activity associated with different emotional states, thereby advancing our knowledge of emotional regulation, disorders, and related phenomena. The essence of emotional EEG patterns lies in elucidating the neural underpinnings of human emotions and cognitive functions, alongside their extensive utility in diverse fields like medicine, psychology, and engineering, providing valuable insights and support for enhancing human lives and well-being [3].

Deep learning models play a pivotal role in learning sophisticated feature representations from EEG signals, thereby enhancing performance in tasks such as emotion recognition and brain–computer interfaces [4]. These models autonomously learn abstract features embedded within EEG signals, effectively capturing crucial information. Given the complex nonlinear nature of EEG signals, traditional emotion recognition methods relying on manual feature extraction and classification may struggle to capture intricate patterns and details [5]. Deep learning models, on the other hand, autonomously extract features from raw data and represent them at higher abstraction levels, thereby boosting accuracy and efficiency in emotion recognition tasks [6].

Moreover, deep learning models can optimize model structures and algorithms, leading to higher running speeds and real-time performance, thus making emotion recognition systems more practical for real-world applications [7]. For instance, the integration of deep learning with EEG technology represents a promising avenue for advancing emotion recognition capabilities and improving their feasibility in practical settings. For example, Chen et al. introduced a Graph Convolutional Network (GCN) based on Ordinary Differential Equations (ODE), which significantly enhances performance in handling the relationships among EEG channels [8]. In another instance, Xu et al. presented a method leveraging spiking neural networks to achieve high accuracy in recognizing emotions within EEG datasets [9].

EEG signals are known for their inherent complexity and nonlinearity, making them challenging to analyze directly. Entropy serves as a valuable metric to quantify this complexity, offering a quantitative measure of the irregularity within the signal [10]. A higher entropy value signifies increased unpredictability and complexity in the signal, while a lower entropy value indicates a more structured and ordered signal. Leveraging entropy as a feature extraction tool facilitates the conversion of raw EEG signals into numerical representations, thereby enabling various applications such as emotion classification and cognitive state recognition [11]. For instance, Lu et al. proposed a novel EEG-based emotion recognition approach that utilizes dynamic entropy measures, demonstrating robust generality in emotion recognition tasks [12]. Additionally, in Patel’s research, a novel classification method was investigated, leveraging Tsallis entropy in conjunction with the K-Nearest Neighbors (KNN) classifier [10].

Examining entropy provides insights into the dynamic characteristics of the nervous system. Differential entropy (DE) offers a means to capture the temporal variations present in signals. Given that EEG signals exhibit dynamic changes over time, employing DE proves particularly advantageous, as it offers a more precise depiction of this evolving process. Moreover, DE surpasses standard entropy by accounting for signal variations over time, thereby enhancing feature discrimination and proving especially beneficial for complex emotional and cognitive state recognition tasks [13].

In our study, we propose a novel approach to train and classify EEG emotion states by integrating ResNet18 network architecture with DE. ResNet18 is adept at capturing both temporal and spatial characteristics of signals, with its cross-layer connections designed to capture spatiotemporal features across different levels. By incorporating DE alongside ResNet18, we aim to better model the temporal dynamics inherent in EEG signals, thereby enhancing the network’s ability to perceive temporal features within the signals. This synergistic approach allows for a more comprehensive understanding of signal dynamics, ultimately improving the network’s performance in analyzing EEG data for emotion classification and cognitive state recognition. Combining ResNet18 with DE for training and classifying EEG signals enables us to harness the strengths of deep learning and information theory synergistically.

Our approach merges the ResNet18 network and the DE measures to enhance the classification performance of human emotion states significantly. In essence, our contributions are as follows:

(1)

A framework for classifying multiple distinct human emotions from EEG signals is proposed, capable of identifying five distinct human emotions (happiness, sadness, fear, disgust, and neutrality).

(2)

A high-accuracy classification performance for the different emotional states of happiness, sadness, fear, disgust, and neutrality is achieved, by exploiting EEG differential entropy measures and ResNet18.

The remaining sections of our paper are structured as follows. The second part details the method of the DE combing with ResNet18 network. The third part presents the outcomes of data analysis and EEG-based emotion classification experiments. The fourth part provides a comparative analysis of our method with existing related research, showcasing experimental results. The final part concludes our findings and contributions.

2. Method

2.1. Proposed Framework for Classifying Multiple Distinct Human Emotions from EEG Signals

As depicted in Figure 1, the experimental subjects watched emotion-inducing videos corresponding to different emotions, allowing them to reach corresponding emotional states and collect corresponding 62 channel EEG signals in real time. The collected EEG signals undergo initial preprocessing steps, including downsampling, bandpass filtering, and rectifying artifacts. Subsequently, DE is extracted from the preprocessed signals. Subsequently, this extracted measure of DE serves as feature input to the ResNet18 network, where the classification process is trained and executed to obtain the final five types of human emotions (happiness, sadness, fear, disgust, and neutral).

2.2. Experimental Data

The SEED-V database delved into an examination of five distinct emotions—happiness, sadness, fear, disgust, and neutral. A curated selection of 15 film clips, with 3 clips representing each emotion, served as stimuli. As shown in Figure 2, five specific emotional stimulation videos correspond to five different emotions, namely fear, happiness, disgust, sadness, and neutral. Each of the 16 subjects, comprising 6 males and 10 females, partook in the experiment 3 times, with each session spanning 50 min. The ages of the 16 subjects were 21, 21, 23, 21, 20, 23, 23, 21, 23, 24, 24, 20, 22, 22, 19, and 19, respectively. The average age is 21.63, with a standard deviation of 1.63. Throughout the experiment, a meticulously designed protocol was adhered to, as follows: a 15 s initial pause preceded each stimulus, followed by 2 to 4 min of stimulus exposure, with intervals lasting either 15 or 30 s, depending on the emotion portrayed [14].

EEG signals were captured using a sophisticated 62-channel EEG device with a sampling frequency set at 1000 Hz, later downsampled to a 200 Hz rate to facilitate processing. Preprocessing of EEG data involved the application of a band-pass filter spanning 1 to 75 Hz, aimed at eliminating noise and artifacts. Subsequent to the experiments, participants engaged in a self-assessment segment, where they were prompted to rate the impact of the stimuli on a scale ranging from 0 to 5. A score of 5 indicated the most favorable response, while 0 denoted the least favorable.

2.3. Feature Extraction of Differential Entropy Measures

Given the non-Gaussian distribution of EEG signals, utilizing DE emerges as a potent technique for extracting features from such signals. This method aptly captures the complexity and uncertainty inherent in EEG signals during feature extraction. Lower entropy values signify a higher degree of signal orderliness and predictability, while higher entropy values indicate greater signal complexity and unpredictability.

As a method for feature extraction, entropy measurement offers a means to characterize the diversity and intricacy of information embedded within EEG signals, facilitating the extraction of pertinent features crucial for subsequent analysis and classification. Specifically, the employment of DE enables us to quantify the complexity and stochasticity inherent in EEG signals. Through the computation of DE, we gain insights into the temporal or spatial uncertainty present in the signals, alongside a quantitative assessment of the informational content they carry. Moreover, by examining the dynamic attributes of the signal, we can derive valuable features conducive to further analysis and classification endeavors.

As illustrated in Figure 3, the initial EEG signal collected comprises data from 62 channels, which underwent preprocessing. Initially, the EEG data were downsampled to 200 Hz and then subjected to a bandpass filter ranging from 0 to 75 Hz to eliminate any signal noise. We identified and rectified artifacts in the EEG signals stemming from electrode displacement and external interference.

After preprocessing the EEG signals from 62 channels, we utilized a 256-point Short-Time Fourier Transform technique along with a 1 s non-overlapping Hanning window to extract five distinct frequency bands from the following preprocessed EEG signals: delta (1–4 Hz), theta (4–8 Hz), alpha (8–14 Hz), beta (14–31 Hz), and gamma (31–50 Hz). From each frequency band, the DE feature is computed, culminating in a total of 310 DE features (62 channels × 5 frequency bands). The calculation formula for DE is outlined as follows:

$$DE=-{\displaystyle {\int }_{\infty }^{\infty }P(x)\ln (P(x))dx}$$

(1)

We assume that the EEG signals follow a Gaussian distribution: $x~N(\mu ,{\sigma }^2)$. Consequently, the calculation of the DE features can be simplified:

$$DE=-{\displaystyle {\int }_{\infty }^{\infty }\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \frac{{(x-\mu )}^2}{2{\sigma }^2}}\log (\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \frac{{(x-\mu )}^2}{2{\sigma }^2})dx=\frac{1}{2}\log (2\pi e{\sigma }^2)$$

(2)

2.4. Pattern Classifier Based on ResNet18

To establish an end-to-end model for deep feature extraction and classification of EEG data, we employed the ResNet18 network to process the extracted features of DE. Initially, the DE feature, sized at 310, undergoes reshaping to (62,5), integrating data from 62 channels scanned using ESI. As illustrated in Figure 4a, within the overall model architecture, the input size is structured as 5 features with 62 channels, wherein conv represents the convolutional layer, Max pool signifies the maximum pooling layer, AVG pool denotes the average pooling layer, and FC denotes the fully connected layer. The multi-layer convolutional structure amalgamates DE representations from diverse frequency bands and channels to learn multi-level features in EEG data. Concurrently, residual connections facilitate the direct propagation of shallow information into deeper networks, thereby retaining crucial EEG data. Subsequently, the deeply extracted DE features are employed for the classification of 5 distinct emotions through a fully connected layer.

Table 1. The details of structure of ResNet18-based emotion recognition method.

Module Level	Module/Layer Type	Configurations
0	Input features	Channel = 62 Frequency bands = 5 Feature size = 62 × 5
1	Input conv layer	In_channel = 62 Out_channel = 64 Kernel = 7
2	Maxpool	Kernel = 3
3	Conv layers #1	In_channel = 64 Out_channel = 64 Kernel = 3
4	Conv layers #2	In_channel = 64 Out_channel = 128 Kernel = 3
5	Conv layers #3	In_channel = 128 Out_channel = 256 Kernel = 3
6	Conv layers #4	In_channelg = 256 Out_channel = 512 Kernel = 3
7	Avgpool	output_size = 1
8	FC	In_feature = 512 Out_feature = 5

shows the details of the emotion classification method based on the ResNet18. The model computes the output probability for each emotion type post multi-layer convolution. Throughout the network’s training process, cross-entropy loss serves as the optimization criterion, facilitating the refinement of weights and other parameters in the model.

The primary network utilized in this study comprises multiple analogous residual blocks linked sequentially, as illustrated in Figure 4b. Each residual block encompasses two paths: F(x), and x. The F(x) path, serving to adjust residuals, is termed the residual path, while the x path represents an identity mapping, referred to as the shortcut path. The residual path comprises two convolutional layers with a kernel size of 3, followed by a ReLU activation layer. The shortcut path can be categorized into two scenarios: one involves directly outputting the input x, while the other necessitates a 1 × 1 convolution for upsampling or downsampling, ensuring consistency in the shape of the shortcut output with that of the F(x) path output. Upon combining the residual and shortcut paths, they traverse through the ReLU activation layer and function as input for the subsequent residual block.

In this study, we employed the Adam (Adaptive Moment Estimation) optimizer to update the parameters of the ResNet network. In our experiment, we configured the parameters of the Adam optimizer as follows: ${\beta }_1=0.9$, ${\beta }_2=0.999$, learning rate = 1 × 10⁻⁴, weight decay = 1 × 10⁻². The Adam optimizer [15] is an optimization algorithm that amalgamates momentum and adaptive learning rate techniques, facilitating the discovery of superior solutions within the parameter space. Its update rules involve computing the moving average of the first and second moments of the gradient, thereby correcting their deviations to enable adaptive parameter updates.

3. Results

In our experiment, we commenced by computing the confusion matrix using EEG data from 16 participants in the SEED-V dataset, thereby elucidating the precise classification scenario. Subsequently, our investigation encompassed the classification of five classes of human emotions (happiness, sadness, fear, disgust, and neutral) based on EEG signals from the aforementioned 16 individuals, followed by an in-depth analysis of the classification outcomes. To visually represent the classification of EEG emotion states, we undertook a two-dimensional mapping of the classification results and portrayed them via graphical imagery. Additionally, we computed and showcased ROC curves for the five categories of emotional states, thereby facilitating a comprehensive assessment of the efficacy of the proposed network classification model.

We utilized the proposed network to classify the five EEG emotional states of 16 participants in the dataset and conducted a comparative analysis of the results. Figure 5 illustrates the classification results for the emotional states of the 16 subjects, with accuracy values spanning from 87.64% to 99.21%. Notably, the 5th individual attained the lowest classification accuracy, while the remaining 15 individuals achieved accuracies exceeding 90%, with 11 individuals surpassing the 95% mark.

In addition to demonstrating recognition accuracy, we present the confusion matrix of the proposed method applied to the SEED-V dataset in Figure 6. This matrix provides insights into the recognition accuracy of the ResNet18 network across different emotional states. Within the confusion matrix, varying color depths indicate different values, transitioning from blue to red with increasing magnitude. The deeper the red color, the higher the corresponding value.

As illustrated in Figure 6, the ResNet18 network achieves recognition accuracies of 86.16%, 90.19%, 94.61%, 96.78%, and 96.13% for the following five emotional states: fear, sadness, neutral, happiness, and disgust, respectively. It is evident that the network exhibits the highest accuracy in recognizing a happy emotional state, while its performance is comparatively lower for fearful emotional states. The accuracy rate for detecting fear emotions is 86.16%, with misclassifications as follows: 3.46% as sad, 1.73% as neutral, 1.8% as happy, and 6.85% as doubt. Notably, fear often tends to be misclassified as doubt due to the similarity in their EEG signals, posing a challenge for accurate differentiation. Moreover, the graphical representation reveals that 2.5% and 0.72% of happy emotional states are misclassified as neutral and disgust emotional states, respectively.

To further underscore the efficacy of the ResNet18 network in EEG emotion classification, we employed a two-dimensional visualization of the classification outcomes. The 512 dimensional features output by the average pooling layer of ResNet18 are mapped to a two-dimensional image using the t-SNE algorithm [14]. Different colors represent different emotions, and different emotions are completely separated without overlapping areas, indicating excellent classification results for the five emotions. Illustrated in Figure 7a–d, we depict the classification results for subjects 2, 5, 6, and 9 utilizing the ResNet18 network across five distinct EEG emotional states.

Initially, emotions in EEG recordings appear intertwined and challenging to discern. However, post-training and classification using the ResNet18 network, Figure 6 reveals a distinct clustering of emotions of the same type. This visual representation demonstrates that EEG signals corresponding to different emotions are more visually distinguishable, thereby corroborating the effectiveness of the ResNet18 network model in EEG emotion classification.

To assess the ResNet18 network’s capability in deciphering EEG emotion states, we conducted an analysis utilizing the ROC curve and AUC (Area Under the ROC Curve) for Subject 4. Each of the five emotion categories was treated as a positive example, while all other categories were deemed negative examples. By calculating the True Positive Rate (TPR) and False Positive Rate (FPR) for each category and plotting the curve, we obtained a comprehensive evaluation model, showcasing its performance across diverse categories.

The multi-class ROC curve provides a holistic overview of performance indicators across all categories through the following two perspectives: micro average, and macro average. The micro average ROC curve computes the overall TPR and FPR by aggregating the number of true and false positive cases across all categories, thus treating each category with equal importance and consolidating their performance into a singular metric. Conversely, the macro average ROC curve evaluates the model’s performance by calculating the TPR and FPR for each category separately and then averaging the values, assigning equal weight to each category.

The significance of the multi-class ROC curve lies in its ability to offer a comprehensive evaluation perspective on the performance of multi-class classification models. By scrutinizing the TPR and FPR of different categories, one can gauge the model’s recognition ability across each category. Both micro average and macro average ROC curves facilitate the comparison of overall performance among different models in multi-class classification tasks.

As illustrated in Figure 8, we chose Subject 4 as the focal point of our experimentation. Across five distinct emotions—disgust, fear, sadness, neutrality, and happiness—the corresponding AUC values were 0.9871, 0.9886, 0.8976, 0.9330, and 0.9718, respectively. A higher AUC value indicates stronger discrimination capability within the network for different emotions.

4. Discussion

In our research, we integrated the ResNet18 network with DE to classify EEG-based human emotion states. DE serves as a valuable tool for quantifying the uncertainty and information content within EEG signals, aiding in the discernment of distinct signal types. ResNet18, renowned for its deep convolutional architecture, boasts formidable feature extraction capabilities. When EEG signals are fed into the ResNet18 network, it adeptly extracts high-level abstract feature representations, enabling the acquisition of intricate insights into the brain activity’s spatiotemporal dynamics. Our findings from emotion recognition using EEG signals underscore the synergistic benefits of this combination, as it enhances both the accuracy and robustness of signal classification through sophisticated EEG signal-processing techniques.

To demonstrate the superior performance of the ResNet18 network in conjunction with the DE method for distinguishing emotion states, we conducted a comparative analysis with recent studies utilizing the same SEED-V dataset. Our evaluation encompassed studies by Zhu, Zhou, Jin, Aslan, Kamble, and Conejo.

In Zhu’s research, an emotion recognition method termed Joint Distributed Instances Represent Transfer (JD-IRT) was employed. This method integrates two core components: Joint Distribution Deep Adaptation (JDDA), and Instance-Representation Transfer (I-RT). Zhou’s study utilized a progressive graph convolution network (PGCN) to learn discriminative EEG features. Jin proposed a Graph Adaptive Semi-supervised Discriminative Subspace Learning (GASDSL) model for EEG-based emotion recognition. Aslan’s study utilized Discrete Wavelet Transform (DWT) and Statistical Moments (SM) methods to extract features from EEG signals, and applied Convolutional Neural Networks (CNN) for classification. Liu’s study employed the Deep Canonical Correlation Analysis (DCCA) method for emotion recognition from EEG signals. Conejo’s study proposed two dimension reduction algorithms based on Local Binary Pattern (LBP) derived from machine learning models, which included electroencephalogram channel selection and conflict learning.

Comparing the results of EEG-based emotion classification within the SEED-V dataset, as depicted in

Table 2. The results of classification accuracy from Zhu’s, Zhou’s, Jin’s, Aslan’s, Liu’s, Conejo’s, and our studies.

Title	Dataset	Methodology	Mean Accuracy	STD	Year
Zhu’s study [16]	SEED-V	JD-IRT	65.91%	12.31%	2023
Zhou’s study [17]	SEED-V	PGCN	71.40%	9.43%	2023
Jin’s study [18]	SEED-V	GASDSL	82.03%	-	2023
Aslan’s study [3]	SEED-V	DWT and SM	82.06%	-	2023
Liu’s study [14]	SEED-V	DCCA	85.30%	5.6%	2022
Conejo’s study [19]	SEED-V	LBP	93.85%	5.64%	2023
Our work	SEED-V	ResNet18-DE	95.61%	4.02%	2024

, the accuracies of EEG emotion classification for Zhu’s, Zhou’s, Jin’s, Aslan’s, Liu’s, and Conejo’s studies were 65.91%, 71.40%, 82.03%, 82.06%, 85.30%, and 93.85%, respectively. In contrast, our proposed method, which combines the ResNet18 network with DE, achieves an accuracy of 95.61% in classifying EEG-based emotion states, significantly enhancing the resolution performance. Through this comparison of classification accuracy, it becomes evident that our method retains more pertinent information while eliminating redundant data, thus leading to a more precise and efficient EEG emotion classification outcome.

However, our research still faces several limitations. Schubling et al. [20] segmented the beta band into low and high beta bands and explored their relationship with emotional processing. Wijk et al. [21] suggested that different frequency ranges within the beta band significantly influence motor control. Borzelli et al. [22] identified increased synchronization among motor units involved in stiffness modulation, supported by cross-muscle coherence analysis, revealing peaks in the β-band, typically linked with cortical activity. These studies are about the impact of frequency band separation on brain activity, and we have not yet considered this factor in our experiments. We plan to incorporate frequency band separation considerations in future experiments. Additionally, Duclos et al. [23] proposed that posture affects emotional experiences similarly to facial expressions, while Carney et al. [24] suggested that body posture can induce physiological and behavioral changes, eliciting various emotions. Buisine et al. [25] explored emotional expression detection through different body positions, emphasizing the close connection between the motor system and emotions. Both facial expressions and body posture significantly influence emotional recognition. Our study utilized the SEED-V open-source dataset without considering facial expressions and body posture. Future experiments will investigate the interplay between emotion and motor control and incorporate these factors. Furthermore, Leonardi et al. [26] proposed brain oscillatory rhythms as a potential measure for stroke recovery, indicating their consistent patterns as a promising tool for predicting rehabilitation outcomes. However, our study did not include patients with neurological conditions, limiting its scope. Future research will involve neurological patients to explore EEG-based emotion recognition.

5. Conclusions

In our study, we propose a novel approach that combines the ResNet18 network with the DE method to classify human emotion states effectively from EEG signals. The DE method, serving as an information measurement technique, offers the ability to precisely quantify the uncertainty and information content inherent in EEG signals. Utilized as input to the network, DE enriches the model with a more comprehensive dataset, thereby enhancing its capacity to understand and interpret EEG signals. Leveraging the ResNet18 network, renowned for its deep convolutional architecture and multi-level feature extraction capabilities, further amplifies our approach’s analytical prowess.

By integrating DE with the ResNet18 network, we are able to unveil the spatiotemporal dynamics within EEG signals, extracting intricate feature representations across various network levels. This amalgamation enables us to more effectively capture the nuanced complexity and abstraction present in EEG data, thereby elevating the accuracy and robustness of our classification system. Our experimental findings unequivocally demonstrate the superior performance achieved with the ResNet18 network in tandem with DE, showcasing its remarkable classification proficiency in human emotional states.

Moreover, our method underwent rigorous comparison against existing approaches using identical datasets, affirming its superior efficacy in emotion recognition from EEG signals. This substantiates the validity and reliability of our proposed methodology. Ultimately, our approach not only represents a highly effective means of classifying EEG-based emotion states, but also holds promising implications for advancing research within the broader domain of EEG pattern analysis.