A Comprehensive Interaction in Multiscale Multichannel EEG Signals for Emotion Recognition

Abstract

Electroencephalogram (EEG) is the most preferred and credible source for emotion recognition, where long-short range features and a multichannel relationship are crucial for performance because numerous physiological components function at various time scales and on different channels. We propose a cascade scale-aware adaptive graph convolutional network and cross-EEG transformer (SAG-CET) to explore the comprehensive interaction between multiscale and multichannel EEG signals with two novel ideas. First, to model the relationship of multichannel EEG signals and enhance signal representation ability, the multiscale EEG signals are fed into a scale-aware adaptive graph convolutional network (SAG) before the CET model. Second, the cross-EEG transformer (CET), is used to explicitly capture multiscale features as well as their correlations. The CET consists of two self-attention encoders for gathering features from long-short time series and a cross-attention module to integrate multiscale class tokens. Our experiments show that CET significantly outperforms a vanilla unitary transformer, and the SAG module brings visible gains. Our methods also outperform state-of-the-art methods in subject-dependent tasks with 98.89%/98.92% in accuracy for valence/arousal on DEAP and 99.08%/99.21% on DREAMER.

1. Introduction

Affective computing is an umbrella term for human emotion, sentiment, and emotion recognition. As emotion affects human daily behaviors and cognitive activities, emotion recognition plays a crucial role in research fields such as artificial intelligence, medical and health [1,2,3,4,5,6], and brain–computer interfaces (BCI). Generally, both nonphysiological and physiological signals can be used to recognize human emotion [7]. The study conducted by Cacioppo et al. [8] indicates that changes in emotions can lead to physiological variations, including facial muscle movements, brain activity (EEG), and autonomic nervous system (ANS) activity. In brain activity, highly excited or anxious emotional states result in higher-frequency, more intense, and more irregular electrical activity in the brain. Conversely, relaxed and calm emotional states may manifest as lower-frequency, weaker, and more regular electrical activity. Compared to nonphysiological signals such as facial expression, body behavior, speech signals, and textual information, physiological signals have the advantage of being difficult to fake [9], making them more credible for emotion recognition. Among physiological signals, the EEG signal directly reflects brain activity during an emotional response, which is proved to be more effective than other noninvasive physiological signals, such as electrocardiogram (ECG), electrooculogram (EOG), galvanic skin response (GSR), electromyogram (EMG), humidity, and temperature.

In the early years, most of the works decoupled the EEG-based emotion recognition as a feature extraction stage and a classification stage. Feature extraction is the vital stage of emotion recognition, which can be conducted on both the temporal domain and the spectral domain. Temporal domain features mainly reflect the temporal information of EEG signals, where typical ones include sample entropy [10] and fractal dimension feature [11]. Spectral domain features aim to capture emotion information in different frequency bands, where commonly used ones include differential entropy (DE) [12], power spectral density (PSD) [13], etc. These features mainly rely on researchers’ careful design with professional knowledge, which can be biased and limited in representative ability.

Recently, with the great success of deep learning methods in most of the research fields [14,15,16], deep learning-based EEG emotion recognition methods, which can unify the feature extraction and classification stages in an end-to-end manner, emerged in large numbers. Regarding the multiple channels of EEG signals as spatial information, a straightforward scheme is to apply 2D convolutional neural networks (CNNs) to these spatial–temporal series [17,18]. Generally, a recurrent neural network (RNN) or long short-term memory network (LSTM) can be further used to gather temporal information on CNN feature maps [19,20,21,22]. Optionally, transformers [6,23] or 3D CNNs can be also applied if EEG signals are viewed as temporal segments [24]. As the EEG signals are discontinuous in the spatial domain, it is suggested that it may be not suitable to apply CNNs in this way on images. More recently, there is a trend to build models based on graph theory, where the graph is mainly designed or learnable, to describe the relation of multiple channels [25,26,27,28,29].

Among the aforementioned methods, almost all of them do not pay attention to multiscale EEG information. However, the physical sciences have indicated that numerous physiological components function at various time scales [30,31], suggesting the importance of multiscale information for EEG signals. To this end, there exist some other studies which focus on multiscale EEG features-based emotion recognition, where multiscale permutation entropy (MPE) and multiscale convolutional kernels are commonly used [32,33,34]. Overall, there is rare exploration on the views of both multiscale information and spatial relationships.

In this paper, we jointly consider the multiscale information and the multichannel relationship of raw EEG signals and propose a conceptually new yet simple method, termed the cascade scale-aware adaptive graph and cross-EEG transformer (SAG-CET). Unlike the recent transformer-based EEG emotion recognition methods that make efforts on single-transformer architecture [6,23], our SAG-CET builds a scale-aware adaptive GCN and a dual-stream transformer to integrate both the interchannel relation and multiscale information. Specifically, EEG signals with multiple channels are first split into nonoverlapping multichannel samples, as is common, and then we apply two division schemes on samples to generate short-scale and long-scale patches. After linear projection, both kinds of patches are fed into the SAG to enhance their representative abilities from the relation of different EEG channels. Subsequently, these patches along with two class tokens are fed into the CET. In the CET, different scale patches and tokens are first encoded by separated self-attention (SA) encoders, and then the class tokens are exchanged twice in a cross-attention operation to gather interscale correlation information. We apply several CETs to enhance model capacity and performance and finally take both class tokens for classification.

Additionally, to deeply investigate the effects of different scale information, we conduct extensive evaluations on DEAP and DREAMER to answer the following questions: (1) How does signal scale affect emotion recognition? (2) How does spatial correlation affect emotion recognition? (3) What are the optimal scales for SAG-CET? We observe that our SAG-CET significantly outperforms unitary transformer-based EEG emotion recognition yet is better than feature concatenation strategy and late fusion. We finally set up new state-of-the-art performance on two popular EEG emotion datasets. On DEAP, we achieve 98.89% and 98.92% in average subject-dependent accuracy of valence and arousal, and the numbers are 99.03% and 99.21% on DREAMER.

Our contributions can be summarized as follows:

• We propose to jointly consider the spatial relationship and temporal scale for EEG-based emotion recognition with a novel cascade framework.
• With multiscale EEG patch signals, we introduce the SAG to learn the relationship of multiple channels and propose the CET, which is superior to other fusion strategies, to better integrate multiscale representations.
• We conduct extensive experiments on the issues of spatial relationship and temporal scale and set up new state-of-the-art performance on two popular EEG emotion datasets.

2. Related Work

2.1. Valence–Arousal Model

The dimensional theory proposed by Russell et al. [35] categorizes emotions based on dimensional space. This theory suggests that emotions are constantly changing. One approach represents emotions using two variables, valence and arousal. In most emotion recognition studies, different emotions are mainly classified according to the valence–arousal emotional model. As shown in Figure 1, the vertical axis signifies arousal, which describes the progression of emotions from calm to excitement. The horizontal axis represents valence, reflecting emotions from low to high levels of positivity. Different emotions can be indicated by various coordinates on the graph; for example, excitement corresponds to positive arousal and positive valence, while sadness corresponds to negative arousal and negative valence.

2.2. EEG Emotion Recognition

The initial emotion recognition is mainly dominated by traditional machine learning. Li et al. [36] selected the frequency band in the spectral domain to extract relevant features and used a common spatial pattern and support vector machine (SVM) for classification. Bazgir et al. [37] used wavelet transform and principal component analysis to extract the frequency domain features of EEG signals and, respectively, used machine learning methods such as k-nearest neighbor (KNN) SVM to classify emotions. Smith K. Khare et al. [38] proposed optimized variational mode decomposition for emotion recognition using single-channel EEG signals. These methods based on traditional machine learning require good feature design methods, and as the amount of data increases, the performance of the model is affected.

The subsequently developed deep learning algorithms performed well despite the increasing data. For example, Zhong et al. [39] proposed a regionally related and attention-driven bidirectional LSTM network (RA-BiLSTM) for classifying brain activity induced by images. Li et al. [40] designed a hybrid deep learning model that combines CNN and RNN to extract features. These methods achieved good results but encountered new problems: Existing deep learning methods cannot focus on the important characteristic of long-range correlation in the temporal scale of EEG signals. To solve the aforementioned issue, researchers have attempted to use transformers. Li et al. [6] proposed an automatic transformer neural architectures search (TNAS) framework based on a multiobjective evolution algorithm (MOEA) for EEG-based emotion recognition.

2.3. The Usage of Multichannel and Multiscale Information

The spatial distribution of EEG electrodes is adjacent. Due to the limitation of insufficient spatial resolution of EEG electrodes, the collected EEG signal of one electrode will be affected by the EEG signals from other electrodes. Therefore, it is necessary to model the influence of different spatial regions using spatial relationships [41]. To this end, some researchers process EEG signals in a way similar to the field of computer vision to enable the model to automatically capture spatial features by convolution operation, as shown in the middle of Figure 2. For example, Tao et al. [42] proposed ACRNN, which maps the EEG signals onto a matrix similar to an image to represent the spatial relationships between multiple channels. However, the problem is that the interpolation of 2D matrix mapping introduces noise at discontinuous points between different EEG channels and cannot effectively represent the relationship between multiple channels. Some researchers resorted to graph theory to construct the relationship of multiple channels, as shown in the right of Figure 2. For example, Jia et al. [43] proposed a novel heterogeneous graph recurrent neural network (HetEmotionNet), fusing multimodal physiological signals for emotion recognition.

In addition to using graph theory to model the relations of multiple channels, the utilization of multiscale features also greatly improves the performance [30,31]. Jomaa et al. [44] introduced an MPE-based method that measures complexity in nonstationary multivariate signals. Su et al. [45] proposed a 3D CNN model with multiscale convolutional kernels to recognize emotional states. Although the aforementioned methods attempted to utilize the multiscale features of EEG signals and achieved satisfactory performance, features like MPE and multiscale convolution kernels still need predesign and computation. In addition, few studies combine multiscale features with multichannel features constructed from a graph.

Overall, although existing EEG emotion recognition methods have achieved good performance, there is still a lack of comprehensive methods to address the aforementioned issues. Our proposed method, SAG-CET, effectively utilizes a combination of graph theory and a transformer. Additionally, SAG-CET aims to streamline and improve the representation capabilities of EEG signals by capturing inherent time multiscale features.

3. Methodology

3.1. Preliminaries

Vision transformer: A vision transformer (ViT) is a variant architecture of the transformer [16] used in computer vision [14]. Different from the transformer that deals with 1D data in the field of NLP, the VIT applied in an image handles 2D data. The ViT is composed of multiple multihead attention modules and multiple dense layers, which are combined alternately.

The ViT divides a 2D image $x\in {\mathbb{R}}^{H\times W\times C}$ into N patches $x_p\in {\mathbb{R}}^{P_1·P_2·C}$. H, W, and C, respectively, represent the height, width, and number of channels of the input image. $P_1$ and $P_2$, respectively, represent the height and width of the patch. The number of patches is $N=\frac{HW}{P_1{P}_2}$. Then, the  $x_p$ is projected into a d dimension token sequence using a linear projection layer. Meanwhile, a learnable CLS token $x_c\in {\mathbb{R}}^{1\times d}$ is added to the first position of the sequence to represent the global information of the image. Next, a 1D positional embedding vector $P\in {\mathbb{R}}^{(N+1)\times d}$ is embedded into the token sequence to encode positional information. After that, the token sequence is input into L stacked transformer encoders for self-attention calculation. Finally, the first output representation which can achieve global information integration is usually used for classification.

The transformer encoder is a key component of the transformer architecture, consisting of multiple blocks that incorporate multihead self-attention (MSA) and a feed-forward network (FFN). The FFN is composed of two layers of multilayer perceptrons (MLPs), with a RELU activation function following the first linear layer. To ensure stable training and improve performance, layer normalization is applied before the input of each block. Additionally, residual connections are employed in the output of each block to facilitate the flow of information through the network and mitigate the vanishing gradient problem.

The proposed SA encoder of SAG-CET utilizes the same components as a ViT, with the key difference being that while images have height, width, and channel dimensions, brainwave signals have only two dimensions—time and channel number. Therefore, by reshaping the brainwave signals and adding a padding dimension, they can be fully adapted for use with the encoder component of the ViT.

Definition of graph: We define $G=(V,E,A)$ as a graph, where V is the set of nodes. Each element $v_i\in V^C$ in the set represents a node of the graph, where C represents the number of channels and the value of $v_i$ represents the feature of the node. E is a set of edges used to represent the connections between nodes. $A\in R^{C\times C}$ is a learnable adjacency matrix used to represent the connectivity relationship between nodes. $a_{ij}=1$ represents that node i and node j are mutually connected. If a self-loop $a_{ii}=1$ is added to the adjacency matrix, it can be represented in matrix form as $\widehat{A}=A+I$. The identity matrix is represented as $I\in R^{C\times C}$. Define D as the degree matrix, where the degree matrix is a diagonal matrix with values $D_i={\sum }_{j\in C}{a}_{ij}$ on the diagonal and zero in other regions. In the case of the adjacency matrix containing self-loops, $\widehat{D}=D+I$. Define the Laplacian operator to describe the difference between a node and its adjacent nodes in a graph. Its matrix form L is defined as follows:

$$\begin{array}{c}\begin{array}{cc}\hfill \Delta f& =DX-AX=(D-A)X\hfill \\ \hfill L& =D-A\hfill \end{array}\hfill \end{array}$$

(1)

The Laplacian matrix is a matrix representation of a graph, which can be used to capture the relationships between nodes in the graph structure, as shown in Figure 3. In order to ensure the weighted aggregation of the first-order neighbor information of nodes, the weight is inversely proportional to the degree of the node. The normalized Laplacian matrix is defined as

$$\begin{array}{c}{L}_{sym}=I-D^{-\frac{1}{2}}A{D}^{-\frac{1}{2}}\hfill \end{array}$$

(2)

Define the graph Fourier transformation and the graph inverse Fourier transformation as follows:

$$\begin{array}{c}\begin{array}{cc}\hfill \widehat{x}& =U^{T}x\hfill \\ \hfill x& =U\widehat{x}\hfill \end{array}\hfill \end{array}$$

(3)

where U is the eigenvector matrix of the Laplacian matrix.

3.2. Overview of Cascade Scale-Aware Adaptive Graph and Cross-Transformer Method

We propose a novel architecture named the cascade scale-aware adaptive graph and cross-transformer (SAG-CET) method. As illustrated in Figure 4, SAG-CET consists of two primary components: a scale-aware adaptive GCN (SAG) and a cross-EEG transformer (CET).

First, we divide an EEG signal sample into two signal sequences according to two proportions and project them through the FC layer to form temporal patches. For an EEG sample with L length, the default proportions used to divide are L/32 and L/4. Then, we build a spatial electrode graph from the dual stream, where SAG is employed to learn the graph representation that captures the shared spatial correlation of multichannel connections between short and long temporal patches. After that, we use the self-attention encoder and the proposed novel cross-attention module to capture two global representations, i.e., class tokens, from two scale streams. And the class tokens are reprojected into the same dimension and averaged to a final representation for classification. Next, we will introduce the main components of our model in detail.

3.3. Scale-Aware Adaptive GCN

Considering the low spatial resolution of EEG signals and the interaction between multiple channels, it is imperative to account for the spatial correlation between these signals to enhance the performance of the model. Therefore, we propose the SAG to effectively learn and capture the spatial correlation present among various EEG channels. In the SAG, a learnable adjacency matrix $\widehat{A}$ is used to construct the edge connections between nodes, and then a learnable weight matrix W is used to update the weights of the edges. The adjacency matrix $\widehat{A}$ represents the edge connections, and the weight matrix W represents the edge weights, which together define the spatial relationships within the graph.

Spatial electrode graph construction: In the definition of the graph $G=(V,E,A)$, the nodes $v_i\in V^C$ of the spatial electrode graph represent EEG electrode channels, where $v_i$ denotes the amplitude of the EEG signal at a single time step. The $\widehat{A}\in R^{N\times N}$ is a learnable adjacency matrix, and it captures the connectivity between electrode channels in EEG signals. E is a set of edges that connect the EEG electrode channels, and the values of its elements are determined by $\widehat{A}$.

Spatial electrode graph embedding: We learn the embedding of each node in the spatial electrode graph with the SAG, aiming to fully exploit the connections between the dual streams. First, the node sequences from each stream are projected using the functions $f{\left(x\right)}^l$ and $f{\left(x\right)}^s$ to align their dimensions. The dimension alignment functions $f{\left(x\right)}^l$ and $f{\left(x\right)}^s$ are fully connected layers that are used to transform the large-scale signal dimensions into small-scale signal dimensions. Subsequently, the resulting multiscale graph node sequence is fed into the SAG to compute the average weight. More specifically, the graph convolution process for the signals x and y is as follows:

$$\begin{array}{c}x{\ast }_{g}y=U\left(\left(U^{T}x\right)\odot \left(U^{T}y\right)\right)\hfill \end{array}$$

(4)

where ⊙ means the Hadamard product. Suppose $g(·)$ is a filtering function (convolutional kernel); then, the signal x filtered by $g\left(L\right)$ can be expressed as

$$\begin{array}{c}y=g\left(L\right)x=g\left(U\Lambda U^T\right)x=Ug(\Lambda )U^{T}x\hfill \end{array}$$

(5)

where the convolutional kernel $g(\Lambda )=diag\left([\widehat{y_1},\widehat{y_2},\cdots ,\widehat{y_n}]\right)$ is a diagonal matrix. To simplify computations, we use K-order Chebyshev polynomials [46] to replace the polynomial expansion of $g\left(L\right)$. and we replace the convolution kernel with

$$\begin{array}{c}g(\Lambda )=\sum _{k=0}^{K-1}{\theta }_k{T}_k(\Lambda )\hfill \end{array}$$

(6)

Here, the coefficients of the Chebyshev polynomials are given by ${\theta }_k$, and the eigenvector obtained by performing eigenvalue decomposition on the Laplacian matrix is $\Lambda =diag\left([{\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_N]\right)$. $T_k(\Lambda )$ can be calculated according to the following formula:

$$\begin{array}{c}\left\{\begin{array}{c}{T}_0\left(x\right)=1,T_1\left(x\right)=x\hfill \\ T_k\left(x\right)=2x{T}_{k-1}\left(x\right)-T_{k-2}\left(x\right),k\ge 2\hfill \end{array}\right.\hfill \end{array}$$

(7)

According to (6), we can rewrite the graph convolution operation defined in (5) as

$$\begin{array}{c}\begin{array}{cc}\hfill y& =Ug\left(\Lambda \right)U^{T}x\hfill \\ \hfill & =\sum _{k=0}^{K-1}U\left[\begin{array}{ccc}{\theta }_k{T}_k\left({\lambda }_0\right)& \cdots & 0\\ ⋮& \ddots & ⋮\\ 0& \cdots & {\theta }_k{T}_k\left({\lambda }_{N-1}\right)\end{array}\right]U^{T}x\hfill \\ \hfill & =\sum _{k=0}^{K-1}{\theta }_k{T}_k\left(L\right)x\hfill \end{array}\hfill \end{array}$$

(8)

3.4. Cross-EEG Transformer

Due to the physical sciences indicating that numerous physiological components function at multiple scales [30,31,47,48,49] and the ability of attention mechanisms to capture long-range temporal correlations, we propose the CET to integrate EEG signals of varying temporal scales.

In the initial stage, each stream undergoes spatial feature extraction via the SAG to initialize a learnable class token to represent global temporal features. Subsequently, each class token captures temporal features from other patches within the stream through the SA encoder. The CET incorporates multiple fusion and information exchange mechanisms within the cross-attention block, facilitating the class tokens to effectively capture multiscale features within the EEG signals. Finally, the class tokens obtained through the CET are utilized for emotion prediction.

The cross-attention block plays a pivotal role in information exchange between dual streams. Specifically, it facilitates the exchange of information between the class tokens and the temporal patches from the other stream, followed by the projection of the exchanged class tokens back to their respective streams. As a result, each stream is able to incorporate relevant information from the other stream, thereby significantly enhancing its ability to comprehend and handle multiscale inputs. Figure 5 illustrates the cross-attention block for the long stream. First, the class token of the long stream $x_{cls}^l$ is subject to an alignment projection function denoted as $f^l\left(x\right)$, resulting in the transformed vector $x_{cls}^{\prime l}$ as defined in Equation (9). Subsequently, $x_{cls}^{\prime l}$ is utilized to generate the query by means of $W^Q\in R^{C\times (C/h)}$. Simultaneously, $x_{cls}^{\prime l}$ is concatenated with the temporal patches of the short stream $x_p^s$ to yield $x^{\prime l}$, which is employed to derive the key and value via $W^K\in R^{C\times (C/h)}$ and $W^V\in R^{C\times (C/h)}$, respectively. The attention weights are calculated using the query, key, and value.

$$x_{cls}^{\prime l}=f^l\left(x_{cls}^l\right)$$

(9)

$$x^{\prime l}=\left[x_{cls}^{\prime l}\parallel x_p^s\right]$$

(10)

The process of the cross-attention (CA) block can be expressed as

$$\begin{array}{c}\hfill Q=x_{cls}^{\prime l}{W}^Q,K=x^{\prime l}{W}^K,V=x^{\prime l}{W}^V\\ \hfill A=softmax\left(Q{K}^T/\sqrt{C/h}\right),CA\left(x^{\prime l}\right)=AV\end{array}$$

(11)

where C is the dimension of embedding and h is the number of heads. Cross-attention also uses multihead cross-attention (MCA) by referring to the SA mechanism, but unlike a vanilla unitary transformer, it does not use a feed-forward network after MCA. Specifically, the process of MCA can be represented as follows:

$$\begin{array}{c}{y}_{cls}^l=f^l\left(x_{cls}^l\right)+MCA\left(LN\left(\left[f^l\left(x_{cls}^l\right)\parallel x_p^s\right]\right)\right)\\ z^l=\left[g^l\left(y_{cls}^l\right)\parallel x_p^l\right]\end{array}$$

(12)

where $g^l\left(x\right)$ is the inverse projection function, which is used to reproject the class token that has undergone information fusion back to its original dimension for concatenation with temporal patches. Finally, the pseudocode of the SAG-CET is shown in Algorithm 1.

Algorithm 1 Training procedure of SAG-CET

Input:raw signal data D Output:Network ${\theta }^{\left(N\right)}$ Initialize:${\widehat{A}}^{\left(0\right)},{\mathbf{x}}_{cls}^{l\left(0\right)},{\mathbf{x}}_{cls}^{s\left(0\right)}$,${\theta }^{\left(0\right)}$ 1: $Patc{h}_s,Patc{h}_l=\mathrm{SplitPatch}(D,s_{size},l_{size})$ 2: $Nod{e}_s,Nod{e}_l=\mathrm{PatchToNode}(Patc{h}_s,Patc{h}_l)$ 3: $NodeSet=\mathrm{align\_cat}(Nod{e}_s,Nod{e}_l)$ 4: for $i=0$ … $N-1$ do 5:  ${Node}_s^{\prime },{Node}_l^{\prime }=\mathrm{SAG}(NodeSet,{\widehat{A}}^{\left(i\right)})$ 6:  ${Patch}_s^{\prime },Patc{h}_l=\mathrm{NodeToPatch}({Node}_s^{\prime },{Node}_l^{\prime })$ 7:  ${\mathbf{x}}_{cls}^{l\left(i\right)},{\mathbf{x}}_{cls}^{s\left(i\right)}=\mathrm{CET}({Patch}_s^{\prime },{Patch}_l^{\prime },{\mathbf{x}}_{cls}^{l\left(i\right)},{\mathbf{x}}_{cls}^{s\left(i\right)})$ 8:  $output=\mathrm{classifier}({\mathbf{x}}_{cls}^{l\left(i\right)}+{\mathbf{x}}_{cls}^{s\left(i\right)})$ 9:  $AccSub=\mathrm{acc}(label,output)$ 10:   $loss=\mathrm{nn}.\mathrm{CrossEntropyLoss}(output,label)$ 11:   ${\widehat{A}}^{\left(i\right)},{\mathbf{x}}_{cls}^{l\left(i\right)},{\mathbf{x}}_{cls}^{s\left(i\right)}\leftarrow \mathrm{loss}.\mathrm{backward}\left(\right)$ 12: end for 13: return${\theta }^{\left(N\right)}$

4. Experiments

4.1. Datasets and Experiment Setting

We conducted subject-dependent experiments on two widely used public datasets, namely, DEAP [50] and DREAMER [51], which have been extensively studied by researchers in the field of emotion recognition, as shown in

Table 1. The details of DEAP and DREAMER.

Dataset	Subjects	Label	Experiments	Signals (Channel)
DEAP	32	Valence, arousal, dominance	40/subject	EEG (32), EMG (2), EOG (2), GSR (1), BVP (1), temperature (1), respiration (1)
DREAMER	23	Valence, arousal, dominance, liking	18/subject	EEG (14), ECG (2)

. DEAP is a multimodal physiological signals dataset, which comprises recordings from 32 subjects. The dataset includes 32 channels of EEG signals and 8 channels of peripheral physiological signals (PPS), with an EEG signal sampling rate of 512 Hz. The remaining eight channels of PPS consist of signals for EMG, EOG, GSR, blood volume pulse (BVP), temperature, and respiration. The subjects were instructed to rate their valence, arousal, and dominance on a scale of 1 to 9 for the four emotional dimensions. DREAMER includes EEG signals from 14 channels and ECG signals from 2 channels of 23 subjects. The EEG signals were sampled at a rate of 128 Hz. The subjects were asked to rate four emotional dimensions, namely, valence, arousal, dominance, and liking, on a scale of 1 to 5.

To ensure consistency and comparability between the two datasets, we downsampled all EEG signals to 128 Hz and extracted a 60-second segment from each signal. We filtered EEG signals to 4–45 Hz and performed blind source separation to remove EOG artifacts [50,51]. To enhance the characterization ability of the EEG signals, we performed baseline correction on the DEAP and DREAMER datasets by subtracting the average of each second for three and four seconds, respectively [52]. To increase the scale of our training set, we augmented the dataset by extracting nonoverlapping 1-s signal segments as samples [53]. Finally, following the common setting [6,54], we categorized the valence and arousal labels in the DEAP and DREAMER datasets as either high or low using thresholds of 5 and 3, respectively.

For each EEG sample with length L, we split it into long-term patches as L/4 and short-term patches L/32 by default. For our experiments, we set the initial learning rate to 0.001 and the batch size to 64. We utilized Adam as the optimizer and set the ${\beta }_1$ and ${\beta }_2$ parameters to 0.5 and 0.9, respectively. To ensure the reliability of our results, we employed a random 10-fold cross-validation approach and trained our model for 100 epochs for each fold. We evaluated the performance of our model using accuracy metrics. All experiments were conducted on Tesla A100 GPUs.

4.2. Comparisons to State of the Art

On DEAP and DREAMER, we compared our model with the SOTA methods, which use different backbones. The used models contain CRAM [20], ACRNN [42], MMResLSTM [55], DGCNN [56], IAG [26], V-IAG [29], SSTemotionNet [57], HetEmotionNet [43], EeT [54], and TNAS [6], as shown in

Table 2. Comparison of the input features and backbones in state-of-the-art methods.

Model	Feature	Modality	Backbone
CRAM [20]	Raw signal	EEG	CNN + LSTM + attention
SSTemotionNet [57]	Raw signal + DE	EEG	CNN + attention
ACRNN [42]	Raw signal	EEG	CNN + LSTM + attention
DGCNN [56]	PSD	EEG	GCN
IAG [26]	PSD	EEG + PPS	GCN + LSTM
V-IAG [29]	PSD	EEG + PPS	GCN + LSTM
HetEmotionNet [43]	Raw signal + DE	EEG + PPS	GCN + GRU
MMResLSTM [55]	Raw signal	EEG + PPS	LSTM + ResNet
EeT [54]	Raw signal	EEG	Transformer
TNAS [6]	Raw signal	EEG	Transformer
SAG-CET (ours)	Raw signal	EEG	GCN + transformer

. To ensure the comprehensiveness of the comparative experiments, these models adopt different feature selection strategies, including spatial–temporal domain features [20,55], spatial–spectral domain features [26,29,56], and the integration of spatial–spectral–temporal domain features [43,57]. Apart from this, we chose a wide range of models which include architectures featuring different combinations of CNN, RNN, LSTM, GRU, GCN, transformer, and others to comprehensively demonstrate the difference of our model compared to previous works. To ensure a fair comparison, we applied identical data preprocessing steps to both our model and the selected baseline models on both datasets and conducted comparative experiments under the same experiment conditions.

4.3. Experiment on DEAP

Table 3. The mean accuracies (ACC) and standard deviations (STD) on the DEAP dataset.

Model	Valence (%)	Arousal (%)
CRAM [20]	87.09 ± 7.49	84.46 ± 9.27
DGCNN [56]	90.44 ± 3.01	91.70 ± 3.46
MMResLSTM [55]	92.30 ± 1.55	92.87 ± 2.11
EeT [54]	93.34 ± 2.12	92.86 ± 2.35
ACRNN [42]	93.72 ± 3.21	93.38 ± 3.73
SST-EmotionNet [57]	95.54 ± 2.54	95.97 ± 2.86
HetEmotionNet [43]	97.66 ± 1.54	97.30 ± 1.65
TNAS [6]	98.66 ± 0.94	98.68 ± 0.98
SAG-CET (ours)	98.89 ± 0.84	98.92 ± 0.81

presents the average accuracy and standard deviations of the models on the DEAP dataset. Figure 6 shows the average accuracy per subject. Our model achieves an accuracy of 98.89% and 98.92% in valence and arousal, respectively, while the other baseline methods ranged between 84.46% and 98.68%. In these models, TNAS, which has a performance (98.66% for valence, 98.68% for arousal) similar to SAG-CET, combines the advantages of transformer and MOEA to build a backbone network. However, the performance of TNAS on DREAMER is far inferior to SAG-CET, indicating that SAG-CET has better generalization ability and model robustness than TNAS. Additionally, the proposed model demonstrates satisfactory stability, as evidenced by its lowest standard deviation. The results demonstrate that our proposed model achieves better performance than SOTA baselines on DEAP. Figure 7 provides a confusion matrix to illustrate the correctness of the model. The sample size of the DEAP dataset is 60 × 40 × 32, and the sample size of the DREAMER dataset is 60 × 23 × 18.

4.4. Experiment on DREAMER

Table 4. The mean accuracies (ACC) and standard deviations (STD) on the DREAMER dataset.

Model	Valence (%)	Arousal (%)
DGCNN [56]	86.23 ± 3.25	84.54 ± 3.16
IAG [26]	90.75 ± 2.27	91.03 ± 1.65
CRAM [20]	92.27 ± 2.95	93.03 ± 1.87
V-IAG [29]	92.82 ± 2.16	93.09 ± 1.44
TNAS [6]	96.95 ± 3.35	96.41 ± 3.61
ACRNN [42]	97.39 ± 1.37	97.98 ± 1.92
SAG-CET (ours)	99.08 ± 0.51	99.21 ± 0.72

showcases the performance of the proposed model and other baseline models on the DREAMER dataset. Figure 8 shows the average accuracy per subject. In general, our SAG-CET achieves the best performance with an accuracy of (99.08% for valence and 99.21% for arousal) on the DREAMER dataset compared to all previous SOTA methods, with the lowest standard deviation.

Notably, although ACRNN seems to have achieved a slightly lower performance (97.39% for valence, 97.98% for arousal) on the DREAMER than SAG-CET, compared to performance on the DREAMER dataset, SAG-CET far outperforms ACRNN on the DEAP dataset, which indicates that SAG-CET has excellent generalization ability.

4.5. Results Analysis

Figure 9 shows the average weight matrix of MHA of each stream in each subject before performing cross-attention calculation, which indicates the distribution of attention in the input embedding. The i-th row of each attention weight matrix corresponds to the attention distribution of class token of i-th subject over the entire embedding sequence. In the long stream, attention mainly focuses on the middle to the posterior part of temporal patches, while in the short stream, attention focuses on the anterior part of temporal patches. This indicates that the features of the EEG signals in temporal patches at different scales are complementary.

To more intuitively describe the ability of EEG signals to represent emotion at various scales, we explored the impact of different patch sizes on the performance of EEG emotion recognition using the single-stream input transformer. As shown in Figure 10, results indicate that the accuracy gradually improves with the increase in patch size. However, it is worth noting that the growth rate of accuracy is negatively correlated with the patch size. This suggests that although larger patch sizes can capture more temporal information, there is a limit to the improvement in accuracy through this approach. In order to find the optimal scale for using SAG-CET for EEG emotion recognition, we conducted a set of experiments to evaluate the optimal combination of patch proportion in a segment of an EEG signal sample. As shown in Figure 10, when the patch sizes for short-stream and long-stream inputs are, respectively, 1/32 and 1/4 of the EEG signal sample, CET achieves the best performance on both datasets. Furthermore, our proposed CET model overcomes the limitation shown in Figure 10 by fusing features from two different patch sizes, demonstrating the effectiveness of multiscale features in EEG emotion recognition.

4.6. Ablation Study

In order to verify the effectiveness of the proposed model, we conducted two types of ablation studies on two datasets: (1) Ablation studies to validate the effectiveness of the SAG in extracting spatial features. (2) Ablation studies to verify if the dual-stream structure and CET can effectively perform multiscale feature fusion. The single-stream input only uses an SA encoder, and the dual-stream input additionally uses a cross-attention block. The results are as shown in Figure 11 and Figure 12; we have the following observations:

• By comparing the results of experiments conducted solely using long-stream, short-stream, or dual-stream input on two datasets, we can observe that dual-stream input performs better than long-stream or short-stream input, which fully explains the effective promotion with the application of the cross-attention block.
• By comparing the results of experiments conducted on the SAG block, removing the SAG component from the SAG-CET reduces the performance. The CET with SAG outperforms the CET without SAG, which indicates that the SAG is effective in constructing the spatial correlations of multiple channels. Notably, our analysis showed that the best performance was achieved by combining all the aforementioned factors across all the experiments.

4.7. Limitations and Future Work

In this study, as a common setting, we adopt a conventional binary classification to classify the valence and arousal of emotions and only conduct subject-dependent experiments. Thus, the generalization of a subject-independent setting is unknown, and multiclass tasks on valence and arousal can be further explored. In addition, since EEG can represent extensive physiological states, another future work is to adapt our method to other states like mental fatigue as well as sleep monitoring, epilepsy monitoring, depression treatment, and other aspects.

5. Conclusions

In this paper, we propose a novel approach to EEG signal emotion recognition using end-to-end deep learning with an emphasis on multiscale and multichannel feature extraction. The proposed model, SAG-CET, utilizes a scale-aware adaptive GCN, dual-stream input with varying scale EEG signal patches, and a cross-attention block that enables the fusion of features from signals of different scales. Extensive experiments comparing our approach with various backbone network models on the DEAP and DREAMER datasets demonstrated that our model achieves SOTA performance, outperforming existing methods by a significant margin. Additionally, we conducted experiments to demonstrate the complementary nature of signal patches at different temporal segmentation scales. Through extensive experimentation, we discovered the optimal combinations of multiple scale ratios, contributing to the development of a multiscale segmentation strategy in the temporal dimension for EEG signals. Finally, we performed ablation experiments by removing two key modules to demonstrate the effectiveness of the proposed modules.