**1. Introduction**

Affective computing is a branch of artificial intelligence. It is computing that relates to, arises from, or influences emotions [1]. Automatic emotion recognition is an area of study that forms part of affective computing. Research in this area is rapidly evolving thanks to the availability of affordable devices for capturing brain signals, which serve as inputs for systems that decode the relationship between emotions and electroencephalographic (EEG) variations. These devices are called EEG-based brain-computer interfaces (BCIs).

Affective states play an essential role in decision-making. Such states can facilitate or hinder problem-solving. Emotion recognition takes advantage of positive affective states, enhances emotional intelligence, and consequently improves professional and personal success [2]. Moreover, emotion self-awareness can help people manage their mental health and optimize their work performance. Automatic systems can increase our understanding of emotions, and therefore promote effective communication among individuals and human-to-machine information exchanges. Automatic EEG-based emotion recognition could also help enrich people's relationships with their environment. Besides, automatic emotion recognition will play an essential role in artificial intelligence entities designed for human interaction [3].

According to Gartner's 2019 Hype Cycle report on trending research topics, affective computing is at the innovation trigger stage, which is evidenced by the field's copious publications. However, there are still no defined standards for the different components of the systems that recognize emotions using EEG signals, and it is still challenging to detect and classify emotions reliably. Thus, a survey that updates the information in the emotion recognition field, with a focus on new computational developments, is worthwhile.

This work reviews emotion recognition advances using EEG signals and BCI to (1) identify trends in algorithm usage and technology, (2) detect potential errors that must be overcome for better results, and (3) identify possible knowledge gaps in the field. The aim is to distinguish what has already been done in systems implementations and catch a glimpse of what could lie ahead. For context, our study is a survey from 2015 to 2020.

The present article gives an overview of datasets, emotion elicitation methods, feature extraction and selection, classification algorithms, and in general terms, computer intelligence techniques used in this field. We present a brief review of the components of an EEG-based system to recognize emotions and highlight trends showing statistics of their use in the literature. We deliver a compilation of papers describing new implementations, analyzing their inputs, tools, and considered classes. This up-to-date information could be used to discover and sugges<sup>t</sup> new research paths.

The present survey followed the guidelines of [4]. We used Semanticscholar.org for searches of sources because it links to the major databases that contain journals and conferences proceedings. The search criteria were the keywords linked to our review's objectives.

We extracted articles from journals and conferences that present new implementations of computational intelligence techniques. Concretely, the analyzed papers' primary objectives were computational systems that applied algorithms for the detection and classification of emotions using EEG-based BCI devices. Such studies also included performance measures that allowed a comparison of results while taking into account the classified number of emotions.

As a result, we obtained 136 journal articles, 63 conference papers, and 15 reviews. Each whole article was read to have complete information to guide the application of inclusion and exclusion filters. The inclusion criteria were: (1) The articles were published in the considered period in peer-reviewed journals and conferences, (2) they constitute emotion recognition systems that used EEG-based BCI devices with a focus on computational intelligence applications, and (3) they include experimental setups and performance evaluations. Lastly, we applied additional exclusion criteria and eliminated review articles and other studies that have a di fferent perspective as medical studies for diagnosis or assessment.

With these considerations, we selected 36 journal studies and 24 conference papers. From this group, we extracted statistical data about computational techniques to detect trends and perform a comparative analysis. Finally, from these 60 papers, we chose a sample of 31 articles to show a summary of technical details, components, and algorithms. It should be noted that according to generally accepted practices, 31 observations are su fficient for statistically valid conclusions due to the central limit theorem. Then, from this subsample of articles, we obtained some additional data and tendencies.

This document is organized as follows: Section 1 presents an introduction of the topic, with an overview of BCI devices, emotion representations, and correlations among brain locations, frequency bands, and a ffective states. Section 2 shows the structure of EEG-based BCI systems for emotion recognition. Their principal components are revised: (1) Signal acquisition, (2) preprocessing, (3) eature extraction, (4) feature selection, (5) classification, and (6) performance evaluation. Section 3 analyzes the components of our chosen research pieces and discusses trends and challenges. Section 4 presents future work. Section 5 features the conclusions or this survey.

#### *1.1. EEG-Based BCI in Emotion Recognition*

Many studies sugges<sup>t</sup> that emotional states are associated with electrical activity that is produced in the central nervous system. Brain activity can be detected through its electrical signals by sensing its variations, locations, and functional interactions [5] using EEG devices. EEG signals have excellent temporal resolution and are a direct measurement of neuronal activity. These signals cannot be manipulated or simulated to fake an emotional state, so they provide reliable information. The challenge is to decode this information and map it to specific emotions.

One a ffordable and convenient way to detect EEG signals is through EEG-based BCI devices that are non-invasive, low cost, and even wearable, such as helmets and headbands. The development of these tools has facilitated the emergence of abundant research in the emotion recognition field.

Some scientists predict that EEG-based BCI devices will soon improve their usability. Therefore, shortly, they could be used on an everyday basis for emotion detection with several purposes, such as emotion monitoring in health care facilities, gaming and entertainment, teaching-learning scenarios, and for optimizing performance in the workplace [6], among other applications.

#### *1.2. Emotion Representations*

Emotions can be represented using di fferent general models [7]. The most used are the discrete model and the dimensional models. The discrete model identifies basic, innate, and universal emotions from which all other emotions can be derived. Some authors state that these primary emotions are happiness, sadness, anger, surprise, disgust, and fear [8]. Some researchers consider that this model has limitations to represent specific emotions in a broader range of a ffective states.

Alternatively, dimensional models can express complex emotions in a two-dimensional continuous space: Valence-arousal (VA), or in three dimensions: Valence, arousal, and dominance (VAD) [9]. The VA model has valence and arousal as axes. Valence is used to rate positive and negative emotions and ranges from happy to unhappy (or sad). Arousal measures emotions from calm to stimulated (or excited). Three-dimensional models add a dominance axis to evaluate from submissive (powerless) to empowered emotions. This representation distinguishes emotions that are jointly represented in the VA model. For instance, fear and anger have similar valence-arousal representations on the VA plane. Thus, three-dimensional models improve "emotional resolution" through the dominance dimension. In this example, fear is a submissive feeling, but anger requires power [10]. Hence, the dominance dimension improves the di fferentiation between these two emotions.

Figure 1 shows a VA plane with the representation of basic emotions. The horizontal axis corresponds to valence dimensions, from positive to negative emotions. Likewise, the vertical axis corresponds to arousal. These two variables can be thought of as emotional state components [5]. Figure 2 presents the VAD space with a representation of the same basic emotions.

**Figure 1.** Emotional states in the Valence-Arousal space [11].

**Figure 2.** Emotional states in the Valence-Arousal-Dominance space [12].

Table 1 shows that some researchers studying EEG-based functional connectivity in the brain have reported a relationship between specific brain areas and emotional states. Studies that take at-single-electrode-level analysis into account have shown that asymmetric activity at the frontal site in the alpha band is associated with emotion. Ekman and Davidson found that enjoyment generated an activation of the brain's left frontal parts [13]. Another study found a left frontal activity reduction when volunteers adopted fear expressions [14]. Increased power in theta bands at the frontal midline is associated with pleasurable emotions, and the opposite has been observed with unpleasant feelings [15].



Several studies confirm that frequency bands are related to affective responses. However, emotions are complex processes. The authors in [15] assert that the recognition of different emotional states may be more valid if EEG-based functional connectivity is examined, rather than a single analysis at the

electrode level. Correlation, coherence, and phase synchronization indices between EEG electrode pairs are used to estimate functional connectivity between different brain locations. Likewise, differential entropy (DE), and its derivatives like differential asymmetry (DASM), rational asymmetry (RASM), and differential caudality (DCAU) measure functional dissimilarities. Such features are calculated through logarithmic power spectral density for a fixed-length EEG sequence, plus the differences and ratios between DE features of hemispheric asymmetry electrodes [19].

The growing consensus seems to be that a simple mapping between emotions and specific brain structures is inconsistent with observations of different emotions activating the same structure, or one emotion activating several structures [20]. Additionally, functional connectivity between brain regions or signal complexity measures may help to detect and describe emotional states [21].

#### **2. EEG-Based BCI Systems for Emotion Recognition**

Figure 3 presents the structure of an EEG-based BCI system for emotion recognition. The processes of signal acquisition, preprocessing, feature extraction, feature selection, classification, and performance evaluation can be distinguished and will be reviewed in the following subsections.

**Figure 3.** Components of an EEG-based BCI for emotion recognition.

#### *2.1. Signal Acquisition*

Inexpensive wearable EEG helmets and headsets that position noninvasive electrodes along the scalp can efficiently acquire EEG signals. The clinical definition of EEG is an electrical signal recording of brain activity over time. Thus, electrodes capture signals, amplify them, and send them to a computer (or mobile device) for storage and processing. Currently, there are various low-cost EEG-based BCI devices available on the market [22]. However, many current models of EEG-based BCI become incommodious after continued use. Therefore, it is still necessary to improve their usability.

#### 2.1.1. Public Databases

Alternatively, there are also public databases with EEG data for affective information. Table 2 presents a list of available datasets related to emotion recognition. Such datasets are convenient for research, and several emotion recognition studies use them.


**Table 2.** Publicly available datasets.

#### 2.1.2. Emotion Elicitation

The International Affective Picture System (IAPS) [31] and the International Affective Digitized Sound System (IADS) [32] are the most popular resources for emotion elicitation. These datasets provide emotional stimuli in a standardized way. Hence, it is useful for experimental investigations.

IAPS consists of 1200 images divided into 20 sets of 60 photos. Valence and arousal values are tagged for each photograph. IADS' latest version provides 167 digitally recorded natural sounds familiar in daily life, with sounds labeled for valence, arousal, and dominance. Participants labeled the dataset using the Self-Assessment Manikin system [12]. IAPS and IADS stimuli are accessible with labeled information, which is convenient for the construction of a ground-truth for emotion assessment [33].

Other researchers used movie clips, which have also been shown capable of provoking emotions. In [34], the authors state that emotions using visual or auditory stimuli are similar. However, results obtained through affective labeling of multimedia may not be generalizable to more interactive situations or everyday circumstances. Thus, new studies using interactive emotional stimuli to ensure the generalizability of results for BCI would be welcomed.

Numerous experiments stimulated emotions in different settings, but they do not use EEG devices. However, they collected other physiological indicators as heartrate, skin galvanic changes, and respiration rate, among others. Conceptually, such paradigms could be useful if they are replicated for EEG signal acquisition. Possible experiments include stress during interviews for the detection of anger, anxiety, rejection, and depression. Exposure to odorants triggers emotions, such as anger, disgust, fear, happiness, sadness, and surprise. Harassment provokes fear. A threat of short-circuit, or a sudden backward-tilting chair elicits fear. A thread of shock provokes anxiety. Naturally, these EEG-based BCIs experiments should take into account ethical considerations.

To our knowledge, only a few studies have used more interactive conditions where participants played games or used flight simulators to induce emotions [35,36]. Alternatively, some authors have successfully used auto-induced emotions through memory recall [37].
