A Survey on Datasets for Emotion Recognition from Vision: Limitations and In-the-Wild Applicability

Abstract

Emotion recognition is the task of identifying and understanding human emotions from data. In the field of computer vision, there is a growing interest due to the wide range of possible applications in smart cities, health, marketing, and surveillance, among others. To date, several datasets have been proposed to allow techniques to be trained, validated, and finally deployed to production. However, these techniques have several limitations related to the construction of these datasets. In this work, we survey the datasets currently employed in state-of-the-art emotion recognition, to list and discuss their applicability and limitations in real-world scenarios. We propose experiments on the data to extract essential insights related to the provided visual information in each dataset and discuss how they impact the training and validation of techniques. We also investigate the presence of nonverbal cues in the datasets and propose experiments regarding their representativeness, visibility, and data quality. Among other discussions, we show that EMOTIC has more diverse context representations than CAER, however, with conflicting annotations. Finally, we discuss application scenarios and how techniques to approach them could leverage these datasets, suggesting approaches based on findings from these datasets to help guide future research and deployment. With this work we expect to provide a roadmap for upcoming research and experimentation in emotion recognition under real-world conditions.

1. Introduction

The recognition of human behavior is a task that has been of increasing interest to our society in recent years due to its ability to enable a deeper understanding of users in the most diverse scenarios. By monitoring humans, machine intelligence systems can detect patterns in behavior and suggest changes that could improve interaction and quality of experience. One of the many sub-tasks involved in the recognition of behavior is called emotion recognition, which, from multiple definitions in both computer science and psychology research, can be defined as the process of identifying emotions based on nonverbal cues given by the person [1,2,3,4,5,6,7]. The nonverbal aspect is essential in many contexts where the user needs to interact with environments instead of systems, allowing for an unobtrusive experience and a perception of emotion that is not forced.

Especially in the context of smart environments, which can be extended to smart cities, understanding their user is crucial. Given the focus of smart cities on building more attractive and lively spaces, physical sensors and internet-of-things frameworks are typically deployed to measure various aspects of a region, such as weather, pollution, and traffic. However, they need more focus on understanding how their citizens feel near specific areas in the city and how changes could impact their perception and feeling about those spaces [8].

There have been studies focused on understanding the impact of emotion recognition approaches for decision making in these scenarios, mainly to understand the correlation of green spaces, such as public parks and forests, on the urban scene. For instance, the work by [9] studied preferences of ornamental plant species for green spaces by leveraging emotion perception. In addition, the work by [10] used selfies to extract emotion, revealing that visitors from parks near the city center are more susceptible to displaying negative emotions. Other works have focused on different scenarios, such as [11], which investigated the relationship between perceived emotions on the university campus on variables such as environment and habitation availability, and [12], which investigated how sound pollution can affect perceived emotion. Therefore, by focusing on human factors, smart cities can engage with the citizens in a personalized way and have real-time, not modeled, feedback about their services or public spaces.

Along the nonverbal axis, there are many cues that humans can naturally perceive from different points of view. For example, involuntary movements (facial expressions, body language, and eye gaze), voluntary movements (touching and avoidance of touching by the peer), or conversational cues (changes in subject, lies, superficial talk, and variations in speech tonality). Although each sub-group of nonverbal communication has significant research works that propose approaches for understanding them, in this work, we focus on those which can be understood through vision-based approaches and in the wild, discarding, for instance, datasets for emotion recognition based on speech tonality, since in these uncontrolled scenarios people will not always be in conversation with peers and a high-quality recording of the voice would be very difficult to achieve.

The usage of other nonverbal cues is what currently separates the literature into two tasks: facial expression recognition (FER) and emotion recognition. The first task is motivated by the fact that facial expressions carry many discriminative features that we can correlate with emotion. Early approaches for FER focused on lab-controlled scenarios and posed expressions. Throughout the years, competitions such as Emotion Recognition in the Wild (EmotiW) [13] have been driving this task towards less restrictive approaches that could also be deployed in the wild. However, such techniques are still restricted to scenarios with a visible face and are harmed when the face is occluded or not directly aligned with the camera [14,15]. These restrictions motivated researchers to fuse facial expressions with other types of nonverbal cues, these approaches are what is known today as emotion recognition. Approaches such as the work by Kosti et al. [16], context-aware emotion recognition networks (CAER-Net) [17] and global-local attention for emotion recognition network (GLAMOR-Net) [18] proposed the extraction of background features referenced to a context, while approaches such as emotion recognition using adaptive multi-cues (EmotiRAM) [19] and the work by Chen et al. [20] proposed, beyond context, to leverage body language. Therefore, combining multiple cues may lead to more robust techniques since, in case of the occlusion of a specific cue, the model could rely on other cues to make predictions.

As the evaluations on the benchmarks evolve, there is a tendency to deploy these systems in real-life scenarios to contribute to society and generate results outside the academic scenario, in which we usually validate datasets. However, it is well-known that deep learning systems, such as those employed for emotion recognition techniques, depend highly on data availability and quality. Although there are several datasets for this task, each dataset has its particularities and limitations, which could severely harm the capacity of execution in real scenarios. Furthermore, there are limitations regarding sample bias, in which samples present in the dataset are not representative of the real world, and recall bias, which is caused by the way multiple annotators are usually handled in these scenarios.

In this work, we survey datasets currently used for benchmarking techniques for emotion recognition. Specifically, we focus on vision-based datasets containing images or videos for evaluation. Therefore, we do not evaluate datasets focusing on speech tonality, for example. Some of the contributions of our work are:

• We review how the behavioral psychology literature describes emotion perception in humans and correlate this with current datasets (Section 2);
• We survey and list the datasets currently employed for benchmarking in the state-of-the-art techniques (Section 3);
• We explore the difference in the annotations of these datasets and how they could impact training and evaluating techniques (Section 3.2);
• We discuss annotations described using continuous models, such as the valence, arousal and dominance (VAD) model, and how they can harm the ability of a model to understand emotion (Section 3.3);
• We investigate the presence of nonverbal cues beyond facial expression in the datasets of the state-of-the-art techniques and propose experiments regarding their representativeness, visibility, and data quality (Section 3.4);
• We discuss possible application scenarios for emotion recognition extracted from published works and how each dataset can impact positively and negatively according to its features (Section 4).

This survey also differs from other published surveys in emotion recognition, such as [21,22,23,24,25], since we focus on listing, understanding, and discussing datasets and the permissions and limitations they bring to techniques, while surveys usually focus on listing techniques, their limitations, specifications, and results on benchmarks. To the best of the authors’ knowledge, this work is the first survey that focuses on datasets instead of techniques.

2. Perceived Emotion in Humans

Although emotion recognition is a multidisciplinary field of study, it is currently highly dependent on the advances made from research on cognitive psychology. With the rise of this field in the 1950s, theories based on cognitive approaches have gained significant appeal by connecting sentiment and behavior [26]. Current research also proposes a significant amount of sub-fields that study specific behaviors which we can correlate with emotion, such as kinesics [27], haptics (or tactile) communication [28], and proxemics [29]. Undoubtedly, the advances in these fields will directly contribute to new, exciting approaches to emotion recognition using deep learning. Furthermore, given how emotion recognition is a cognitive task, we can develop models that could try to imitate at some level the way that humans perceive emotion.

However, there are already many discoveries and discussions in the literature. Darwin, for instance, in his work The expression of the emotions in man and animals [30], studied how our emotions influence our body movements. This publication is not new but instead goes back one and a half centuries. Other significant researchers, such as Ekman [31] and Wallbott [4,32], have also contributed profoundly to the perception of this field. Therefore, we hypothesize that the limiting factor for the advance of this technology is not related to a lack of computational power or the availability of state-of-the-art deep learning models; but the lack of quality data that researchers could use to train such models. Designing and generating a dataset is expensive, and this factor could be even more impactful for more specific tasks, which is the case for emotion recognition and general fields that need to create descriptions for concepts that are natural and innate to humans.

2.1. The Portrayal of Emotion

The portrayal of emotion comes naturally to humans due to the connection between personality, mood, sentiment, and physical behavior. We communicate our emotions in two primary forms: verbal communication, when one uses words to communicate feelings, and nonverbal communication, in which we give cues of how we feel on different channels and at different intensities. The latter can constitute up to 50% of what we are communicating [3].

Darwin’s work [30] proposed a view regarding the role of body movements and facial expressions in nonverbal communication. Pride, for instance, makes a person exhibit a sense of superiority by erecting their head and body, making them appear as large as possible, as if they were puffed up with pride. In a later study, ref. [33] showed that sighted, blind, and even congenitally blind individuals from multiple cultures portrayed similar behaviors in response to success and failure situations at the Olympic or Paralympic Games. Therefore, this study shows that these expressions are not simply stereotypes mimicked between people; rather, this behavior may be biologically innate to humans. In

Table 1. Body movements observed with emotions [30,32].

Emotion	Description of Body Movements
Joy	Various purposeless movements, jumping, dancing for joy, clapping of hands, stamping while laughing, head nods to and fro, during excessive laughter the whole body is thrown backward and shakes or is almost convulsed, body held erect and head upright (pp. 76, 196, 197, 200, 206, 210, 214)
Sadness	Motionless, passive, head hangs on contracted chest (p. 176)
Pride	Head and body held erect (p. 263)
Shame	Turning away the whole body, more especially the face, avert, bend down, awkward, nervous movements (pp. 320, 328, 329)
Fear	Head sinks between shoulders, motionless or crouches down (pp. 280, 290), convulsive movements, hands alternately clenched and opened with twitching movement, arms thrown wildly over the head, the whole body often turned away or shrinks, arms violently protruded as if to push away, raising both shoulders with the bent arms pressed closely against sides or chest (pp. 291, 305)
Anger/rage	Whole body trembles, intend to push or strike violently away, inanimate objects struck or dashed to the ground, gestures become purposeless or frantic, pacing up and down, shaking fist, head erect, chest well expanded, feet planted firmly on the ground, one or both elbows squared or arms rigidly suspended by the sides, fists are clenched, shoulders squared (pp. 74, 239, 243, 245, 271, 361)
Disgust	Gestures as if to push away or to guard oneself, spitting, arms pressed close to the sides, shoulders raised as when horror is experienced (pp. 257, 260)
Contempt	Turning away of the whole body, snapping one’s fingers (pp. 254, 255, 256)

, we present other examples of correlations between body movements and emotions as proposed by Darwin in his work.

Other works over the years have investigated different approaches related to body language. Ref. [4] proposed an experiment to understand which behavioral cues related to body movement and language people would use to judge emotions from six professional actors hired to play two scenarios. From this study, the judges employed the number of hand movements, head orientation, and behavior of movements, such as an energetic movement, to classify the emotion. Later, ref. [32] proposed another experiment to demonstrate that some movements and postures are, to some degree, specific to certain emotions and can be explained by the intensity of these actions. Their results agreed with Darwin’s evaluation, pointing out that, for example, people feeling ashamed or sad usually have a more collapsed body posture.

Among various other researchers, Ekman’s work to describe facial expressions and their correlation with emotion has been widely accepted and employed in multiple levels for different works descending from affective computing. One of the categories of expressions is called momentary expressions because the information they convey can be captured instantly. This means that one could take a snapshot at any point when the expression is at its apex and this snapshot would easily convey that person’s emotion [31]. This is an argument for datasets based on images, for example. Another possible type of expression is one extended in time, in which a sequence of signals provides the necessary cues. Ref. [34] discusses that emotions such as embarrassment, amusement, and shame have a distinct nonverbal display on facial expressions and other cues related to the facial area. For example, embarrassment is marked by gaze aversion, shifty eyes, speech disturbances, face touches, and nervous smiles. This is an argument for datasets based on videos, for example.

However, given how emotions are intrinsically innate to each person, it is difficult to argue which emotion a specific person is feeling. This is why a discussion regarding perceived emotion is more accurate, meaning understanding how that person exposes their emotion naturally or in a posed manner. Therefore, can there be emotions without facial expressions? As Ekman [31] discussed, individuals may not show any visible evidence of emotion in the face. Tassinary and Cacioppo Tassinary and Cacioppo [35] showed in their work that by employing surface electromyography, they could record slight electrical variations in the face. This means that even when someone does not display emotions in their face, they do so in an unobservable manner.

People can also try to fabricate expressions when they do not feel any emotion, in an attempt to mislead the observer. Evidence suggests that emotions such as enjoyment, anger, fear, and sadness contain muscular actions that most people are unable to perform voluntarily, meaning that, although this is possible, it is difficult to do so in a convincing manner [31,36]. This can be expanded into other aspects of behavior that one may want to disguise. For example, suppose one deliberately wants to hide their visible emotion by forcing expressions on their face, in that case, cues from the felt emotion could be perceived from other sources, especially body language [37].

2.2. Association between Nonverbal Cues

Cues are presented simultaneously, and humans rely on multiple cues to decode emotion. Therefore, instead of designing an approach that focuses on a specific cue, looking at the entire scene may be more informative, including the context in which the person is placed. The context has a direct influence on emotion, and the amount of influence is usually divided into three broad categories by the literature on psychology, discerning personal features, such as demographics and traits, situational features, such as goals and social norms, and cultural features, such as interdependence and nation wealth [38].

Therefore, situational features should be encoded in datasets, given that their reflection of the situation will impact someone’s emotions. The current placement of the person will have a direct impact on their emotions. Examples of placements that could positively impact emotions could be parties, public parks, and natural landscapes, while obscure, dark scenarios could indicate negative emotions. In addition, the presence of other people in the scene could be used as a cue for perceiving emotion. For example, in samples with group interaction, the individual emotions of the group members influence each other [39]. Studies show that perceivers automatically encode such contextual information to make a judgment [6], and that altering this background information could lead to an altered perception of the same facial expression [40].

3. Datasets used in State-of-the-Art Techniques

In this section, we will introduce the datasets currently employed as benchmarks for state-of-the-art techniques. Rather than providing an exhaustive overview of past datasets for emotion recognition, we focus on the recent efforts currently being employed and evaluated in the state-of-the-art techniques. We show samples from these datasets in Figure 1 and give an overview of these datasets in Table 2. As discussed previously, due to the importance of context and other nonverbal cues for perceiving emotion, we will focus our efforts on datasets that explore such features or have them available, discarding, for instance, datasets for facial expression recognition (FER) in this evaluation. Therefore, datasets such as AffectNet [41], the facial expression recognition 2013 (FER2013) dataset [42], and the extended Cohn–Kanade dataset (CK+) [43] will not be considered in this evaluation, neither will techniques that evaluate solely using these datasets, such as the technique proposed by Savchenko et al. [44], the work by Kollias and Zafeiriou [45], distract your attention (DAN) [46], exploiting emotional dependencies with graph convolutional networks for facial expression recognition (Emotion-GCN) [47], the model proposed by Ryumina et al. [48], the model proposed by Aouayeb et al. [49], frame attention networks (FAN) [50], and many others that place themselves as benchmarks for FER.

The Acted Facial Expressions in the Wild (AfeW) dataset [51] was proposed to tackle the limitation imposed by the lack of data from real-world scenarios. At the time of publication, datasets for this task were mainly composed of images recorded in laboratory scenarios with posed expressions. The authors argue that, ideally a dataset for this task would be recorded with spontaneous expressions and in real-world environments, which continues to be difficult today. Therefore, they propose using scenes extracted from movies, that contain environments close to the real world. We show samples of this dataset in Figure 1a.

The authors extracted clips from 54 movies containing subtitles, which were later parsed for expression-related keywords that could assist the labeler. Two labelers annotated each sample with an expression. For each person present in the scene, the labels also contained information on the head pose, age of character, gender, and expression of the person. A static subset, SfeW, was also generated from this dataset. They generated 1426 video clips with sequence lengths from 300 to 5400 ms and with 330 subjects (performers). Later, the AfeW was extended to the AfeW-VA dataset [52], which labeled emotion using a continuous model. We will discuss these annotations in Section refsub:cont.

Table 2. Overview of the datasets commonly used in the state-of-the-art techniques. Each row contains a different dataset. VAD or VA are continuous annotation models. For the demographics, M and F stand for male and female, respectively, while A, T, and C stand for adult, teenager, and children.

Dataset Name	Annotation Formats	Annotators	Samples	Demographics
EMOTIC [53]	Categorical—26 classes Continuous—VAD model	AMT 2	18,316 images	23,788 people—66% M and 34% F; 78% A, 11% T, and 11% C
EMOTIC 1 [16]	Categorical—26 classes Continuous—VAD model	AMT 2	23,571 images	34,320 people—66% M and 34% F; 83% A, 7% T, and 10% C
CAER and CAER-S [17]	Categorical—6 classes + neutral	6 annotators	13,201 video clips 70,000 images	Unknown
iMiGUE [54]	Categorical—2 classes	5 (trained) annotators	359 video clips	72 adults—50% M and 50% F
AfeW [51]	Categorical—6 classes + neutral	2 annotators	1426 video clips	330 subjects aged from 1 to 70 years
AfeW-VA [52]	Continuous—VA model	2 (trained) annotators	600 video clips	240 subjects aged from 8 to 76 years 52% female
BoLD [55]	Categorical—26 classes Continuous—VAD model	AMT 2	26,164 video clips	48,037 instances—71% M and 29% F; 90.7% A, 6.9% T, and 2.5% C
GroupWalk [56]	Categorical—3 classes + neutral	10 annotators	45 video clips	3544 people

¹ A second version of EMOTIC was published in 2019. ² Amazon Mechanical Turk.

Table 2. Overview of the datasets commonly used in the state-of-the-art techniques. Each row contains a different dataset. VAD or VA are continuous annotation models. For the demographics, M and F stand for male and female, respectively, while A, T, and C stand for adult, teenager, and children.

Dataset Name	Annotation Formats	Annotators	Samples	Demographics
EMOTIC [53]	Categorical—26 classes Continuous—VAD model	AMT 2	18,316 images	23,788 people—66% M and 34% F; 78% A, 11% T, and 11% C
EMOTIC 1 [16]	Categorical—26 classes Continuous—VAD model	AMT 2	23,571 images	34,320 people—66% M and 34% F; 83% A, 7% T, and 10% C
CAER and CAER-S [17]	Categorical—6 classes + neutral	6 annotators	13,201 video clips 70,000 images	Unknown
iMiGUE [54]	Categorical—2 classes	5 (trained) annotators	359 video clips	72 adults—50% M and 50% F
AfeW [51]	Categorical—6 classes + neutral	2 annotators	1426 video clips	330 subjects aged from 1 to 70 years
AfeW-VA [52]	Continuous—VA model	2 (trained) annotators	600 video clips	240 subjects aged from 8 to 76 years 52% female
BoLD [55]	Categorical—26 classes Continuous—VAD model	AMT 2	26,164 video clips	48,037 instances—71% M and 29% F; 90.7% A, 6.9% T, and 2.5% C
GroupWalk [56]	Categorical—3 classes + neutral	10 annotators	45 video clips	3544 people

¹ A second version of EMOTIC was published in 2019. ² Amazon Mechanical Turk.

The Emotions in Context (EMOTIC) dataset [53] is a benchmark widely employed in the current literature for emotion recognition. The objective behind the construction of EMOTIC was to gather images and subjects in their natural, unconstrained environments, including contexts. The authors based their collection on well-established datasets such as Microsoft’s common objects in context (MSCOCO) dataset [57] and ADE20K [58], and expanded these samples with images captured using Google Images. The same authors later revisited the dataset in their 2019 work Context based emotion recognition using emotic dataset [16]. Please note that the latter version is discussed in this work. In addition, we show samples of this dataset in Figure 1b.

EMOTIC is a dataset widely used today in the literature due to its proximity to an in-the-wild setting. The authors used the Amazon Mechanical Turk service to bring annotators and ask about their perceived emotions from people and context. A first round of annotations is proposed, in which a single annotator annotates one image. After this, the test and validation set receives up to two and four extra annotations, leading to three and five annotators for these sets, respectively. The 2017 version contains 18,316 different images containing 23,788 annotated subjects, of which 66% are males and 34% are females, according to the authors. They also annotated the perceived age of the subjects, of which 11% were children, 11% were teenagers, and 78% were adults. The 2019 version contains 23,571 images containing 34,320 annotated subjects, with the same gender division as the 2017 version, but containing 10% children, 7% teenagers, and 83% adults.

The Context-Aware Emotion Recognition (CAER) dataset [17] was proposed to solve limitations perceived by the authors on other in and also in EMOTIC. According to the authors, although EMOTIC contains contextual information, they work on a different aspect and propose a large-scale dataset for context-aware emotion recognition with various context information. We propose an experiment to validate this affirmation later in this work. Another limitation, compared to EMOTIC, is that emotions are highly dynamic, and using images could be a limiting factor for techniques. Therefore, the authors propose the CAER dataset, which contains videos, and the CAER-S dataset, for static images, the most commonly used benchmark between these two.

Another main differing factor between CAER and EMOTIC is the sample source. While EMOTIC focuses on reusing images from other popular datasets and complementing them with images from the web, CAER focuses on extracting images from television shows. This yields questions regarding the expressivity of the emotions in the scenes. The psychology literature tackles the problem of posed emotions, and evidence suggests that they are at least an approximation of what is actually felt. Studies show that we can diminish the impact of these problems by using different actors in the experiments, that are unaware of the task—in this case, the actors were unaware that this footage would be used for the emotion recognition task. Therefore, we can consider that the negative impact is low [4,32,59].

The Body Language Dataset (BoLD) [55] has a different focus from EMOTIC and CAER. While also containing contextual information from background scenes, its focus is on the availability and description of body language, another significant nonverbal cue that the current literature has yet to adopt [19,20]. The authors’ proposal concerns a dataset with crowdsourced emotional information from videos that were validated using experiments with action recognition-based methods and Laban movement analysis features.

The construction process of the dataset is based on three stages related to selection (of clips and time period), annotation of pose (pose estimation and tracking), and perceived emotional features. Given the intersection of these cues for activity recognition, the authors choose clips from the AVA dataset [60] for annotation. Each person receives a unique ID number, which is tracked among frames, and has their emotions annotated. Focusing on the visibility of the body, close-up clips are removed.

The dataset contains 26,164 video clips with 48,037 instances, defined as an identifiable character with landmark tracking in a clip. Each instance is annotated by 5 participants, who annotate 20 samples. This dataset is widely representative with varied demographics, especially ethnicity categories that include sub-represented groups such as Hispanic or Latino, Native Hawaiian or Other Pacific Islander, for example. Although the dataset is promising, it is still not widely used in the literature on emotion recognition, even though it could lead to interesting in-the-wild applications.

The GroupWalk dataset [56] comprises videos recorded with stationary cameras in eight real-world scenarios. A total of 10 annotators annotated 3544 agents with visible faces across all videos. This dataset is still to be widely used in the current literature given its focus on a very specific nonverbal cue.

The Micro-Gesture Understanding and Emotion analysis (iMiGUE) dataset [54] focuses on nonverbal body gestures, specifically microgestures, to understand perceived emotion. The dataset contains samples extracted from post-match press conferences with professional athletes, which contain footage from interviews and questions after sports matches. In these cases, the player has little to no time to prepare for the questions. As a result, the expressions and reactions are usually not posed and have insightful reflections on the innate emotion.

The authors collected 359 videos with 258 wins and 101 losses from post-match grand slam tournament interviews, totaling 2092 min in duration. The dataset is identity-free, meaning that biometric data, such as the face and voice, have been masked. They also focus on ethnic diversity and gender balance. Five annotators were responsible for annotating the data, which contained information from body, head, hand, body+hand, and head+hand.

3.1. Techniques in the State of the Art

Recently, datasets recorded in the wild have gained attention for allowing evaluations in uncontrollable scenarios, such as laboratory environments. The AfeW benchmark [51] was one of the first public databases used for this evaluation scenario and was employed as the main dataset for evaluation at the emotion recognition in the wild (EmotiW) [61] challenges. Subsequently, datasets such as EMOTIC [16,53] and CAER [17] have been published to explore the participation of context in emotion recognition deeply. They are the datasets most commonly used today by the state-of-the-art techniques for evaluation.

In

Table 3. State-of-the-art techniques, published from May 2019 until January 2023, and the datasets they used for evaluation. In case of duplicates, such as [18], both datasets were used in the evaluation.

Approach	Available From	Dataset
Wu et al. [62]	May 2019	EMOTIC
Zhang et al. [63]	August 2019	EMOTIC
Mittal et al. [56]	June 2020	EMOTIC
Thuseethan et al. [64]	September 2021	EMOTIC
Peng et al. [65]	December 2021	EMOTIC
Wu et al. [66]	August 2022	EMOTIC
Yang et al. [67]	October 2022	EMOTIC
Lee et al. [17]	October 2019	CAER
Lee et al. [17]	October 2019	CAER-S
Gao et al. [68]	January 2021	CAER-S
Zhao et al. [69]	May 2021	CAER-S
Zhao et al. [70]	July 2021	CAER-S
Le et al. [18]	December 2021	CAER-S
Le et al. [18]	December 2021	NCAER-S 1
Wu et al. [66]	August 2022	CAER-S
Chen et al. [20]	October 2022	CAER-S
Costa et al. [19]	October 2022	CAER-S
Zhou et al. [71]	January 2023	CAER-S

¹ A variation of CAER-S proposed by these authors.

, we survey techniques from May 2019 to January 2023 and compare the participation of EMOTIC and CAER on the evaluation pipeline for these techniques. Although there are other promising datasets, especially BoLD [55], that could be used for emotion recognition, given that most of the techniques focus only on context, it is justified that EMOTIC and CAER still dominate the evaluation focus in the state-of-the-art techniques. iMiGUE [54] and GroupWalk [56] are also not very common as benchmarks, mainly due to their specificity. AfeW and AfeW-VA are also more common in approaches for FER. In these cases, they are listed here for also containing visible context information.

3.2. Emotion Categories and Annotations

Techniques related to affective computing usually focus on some variation of the well-known set of Ekman’s [72] basic emotions, which are Anger, Fear, Sadness, Enjoyment, Disgust, and Surprise. The same applies to datasets related to emotion recognition and its parent fields of study, which usually add the Neutral emotion.

This set of categorical annotations is the case for the CAER dataset [17], in which videos and images are annotated using Ekman’s basic emotions. A limitation, however, in the format of annotation used in both CAER and CAER-S, is that a single annotation is given for the video or image. This means that in images with multiple people, only one label is given, and no bounding box or identifiable information is provided to identify on which person that annotation was made. This harms techniques such as CAER-Net [17], GLAMOR-Net [18], and other techniques trained on this dataset, because the authors usually use the first face detected in the image, as discussed in their research papers. EmotiRAM [19] employs a face selector algorithm that searches for the leading performer in the scene based on the assumption that the annotation would be from them, but is also limited to images with difficult scenarios. We show examples of images with more than one person framed in Figure 2.

For GroupWalk [56], a completely different set of emotions is proposed. The authors proposed using a new set of seven categories that take into consideration the level of the perceived emotion, which can be Happy, Sad, or Angry, combined with the modifiers Somewhat or Extremely, in addition to Neutral.

The authors proposed a different approach for EMOTIC, by defining an extended list of 26 emotional categories also containing Ekman’s basic emotions, leading to 20 novel emotional states for comprehension, which was also employed in BoLD. To define these emotional categories, the authors proposed an approach based on word connections (affiliations and relevance of words) and interdependence of words (psychological and affective meaning) to form word groupings. However, the main difference is that neither EMOTIC or BoLD contain the Neutral category, with the authors arguing that, generally, at least one category can be applicable, even with low intensity.

However, the lack of a neutral annotation group could lead to neutral images being sampled into opposite groups and therefore harm the learning process of a network. For example, research on social cognition points out that humans perceive emotionally neutral faces depending on visible traits; positive traits are correlated with happiness, and traits involving dominance and threat are correlated with anger [73,74]. Therefore, a person that appears to be emotionally stable would be associated with Happy, while a person that appears to be aggressive would be associated with Angry. In this scenario, given a neutral image, and more specifically one with a neutral face, the action of the annotator would likely depend on their perception of the personality traits of the people present in the image instead of the perceived emotions.

Assessing Annotators’ Agreement

Could a high number of classes impact the agreement between annotators? We hypothesize that the various possibilities of classes could lead to uncertainty in annotation, leading to disagreement among annotators. Therefore, as an example, we investigate the agreement among annotators on the EMOTIC dataset.

The authors also proposed a study to assess the level of agreement among annotators. They employed a quantitative metric known as the Fleiss’s Kappa measure [75], which evaluates the reliability of agreement among a fixed number of raters in assigning categorical ratings. They showed that more than 50% of images had $\kappa >0.30$, indicating that, in these cases, the annotations were better than random. As stated in Fleiss’s work, it is reasonable to interpret the absence of agreement among raters as their inability to distinguish subjects—in this case, different emotions. As is also proposed in Fleiss’s work, the authors did not evaluate the agreement by category.

We proposed an experiment to overview the agreement among annotators. We empirically chose to focus on the test set, given that this is how current state-of-the-art approaches evaluate their results. For this experiment, given a set of images of the test set $I=\{I_1,\dots ,I_n\}$, containing a set of persons $P=\{P_1,\dots ,P_n\}$, we loop each $P_i$ for each $I_i$ and store their annotations if the number of annotations for that person is higher than one. We then count the co-occurrence of each emotion for that sample in pairs and store this information in a co-occurrence matrix. For example, if the annotations for $I_1;P_1$ are Happy, Engagement from annotator 1 and Peace from annotator 2, we would increment the co-occurrences of (Happy, Engagement), (Happy, Peace), (Engagement, Happy), (Engagement, Peace), (Peace, Happy), and (Peace, Engagement) by one. Therefore, we construct a database containing the co-occurrence of the annotations given by all annotators for each image and person. We display a visualization of this data in Figure 3 as a confusion matrix, to allow an easier visualization normalized by emotion. This means that, for a pair of emotions, the value shown indicates the co-occurrence of those emotions. If the value is higher than one, the annotators agreed more with this specific pair instead of agreeing on the emotion. If the value is less than one, they agreed less. Given the dataset’s license, users must provide their copy of EMOTIC’s annotation to reproduce this result.

From this overview, we can see a general agreement between classes. For example, the co-occurrence between Affection and Happiness is $0.99$ (1879 samples), indicating a strong agreement between annotations, as there is between Suffering and Sadness, with a $0.77$ agreement (426 samples). We can also see, however, the odd distribution of Engagement, which, by the authors’ definition, is paying attention to something; absorbed into something; curious; interested [16]. Sensitivity, which is defined as the feeling of being physically or emotionally wounded; feeling delicate or vulnerable shares co-occurrences with Happiness ($0.42$; 101 samples) and Sadness ($0.59$; 142 samples), even with these two having opposite definitions. Finally, we can also see outliers in this data, composed of directly opposite emotions. For example, Disconnection and Engagement ($0.68$; 1365 samples), Fear and Confidence ($0.35$; 104 samples), Fear and Excitement ($0.44$; 131 samples), and Pain and Happiness ($0.23$; 50 samples). A similar experiment was performed by the authors and is published in their reference paper, but using a different approach than ours. In our approach, we look directly at the co-occurrence of each pair of emotions and compute the average based on the number of samples, which reveals insights that were not present in their evaluation. These insights should be used by scientists and engineers when developing their applications to be aware of limitations present in the dataset.

3.3. Continuous Annotations

Some datasets also annotate samples with continuous dimensions instead of only discrete categories. For example, the VAD model [76] is usually employed for emotion recognition. In this model, emotions are placed in a three-dimensional space: valence, arousal, and dominance. The valence (V) axis determines whether an emotion is pleasant or unpleasant to the perceiver, therefore distinguishing between positive and negative emotions. The arousal (A) axis differentiates between active and passive emotions. Finally, the dominance (D) axis represents the control and dominance over the nature of the emotion. Therefore, each emotion can be represented as a linear combination of those three components [77].

The AfeW-VA dataset [52,78] contains annotations of valence and arousal for 600 clips, which were also, in part, used in AfeW, to create an expanded dataset. The values for each axis of the model range within [−10, 10]. However, as exposed in their work, a significant number of the annotations lie around the neutral value for valence and arousal, indicating that a significant part of the dataset contains neutral expressions. A possible reason for this behavior is that the entire dataset was annotated by just two individuals who, although certified on the facial action units coding system, could be suffering from annotator burnout [79].

The EMOTIC dataset also contains annotations using the VAD model, ranging within [1, 10]. However, unlike AfeW-VA, the authors chose to rely on crowdsourced annotations powered by the Amazon Mechanical Turk (AMT) platform, with restrictions to discard annotations that were, in their opinion, not compatible with their discrete annotations. These control images would appear once for every 18 images shown to the annotator. However, as reported in their work, the metrics related to annotation consistency point to disagreement in some cases. For example, the standard deviation for the dominance dimension is $2.12$, which can be considered high given that the values range within [1, 10]. For valence and arousal, the standard deviations are $1.41$ and $0.70$, respectively. Finally, the score distribution for each dimension is higher than AfeW-VA’s, indicating that the perceived emotions are more diverse from the annotators’ points of view. We extract the mean for each annotated emotion’s valence, arousal, and dominance axis on EMOTIC, and we show this visualization in Figure 4. This figure shows that, as expected, we have a cluster of positive and negative emotions. Sadness and Suffering, for example, have almost the same VAD mean. At the same time, Pleasure and Happiness are also close. This visualization also points out annotations that become confused in this form of annotation, such as Peace and Fear.

Finally, the community does not agree that the VAD model is a good approach to representing emotions. It is challenging to represent emotional categories with numbers, Figure 4, given that this is different from how humans naturally perceive emotions, even more so when the emotion needs to be divided into different categories. The representation is also deeply intimate and changes from person to person and culture to culture. In datasets with continuous and categorical annotations, such as EMOTIC, one might employ weights between each annotation format to control their participation in calculating loss, for example.

3.4. Presence of Nonverbal Cues

Except for the AfeW and AfeW-VA datasets, all the other databases surveyed in this work were designed for the presence of nonverbal cues other than facial expressions in their samples. For example, EMOTIC and CAER/CAER-S focus on context; BoLD focuses on body language; GroupWalk focuses on gaits; and iMiGUE focuses on microexpressions of the body, hands, face, and the combination of them. However, it is possible to extract other nonverbal cues from these datasets, even if these were not the focus of their design. We list the presence and visibility of nonverbal cues in

Table 4. Presence and visibility of nonverbal cues in the datasets currently employed by the state-of-the-art techniques. For each dataset (rows), we classify the cues (columns) as missing, somewhat present, or present, or annotated for when the cue is not only visible but annotated by humans.

Dataset Name	Cue
	Facial Expression	Context or Background	Body Language	Others
EMOTIC [16,53]	Somewhat present	Present	Somewhat present	N/A
CAER/CAER-S [17]	Present	Present	Somewhat present	N/A
iMiGUE [54]	Present	Missing	Annotated	Annotated microgestures
AfeW/AfeW-VA [51,52]	Present	Somewhat present	Somewhat present	N/A
BoLD [55]	Present	Present	Annotated	N/A
GroupWalk [56]	Somewhat present	Somewhat present	Somewhat present	Gait is visible but not annotated

.

Given how facial expressions are severely significant for emotion recognition, it is important to have good crops of the face available, which is the case for most of the samples from these datasets. For EMOTIC, however, it is more common to see images with severe facial occlusion, as shown in Figure 5. For the other datasets, since the data were obtained from television clips or movies, it is more common for people to be aligned with the camera. For GroupWalk, given how security cameras and handheld devices are also used to record the clips, it is common that sometimes the faces will be distant from the camera, making it difficult to explore this cue.

3.4.1. Context Variability

Context information is also available in most samples of the datasets selected for evaluation in this survey. Even for datasets such as AfeW and AfeW-VA, which are commonly used for FER, background and scene information is available and can be used to leverage context. However, when proposing the CAER dataset, the authors argue that the list of datasets available until the date of publication (including EMOTIC) did not contain a dataset with multiple context information. When exploring CAER-S, we note that, even though the source of the samples is 79 television shows, a significant number of frames contain repeated background information, which does not occur in EMOTIC at the same significance due to the source of the data.

To validate the variability in background information (context) in EMOTIC and CAER-S, we propose an experiment using open-source code for scene recognition. For this, we use the work by [80], which extracts features from images for classification. Given an image I, we feed it to the pipeline of the proposed network to extract the scene label s, which describes at some level the context from the input image $I_c$. Please note that, although this technique is compatible with the state of the art with a sufficient Top@1 score, our intent is not to quantify the different context information to disclose how many images have a specific background, but rather to understand if these images are different enough to have different classifications among the dataset.

First, we modified the data loaders proposed by the authors, to load images from the CAER-S and EMOTIC datasets separately. After this, we employed the same transforms used in their evaluation to extract comparable information. We decided empirically to use the model pre-trained on the Places 365 dataset [81] given its high data variability. After this, we sampled each image and extracted its Top@1 classification, storing it in a file for further processing. We show the results of this experiment in Figure 6. This graph shows that the classifications given by [80] are more grouped in CAER-S than in EMOTIC, implying that the images in this dataset are less diverse. Please note that, although the datasets contain a different number of samples, our investigation is regarding data distribution. A dataset with images containing diverse backgrounds would be more distributed among this graph, as happens with EMOTIC, independent of the number of samples present in it. Finally, this experiment complements individual observations on the dataset, indicating that CAER and, subsequently, CAER-S do not have higher context variability than EMOTIC.

3.4.2. Body Keypoints Visibility

Another essential cue that is present in these datasets is the body. Even though only specific datasets such as iMiGUE have annotated body language (see Table 3), scientists and engineers can extract insights from this cue by using different types of approaches. Ref. [19], for example, employed a body encoding stream that used a simple model proposed by [82] until its last convolutional layer, allowing the network to correlate the internal representations of this model and emotion. After that, more robust approaches were proposed, which focused on an activity recognition pipeline, such as [20,56]. We propose an experiment to assess the visibility of the body keypoints on images from three datasets: EMOTIC and CAER, which are the main datasets used today in the literature, and also BoLD, given its body language focus.

In this experiment, we loop through every image of the datasets, except for BoLD for which we choose a single random frame from every video. This different approach is motivated by some reasons, such as (a) there is low variability between frames in each video of BoLD and (b) the high number of recorded frames would make this experiment unfeasible due to the high processing time. Please note that, although BoLD has ground-truth body keypoint annotation, we chose not to use it in our evaluation to keep a comparable basis regarding body keypoint visibility.

We use you only look once (YOLO) [83] for each image or video frame to detect the people present on the scene. EMOTIC has ground-truth annotations regarding this aspect. Again, we chose not to use them in our evaluation to keep a comparable basis among datasets. For each detected person, we feed the cropped region to MediaPipe [84] to extract the detected pose. We empirically chose MediaPipe for this experiment due to its capability of also predicting keypoint visibility. Next, we sum the visibility of each of the 33 keypoints for each dataset. Finally, we plot these points using a spatial distribution representing a neutral pose body. For the drawing, we normalized the keypoint visibility regarding the maximum value stored to allow better comparison between points in the same and different datasets. We show the results of this experiment in Figure 7.

From the results of this experiment, we can notice that CAER-S is the dataset with the least visibility regarding the lower body parts. Given how the dataset is built upon video clips from television shows, it is common to have the lead performer centered on the screen and only show the lower body parts when relevant to the story. BoLD, although being focused on body language, also has less visibility for the lower body parts when compared to EMOTIC. Please note that BoLD provides body annotations and that this result may differ when theses annotations are used. Finally, EMOTIC has a well-distributed visibility of the whole body, which is justified by the multiple camera placements present in the dataset. All of the datasets have good visibility for the face, hands, and arms.

4. Discussion

Scientists and engineers currently use all of the datasets listed in this work to develop emotion recognition models and pipelines. However, in which scenarios could techniques trained using these datasets be deployed? Models, no matter how deep or robust, are still limited by the quality and availability of data. Therefore, based on features extracted from these datasets, we propose a discussion regarding application scenarios and gaps that we could expect in these approaches. Finally, before working on the construction and deployment of the technology, one should compare the requirements with the details described in each dataset above. Researchers should also know that these datasets were not built toward a specific requirement. The best practice is to have a dataset to fine-tune techniques for each specific need. We discuss below the possible application scenarios and limitations of each dataset, while also summarizing this discussion in

Table 5. Summary of the suggested application scenarios of each dataset and their possible limitations.

Dataset	Suggested Application Scenarios	Possible Limitations
EMOTIC [16,53]	Images in which context is highly representative and/or highly variable. Content published on social networks. Scenes in which the face can be occluded. Images sampled from cameras aligned with the ground. Applications that require a deeper comprehension of sentiments instead of a direct emotion classification.	Lack of neutral classification may generate noise in some applications. Lack of ethnic representation means that techniques could be biased in some regions
CAER/CAER-S [17]	Images or videos in which the face will mostly be aligned to the camera. Interviews, one-on-one meetings, reactions, or other scenarios where the camera will be mounted at eye level. Feedback for retailers on individual spaces or services. Generic applications in which a direct classification is sufficient. Affective computing on notebooks, smartphones, tablets, or other mobile devices with a front-facing camera.	Prejudicial when the camera has a different viewpoint than the one present in the dataset (for example, top-down). Prejudicial when the person is not directly aligned to the camera. Lack of ethnic representation means that techniques could be biased in some regions.
iMiGUE [54]	Spontaneous scenarios in which subtle cues need to be considered. Interviews, one-on-one meetings, reactions, or other scenarios where the camera will be mounted at eye level. Applications with high reliance on body language and gestures.	By itself, the dataset can be harmful for scenarios in which facial expressions are considered important. Prejudicial when the person is not directly aligned to the camera. No direct emotion representation is present in the dataset.
AfeW/AfeW-VA [51,52]	Scenes with low context contribution. Affective computing on notebooks, smartphones, tablets, or other mobile devices with a front-facing camera. Feedback for retailers on individual spaces or services. Scenes in which subtle facial expressions do not need to be considered	Lack of ethnic representation means that techniques could be biased in some regions. Low image resolution might restrict models in learning subtle cues.
BoLD [55]	Images or videos with representative body language. Applications with a deeper emotion representation related to multiple cues simultaneously. Social interactions. Applications that are expected to work with people from diverse ethnicities. Images sampled from cameras aligned with the ground. Scenes in which the face can be occluded.	Low resolution in some samples could impact learning from facial expressions. Lack of neutral classification may generate noise in some applications.
GroupWalk [56]	Scenarios in which a distant emotional overview is sufficient. Security cameras. High-level descriptions of emotions in a (smart) space. Interactions in which more descriptive emotions are not needed.	Noise from the dataset could be prejudicial to the model. Gait is not always representative of emotion in this viewpoint. Poor presence of other nonverbal cues. Generic emotion categories.

.

4.1. Application Scenarios

Smart environments and spaces. A trending topic in the literature is extracting semantics-related people to improve decision making and extract insights from public and private spaces. Urbanists have been working on managing public spaces and proposing interventions based on how the local community relates to that place. Usually, this is performed in person by observing the region and registering the usage by residents. Researchers have been proposing the usage of pedestrian tracking [85] to understand this relation and plan interventions. This scenario could be improved by understanding emotion. Other research has also been proposed with the objective of correlating emotions and environments [9,10,12]. In these scenarios, surveillance cameras that are already installed could be reused to lower costs, for example. These cameras are usually placed with a top-down view, in which one can see the entire person, but only sometimes can their face be seen with clarity. In these scenarios, one should not depend too much on facial expression recognition but focus on other cues to extract emotion. Context is also not very representative since, in these views, the context is not prone to change and would usually only be the floor in the scene. GroupWalk and BoLD could be suggested for these application scenarios. According to the need for emotional representation, GroupWalk could be used to train or evaluate the model or pipeline, bearing in mind that the image quality could vary between recordings. The body language annotations in BoLD could be useful when a more detailed emotional representation is needed. For example, how groups of people are behaving in that region of the environment.

Another possibility is related to retail or private businesses. Decision makers could place cameras with a more frontal view of people and explore the facial cues, for example. Cameras placed on the cashier and on the store exit could extract information about the user’s experience in their store. For this case, AfeW, CAER, and CAER-S could be employed, given that they will always have a good facial crop of the performers in the set and that the body will only sometimes be evident, and because the context would not vary much. In cases where context variability is expected, EMOTIC could be helpful, given the different types of semantics that can be extracted from it. Furthermore, training on this dataset could help towards a robust model in cases where faces will not be visible.

Affective and assistive robotics. Another topic that has gained interest in the literature is assistive robotics. Currently, we have tasks from competitions in which the robot should follow someone around and help them with some tasks. However, these could also be extended into a more affective sense, allowing the development of affective robots to help people who need special care.

In [86], the authors proposed CuDDLer, a social robot that was designed as a companion for children. The robot can extract a child’s emotions and react accordingly. For example, suppose the child is sad or even crying. In that case, the robot could respond with emotional reactions to help build positive feelings. In this case, affective robots could be powered with multi-cue emotion recognition models that would help them respond better by understanding the context around them. Models for this task could be trained using EMOTIC to allow context encoding and semantics representation [66]. Of course, one cannot expect a child or a person that must be accompanied to look at the robot all of the time, so techniques must be robust to facial occlusion.

Emotion tracking and mental health. Systems that track mood and emotion are more common today, mainly due to the recent discussions regarding the importance of mental health, conditions that have increased recently due to the COVID-19 outbreak, which generated outcomes that could last for years. Although clinicians can give mental support through consultations, patients can sometimes feel uncomfortable exposing their feelings verbally [87]. Telemedicine systems that monitor a patient’s well-being could be expanded to also monitor their moods and emotion over time. EMOTIC could also be employed here, given its high context availability and people who are not always facing the camera, to track mood from photos from the gallery and cameras placed in the person’s house. For example, the techniques could benefit from a high-level context description to correlate house objects with emotions.

4.2. Limitations

Specific tasks. Two of the datasets surveyed in this work (iMiGUE and GroupWalk) were proposed to solve specific tasks. How iMiGUE data was sampled places it close to the optimal scenario of extracting emotion from interviews. However, the simplified category representation used by the authors limits techniques related to the overall perception of emotion, narrowing them down to only “positive” or “negative” representations. The identity-free aspect of the dataset also negatively impacts how techniques could learn to represent these other emotion categories since, for example, the facial expression is hidden for protection. Therefore, the dataset limits the techniques to a specific task, which could not be expanded for similar applications such as assessing comfort in job interviews or evaluating the quality of customer care services, due to these imposed restrictions.

Other limitations are present in the GroupWalk dataset, related to sample quality and image resolution. Although the variation in camera placements is a feature that would add robustness to techniques, in some cases, the camera appears to be handheld. We can notice distortions in the videos, probably caused by stabilization software to remove motion blur. Besides the noise and low image quality in some samples, bounding boxes are also annotated directly to the image. Researchers and engineers have been working with data augmentation and proved that employing techniques such as random crops, rotations, and noise can improve the overall performance of the image. However, when these artifacts are written directly to the image, the accuracy of the models could be harmed.

We still lack datasets for other specific tasks in the literature. For example, for in-the-wild scenarios, a viewpoint from a security camera would be ideal since it would allow the reuse of this footage for emotion recognition. A possibility would be to augment the data to be close to this scenario; however, this is difficult when using 2D data such as images and 2D body landmarks.

Labels. Another limitation in this field of study is related to the labeling of datasets. From this survey, we show that there is at least one significant problem related to the labeling of data that could be harmful to techniques. EMOTIC and BoLD, datasets that employ the annotation format of 26 categories, do not add a neutrality class but rather rely on categories such as Disconnection or Peace that could also be relatable to Neutral. However, two problems arise with this approach: first, as we discussed before, visible traits of the person sampled could lead to a correlation with both positive and negative emotions. Second, the high number of possible categories has a direct impact on annotators’ agreement.

On the other hand, the CAER and CAER-S datasets employ an annotation format of seven categories, one related to neutrality. However, their annotations are not as structured as in EMOTIC, in which one can access individual emotions and demographics for each person in the image. In multiple cases, more than one person is present in the scene, and it is difficult to correlate the annotated sample.

Lack of consistency in labeling. We also show that there is little concordance among datasets related to which classes should be employed, as we show in

Table 6. Emotion categories employed in the datasets reviewed in this work.

EMOTIC and BoLD	CAER and AfeW	iMiGUE	GroupWalk
1. Peace: well being and relaxed; no worry; having positive thoughts or sensations; satisfied 2. Affection: fond feelings; love; tenderness 3. Esteem: feelings of favorable opinion or judgment; respect; admiration; gratefulness 4. Anticipation: state of looking forward; hoping for or getting prepared for possible future events 5. Engagement: paying attention to something; absorbed into something; curious; interested 6. Confidence: feeling of being certain; conviction that an outcome will be favorable; encouraged; proud 7. Happiness: feeling delighted; feeling enjoyment or amusement 8. Pleasure: feeling of delight in the senses 9. Excitement: feeling enthusiasm; stimulated; energetic 10. Surprise: sudden discovery of something unexpected 11. Sympathy: state of sharing others emotions, goals or troubles; supportive; compassionate 12. Doubt/Confusion: difficulty to understand or decide; thinking about different options 13. Disconnection: feeling not interested in the main event of the surroundings; indifferent; bored; distracted 14. Fatigue: weariness; tiredness; sleepy 15. Embarrassment: feeling ashamed or guilty 16. Yearning: strong desire to have something; jealous; envious; lust 17. Disapproval: feeling that something is wrong or reprehensible; contempt; hostile 18. Aversion: feeling disgust, dislike, repulsion; feeling hate 19. Annoyance: bothered by something or someone; irritated; impatient; frustrated 20. Anger: intense displeasure or rage; furious; resentful 21. Sensitivity: feeling of being physically or emotionally wounded; feeling delicate or vulnerable 22. Sadness: feeling unhappy, sorrow, disappointed, or discouraged 23. Disquietment: nervous; worried; upset; anxious; tense; pressured; alarmed 24. Fear: feeling suspicious or afraid of danger, threat, evil or pain; horror 25. Pain: physical suffering 26. Suffering: psychological or emotional pain; distressed; anguished	1. Anger 2. Disgust 3. Fear 4. Happiness (Happy) 5. Sadness (Sad) 6. Surprise 7. Neutral	1. Positive 2. Negative	1. Somewhat happy 2. Extremely happy 3. Somewhat sad 4. Extremely sad 5. Somewhat angry 6. Extremely angry 7. Neutral

. This is a problem because it limits the possible datasets that one may use regarding a specific application scenario. For example, for an application more closely related to iMiGUE or CAER-S, one may not be able to extract representations related to Suffering, which is present in EMOTIC or BoLD.

We may argue that this could be achieved through transfer learning approaches such as fine-tuning the model to be able to generate predictions related to a dataset after learning representations from other datasets. Although there is no domain mismatch in this scenario, the data augmentations that are performed by techniques could be harmful to performance. As [88] discusses, several pre-training factors impact the accuracy of a model, such as data augmentation and the amount of data used in the pre-training. Fine-tuning can also lead to overfitting [89] and catastrophic forgetting [90,91], as well as other problems related to training procedures, such as the investment of resources. Finally, the argument that one may pre-train using dataset X to learn their representations and then fine-tune using dataset Y to achieve that set of categories is not sustained by the current literature, and more experiments are needed in order to validate this hypothesis.

Lack of ontologies to represent emotion. An ontology defines a set of primitives related to a specific domain of knowledge. Although there exist ontologies related to emotions, such as MFOEM [92], they are not used today as a baseline to define the emotion categories and what they represent. For example, Surprise, which MFOEM defines as an emotion caused by encountering unexpected events, can have a positive and a negative modifier regarding the person’s evaluation of the event, something that is not present in datasets today. For EMOTIC and BoLD, we have emotions that are closely related, such as Happiness and Pleasure. However, MFOEM suggests that Pleasure can also be correlated with satisfaction, which could help differentiate between these two classes more deeply. Therefore, by employing ontologies for emotion representation, one could merge the wide range of used label sets and have ground-truth descriptions of them from the literature.

5. Conclusions

In this work, we present an evaluation of datasets for vision-based emotion recognition, focusing on datasets that allow more than one visible cue simultaneously. We propose experiments to explore the data and correlate it with findings from the psychology literature.

From these experiments, among other insights and highlights, we show that the difference between the two datasets most applied in the state-of-the-art techniques is significant, due to the many different features employed in them. The CAER and CAER-S datasets provide a more stable benchmark, with images with faces centered in relation to the camera. In contrast, EMOTIC provides a benchmark that is more unpredictable. Both datasets employ an annotation format that can be prejudicial on some level, according to the planned application scenario. EMOTIC uses 26 categories to represent emotion, which does not include neutrality. In this case, the high number of possible categories could impact annotator agreement and harm some applications. While CAER uses a reduced set of categories, comprised of Ekman’s six emotional categories and a neutrality label, they also lack descriptions regarding to whom that annotation was attached in the scene.

We also evaluate datasets that are less common in the literature, such as BoLD, iMiGUE, and GroupWalk, which are focused on specific cues and that could be very aligned with some application scenarios. Finally, we discuss these scenarios and suggest approaches based on findings from these datasets to help guide future research and deployment.

With this work, we expect to provide a roadmap for upcoming research and experimentation in emotion recognition under real-world conditions. By presenting our proposed investigations and experiments, we hope to assist decision makers in selecting the most suitable dataset for training and validation. Furthermore, deep learning often demands a significant amount of energy, which can have adverse environmental effects. Thus, we anticipate that this guidance will enable informed decisions regarding data quality and availability, mitigating the necessity of training or evaluating using multiple datasets to validate concepts.