The Effect of Emotional Intelligence on the Accuracy of Facial Expression Recognition in the Valence–Arousal Space

Abstract

Facial expression recognition (FER) plays a pivotal role in affective computing and human–computer interaction by enabling machines to interpret human emotions. However, conventional FER models often overlook individual differences in emotional intelligence (EI), which may significantly influence how emotions are perceived and expressed. This study investigates the effect of EI on facial expression recognition accuracy within the valence–arousal space. Participants were divided into high and low EI groups based on a composite score derived from the Tromsø Social Intelligence Scale and performance-based emotion tasks. Five deep learning models (EfficientNetV2-L/S, MaxViT-B/T, and VGG16) were trained on the AffectNet dataset and evaluated using facial expression data collected from participants. Emotional states were predicted as continuous valence and arousal values, which were then mapped onto discrete emotion categories for interpretability. The results indicated that individuals with higher EI achieved significantly greater recognition accuracy, particularly for emotions requiring contextual understanding (e.g., anger, sadness, and happiness), while fear was better recognized by individuals with lower EI. These findings highlight the role of emotional intelligence in modulating FER performance and suggest that integrating EI-related features into valence–arousal-based models could enhance the adaptiveness of affective computing systems.

1. Introduction

Facial expression recognition (FER) refers to the computational process of automatically identifying human emotions through facial expressions captured from static images or dynamic video sequences. Facial expressions represent one of the most immediate and universally recognized modes of emotional communication. Consequently, FER has become a core component in applications including human–computer interaction (HCI), affective computing, healthcare, and education [1,2].

Recent advances in computer vision and deep learning have significantly enhanced FER performance by enabling models to learn expressive features directly from raw data, surpassing traditional handcrafted approaches [2]. However, most FER systems continue to rely on categorical models that classify expressions into discrete labels such as happiness, anger, or sadness [3,4]. While effective in controlled settings, these models struggle to represent the continuous, dynamic, and subjective nature of human emotions [5,6].

Emotion perception is inherently subjective and varies across individuals due to differences in cultural norms, emotion regulation tendencies, and personal appraisals [7,8,9]. In addition to these psychological mechanisms, Cabanac [10] proposes a hedonic-centered theory of emotion, defining emotions as mental experiences with high intensity and either a pleasant or unpleasant hedonic tone. From this perspective, emotion serves as a shared evaluative mechanism to compare motivations and guide behavior. This definition aligns with the dimensional view of affect and underscores the need for FER systems to move beyond fixed categories and instead adopt dimensional models that reflect emotional nuance.

In this context, researchers have increasingly turned to dimensional emotion models, particularly Russell’s circumplex model of affect [11], which represents emotions along two continuous dimensions: valence (degree of pleasure) and arousal (level of activation). Valence represents the degree of pleasure or displeasure, while arousal reflects the level of activation or energy. These models describe emotions along continuous dimensions, offering a more flexible and detailed approach. For example, Le et al. [12] proposed a model using Label Distribution Learning (LDL) that maps facial expressions onto the valence–arousal space, representing emotional states as probabilistic distributions over the valence–arousal space. Hwooi et al. [13] employed regression models to predict continuous valence–arousal values from facial expressions. Dimensional models allow for more nuanced emotional representation compared to rigid categorical labels and regression-based FER models allow for finer predictions of emotional states and better capture the subjectivity inherent in emotion perception [14].

In addition to dimensionality, individual differences must be considered for more ecologically valid FER systems. A key trait in this regard is emotional intelligence (EI)—commonly defined as the ability to perceive, understand, regulate, and effectively utilize emotions [15]. EI is closely linked to social intelligence (SI)—the capacity to understand and navigate inter-personal relationships effectively [16]. Emotional perception is inherently social because emotions are shaped and regulated within interpersonal contexts, influenced by cultural norms, social roles, and group identities [17,18]. For example, individuals tend to recognize emotions more accurately within their own cultural or in-group contexts [19,20], and higher EI individuals tend to recognize emotions more accurately, even in ambiguous contexts such as masked faces [21]. Salovey and Mayer [22] conceptualize EI as a subset of social intelligence, emphasizing the role of emotional awareness and regulation in guiding thought and behavior. According to Bar-On and Goleman, EI encompasses a range of non-cognitive capabilities, including emotion perception, stress regulation, and interpersonal adaptability, which are not only innate but also trainable [23].

Neuroscientific and psychological studies suggest that individuals with higher EI demonstrate enhanced emotion recognition performance. This is supported by stronger neural integration across socioemotional brain networks, including the amygdala, insula, and prefrontal cortex [24,25,26]. The vmPFC supports the somatic marker hypothesis, the anterior insula contributes to interoceptive and social emotion processing, and the amygdala is essential for recognizing subtle facial cues [27,28,29]. Recent findings further highlight the dynamic interactions among the vmPFC, ACC, and insular cortex in regulating emotion and social perception. Collectively, these regions support interoceptive awareness, social decision-making, and emotional regulation [30].

Despite the growing evidence of EI’s importance, few FER studies have empirically examined how EI modulates emotion recognition performance. Instead, most research focuses on improving FER through model architecture or dataset expansion, often overlooking how personal traits shape emotional perception [12]. This gap highlights the need for FER frameworks that account for individual differences, particularly through EI.

To address this, the present study investigates the relationship between emotional intelligence and facial expression recognition accuracy using a regression-based, dimensional FER framework grounded in Russell’s model. Participants were grouped into high-EI and low-EI cohorts based on their scores from the Tromsø Social Intelligence Scale (TSIS) [31] and a task of emotion recognition and expression. Although the TSIS was originally developed to assess social intelligence, specific questions may be used as a proxy for EI due to its conceptual overlap [32]. These questions measure individuals’ ability to identify and interpret emotional expressions in others, a core component of EI. This ability corresponds closely to the “Perceiving Emotions” component in Mayer and Salovey’s four-branch model and is conceptually related to the empathy and social awareness competencies described in Goleman’s emotional intelligence framework [23]. Crowne [32] emphasized the considerable conceptual overlap between SI and EI in terms of interpersonal sensitivity, decoding of nonverbal emotional cues, and emotional regulation.

To ensure consistency and generalizability in evaluating the effect of emotional intelligence, five widely used FER models—EfficientNetV2-L, EfficientNetV2-S [33], MaxViT-B, MaxViT-T [34], and VGG16 [35]—were trained under identical conditions using the AffectNet dataset [36]. Each model was tested on facial expression data from both low and high EI groups. A regression approach was adopted to estimate valence–arousal values and detect subtle differences in recognition performance between groups.

This study aims to provide empirical insight into how individual emotional traits, particularly EI, influence facial expression recognition. By incorporating personal variation into FER analysis, the research contributes to the development of more adaptive, context-aware affective computing systems.

2. Related Works

2.1. Categorical Emotion Recognition Models

A commonly adopted approach in facial expression recognition (FER) involves the use of categorical models that classify expressions into discrete emotional labels such as happiness, anger, and sadness. Distract your Attention Network (DAN) [3] enhances classification performance by employing spatial and semantic attention mechanisms to selectively focus on relevant facial regions. POSTER++ [4] integrates facial landmark information and temporal attention through a transformer-based architecture, achieving improved classification in dynamic contexts. PACVT [37] incorporates contextual visual features by using a vision transformer that jointly encodes facial expressions and scene context to disambiguate emotional meaning.

While these models achieve high classification accuracy, they are limited by their reliance on fixed emotional categories and struggle to account for the nuanced, continuous, and subjective nature of affect.

2.2. Dimensional Emotion Recognition Models

Recent research in facial expression recognition (FER) has shifted from traditional categorical approaches to dimensional emotion modeling, particularly using the valence–arousal framework. This shift allows for more nuanced and continuous representations of affective states. Several recent studies have proposed deep learning-based approaches aligned with this framework.

Le et al. proposed Label Distribution Learning for Valence–Arousal (LDL-VA) which represents emotional states as label distributions in the valence–arousal space rather than point estimates [12]. Their approach captures the inherent ambiguity and inter-rater variability in emotion annotation and showed robust performance on the AffectNet dataset, achieving higher CCC scores than baseline regression models.

FATAUVA-Net, introduced by Chang et al., employed a multi-task learning approach that combines facial action unit (AU) detection with valence–arousal prediction [38]. By integrating facial muscle activity, the model leverages both local facial features and global emotional context, enhancing prediction accuracy. This approach acknowledges that facial expressions are composed of fine-grained muscle movements and reflects that structure in its architecture.

Hwooi et al. proposed a regression-based approach that estimates valence and arousal by training individual models per discrete emotion class [13]. Their model combines CNN-based feature extraction (e.g., InceptionResNetV2, DenseNet201) with regression networks and maps discrete emotion labels to the continuous emotional space. Notably, it outperformed existing methods on AffectNet and showed strong generalization on the Aff-Wild2 dataset, particularly for valence prediction.

Finally, Circumplex Affect Guided Expression Inference (CAGE) offers an advanced framework for dimensional FER by embedding expressions directly into the valence–arousal space [39]. It integrates contextual embeddings and adversarial training to align the predicted affective space with Russell’s circumplex model. In comparative evaluations, CAGE achieved state-of-the-art performance on AffectNet’s continuous label subset, making it a strong baseline in dimensional affect inference.

Collectively, these models enhance dimensional emotion recognition by incorporating facial structure, temporal dynamics, and distributional learning. However, they generally overlook individual affective traits such as emotional intelligence (EI), which are known to modulate emotion perception. This study addresses this gap by applying a regression-based dimensional model to examine the influence of EI on recognition performance.

2.3. Multimodal Emotion Recognition Approaches

FER systems have also been extended to include multimodal input sources such as physiological signals and contextual information. Soleymani et al. [40] incorporate EEG, eye gaze, and pupillary responses to improve affect prediction during video watching, highlighting the value of combining neural and behavioral signals to account for emotion’s subjective nature. The ABAW Challenge [41] establishes a comprehensive benchmark that combines valence–arousal estimation, expression recognition, and action unit detection in a multi-task framework. These efforts emphasize the importance of analyzing emotions across multiple affective dimensions and modalities. One study [42] further integrates facial expression features with physiological signals—namely video-derived iPPG and heart rate variability (HRV)—using a 3D-SE-Xception network and MTTS-CAN.

While this approach achieved improved recognition performance compared to unimodal models, it suffered from synchronization challenges across modalities and performance degradation when multiple signals were combined.

While recent FER studies have advanced emotion recognition through dimensional modeling and multimodal integration, they largely overlook individual differences such as emotional intelligence (EI). Given EI’s documented role in emotion perception, its absence limits the adaptability of current models. This study addresses that gap by examining how EI influences dimensional FER accuracy, aiming to inform the development of more personalized and context-aware affective systems.

3. Materials and Methods

3.1. Dataset

AffectNet is one of the most comprehensive datasets for facial expression recognition (FER), containing over one million images collected from the web [36]. It includes both discrete emotion labels (e.g., happiness, sadness, and anger) and continuous valence–arousal annotations, making it a widely used benchmark for emotion analysis models. The dataset was designed to address the limitations of previous FER datasets, which often lacked diversity and contained a limited number of annotated expressions. AffectNet provides a large-scale, in-the-wild dataset that captures a wide variety of facial expressions across different ethnicities, lighting conditions, and head poses.

In the context of this study, AffectNet was chosen as the training dataset for deep learning models due to its high-quality annotations and comprehensive coverage of facial emotions. Specifically, a refined subset called AffectNet-8 was used, which includes only the samples with valid valence–arousal labels. The dataset was divided into training and validation sets to support model development and evaluation.

For model evaluation, the median valence and arousal values of each discrete emotion category were calculated from the training set and used as reference points for assessing prediction error (Figure 1). As contentment is not included in the AffectNet dataset, its reference coordinates were derived from prior literature, specifically based on Figs. 2 and 8 from the AffectNet study. These coordinates served as the ground truth for evaluating the model’s ability to estimate emotional states along the valence–arousal continuum.

In addition, Figure 2 presents representative facial images corresponding to each of the eight emotion categories used in this study, illustrating the main visual differences across classes.

3.2. Model Architecture

This study employed five widely used deep learning models to evaluate facial expression recognition (FER) performance in the valence–arousal space: EfficientNetV2-L, EfficientNetV2-S [33], MaxViT-B, MaxViT-T [34], and VGG16 [35].

EfficientNetV2 is a convolutional neural network (CNN) architecture that enhances training speed, accuracy, and efficiency by using a compound scaling method to jointly optimize the model’s depth, width, and resolution. It also integrates fused-MBConv blocks for faster convergence and better performance on small-scale features. The large variant (EfficientNetV2-L) offers high accuracy on large datasets such as AffectNet, while the small variant (EfficientNetV2-S) provides a lightweight, faster alternative with minimal compromise in accuracy. By comparing these two, the study examines whether EI-based performance differences are consistent across model sizes.

MaxViT is a hybrid vision transformer that combines convolutional blocks with multi-axis attention, enabling efficient modeling of both local and global feature dependencies. It introduces a block-wise structure that applies both grid-based and dilated attention to balance spatial precision and long-range context understanding. The base version (MaxViT-B) is designed for high-capacity learning, while MaxViT-T is a compact version aimed at faster inference with reduced resource requirements.

VGG16 is a classical CNN architecture composed of 13 convolutional layers and three fully connected layers, using uniform 3 × 3 kernels and a simple sequential design. Although it lacks the efficiency of more modern architectures, it remains a reliable benchmark due to its strong feature representation capabilities. In this study, VGG16 serves as a baseline for evaluating whether the effect of emotional intelligence on model performance is consistent across different network architectures.

All models were initialized with ImageNet [43] pre-trained weights and fine-tuned on the AffectNet dataset [36]. To ensure fair comparison, they were trained under identical conditions, including input size, optimizer, batch size, and loss function. Each model outputs two continuous values corresponding to valence and arousal for a given facial image.

3.3. Reference Framework

Circumplex Affect Guided Expression Inference (CAGE) is a facial expression recognition model that incorporates contextual and affective cues within Russell’s circumplex model of affect [39]. Rather than treating emotions as discrete classes, CAGE projects expressions into a continuous two-dimensional valence–arousal space. It enhances dimensional emotion recognition by integrating an adversarial loss framework and affective priors that preserve the semantic structure of emotional states.

This model achieved state-of-the-art performance on the AffectNet dataset in valence–arousal regression tasks, particularly for in-the-wild facial images. Its robust mapping between facial features and dimensional affective coordinates makes it a valuable reference for studies adopting continuous emotion modeling.

In this study, CAGE serves as a reference model for analyzing how valence–arousal-based inferences can be influenced by individual differences in EI. Given that EI affects how individuals interpret and express emotions, the incorporation of a model that aligns with the circumplex theory of affect allows for a deeper exploration of how EI modulates affect perception. CAGE’s framework also provides a structured way to assess whether EI levels contribute to systematic biases in facial emotion recognition.

3.4. Participants and EI Grouping

A total of 34 participants (age range: 20–52 years; mean age = 32.85, SD = 8.89) took part in the study. The sample comprised 14 males and 20 females. Participants were recruited through public advertisements and volunteered to participate in the study. Inclusion criteria required that participants had no known history or family history of facial muscular disorders, no issues with vocabulary comprehension, and corrected visual acuity of 0.6 or higher. There were no restrictions regarding participants’ age or educational background. All participants provided informed consent and received a nominal compensation upon completing the experiment.

3.5. Experimental Task and Procedure

The experiment was conducted remotely using the Social EQ mobile application developed by Emotionist, Inc. (Seoul, Republic of Korea) [44]. All tasks were completed by participants in their own living environments using personal smartphones. The procedure consisted of three sequential stages: emotional intelligence (EI) assessment, emotion recognition, and emotion expression.

Step 1: Emotional Intelligence Assessment

Participants first completed the Tromsø Social Intelligence Scale (TSIS), a self-report questionnaire designed to measure social and emotional sensitivity [31]. The questionnaire was administered through the mobile app and included items rated on a 7-point Likert scale (1: strongly disagree to 7: strongly agree). Example items include the following:

• I am often surprised by the reactions of others to my behavior.
• When I express my thoughts, people often respond with anger or irritation.
• I sometimes realize that people are unpredictable.
• I am often surprised by the behavior of others.
• I inadvertently hurt someone’s feelings.
• I sometimes find it hard to understand the choices others make.

Step 2: Emotion Recognition Task

Participants were presented with images depicting facial expressions on their smartphone screens and asked to select the corresponding emotion from a predefined list. The task consisted of seven items, with each item displayed for a maximum of 5 s. The emotion recognition (perception) score was calculated as the percentage of correct responses out of the total items.

Step 3: Emotion Expression Task

Participants were instructed to mimic facial expressions shown in example images as accurately as possible. Each trial lasted up to 10 s, during which participants prepared and held their expression. At the end of each trial, the smartphone’s front-facing camera automatically captured the participant’s face. These images were later preprocessed and used as the input for deep learning models to predict valence and arousal values. The emotion expression score was computed by comparing the predicted emotion label, inferred by the app’s built-in emotion recognition model, with the intended target emotion.

Each of the three components—TSIS, emotion recognition, and emotion expression—was rescaled to a 100-point scale, resulting in a maximum total EI score of 300. This composite score served as the basis for subsequent analysis of group differences in facial expression recognition performance. All responses and captured images were anonymized and securely stored for subsequent analysis. The overall flow of the three-stage experiment, including data collection and model-based emotion recognition, is illustrated in Figure 3.

3.6. Evaluation Metrics

To evaluate model performance in predicting continuous emotional states, this study utilized a set of regression-based evaluation metrics. Five evaluation metrics were used to quantify prediction accuracy and agreement between predicted and reference values: mean squared error (MSE), root mean squared error (RMSE), concordance correlation coefficient (CCC), Pearson correlation coefficient (Corr), and sign agreement metric (SAGR). In addition, kernel density estimation (KDE) was performed to analyze the distributional characteristics of the predicted values. From this, both mean log density and peak density were computed to assess the consistency and concentration of emotion predictions. Below are detailed explanations of each metric, along with their mathematical formulations. Mean squared error (MSE) quantifies the average squared difference between predicted and true values. A lower MSE indicates that the model’s predictions are closer to the actual values, making it a widely used loss function for regression tasks.

$$\mathrm{M}\mathrm{S}\mathrm{E}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-{\widehat{y}}_i)}^2$$

(1)

Root mean squared error (RMSE) is the square root of the MSE, making it more interpretable since it is in the same unit as the target variable. Lower RMSE values indicate better predictive performance.

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{(y_i-{\widehat{y}}_i)}^2}$$

(2)

N is the number of samples and $y_i$ is the ground truth value. ${\widehat{y}}_i$ is the predicted value.

Concordance correlation coefficient (CCC) evaluates how well the predicted values agree with the true values by considering both precision and accuracy. It is a robust metric that accounts for scale shifts and biases. A higher CCC value (closer to 1) indicates better agreement between predictions and actual values.

$$\mathrm{C}\mathrm{C}\mathrm{C}={\displaystyle \frac{2\rho {\sigma }_y{\sigma }_{\widehat{y}}}{{\sigma }_y^2+{\sigma }_{\widehat{y}}^2+{({\mu }_y-{\mu }_{\widehat{y}})}^2}}$$

(3)

$\rho$ is the Pearson correlation coefficient between predicted and true values. ${\sigma }_y$ and ${\sigma }_{\widehat{y}}$ are the standard deviations of the true and predicted values, respectively. ${\mu }_y$ and ${\mu }_{\widehat{y}}$ are the means of the true and predicted values.

Correlation coefficient (Corr) measures the strength and direction of the linear relationship between the predicted and true values. It ranges from −1 to +1, where values close to +1 indicate a strong positive relationship, whereas values near −1 indicate a strong negative relationship, and 0 indicates no correlation.

$$\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}={\displaystyle \frac{\sum (y_i-{\mu }_y)({\widehat{y}}_i-{\mu }_{\widehat{y}})}{\sqrt{\sum {(y_i-{\mu }_y)}^2}\sqrt{\sum {({\widehat{y}}_i-{\mu }_{\widehat{y}})}^2}}}$$

(4)

Sign agreement ratio (SAGR) represents the proportion of instances where the predicted and actual values have the same sign. This metric is particularly useful in applications where correctly predicting the direction of change is more important than the magnitude. A higher SAGR value indicates better performance in predicting the correct direction of emotions.

$$\mathrm{S}\mathrm{A}\mathrm{G}\mathrm{R}={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}1\left(sign\right(y_i)=sign({\widehat{y}}_i))$$

(5)

1(⋅) is an indicator function that returns 1 if the specified condition is true and 0 otherwise. $sign\left(y_i\right)$ and $sign\left({\widehat{y}}_i\right)$ represent the signs of the actual and predicted values, respectively.

Mean distance measures the mean absolute distance between predicted and true values. It provides a direct interpretation of prediction deviation in the original scale of the data. A lower mean distance indicates better model performance.

$$\mathrm{M}\mathrm{e}\mathrm{a}\mathrm{n} \mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left|y_i-{\widehat{y}}_i\right|$$

(6)

Kernel density estimation (KDE) is a non-parametric way to estimate the probability density function of a random variable, based on observed data. It is often used to visualize or evaluate how predicted values are distributed in relation to true data distributions. Mean log density quantifies how concentrated the predicted values are within the estimated density function. A higher value indicates that the predictions are more tightly clustered in high-density areas, suggesting greater consistency and confidence in the predictions.

$$\widehat{f}\left(x\right)={\displaystyle \frac{1}{Nh}}\sum _{i=1}^{N}K\left({\displaystyle \frac{x-x_i}{h}}\right)$$

(7)

K(⋅) is the kernel function (typically Gaussian). h is the bandwidth parameter and $x_i$ are the observed data points.

Peak density refers to the highest value of the estimated probability density function of the predicted values. It represents the peak of the distribution and provides a measure of how sharply the predictions are concentrated around a particular value. A higher peak density indicates that the predicted values are strongly clustered around a central mode. $\widehat{f}\left(x\right)$ is the kernel density estimation function.

$$max\widehat{f}\left(x\right)$$

(8)

All metrics were calculated separately for valence and arousal, and performance was averaged across all test samples. These metrics were also used to compare model performance across emotional intelligence (EI) groups to examine whether EI levels influence the accuracy of emotion recognition in continuous affective space.

4. Analysis

This chapter outlines the analytical procedures undertaken to evaluate model performance and assess the influence of emotional intelligence (EI) on facial expression recognition. The analysis was conducted from three perspectives: (1) evaluation of deep learning-based emotion recognition models, (2) statistical grouping based on EI scores, and (3) performance comparison across EI groups.

4.1. Deep Learning-Based Emotion Recognition

The first analysis focused on evaluating the performance of five deep learning models—EfficientNetV2-L, EfficientNetV2-S, MaxViT-B, MaxViT-T, and VGG16—on facial emotion recognition. All models were trained using the AffectNet dataset, and inference was performed on facial expression images collected from 34 participants.

Valence and arousal were predicted independently, following a regression-based approach. For each dimension, five evaluation metrics were computed: mean squared error (MSE), root mean squared error (RMSE), concordance correlation coefficient (CCC), Pearson correlation coefficient (Corr), and sign agreement metric (SAGR). Model performance was assessed both at the individual level and through metrics averaged across all models. These results were also compared with the performance reported in previous studies using the same dataset, to contextualize the effectiveness of the selected models.

4.2. EI Grouping and Statistical Analysis

To investigate the role of emotional intelligence in facial emotion recognition, EI scores were computed for each participant based on three components: the Tromsø Social Intelligence Scale (TSIS), an emotion recognition task, and an emotion expression task. Each component was standardized to a 100-point scale, and the total score was calculated out of 300.

Subsequently, all EI-related scores were standardized and subjected to reliability analysis using Cronbach’s alpha. Dimensionality reduction was performed through principal component analysis (PCA), and participants were grouped into two clusters using k-means clustering (k = 2). An independent sample t-test was conducted to confirm significant differences between the resulting EI groups.

4.3. EI-Based Comparison of Emotion Prediction

To evaluate whether EI levels influenced model performance, the predicted valence and arousal values for each participant were compared against reference values for each target emotion. These reference values were defined as the median valence and arousal coordinates of each emotion category calculated from the training set. For the emotion category contentment, which is not included in the AffectNet dataset, external coordinates from previous literature were used [36].

Group-level comparisons were conducted between high-EI and low-EI participants. Among the performance metrics, MSE and RMSE were statistically compared between groups using independent sample t-tests or Mann–Whitney U-tests, depending on normality. SAGR, being a categorical agreement metric, was analyzed using a chi-square test. CCC and Corr were not subjected to inferential tests, as they represent correlation-based measures; instead, they were compared descriptively through mean value comparisons across groups.

Two approaches were used for aggregating prediction errors: (1) computing the mean error across all emotion categories and (2) analyzing error metrics separately for each emotion category. To further examine distributional consistency in model outputs, kernel density estimation (KDE) was performed. From the KDE curves, mean log density and peak density were computed for each participant to assess the concentration and dispersion of the predicted emotional states.

5. Results

5.1. Deep Learning Recognition Results

Table 1. Comparison of Performance Metrics Between AffectNet, CAGE, and This Study’s Emotion Recognition Results.

Work	AffectNet		CAGE		This Work
Level	Valence	Arousal	Total 1	Total 1	Valence	Arousal
MAE	-	-	0.239	0.235	-	-
MSE	-	-	0.103	0.102	0.097	0.171
RMSE	0.37	0.41	0.321	0.320	0.225	0.299
CCC	0.60	0.34	0.782	0.784	0.780	0.495
Corr	0.66	0.54	-	-	0.807	0.565
SAGR	0.74	0.65	-	-	0.759	0.781

¹ “Total” represents the mean performance metric across valence and arousal.

summarizes the performance metrics of continuous emotion regression models trained on the AffectNet-8 dataset. In the original AffectNet study [36], AlexNet was used to validate the dataset, and its performance was measured using the continuous emotion labels from AffectNet-8. The CAGE study [39] trained EfficientNetV2-S and MaxViT-T on the same dataset and reported their respective performance results. In our study, five different deep learning models, as described in Section 2.3, were also trained using AffectNet-8’s continuous emotion labels, and the mean performance across these models is reported in the table. To improve readability, only the key metrics are included, with model and dataset descriptions provided in the main text.

When evaluated using the same performance metrics as those employed in the AffectNet study, the results of this study demonstrate comparable or slightly superior performance. Additionally, when compared to the results of the two models presented in CAGE, the performance of this study remains largely consistent. However, in the arousal domain, the CCC performance metric is approximately 0.3 lower than that reported in CAGE, whereas it exhibits an improvement of about 0.15 compared to the results from AffectNet.

5.2. Group Classification Based on Emotional Intelligence Scores and Determination of Emotion-Specific True Values

In this study, four EI-related items were used as factors for statistical analysis. The reliability analysis yielded a Cronbach’s alpha of 0.776 (N = 34), indicating acceptable internal consistency. Principal component analysis (PCA) was conducted on the four emotional intelligence-related variables (EI, TSIS, recognition, and expression), and two components were extracted, explaining 86.171% of the total variance. Based on the elbow method, the number of clusters for k-means clustering was set to k = 2, as the inflection point was observed at this value. The clustering algorithm classified all 34 participants into two groups of equal size (N = 17 each) without manual adjustment. One cluster exhibited higher standardized scores across all four variables and was labeled the high EI group, while the other showed lower scores and was labeled the low EI group.

The results of the independent sample tests on the raw emotional intelligence scores for each group are presented in Figure 4. Among the four factors, the emotion recognition score did not satisfy the normality assumption, and the Mann–Whitney U-test was conducted, revealing that the high EI group scored significantly higher (U = 34, z = −3.918, p = 0.000, r = 0.95). For the remaining three factors, normality was met, and t-tests were performed. The results showed that the high EI group had significantly higher scores than the low EI group for EI score (t(32) = 6.603, p = 0.000, d = 1.17), TSIS score (t(32) = 2.855, p = 0.008, d = 0.505), and emotion expression score (t(32) = 3.445, p = 0.002, d = 0.609).

5.3. Performance Comparison of Facial Recognition Models Based on Emotional Intelligence Levels

The predicted emotion coordinates from the test dataset, obtained using five deep learning models, are presented in Figure 5. As described in Section 3, the mean performance metrics for the high EI group and low EI group were computed, and statistical analyses were conducted on the MSE, RMSE, and SAGR metrics. Additionally, when analyzing by emotion category, the mean distance between actual and predicted values, as well as the density distribution, was calculated to comprehensively assess recognition accuracy for each emotion. This approach allowed for a thorough analysis of differences in prediction accuracy and distribution characteristics of facial recognition models based on emotional intelligence levels.

5.3.1. Analysis Results for All Emotions

Table 2. Mean Performance Metrics and Independent Test Results by EI Level for Each Model.

Model 1		Group	MSE						RMSE						CCC	Corr	SAGR
			Mean	SD	U	z	p 2	r	Mean	SD	U	z	p 2	r
Valence + Arousal	L	Low	0.133	0.148	10,638	2.143	0.032	0.18	0.268	0.188	10,704	2.245	0.025	0.19	0.662	0.720	75.7
		High	0.115	0.156					0.229	0.207					0.737	0.760	80.1
	S	Low	0.144	0.158	10,518	1.958	0.050	0.17	0.281	0.192	10,522	1.964	0.050	0.17	0.634	0.689	74.6
		High	0.127	0.170					0.246	0.214					0.720	0.737	80.9
	B	Low	0.144	0.149	10,763	2.336	0.020	0.20	0.290	0.187	10,775	2.354	0.019	0.20	0.615	0.693	73.5
		High	0.121	0.155					0.243	0.208					0.709	0.746	80.5
	T	Low	0.142	0.158	10,316	1.646	0.100	0.14	0.283	0.195	10,327	1.663	0.096	0.14	0.631	0.695	74.6
		High	0.124	0.155					0.248	0.206					0.712	0.740	80.9
	V	Low	0.148	0.158	10,643	2.151	0.032	0.18	0.288	0.197	10,563	0.210	0.043	0.18	0.616	0.680	70.6
		High	0.126	0.162					0.247	0.210					0.702	0.737	78.3
Valence	L	Low	0.102	0.198	10,751	2.317	0.021	0.20	0.232	0.221	10,751	2.317	0.021	0.20	0.759	0.804	74.3
		High	0.068	0.145					0.179	0.189					0.862	0.869	80.1
	S	Low	0.109	0.214	10,296	1.616	0.106	0.14	0.241	0.226	10,296	1.616	0.106	0.14	0.746	0.781	73.5
		High	0.076	0.142					0.199	0.191					0.849	0.850	80.1
	B	Low	0.121	0.214	10,818	2.420	0.016	0.21	0.263	0.229	10,818	2.420	0.016	0.21	0.703	0.755	71.3
		High	0.083	0.153					0.206	0.202					0.821	0.836	78.7
	T	Low	0.120	0.224	10,528	1.973	0.048	0.17	0.256	0.234	10,528	1.973	0.048	0.17	0.708	0.756	73.5
		High	0.085	0.168					0.208	0.206					0.817	0.831	77.9
	V	Low	0.121	0.216	10,629	2.129	0.033	0.18	0.260	0.232	10,629	2.129	0.033	0.18	0.710	0.750	70.6
		High	0.082	0.156					0.208	0.198					0.826	0.836	78.7
Arousal	L	Low	0.163	0.207	10,214	1.489	0.136	0.13	0.304	0.267	10,214	1.489	0.136	0.13	0.503	0.577	77.2
		High	0.163	0.234					0.278	0.294					0.535	0.577	80.1
	S	Low	0.179	0.219	10,127	1.355	0.175	0.12	0.320	0.320	10,127	1.355	0.175	0.12	0.459	0.534	75.7
		High	0.177	0.261					0.294	0.294					0.515	0.551	81.6
	B	Low	0.179	0.219	9991	1.145	0.252	0.10	0.316	0.259	9991	1.145	0.252	0.10	0.457	0.570	75.7
		High	0.177	0.261					0.279	0.287					0.518	0.586	82.4
	T	Low	0.164	0.203	9839	0.911	0.362	0.08	0.309	0.263	9839	0.911	0.362	0.08	0.485	0.568	75.7
		High	0.162	0.223					0.287	0.283					0.532	0.579	83.8
	V	Low	0.174	0.212	9947	1.078	0.281	0.09	0.316	0.274	9947	1.078	0.281	0.09	0.447	0.540	70.6
		High	0.169	0.243					0.285	0.298					0.498	0.566	77.9

¹ Models are abbreviated as follows: L = EfficientNetV2-L, S = EfficientNetV2-S, B = MaxViT-B, T = MaxViT-T, V = VGG16. ² Bold indicates statistically significant differences (p < 0.05).

presents the mean performance metrics and independent test results by emotional intelligence (EI) level for each model. The five deep learning models used in this analysis—EfficientNetV2-L, EfficientNetV2-S, MaxViT-B, MaxViT-T, and VGG16—were trained using the same dataset, AffectNet-8 with continuous emotion labels. In the table, model names are abbreviated for simplicity as L (EfficientNetV2-L), S (EfficientNetV2-S), B (MaxViT-B), T (MaxViT-T), and V (VGG16).

Based on the prediction values from these models, performance metrics were calculated separately for the low and high EI groups. Since the normality assumption was not met for the MSE and RMSE metrics, the Mann–Whitney U-test was applied, while SAGR was analyzed using a chi-square test. The table presents the mean performance metrics computed across both the valence and arousal dimensions.

The key message from Table 2 is that across all five models, participants with higher EI consistently exhibited better prediction performance than those with lower EI. Specifically, the mean MSE and RMSE values were lower in the high EI group, whereas CCC, Corr, and SAGR values were higher. Statistically significant group differences in MSE and RMSE were observed in all models except for the MaxViT-T model. Although the SAGR metric did not reach significance, the high EI group exhibited a mean increase of 8.63%. Additionally, CCC and Corr were, on mean, 13.40% and 7.01% higher, respectively, in the high EI group.

When broken down by emotion dimension, the valence results show significant differences between EI groups in all models except EfficientNetV2-S, with the high EI group outperforming the low EI group across most metrics. For example, the mean CCC and Corr values were 15.81% and 9.81% higher, respectively, in the high EI group. While the SAGR difference was not statistically significant, it still showed an 8.92% improvement.

In contrast, the arousal results did not reveal statistically significant differences for any model. Nonetheless, performance still favored the high EI group, which achieved 1.27% lower MSE, 9.06% lower RMSE, 8.29% higher SAGR, 10.60% higher CCC, and 2.55% higher Corr on mean.

5.3.2. Analysis Results for Individual Emotions

The performance comparison for individual emotions was primarily conducted using MSE and RMSE metrics, as these allowed for statistical testing, while SAGR was excluded due to the absence of statistically significant results in the chi-square analysis.

Figure 6 presents scatter plots of predicted valence–arousal coordinates for each emotion, visualized by five different models. Each emotion is distinguished by color, and brightness is adjusted according to EI level, providing a visual comparison of prediction patterns between the low and high EI groups. The mean distances between the predicted coordinates and the true values, along with the results of independent t-tests comparing the two EI groups, are summarized in

Table 3. Mean Distance Between Predicted Coordinates and True Values for Each Emotion by Emotional Intelligence Level and Results of Independent Tests.

Title	Emotion	Group	Mean	SD	t	df	p	Cohen’s d
Entire model	Neutral	Low	0.191	0.115	2.900	168	0.004	0.45
		High	0.148	0.072
	Surprise	Low	0.208	0.200	3.423	168	0.001	0.53
		High	0.124	0.103
	Disgust	Low	0.498	0.281	2.589	168	0.011	0.40
		High	0.397	0.225
	Fear	Low	0.524	0.208	−4.850	168	0.000	0.75
		High	0.714	0.288
	Anger	Low	0.671	0.260	4.603	168	0.000	0.71
		High	0.457	0.337
	Sadness	Low	0.463	0.235	4.153	168	0.000	0.64
		High	0.317	0.218
	Happiness	Low	0.215	0.202	4.303	168	0.000	0.66
		High	0.104	0.120
	Contentment	Low	0.783	0.092	0.472	168	0.638	0.07
		High	0.776	0.102
Valid model	Fear	Low	0.524	0.208	−4.850	168	0.000	0.75
		High	0.712	0.288
	Anger	Low	0.685	0.264	3.718	100	0.000	0.74
		High	0.455	0.350
	Sadness	Low	0.460	0.235	4.002	134	0.000	0.69
		High	0.304	0.213
	Happiness	Low	0.220	0.195	2.885	66	0.005	0.71
		High	0.106	0.119
	Contentment	Low	0.779	0.102	0.867	32	0.392	0.31
		High	0.754	0.070

Bold indicates statistically significant differences (p < 0.05).

.

Figure 7 displays the kernel density estimation (KDE) plot of the predicted valence–arousal distributions for each emotion, with separated subplots for the two EI groups. Each plot includes two distribution-based metrics: mean log density, representing the overall spread of predicted values (with more negative values indicating broader dispersion), and peak density, representing the maximum local concentration of predictions (with higher values corresponding to higher density in a specific region).

For neutral, the mean distance was 0.191 ± 0.115 for the low EI group and 0.148 ± 0.072 for the high EI group. In surprise, the respective values were 0.208 ± 0.200 and 0.124 ± 0.103. For disgust, 0.498 ± 0.281 and 0.397 ± 0.225. In these three emotions, the mean log density was higher in the low EI group, while the peak density was greater in the high EI group.

For fear, the low EI group had a mean distance of 0.524 ± 0.208, while the high EI group recorded 0.714 ± 0.288. In anger, the values were 0.671 ± 0.260 (low) and 0.457 ± 0.337 (high), and for sadness, 0.463 ± 0.235 (low) and 0.317 ± 0.218 (high). In these three cases, the mean log density was higher in the high EI group, whereas the peak density was higher in the low EI group.

For happiness, the mean distance was 0.215 ± 0.202 for the low EI group and 0.104 ± 0.120 for the high EI group. The mean log density was higher in the low group, whereas the peak density was higher in the high group. In contentment, the mean distance was similar between groups: −0.783 ± 0.092 (low) and 0.776 ± 0.102 (high). The mean log density values were −19.892 (low) and −19.900 (high), and the peak density was slightly higher in the low EI group.

Independent t-tests comparing the mean distances between the two EI groups indicated statistically significant differences for neutral, surprise, disgust, anger, sadness, and happiness, with the high EI group showing lower mean distances from the true values. Conversely, for fear, the low EI group demonstrated lower distances, while contentment showed no statistically significant differences between groups.

Figure 6 presents scatter plots illustrating individual distributions of predicted valence–arousal coordinates, enabling visual comparisons of prediction patterns between the EI groups. Figure 7 emphasizes the density distributions of these predicted coordinates, highlighting the degree of prediction concentration or dispersion. In contrast, Figure 8 specifically visualizes emotions (fear, anger, sadness, happiness, and contentment) that exhibited statistically significant group differences in prediction accuracy as detailed in

Table 4. Summary of statistical comparisons of prediction errors between high and low EI groups across models and emotion categories.

Model 1	Group 2	Valence + Arousal		Valence		Arousal
		MSE	RMSE	MSE	RMSE	MSE	RMSE
L	Low	Fear * t(32) = 2.289, p = 0.029, d = 0.78	Fear * t(32) = 2.334, p = 0.026, d = 0.80
	High	Anger * U = 204, z = 2.049, p = 0.041, r = 0.50	Anger * U = 215, z = 2.428, p = 0.014, r = 0.59		Sadness * t(32) = −2.107, p = 0.043, d = 0.73	Anger * U = 202, z = 1.981, p = 0.049, r = 0.48	Anger * U = 202, z = 1.981, p = 0.049, r = 0.48
S	Low	Fear * t(24.202) = 2.759, p = 0.011, d = 0.95	Fear ** t(24.202) = 2.848, p = 0.009, d = 0.97	Fear ** U = 66, z = −2.704, p = 0.006, r = 0.66	Fear ** t(24.011) = 3.098, p = 0.005, d = 1.06	Fear * t(32) = 2.097, p = 0.044, d = 0.72
	High	Sadness * U = 121, z = 2.325, p = 0.020, r = 0.56	Sadness * t(32) = −2.464, p = 0.019, d = 0.85	Sadness * U = 205, z = 2.084, p = 0.038, r = 0.51	Sadness * U = 205, z = 2.084, p = 0.038, r = 0.51	Sadness * U = 206, z = 2.118, p = 0.034, r = 0.51	Sadness * U = 206, z = 2.118, p = 0.034, r = 0.51
			Anger * U = 204, z = 2.049, p = 0.041, r = 0.50			Contentment * t(32) = −2.207, p = 0.035, d = 0.75	Contentment * t(32) = −2.195, p = 0.036, d = 0.78
B	Low	Fear * t(24.502) = 2.447, p = 0.022, d = 0.84	Fear * t(21.880) = 2.366, p = 0.027, d = 0.81	Fear * U = 86, z = −2.015, p = 0.045, r = 0.49	Fear * U = 86, z = −2.015, p = 0.045, r = 0.49
	High	Sadness * U = 206, z = 2.118, p = 0.034, r = 0.51	Sadness * t(32) = −2.499, p = 0.018, d = 0.85	Sadness * U = 211, z = 2.290, p = 0.022, r = 0.56	Sadness * U = 211, z = 2.290, p = 0.022, r = 0.56
				Happiness * U = 213, z = 2.359, p = 0.018, r = 0.57	Happiness * U = 213, z = 2.359, p = 0.018, r = 0.57
T	Low	Fear * t(32) = 2.188, p = 0.036, d = 0.75	Fear * t(24.776) = 2.160, p = 0.041, d = 0.74	Fear * U = 77, z = −2.325, p = 0.020, r = 0.564	Fear * U = 77, z = −2.325, p = 0.020, r = 0.564
	High
V	Low	Fear * U = 82, z = −2.153, p = 0.031, r = 0.52	Fear ** t(32) = 2.767, p = 0.009, d = 0.95	Fear * U = 83, z = −2.118, p = 0.034, r = 0.51	Fear * t(32) = 2.412, p = 0.022, d = 0.83
	High	Happiness * U = 202, z = 1.981, p = 0.049, r = 0.48	Sadness * U = 204, z = 2.049, p = 0.041, r = 0.50	Happiness * U = 218, z = 2.532, p = 0.011, r = 0.61	Happiness * U = 218, z = 2.532, p = 0.011, r = 0.61	Sadness * U = 217, z = 2.497, p = 0.012, r = 0.43	Sadness * t(32) = −2.678, p = 0.012, d = 0.92
			Anger * t(32) = −2.161, p = 0.038, d = 0.74

¹ Models are abbreviated as follows: L = EfficientNetV2-L, S = EfficientNetV2-S, B = MaxViT-B, T = MaxViT-T, V = VGG16. ² The “Group” column indicates which group had lower prediction errors for each emotion. Asterisks denote significance levels (* p  < 0.05; ** p  <  0.01).

. Table 4 summarizes statistical analyses of prediction errors (MSE and RMSE) across emotions, models, and EI groups. Generally, the high EI group demonstrated lower errors across most emotions. However, fear consistently showed lower errors in the low EI group across all models.

Significant group differences were observed in the mean performance metrics (MSE and RMSE) across valence and arousal dimensions for the emotions fear, sadness, anger, and happiness. Notably, all five models showed significantly lower prediction errors for fear in the low EI group. In contrast, for sadness and anger, the high EI group generally demonstrated better performance, with several models showing statistically significant differences. For happiness, significant results favoring the high EI group in specific models (VGG16 and MaxViT-B).

Dimension-specific analyses indicated statistically significant differences for fear, anger, sadness, happiness, and contentment. In the valence dimension, significant differences emerged for fear, sadness, and happiness, with the low EI group performing better for fear and the high EI group performing better for sadness and happiness. In the arousal dimension, significant differences were observed for fear, sadness, anger, and contentment, with the high EI group performing better in all emotions except fear.

Overall, when averaging the performance metrics across the valence and arousal dimensions, significant differences were observed for fear, sadness, anger, and happiness. In the separate dimension-specific analyses, fear, sadness, and happiness showed significant differences in the valence dimension, while fear, sadness, anger, and contentment exhibited significant differences in the arousal dimension. Notably, fear consistently demonstrated significantly lower errors in the low EI group across all models, whereas anger, sadness, and happiness predominantly showed lower errors in the high EI group.

Figure 8 visualizes the predicted valence–arousal coordinates for emotions with statistically significant differences in Table 4, distinguishing predictions by EI levels. Correspondingly, Table 3 presents the true coordinates, the mean distances between the predicted and true coordinates for both EI groups, and the results of independent t-tests.

Specifically, for fear, the low EI group had a mean distance of 0.524 ± 0.208, significantly smaller than the high EI group’s mean distance of 0.712 ± 0.288. Conversely, for anger, the high EI group exhibited a smaller mean distance (0.455 ± 0.350) compared to the low EI group (0.685 ± 0.264). Similarly in sadness, the high EI group had a lower mean distance (0.304 ± 0.213) relative to the low EI group (0.460 ± 0.235). For these three emotions, Figure 7 indicates that the high EI group had higher mean log densities, whereas the low EI group exhibited greater peak densities. For happiness, the high EI group showed a notably smaller mean distance (0.106 ± 0.119) compared to the low EI group (0.220 ± 0.195). Here, the mean log density was higher in the low EI group, while peak density was greater in the high EI group (refer to Figure 7). In contentment, the distances between groups were similar, with 0.779 ± 0.102 for the low EI group and 0.754 ± 0.070 for the high EI group. KDE metrics were nearly identical between groups, though peak density was slightly higher in the high EI group.

Independent t-tests analyzing the differences in mean distance revealed that for fear, the low EI group had significantly smaller prediction errors. In contrast, for anger, sadness, and happiness, the high EI group made predictions that were significantly closer to the true values. Meanwhile, contentment did not show a statistically significant difference between the two groups in terms of prediction distance.

6. Discussion

In this study, emotional intelligence (EI) was measured using TSIS questionnaire items along with emotion recognition and expression tasks. Participants were grouped based on their EI levels, and differences in emotion recognition accuracy between these groups were analyzed.

6.1. Overall Emotion Recognition

When aggregating prediction results across all emotions, the high EI group demonstrated superior performance compared to the low EI group in terms of the mean valence and arousal metrics, valence-specific performance, and arousal-specific performance (Table 2). This trend was consistently observed across all five models and five performance metrics, supporting the hypothesis that individuals with higher emotional intelligence exhibit greater accuracy in recognizing emotions from facial expressions. However, while MSE and RMSE values exhibited statistically significant differences in most models for valence and the combined valence–arousal metrics, no statistically significant differences were found in arousal-specific performance across any models. These results appear to stem from the inherent characteristics of emotion data.

As illustrated in Figure 5, the distribution of emotion recognition results indicates that data dispersion differs between valence and arousal dimensions. Specifically, while the valence distribution is broadly spread across both groups, the arousal dimension exhibits a more skewed pattern. This may be attributed to the limited representation of the ‘non-aroused’ domain in facial expression datasets collected based on Paul Ekman’s six basic emotions [45,46,47]. Although this study attempted to address this limitation by incorporating the contentment label, the emotion was frequently misclassified as happiness. This outcome may reflect definitional inconsistencies in how contentment is represented across datasets, particularly between AffectNet and the present study. The conceptual ambiguity and data imbalance surrounding contentment likely contributed to limited model performance in this region. A more detailed discussion of these challenges is provided in Section 6.4.

Analysis of the prediction results by emotion category, as shown in Figure 6, indicates that differences in distribution between the high and low EI group were more pronounced in the valence dimension. In most emotions, the peak density point of the high EI group tended to be positioned at more extreme valence values compared to the low EI group. This suggests that perceptions of positive and negative valence may be influenced by individual subjective judgments, and that the interpretation of contextual situations in which emotions are elicited can affect both emotion recognition and expression in the valence dimension [5,8,9,45].

Notably, the high EI group demonstrated a greater ability to accurately interpret the context in which emotions arise, which likely enabled them to assess and express emotions with greater clarity [32,48,49].

6.2. Neutral, Surprise, Disgust

For neutral, surprise, and disgust, no statistically significant differences in prediction errors were observed across all models between EI levels. This suggests that these emotions are less influenced by EI and are generally recognized with similar accuracy across individuals. These emotions are typically elicited by immediate, stimulus-driven responses and are characterized by distinct facial expressions. Because of their visual clarity, they can be accurately decoded without requiring extensive contextual interpretation, thereby reducing the influence of EI on recognition performance.

Neutral faces are usually interpreted as emotionally blank; however, this appearance, while seemingly devoid of emotion, can be perceived as emotionally meaningful depending on the surrounding context. Prior studies have shown that even neutral expressions may carry emotional significance, particularly when presented in emotionally ambiguous or socially charged situations [50,51].

Surprise is associated with clearly identifiable facial features, such as widened eyes and raised eyebrows [52]. According to cognitive evolutionary theory, it arises from a discrepancy between expected and actual events, and this schema-discrepant nature leads to immediate attentional shifts and brief processing delays [53,54]. These reactions are largely automatic, enabling consistent recognition across individuals regardless of EI levels. This likely explains why both groups in this study exhibited tightly clustered prediction distributions around the true values.

In contrast, disgust showed broader prediction distribution patterns across both groups. Although generally evoked by strong aversive stimuli, it is also shaped by individual differences in sensitivity and contextual interpretation [55]. This reflects the subjective nature of disgust elicitors—what one person finds repulsive may not elicit the same emotional response in another. While statistical tests did not reveal significant group-level differences, the high EI group demonstrated smaller mean distances from the true values and higher peak density, suggesting more focused and accurate predictions. Neuroimaging studies have shown that the experience of disgust engages distributed brain systems, particularly the insula and anterior cingulate cortex, which support both visceral reactions and moral evaluations [30,55]. This neural complexity may enable individuals with higher EI, who are better at integrating contextual cues, to interpret disgust expressions more precisely—even in the absence of statistically significant group differences.

In sum, although neutral, surprise, and disgust are not strongly modulated by EI in terms of statistical recognition accuracy, subtle trends in prediction distributions suggest that higher EI may still contribute to more nuanced perception and expression of these emotions. This supports the broader conclusion that emotional intelligence enhances emotion recognition, particularly under conditions of ambiguity or subjectivity.

6.3. Fear, Anger, Sadness

Emotions such as fear, anger, and sadness share high arousal and negative valence characteristics but differ in how they are experienced and interpreted. Their recognition often depends on context and social understanding rather than facial features alone. EI is central to the recognition of these complex emotional states.

Statistical analysis revealed that the high EI group had lower prediction errors and smaller distances from true values across at least four models for all three emotions. They also showed higher log density and more dispersed prediction patterns, suggesting greater sensitivity to contextual cues, supporting previous findings that high EI enhances social–emotional judgment [56,57,58].

For anger, recognition accuracy was significantly higher in the high EI group. Anger typically arises in response to perceived social violations or moral injustice. Peléšková et al. (2024) describe anger as a goal-directed response to social threats or frustrations, requiring cognitive appraisal of intent and fairness [59]. Stemmler et al. (2001) further showed that physiological responses to anger differ based on situational attributions like control and blame. High EI individuals, better at integrating these social and moral cues, are thus more accurate in interpreting anger as a context-dependent emotional signal rather than a mere expression of hostility [60].

Sadness, often tied to loss or empathic concern, is typically expressed more subtly, requiring emotional attunement and perspective-taking for accurate recognition. Ridley et al. (2025) found that individuals with higher empathy—which closely aligns with EI—are more responsive to displays of sadness, even attributing greater trust to sad individuals [61]. Qadeer et al. (2025) identified key empathy-related brain areas—such as the anterior insula and anterior cingulate cortex—as active during sadness perception, suggesting that high EI individuals engage these neural systems to interpret sadness more precisely [62].

Fear showed a contrasting trend. Prediction accuracy was higher in the low EI group, whose results were tightly clustered along the arousal axis. The high EI group’s predictions were more varied across both valence and arousal, indicating a broader interpretation range. This reflects fear’s dual nature—as both an automatic survival response and a socially modulated emotion. LeDoux’s dual-route model explains this with a fast amygdala-based pathway and a slower, cognitive prefrontal route [63]. Pessoa’s network model supports this further [26], proposing that high EI individuals recruit distributed neural networks for more nuanced evaluation of emotional meaning.

In sum, anger and sadness are more accurately recognized by individuals with high EI, due to their strengths in empathy, contextual reasoning, and emotion regulation. Fear, while also complex, may be more readily recognized by low EI individuals through rapid, arousal-based processing. These findings illustrate how EI shapes the way individuals perceive and respond to negative emotions, depending on the cognitive and neural demands of each affective state.

6.4. Happiness, Contentment

Happiness and contentment are emotions classified under high valence levels, and both showed statistically significant differences in mean error between actual and predicted values in only a few models. However, the statistical analysis of mean distance revealed different patterns. For happiness, the high EI group had a significantly smaller mean distance than the low EI group. In contrast, for contentment, no statistically significant group difference was observed, and their mean log and peak densities were also similar.

Happiness is characterized by distinct facial muscle movements, such as upward-curving lips and slight drooping of the eyebrows and outer corners of the eyes. As the only positive emotion among Ekman’s six basic emotions [47,52,64], its clear visual features make it easily recognizable and expressible. Consequently, deep learning models predicted happiness with high accuracy across EI levels.

However, happiness is not solely determined by visible features like smiling. Contextual cues and eye muscle involvement often differentiate genuine from fake smiles [65,66]. This suggests that accurate recognition of happiness still involves social and emotional interpretation. In our results, although models generally performed well for happiness, the high EI group showed more tightly clustered predictions and smaller mean distances, indicating enhanced recognition and expression. This supports the idea that higher EI contributes to more accurate emotional interpretation, particularly for positive emotions in social contexts [65,66].

In contrast, for contentment, only the arousal dimension showed a statistically significant difference, and this was limited to the EfficientNetV2-S model. As illustrated in Figure 8, contentment was distributed within a narrow arousal range, making the statistical results sensitive to minor variations [67]. Given this constrained distribution, it is difficult to draw definitive conclusions regarding EI-related differences in recognition performance.

This limitation may be partly due to the underrepresentation of contentment in the training dataset, as well as inconsistencies in how the emotion is defined across different sources. While AffectNet conceptualizes contentment as a low-arousal state characterized by relaxed facial features such as closed eyes and a neutral mouth [36], other studies describe it as a calm yet positive emotional state located in a region of high valence and moderate relaxed [6,13,46]. Russell’s circumplex model further illustrates variability in the positioning of low-arousal positive emotions like contentment, often overlapping with relaxation or happiness [11].

Previous research has also emphasized that emotion concepts such as contentment are embedded in folk psychological frameworks and are subject to variation across individuals, languages, and cultures [6,19]. These definitional inconsistencies complicate the operationalization of contentment in facial expression datasets and hinder the ability of models to distinguish it from similar emotions like happiness.

Overall, the conceptual ambiguity surrounding contentment may have contributed both to its frequent misclassification and to the lack of statistically significant group-level differences based on emotional intelligence. These findings support the view that contentment, unlike basic emotions, is a more nuanced and culturally shaped emotional state that presents challenges for standardized recognition and computational modeling.

Therefore, it is difficult to draw definitive conclusions regarding the impact of EI on the accuracy of contentment recognition. Among positive valence emotions, happiness appears to be the emotion that exhibits a significant relationship between EI level and emotion recognition accuracy.

7. Conclusions

This study examined how emotional intelligence (EI) influences facial expression recognition (FER), using a composite EI score derived from the Tromsø Social Intelligence Scale and two performance-based tasks. Participants with higher EI demonstrated greater accuracy in recognizing expressions, particularly those requiring contextual understanding such as anger, sadness, and happiness. Interestingly, participants with lower EI performed better in recognizing fear. This may reflect the automatic nature of fear processing, which is closely tied to rapid amygdala responses. Higher-EI individuals may regulate or suppress fear-related cues, whereas lower-EI individuals may exhibit more immediate emotional reactions, leading to greater sensitivity to fear expressions. Fear, anger, and sadness were especially sensitive to EI differences, with happiness serving as a useful supplementary marker. These findings suggest that EI affects not only the outcome of emotion recognition but also the underlying processing strategies. From an application perspective, training emotion recognition models with greater emphasis on these key emotions—especially from individuals with high EI—could improve robustness, particularly when data are limited. The contentment was often misclassified as happiness, likely due to its absence in the AffectNet dataset and the subtle facial distinction between the two. This misclassification underscores the limitations of current FER systems in handling nuanced emotions and highlights the need for a broader emotional taxonomy.

Despite its contributions, this study has several limitations. First, the use of a mobile application-based experimental design allowed for naturalistic data collection but limited experimental control. Environmental variables such as lighting and pose variation may have introduced noise into the facial data. Second, the sample consisted of only 34 Korean adults, limiting the generalizability of the results and potentially introducing cultural bias. Additionally, the estimation of contentment’s emotional coordinates based on external sources may have influenced the model’s interpretability for this emotion. Third, EI was measured using a composite score derived from selected TSIS items and task performance, which may not fully capture the multidimensional structure of EI as defined in the standardized framework. Future research should incorporate standardized EI instruments, multimodal affective data, and more controlled experimental conditions to reduce environmental noise during data collection. Expanding the study population to include diverse racial and cultural groups will also be crucial for validating the consistency and generalizability of the findings. Moreover, utilizing diverse datasets and refining emotion labels—particularly to better capture non-aroused emotional states such as contentment—will contribute to the development of more robust and context-aware FER systems.