Towards Implementation of Emotional Intelligence in Human–Machine Collaborative Systems

Abstract

Social awareness and relationship management components can be seen as a form of emotional intelligence. In the present work, we propose task-related adaptation on the machine side that accounts for a person’s momentous cognitive and emotional state. We validate the practical significance of the proposed approach in person-specific and person-independent setups. The analysis of results in the person-specific setup shows that the individual optimal performance curves for that person, according to the Yerkes–Dodson law, are displaced. Awareness of these curves allows for automated recognition of specific user profiles, real-time monitoring of the momentous condition, and activating a particular relationship management strategy. This is especially important when a deviation is detected caused by a change in the person’s state of mind under the influence of known or unknown factors.

1. Introduction

The evolution of human–machine collaborative systems undoubtedly demands the creation of advanced human-friendly artificial intelligence (AI) architectures capable of person-specific adaptations, endowing the machines with the ability to adequately interpret human inputs, intentions, actions, and behaviors, allowing them to make decisions that are transparent to humans and also conform to societal and moral constraints. These are required for achieving the acceptability of technology and simultaneously for managing the effectiveness of collaborative systems.

Recent scientific studies in this regard are focusing on the intelligence of the machines, presented by their so-called cognitive component, including the ability to acquire information necessary for the achievement of a specific goal, processing this information through various machine learning (ML) methods, shaping a domain-specific knowledge, and as a result, implementation of respective action. The effectiveness of the ML process, measured through feedback from the machine, forms its individual intelligence quotient (IQ) [1], just like the IQ scoring for human individuals.

Human general intelligence also has a second component, emotional intelligence (EI), formulated and studied first by J. Miller and P. Salovey [2] and additionally developed by D. Goleman [3]. EI describes the ability to understand and manage one’s own emotions and the ability to perceive and interpret the emotional states of surrounding individuals correctly and subsequently to shape proper behavior toward them to achieve successful collaboration, i.e., the degree of EI or the emotional quotient (EQ) is directly related to the ability to collaborate. The contemporary understanding of EI from the perspective of psychology includes four components and the corresponding abilities:

• Self-awareness.
• Self-management.
• Social awareness.
• Relationship management.

From the perspective of computer science and human–machine interaction (HMI), these could be interpreted as the following.

Self-awareness: As the machine does not possess inherent feelings and emotions to be aware of, this component represents the ability of the machine to interpret each situation inherent in the dynamics of the collaboration process: the performance of the system (including the individual performance of the human user, participating in the system); phase/stage of the collaborative task; detected deviations, etc.

Self-management: Self-management relates to the machine’s adaptive abilities in response to the interpreted situation regarding the parameters of the collaborative task.

Social awareness: This component represents the ability of the machine to perceive and correctly interpret the emotional and cognitive states of its human partner during the implementation of a particular task, as well as to assess and forecast the influence of the changes in these human states on the task results.

Relationship management: Relationship management relates to the ability of the machine to adaptively change its behavior to ensure a positive impact on the human user (respectively, to the HMI collaborative system), relevant to their emotional and cognitive dynamics.

The latter two components are of particular interest in the current research. Concerning the social awareness of a machine, we consider its abilities for verbal and non-verbal perception (recognition and interpretation) of the human state through various technologies. Specifically, the verbal category involves natural language understanding and processing based on the recent advances in large language models (LLMs) [4,5,6,7,8,9,10,11], speech-to-text [12] and text-to-speech [13] conversion, translation [14], etc. The non-verbal category concerns cognitive state detection methods based on biosensors and visual and tactile input. The diversity of contemporary technologies providing the means of such awareness on the machine side includes:

• Eye tracking—for attention recognition and state detection, by assessment of some parameters [15], e.g., pupil dilation [16,17], saccade and fixation duration [18];
• Facial expression recognition—for emotional and cognitive state detection [19,20,21];
• Electrocardiography (ECG), electrodermal activity (EDA) [22,23,24,25,26], and skin temperature sensors [27]—for assessment of the human body’s physiological reactions related to stress, anxiety, agitation, cognitive load, etc. It was also shown that the detection of heart rate variability could be based on photoplethysmography (PPG) sensors integrated into wearable devices [28,29], as well as on traditional electrode-based technology integrated into plasters [30] or smart textiles [31,32];
• Electroencephalography (EEG) [33,34,35,36,37,38]—for capturing the cognition processes, even in cases of a lack of behavioral reaction caused by a specific stimulus. The potential of this technology is promising, primarily through the development of brain–computer interfaces (BCIs) [39,40,41,42] in that number those with direct brain connection [43].
• Equipment-extensive technologies for human state assessment, such as functional near-infrared spectroscopy (fNIRS) [44,45], capturing the hemodynamics in different parts of the brain; magnetic resonance imaging (MRI) [46], enabling the creation of a digital twin of a human brain [47]; electromyographic sensors [48,49,50] for detection as well as activation of controlled activities of a particular muscle group.

Despite these and other technological advances, recent research on integrating human emotional and cognitive states in adaptive HMI is still fragmented, and social awareness and especially relationship management, based on this awareness, remain neglected. Accounting for this gap, the reported research offers an advance toward enabling social awareness of machines, which opens the opportunity for integrating relationship management strategies. This article presents recent advances in integrating a multimodal emotion recognition component in the intelligent human–machine interface framework—iHMIfr [51], which enables adaptation on the machine side for the implementation of intelligent performance management functionality. The experimental evaluation in a person-independent and person-specific scenario reported in Section 3 validates the proposed approach.

2. Materials and Methods

2.1. Background

The research reported here builds on the conceptual architecture of the intelligent human–machine interface framework (IHMIfr) [51], the workflow adaptation for intelligent human–machine interfaces [52], and the implemented task-related transformation of a collaborative system [53]. The proposed unified view contributes to creating social awareness and relationship management strategies for machines. The proposed advance, which in this research is seen as the second phase, is focused on including a representative user model based on assessing the human state of mind in consistency with iHMIfr. The concept of the experimental validation was upgraded as a second phase of the task-related adaptation presented in [53].

2.2. Research Design

In the following, we present the conceptual design (Section 2.2.1), the experimental protocol, and the dataset used for its experimental validation (Section 2.2.2, Section 2.2.3 and Section 2.2.4).

2.2.1. Conceptual Design

The collaborative system implemented in the first phase of our research was based on using two components: the application and the decision manager. The simplified workflow concerning the decision for adaptation is shown in Figure 1. At this end, a methodology for task-related ex-post transformation was formulated, including a definition of success criteria based on a particular task, a definition of task parameters (speed, complexity, etc.) thresholds, a description of task adaptation strategy based on an estimation of the current performance of the user, and implementation of the selected strategy in the application manager.

The advance proposed here includes a representative user model that consistently assesses the human state of mind with iHMIfr. The human emotional and cognitive state is then interpreted, and a profiling strategy is applied to achieve personalized adaptation. In the present work, the analysis and interpretation of the momentous cognitive state are based on well-known psychological concepts, including the Yerkes–Dodson law and statistical data processing. The methodology followed at this phase includes the acquisition of multimodal input signals during the implementation of the collaborative task; interpretation and assessment of emotional and cognitive states and their registration and monitoring in the human model; definition of criteria for assessment of the emotional and cognitive state; creation of a personalized profile of the users, based on the selected criteria; design of adaptive strategy, based on the current value of the selected criteria; and implementation of the strategy in the application manager.

2.2.2. Experimental Setup

A purposely developed web-based application was used to acquire user data for experimental validation. The web application is coded in JavaScript v. ES12 as a primary language, and ReactJS v. 18.2.0 is used for the reactive rendering of components. Styling is done with Cascading Style Sheets (CSS) in combination with the Material UI library3. The browser’s local memory is used for real-time cashing, and after the end of the task, all the data are saved in a .csv file format.

The input, as seen in Figure 1, is based on the MorphCast platform v.1.16 [54] through the integration of the SDK offered by the manufacturer. MorphCast provides real-time measurements of various psychological parameters, such as emotional valence, arousal, attention, and basic emotional states. These serve as multimodal input streams for user data modeling and interpretation components.

2.2.3. Dataset

Twenty-one volunteers took part in the experiment. All of them were students or researchers at the Technical University of Varna. All participants were informed in detail about the study’s substance and the data collected throughout the experiment. Each one signed written informed consent for inclusion before initiating their participation. In addition, all of the participants completed a questionnaire including data about their age, gender, handedness (left or right), experience in computer games (rarely/moderately/often), weekly time spent on computer/mobile phone/other gaming devices games (under 10 h/between 10 and 20 h/over 20 h), night sleeping time—usually and the last night (in hours) and self-assessment of the current working capacity/efficiency (low/medium/high).

The data were collected from all 21 participants from the Emotional and Cognitive State Tracking Dataset (ETICS). It consists of three .csv files for each individualization level from the second part of the test and correspondingly three files for the third part (with task adaptation), containing:

• Timestamp/stimuli/answer (user response)/performance/speed/reaction time—from the application manager.
• Timestamp/arousal/valence/attention/angry/disgust/fear/happy/neutral/sad/surprise—from MorphCast platform.
• Data fusion of the above, matched by timestamp in the moment of user response.

In addition, synchronized high-resolution video and audio from the PC camera, capturing the user’s face (and face expressions) and screen capture video from the application (the stimuli of the cognitive test), are available in .mp4 file format. Additional information about our research and the presented dataset can be found on Supplementary Materials.

2.2.4. Experimental Protocol

The web-based application and data acquisition platform (Section 2.2.2) were installed on a computer equipped with a high-resolution camera. Each volunteer was left alone during recordings, except the experiment manager, who remained in the room. Each participant completed the informed consent and the questionnaire (Section 2.2.3).

The experimental protocol consisted of three parts—introduction to the rules, individualization phase, and testing phase. The task that the volunteers had to fulfill represented a cognitive T-load D-back test proposed in [55] and explained in detail in [53]. In brief, the T-load D-back test is a two-component task in which the machine provides the user with stimuli from two characters: letters and numbers. The submission sequence is letter–number–letter–number. The user must respond to the numbers, whether they are even or not, by pressing the keys. Regarding letters, a reaction (again by pressing a key) is necessary only if the current letter is the same as the previous one. The first part introduces the participants to the task’s separate components (allowing them to try and receive feedback about the success) and their combination (i.e., the actual test).

The individualization of the task (i.e., the second part) is tailored so that it is initially adapts to the individual characteristics of the specific person, such as cognitive capacity, emotional involvement, concentration, level of fatigue, etc. Individualization consists of successive blocks of the same number of symbols, with the task speed increasing with each subsequent block. The goal is to detect for each user a certain threshold at which the performance drops below a preliminarily defined success criterion (85% in this case). A score above 85% is considered task completion. Each user’s individualized (in terms of speed) charge is then used to place them in a high-stress (high workload) situation where the probability of errors is high. Task-specific parameters—current stage (level speed), performance (% errors), and response time (in milliseconds)—are considered, as they are necessary in forming the decision for task-related adaptation. During the individualization stage, there is a minimum break of 1 min after each level. The participant can prolong the break at their discretion.

The third part consists of a 15-minute test with a constant task-related adaptation based on the current performance (estimated as the average on each sequence of 10 stimuli) and the selected settings of the application adaptive strategy. The selected, in this case, adaptive process is purely reflective, as follows: when the current performance is below the success criterion (85%), the application manager slows the process (by adding 100 ms between the stimuli); if the performance drops below a critical threshold (50%), the speed is decreased by a double step (200 ms); when the performance is above the success criterion, the speed remains the same; and when the performance is above the criterion for excellent performance (95%), the speed increases with a single step (100 ms) to raise the efficiency of the HMI system.

During the second and third parts of the test, data fusion on the decision level is performed, matching the data from the application manager and the data from the MorphCast platform and synchronizing them by timestamps.

3. Results

The experimental results reported here have been obtained using the RapidMiner platform for data science and machine learning [56].

3.1. Results in Person-Independent Scenario

In this scenario, all the fused .csv files pertaining to the 21 volunteers are united in a common dataset. Some of the statistical parameters calculated in relation to the data attributes are presented in

Table 1. Statistical analysis in the person-independent scenario.

Name	Type	Histogram	Min	Max	Average	Deviation
Av. Perf	Real		0.400	1	0.879	0.116
speed	Integer		600	1600	985.401	176.338
reaction time	Integer		4	1380	612.346	165.508
arousal	Real		−0.775	0.389	−0.391	0.223
valence	Real		−0.866	0.462	−0.311	0.101
attention	Real		0	1	0.495	0.401

.

First, we need to acknowledge that the observed average value in terms of user effectiveness was impressive—0.879 ± 0.116, i.e., 87.9%. The system’s adaptability can explain the high average value of this parameter concerning the task. Depending on the estimation of the current performance (averaged for the last 10 stimuli, according to the adaptive strategy chosen in the case), the time step was dynamic and readjusted. The low standard deviation value of ±0.116 among different volunteers indicates that the system adapted accordingly to each person’s individuality. The mean interstimulus interval (defining the speed of stimuli appearance) is 965.401 ms, and the mean user response time is 612.346 ms, with a variance of 160–170 ms due to individual participant differences.

The average for all participants’ estimated arousal and emotional valence are −0.391 ± 0.223 and −0.311 ± 0.101, respectively. The standard deviation values indicate a certain degree of similarity among different people’s cognitive and emotional states due to their involvement in the same task in a common setup.

Because values of attention tracking were normalized to the interval [0, 1], an average value of about 0.5 was observed, which is in the center of this interval. At the same time, the histogram shows a non-Gaussian distribution of values, with two large clusters around 0 and 1. These extreme values correspond to the momentary values of attention and inattention during the task execution. The intermediate values most probably correspond to transition episodes between the two extremes, and thus are relatively rare.

Of considerable interest in our analysis is the interrelationship between cognitive and emotional states on the one hand and the participant’s performance on the other. In cognitive psychology, this relationship is represented by the Yerkes–Dodson law, shown in Figure 2.

The experimental results have been analyzed in two different ways. In the first, the arousal values were juxtaposed against the cumulative performance following the data processing algorithm:

• Data extraction from all participants in the experiment (the sample).
• Combining the data (in RapidMiner—through the “Append” operator).
• Sorting the data in ascending order concerning the attribute “arousal”.
• Setting the values of the “success” attribute as follows: 1 on success and −1 on failure.
• Creating a new attribute, “cumulative performance”, and applying a cumulative function about the “performance” attribute (taking values −1 and 1).
• Using the cumulative performance rate values to establish arousal dependence.

The result is shown in Figure 3, where the interrelationship between arousal and cumulative performance is in good agreement with the Yerkes–Dodson law. At the same time, the adaptability of the task (through the preset strategy) ensures movement along the curve in the zone of optimal performance and does not let it drop.

The second analysis considers the users’ performance as consisting of two components—their attention and personal abilities to implement a particular task. While the latter depends strongly on the user’s personality (and is expected to show different patterns), the first could represent the cognitive part of the performance, i.e., it should also follow the Yerkes–Dodson law. In Figure 4, we show the obtained relationship between the arousal and attention values.

As shown in the figure, at low arousal values, attention is very close to zero (i.e., absent), then, as arousal increases, it also increases until reaching saturation (values close to 1). The graph is in good agreement with the Yerkes–Dodson law, as the optimal average level of arousal to maintain the maximum degree of attention, based on the considered data set, can be determined in the interval [−0.25; 0.25].

3.2. Results in Person-Specific Scenario

In the person-specific scenario, the log files were processed per participant to extract regularities related to creating adequate user profiles. This included calculating and visualizing specific parameters and their subsequent comparison among the participants. We aimed to investigate the availability of characteristics relevant to factors such as gender, age, acquired skills, etc.

3.2.1. Valence–Arousal Relationship

The relationship between emotional valence and arousal was investigated to analyze the participants’ emotional states. It is presented in Figure 5, and the individual participants are distinguished by color. For comparison, Figure 6 shows a column graph of the average performance rate and the average level of attention, again by participants. When analyzing the data, it was observed that the participants with the highest average levels of attention (id_3, id_6, id_8, id_9, id_11, id_12, id_16, id_18, id_20) are predominantly present in the fourth quadrant of the valence—arousal scale, i.e., they are closer to conditions associated with stress. Participants with low levels of attention (id_1, id_4, id_15) mostly fill the second and third quadrants (probably due to their experience in computer games and acceptance of the test as a similar activity and due to the individual specificity of their perceptions). The routineness of task solving, especially for id_1 and id_4, was also confirmed by the high-performance rates accompanied by the lowest reaction times (Figure 7), with a low degree of dispersion presented by the standard deviation (Figure 8). In the first quadrant, there are relatively few data belonging to participants from all groups, and they are somewhat sporadic.

3.2.2. Arousal–Performance Relationship

As in the case of person-independent analysis, the relationship between emotional arousal and performance is established for the cumulative performance or through the attention component. In the first case, the following algorithm is used:

• Data extraction from all participants in the experiment.
• Sorting of the data of each participant in ascending order regarding the attribute “arousal”.
• Setting the values of the “performance” attribute as follows: 1 on success and −1 on failure.
• Creating a new attribute, “cumulative performance rate”, and applying a cumulative function concerning the “success rate” attribute (accepting values −1 and 1).
• Creating a new attribute “id_” and setting unique values for each participant.
• Combining the data (in RapidMiner—by using the “Append” operator).
• Use of cumulative performance rate values to establish dependence on arousal.

The results obtained by implementing this algorithm are shown in Figure 9.

The analysis of results per participant shows the individual optimal performance curves for that person according to the Yerkes–Dodson law. These curves create opportunities for recognition by the machine of specific profiles of the participants, as well as for real-time monitoring and activating a particular type of strategy when a deviation is detected. A change in the person’s state of mind under the influence of some factor might cause variations that can be detected automatically.

3.2.3. Arousal–Attention Relationship

Another exciting aspect of the participant analysis is related to the arousal–attention relationship. This dependence is presented in Figure 10, with each participant’s data in a different color.

As in the person-independent analysis, the relation between arousal and attention again conformed to the Yerkes–Dodson law. However, a significant scatter of these data is also visible. Presenting the data by participants in the experiment and comparing them with the data from the conducted survey, some exciting regularities were revealed. As shown in Figure 11, a significant portion of the scattered data belongs to two profiles. After analysis of the survey data, it was clarified that these two profiles belong to participants with a dominant left hand.

Although all participants differed significantly in the levels of emotional arousal within the test, most of the observed curves follow a trajectory compliant with the Yerkes–Dodson law. The two profiles of left-handed participants differ significantly, which warrants a machine learning model to be elaborated for recognizing atypical profiles, such as a person’s left-handedness, as one of these atypical manifestations. The availability of potential atypicality linked to conditions such as dyslexia, autism, etc. was not investigated in our study.

4. Conclusions

The present research contributes toward developing an emotion modeling component to enhance machines with emotional intelligence functionality. Our concept builds on multimodal input concerning the user’s momentous cognitive and emotional conditions and the intelligent human–machine interface framework—iHMIfr. The experimental validation confirms that the automated recognition of mental states could help achieve awareness about the human condition in collaborative human–machine interaction.

The detection and profiling of personal traits during the execution of a given task and the corresponding activation of adequate adaptive strategies establish the ground for machine-to-human relationship management. Achieving social awareness and relationship management functionality, accompanied by a task-related adaptation (based on self-awareness and self-management on the machine side), contributes directly towards machine emotional intelligence.