Ubiquitous Technologies for Emotion Recognition

Abstract

Emotions play a very important role in how we think and behave. As such, the emotions we feel every day can compel us to act and influence the decisions and plans we make about our lives. Being able to measure, analyze, and better comprehend how or why our emotions may change is thus of much relevance to understand human behavior and its consequences. Despite the great efforts made in the past in the study of human emotions, it is only now with the advent of wearable, mobile, and ubiquitous technologies that we can aim at sensing and recognizing emotions, continuously and in the wild. This Special Issue aims at bringing together the latest experiences, findings, and developments regarding ubiquitous sensing, modeling, and recognition of human emotions.

There is an increased interest in studying facial expressions and how they relate to emotions. Lately, some research has been carried out to enhance in-vehicle interactions for drivers. This increased interest has been mainly focused at increasing road safety such as driver drowsiness, distracted driving, or detecting drivers’ stress. In [1], they proposed a novel contactless method for detecting stress in drivers. This method is based on the extraction of features from thermal infrared imaging, which were obtained in a controlled environment with 10 subjects. In their work, they compared their results with a stress index obtained from electrocardiography (ECG). They used a binary classifier to discriminate among drivers with stress vs. those with no stress, resulting in classification performance with an AUC of 0.80, a sensitivity of 77%, and a specificity of 78%. The results are promising towards identifying driver states that can derive in situations that can be of interest for road safety. In addition to this work, the authors of [2] implemented a system to monitor the driver’s facial expressions to assess health risks such as severe pain episodes while driving. The proposed approach is based on line-segment feature analysis-convolutional recurrent neural network (LFA-CRNN), which had the highest accuracy at approximately 98.92%. The results of this work aim at alerting the driver to risk-related matters while driving.

Furthermore, using facial expressions, Ref. [3] present a novel system for automatically detecting pain in babies using image analysis. They implemented a classifier based on Support Vector Machines (SVM) using the following texture descriptors for pain detection: Local Binary Patterns, Local Ternary Patterns, and Radon Barcodes. Using the Infant COPE database, the authors obtained accuracy around 95%. The approaches of [2,3] both use facial expressions to detect pain, although they use different databases and features. Still, these works provide valuable results that can be applied in systems for increasing and monitoring safety and health.

Image-based emotion recognition is mostly based on the analysis of whole-face images. However, some approaches put the focus on core facial areas (eyes, nose, mouth, etc.), which normally encode most of the affective information. In [4], the authors multi-block deep learning techniques which are trained on core facial areas. They show its application in an app for interview self-management. The same principle is used in [5]. Here, the authors propose the use of convolutional neural networks for the automatic recognition of micro-expressions in real-time. Although the accuracy results are far from optimal, they appear quite promising, especially concerning the time taken to recognize the emotion (a few milliseconds).

Regular RGB cameras are most frequently used in facial expression-based emotion recognition. However, there is a growing bulk of research devoted to using thermal infrared cameras to this very end. In [6], the authors review the application of thermal infrared imaging to determine affective responses ascribed to physiological modulations. Their review outlines the main advantages and challenges of thermal imaging-based affective computing, with a particular focus on its use for human–robot interaction applications.

The automatic recognition of emotions has been largely based, according to the literature, on image or video data. Nonetheless, other types of data are used to identify people’s affective states. For example, Ref. [7] analyzes the audio data generated during phone calls in a call center. The authors use speech emotion recognition to determine the call urgency, giving greater priority to calls featuring emotions such as fear, anger and sadness, and less priority to calls featuring neutral speech and happiness. The results show a significant reduction in waiting time for calls estimated as more urgent, especially those calls presenting emotions of fear and anger.

Electroencephalogram (EEG) signals are another alternate data used for emotion recognition. EEG-based emotion recognition is particularly considered due to its effectiveness compared to body expressions and other physiological signals. In [8], the authors study the use of various statistical techniques, namely logistic regression, Gaussian kernels and Laplacian priors to automatically recognize a set of emotions. They also investigate the critical frequency bands in emotion recognition, concluding a superiority of gamma and beta bands while classifying emotions. The great volume of data generated by EEG approaches makes it particularly interesting to consider using deep learning techniques. In [9], the authors precisely use such techniques with the aim to help determining the attitude of consumers towards a product. The authors prototype their solution with a well-known emotion recognition dataset and they show accuracy results comparable to those obtained via traditional machine learning approaches.