Predicting Activity in Brain Areas Associated with Emotion Processing Using Multimodal Behavioral Signals

Abstract

Artificial agents are expected to increasingly interact with humans and to demonstrate multimodal adaptive emotional responses. Such social integration requires both perception and production mechanisms, thus enabling a more realistic approach to emotional alignment than existing systems. Indeed, existing emotion recognition methods rely on behavioral signals, predominantly facial expressions, as well as non-invasive brain recordings, such as Electroencephalograms (EEGs) and functional Magnetic Resonance Imaging (fMRI), to identify humans’ emotions, but accurate labeling remains a challenge. This paper introduces a novel approach examining how behavioral and physiological signals can be used to predict activity in emotion-related regions of the brain. To this end, we propose a multimodal deep learning network that processes two categories of signals recorded alongside brain activity during conversations: two behavioral signals (video and audio) and one physiological signal (blood pulse). Our network enables (1) the prediction of brain activity from these multimodal inputs, and (2) the assessment of our model’s performance depending on the nature of interlocutor (human or robot) and the brain region of interest. Results demonstrate that the proposed architecture outperforms existing models in anterior insula and hypothalamus regions, for interactions with a human or a robot. An ablation study evaluating subsets of input modalities indicates that local brain activity prediction was reduced when one or two modalities are omitted. However, they also revealed that the physiological data (blood pulse) achieve similar levels of predictions alone compared to the full model, further underscoring the importance of somatic markers in the central nervous system’s processing of social emotions.

1. Introduction

The process of emotion recognition requires the development of models capable of predicting emotional states from various types of signals. Some are directly observable, such as audios and/or videos, while others, such as physiological time-series, require more specific recording devices. Deep learning models have been shown to be effective in many cases and have the potential to improve our understanding of emotions and how they are represented in different types of conversational signals [1]. The ability to detect and recognize human emotions is a crucial aspect of many use cases, especially in the context of human–machine interactions. It represents an important research area of affective computing, which focuses on designing and developing computational systems that can recognize and imitate human affective behavior [2].

Emotion recognition has many applications. For example, in education, the use of emotion-aware robots has been shown to be effective in enhancing student engagement and learning outcomes [3]. Similarly, in the field of mental health and cognitive disorders, social robots have been explored as a potential tool for delivering therapy and providing emotional support to individuals suffering from mental health conditions [4]. Also, emotion recognition can improve the effectiveness of human–robot collaboration [5]. Overall, recognizing human emotions is crucial for developing more effective and efficient human–machine interaction systems. Further research in this field is essential to fully understand the potential applications and benefits of emotion recognition technology.

The process of emotion recognition can be improved by incorporating multiple modalities. Combining different signals, such as images, audio, and text, can provide complementary information that can be used to make more informed decisions. Multimodal learning involves relating features from these different modalities to create a shared representation that an artificial neural network can exploit to make intelligent decisions. In general, multimodal learning approaches are capable of more robust inference compared to unimodal approaches [6].

Existing emotion recognition methods predominantly leverage multimodal behavioral signals such as facial expressions, speech patterns, or rely on non-invasive brain recordings like EEG and fMRI to predict emotions [7,8,9,10]. Although these approaches have shown promising results, accurate emotion labeling remains a persistent challenge due to subjective variability and the complexity of emotional experiences. It should be noted though that there is no “neural signature” for single emotions, but accumulated research has led to identifying a subset of brain regions regularly associated with several emotional processes [11,12]. A key limitation in existing research is the lack of a comprehensive framework that examines the interaction between behavioral and physiological signals to influence brain activity within emotion-related regions. To address this gap, we present a novel multimodal deep learning framework that explores how external signals affect brain activity during emotional processing. Specifically, our model processes two categories of signals recorded alongside brain activity during conversations with either a human or a robot: (1) behavioral signals, including video and audio features, and (2) physiological signals, such as blood pulse measurements. Using this multimodal input, our approach aims to achieve two key objectives: predicting brain activity patterns in response to different emotional stimuli and evaluating the impact of the nature of the interlocutor (human or robot). Most emotion detection systems currently depend on audio-visual data and are trained using manually labeled facial expressions as ground truth [7]. However, facial expressions may not always accurately represent real emotions. In this article, we argue that the most reliable approach to approximate real emotions is through brain activity, since objectively measuring emotions presents significant challenges. Recent advancements in brain activity recording technologies can aid in this regard, by utilizing systems based on EEG, fMRI, or Magnetoencephalography (MEG).

Overall, this article presents an investigation of the possibility of predicting activity in emotion-related brain areas as a proxy for emotion recognition. This approach accounts for the complex and dynamic nature of human interaction by combining conversational and physiological signals, as well as considering the temporal aspect of the data. The proposed deep learning architecture is simple yet effective compared to existing multimodal architectures. The main assumption of our work is that brain activation is a more relevant and accurate representation of the emotional states of individuals than classical approaches solely based on visual expressions. Figure 1 illustrates the interaction protocol, the recorded signals, and the approach described in this manuscript. The addition of physiological data, namely, the blood pulse signals recorded with a photoplethysmograph, is particularly important because the autonomous nervous system is considered to be associated with emotional arousal [13]. Therefore, we expect that the use of blood pulse will significantly improve the capacity to predict the response in brain areas associated with emotional processes.

The rest of the paper is organized as follows: Section 2 presents related work. Section 3 describes the proposed methodology. In Section 4, we focus on the experiment and results. Section 5 discusses the results obtained and the implications for affective neuroscience. And Section 6 contains concluding remarks, limitations, and future perspectives relevant to the current work.

2. Related Work

Human emotional expression is inherently multimodal, relying on a complex interplay of physiological changes in addition to verbal and non-verbal cues, such as in changes blood pulse, voice prosody, and facial expressions (e.g., raised cheeks). Each modality contributes uniquely to emotion, making emotion recognition a challenging task for machine learning [14]. In this section, we review related work on multimodal learning for emotion recognition, focusing on deep learning approaches and fusion techniques.

Multimodal learning has shown its effectiveness in tackling traditional machine learning challenges. In particular, it has been widely used to enhance prediction accuracy in classification and regression tasks [15,16]. In recent years, deep learning has been successfully applied to process multimodal data, as well as learn correlations and joint representations across different modalities [17,18]. However, achieving optimal predictive performance requires large-scale datasets [19,20]. This need is particularly important in multimodal prediction tasks, such as emotion recognition, where the effective integration of different modalities is essential. In [21], for example, a large-scale database named ${CAS\left(ME\right)}^3$, comprising visual, voice, and physiological signals, was introduced and processed through deep learning models for multimodal emotion recognition. Integrating data from multiple modalities, as demonstrated in [8,9,10,22], can provide complementary information and significantly enhance emotion recognition performances. However, this integration process poses challenges due to the unique statistical properties and the complex interplay among low-level features associated with each modality [23].

Previous studies have investigated various strategies for merging multiple modalities, which can be categorized into three main approaches: early fusion, which concatenates features from different modalities as input to a machine learning model [24]; model-level fusion, which combines the hidden representations of each modality during training [23]; and late fusion, a method that applies models to each modality and then aggregates prediction scores according to specific criteria [25]. Temporal learning is also valuable for emotion recognition, which involves analyzing time-varying data such as speech and video [26,27]. Multimodal deep learning models can capture this temporal information by employing Recurrent Neural Networks (RNNs) or Transformers, thereby accounting for the dynamic nature of emotions over time.

Several approaches have been proposed in this context, particularly leveraging the model-level strategy. In [15], cross-modal shared representation learning and multimodal fusion achieved effective emotion classification. Additionally, Srivastava and Salakhutdinov [16] introduced Multimodal Deep Boltzmann Machines to learn a joint density model over a space of multimodal inputs, demonstrating the utility of fusing higher-level representations of disparate modalities (images and text) within a deep learning framework. In a detailed model, Krishna and Patil [28] used a Convolutional Neural Network (CNN) with a cross-modal focus for emotion classification. Notably, the model proposed in [27] predicts multiple emotion labels (Anger, Disgust, Fear, Happiness, Neutral, Sad, and Surprise). This model was trained on short emotional video clips sourced from movies and TV shows, combining acoustic and visual information by applying Mel-Frequency Cepstral Coefficients (MFCCs) and DenseFace [29]. It then utilizes a Long Short-Term Memory (LSTM) network with an early fusion strategy, followed by feed-forward layers to make the predictions. Another approach, proposed in [30], uses a multimodal strategy to integrate audio, visual, and text recorded from video elements: speech, faces, and subtitles, respectively. The performance of various feature combinations is assessed using multiple CNN architectures, including Inception and ResNet, for visual feature extraction. Meanwhile, Bidirectional Encoder Representations from Transformers (BERTs) and spectral analysis are applied to textual and audio data, respectively. A fusion process employs Bidirectional LSTM networks to capture temporal dependencies and combine extracted features. Another similar architecture was proposed in [31] to classify four emotions (sad, excited, neutral, and angry). During the feature extraction step, a combination of models, including CNNs, LSTMs, and attention mechanisms, was utilized. Finally, fully connected layers were applied to learn from the fusion of acoustic and textual features and to perform emotion classification.

Other approaches have employed attention-based neural networks for model-level fusion [32,33]. For instance, in [32], the authors integrated multimodal features by assigning attention mechanisms to each modality, yielding promising results for emotion recognition. A noteworthy study by [33] introduced Transformers for multimodal learning, focusing on unaligned multimodal language sequences, and presented an effective method for processing and combining information from diverse modalities. The use of Transformers for multimodal self-supervised learning from video, audio, and text (VATT) was recently investigated in [34], where the VATT model unifies these three modalities within a single network. Each modality is processed separately through the embedding stage of the Transformer, and self-supervised learning enables training on an extensive dataset, resulting in representations applicable to a wide range of tasks, including emotion recognition.

Despite significant advances in emotion recognition using multimodal data and deep learning models, our work draws inspiration from fundamental studies [15,16] that demonstrate effective strategies for multimodal integration and fusion. We leverage the complementary strengths of multiple modalities, specifically conversational signals and physiological data, to enhance predictive accuracy. Furthermore, our proposed model incorporates temporal modelling to integrate information from conversational cues, physiological signals, and neural data, an aspect that has been relatively under-emphasized in existing models [30,31]. In contrast, these existing models typically rely on classifying emotions from facial expressions. Our proposed approach aims to predict a continuous signal corresponding to the activation of brain regions associated with genuine emotional states directly from neural activity. To achieve this, we employ a Transformer encoder in which attention mechanisms effectively identify and weigh the most salient features. The experimental results indicate that our proposed model outperforms existing models. Furthermore, ablation studies evaluating the contributions of each modality underscore the critical role of physiological data in elucidating brain activity.

3. Methods

This research aims to develop a method for predicting brain activity in regions involved in emotion processing, using multimodal conversational behavioral signals completed by peripheral physiological recordings. This approach enables the inference of a participant’s emotional state during a conversation, whether the interlocutor is human or robotic. Table 1 provides details of the specific brain regions examined in this study. We focus on the stimuli perceived by the participant, including visual and auditory inputs, as well as the participant’s blood pulse, resulting in the following modalities: video (of the interlocutor), audio (of both the participant and the interlocutor), and blood pulse (of the participant). In this section, we first outline the importance of predicting brain activity for emotion recognition before detailing the proposed methodology.

3.1. The Importance of Predicting Brain Activity for Emotion Recognition

Analyzing brain activity helps improve our understanding of human cognitive processes through the application of domain knowledge. Neuroscience has conducted numerous studies aimed at identifying brain regions associated with emotions [35]. Therefore, the chosen brain areas are directly linked to real emotions based on well-established neuroscience experiments. Using deep learning to connect with these studies is a logical step in approaching human behavior through artificial intelligence [36]. Many studies rely on facial expressions of emotion to evaluate emotional states; however, it is important to recognize that these expressions are primarily a proxy. Like other mental states, the emotional states that are actually experienced remain hidden, as they cannot be directly measured using objective tools. Instead, we focus on analyzing brain activity, which we postulate offers a more accurate representation of emotions. Finally, through prediction of brain activity, the predictions obtained can be used as a latent space to infer other forms of emotion. For example, a decoder network could be employed to reconstruct the next likely emotional states from brain activity. As suggested by recent studies [37,38,39,40,41], this approach is related to recent methods that aim to reconstruct original stimuli from brain activity.

Table 1. Studied brain areas and their corresponding Brainnetome atlas codes, to which we added the hypothalamus (see text).

Abbrev	ROIs	Brainnetome Code
l,r vA	left and right Ventral Agranular insula	165/166
l,r dA	left and right Dorsal Agranular insula	167/168
l,r vD	left and right Ventral Dysgranular insula	169/170
l,r dD	left and right Dorsal Dysgranular insula	173/174
l,r dG	left and right Dorsal Granular insula	171/172
l,r H	left and right Hypergranular insula	163/164
l,r MA	left and right Medial Amygdala	211/212
l,r LA	left and right Lateral Amygdala	213/214
Hy	Hypothalamus	[42]

Table 1. Studied brain areas and their corresponding Brainnetome atlas codes, to which we added the hypothalamus (see text).

Abbrev	ROIs	Brainnetome Code
l,r vA	left and right Ventral Agranular insula	165/166
l,r dA	left and right Dorsal Agranular insula	167/168
l,r vD	left and right Ventral Dysgranular insula	169/170
l,r dD	left and right Dorsal Dysgranular insula	173/174
l,r dG	left and right Dorsal Granular insula	171/172
l,r H	left and right Hypergranular insula	163/164
l,r MA	left and right Medial Amygdala	211/212
l,r LA	left and right Lateral Amygdala	213/214
Hy	Hypothalamus	[42]

3.2. Proposed Approach

In this section, we present an overview of our proposed multimodal architecture, which aims to predict activity in brain regions associated with emotional states using three input modalities: audio, video, and blood pulse. First, we embed and encode each modality independently using pre-trained models and Transformer encoders. Secondly, we unify the multimodal features using late fusion. Finally, the encoder outputs are fed into a dense network that performs the classification task. The architecture of our model is shown in Figure 2.

3.2.1. Embedding

In the following, we provide a detailed description of the embedding layers used in the network for audio and video inputs. It should be noted that the raw blood pulse signal is obtained as a time series and does not require a specific embedding.

• Audio embedding: Spectral features are extracted from the raw audio signal using Mel-Frequency Cepstral Coefficients. These techniques aim to capture the characteristics of the vocal tract that are relevant for distinguishing different sounds, speech, or speakers.
• Video embedding: We extract visual features using FaceNet, a deep learning-based facial recognition system developed by Schroff et al. (2015) [29]. The architecture consists of a CNN model followed by a few fully connected layers. The CNN is designed to learn a representation of facial images by extracting key features, such as facial landmarks, textures, and distinctive patterns, which are essential to differentiate between faces. The fully connected layers project these features into an embedding that is invariant to variations in lighting, pose, and facial expressions.

3.2.2. Network Formulation

The model operates temporally, depending on the frequency of fMRI recordings ($1.2$ s). Since our data are organized into 1 min conversations, the model provides several predictions for each conversation by considering a sequence of past frames from the multimodal signals over a period $\tau$ (6 s, i.e., 5 time steps). This period of 6 s corresponds approximately to the maximum length of the activation window of the hemodynamic response function, which characterizes the Blood Oxygen Level-Dependent (BOLD) signal used to measure fMRI brain activity, and peaks between 3 and 5 s after a triggering stimulus [43]. Formally, the function of our network can be expressed by Equation (1).

$$\begin{array}{c}\hfill Y_t=F(X_v^T,X_a^T,X_b^T)+{\epsilon }_t.\end{array}$$

(1)

where F is the function of the model, and $Y_t$ represents the target variable (brain area activation). Each variable ${\left\{X_i^T\right\}}_{i\in \{v,a,b\}}$ is represented as a $2D$ temporal tensor in ${\mathbb{R}}^{T\times d_i}$ where T is the sequence length, and $d_i$ is the embedding dimension for each input modality. The ${\epsilon }_t$ represents the error vector of the model.

The input variable $X_i^T$ is defined as the sequence ${\left\{X_i\right\}}_j^{t-\tau :t}$, where $j^{th}$ element corresponds to the input sequence of values between t and $t-\tau$. Prior to the embedding layer, all input signals have the same duration but different frequencies. Next, an embedding layer $\left(L_{emb,i}\right)$ is applied to each input $X_i^T$ to unify their representation.

$$\begin{array}{cc}\hfill [I_v,I_a,I_b]& =L_{emb}[X_v^T,X_a^T,X_b^T]\hfill \\ & =[L_{emb,v}\left(X_v^T\right),L_{emb,a}\left(X_a^T\right),L_{emb,b}\left(X_b^T\right)].\hfill \end{array}$$

After the embedding, a sampling block is applied to resample the obtained features along the time axis, ensuring that all features have the same number of time steps T. Separate Transformer encoders are then applied to the sequences of each embedded modality, such that

$$\begin{array}{c}\hfill [H_v^{layer},H_a^{layer},H_b^{layer}]=[Encoder\left(I_v\right),Encoder\left(I_a\right),Encoder\left(I_b\right)].\end{array}$$

The outputs from the three modalities are then concatenated using feature-level fusion. This process is expressed in Equation (2), where H represents the output of the concatenated layers.

$$H=Concat[H_v^{layer},H_a^{layer},H_b^{layer}].$$

(2)

Finally, a fully connected layer is applied to the concatenated features $\widehat{Y_t}=FC\left(H\right)$, where the output layer contains a linear activation function and we adopt the Mean Squared Error (MSE) as a loss function.

4. Experiments and Results

In this section, we describe the datasets used and provide details of the experimental setup. Following this, we present the results for the HHI and HRI experiments separately, facilitating a comprehensive comparative analysis.

4.1. Dataset Description

The corpus utilized in this study was collected during an fMRI experiment, as described in [12], where the authors detail the acquisition process of the conversational signals. In addition to the conversational signals, this study also analyzed blood pulse, a physiological signal that was recorded during the experiment using the scanner’s photoplethysmograph. It was captured using a photoplethysmography device placed on the tip of the participant’s left index finger. The device was wirelessly connected via Bluetooth, and data were continuously acquired at a frequency of 200 Hz. The synchronization of the recording was inherent to the scanner’s procedures, aligning with the time intervals of the fMRI recordings. The corpus consists of 25 participants and four sessions. In each session, participants engage in six conversations of 60 s duration: three with a human and three with a conversational robot, alternately. The brain areas studied are listed in Table 1.

The selected areas are directly associated with positive and negative emotions. These include the insular regions in both the left and right hemispheres, as well as regions within the amygdala and hypothalamus. These regions, with the notable exception of the hypothalamus mask taken from [42], are identified using the Brainnetome atlas [44]. The amygdala consists of medial and lateral parts, while the insular cortex is subdivided into ventral and dorsal agranular, dysgranular, granular, and hypergranular areas. The hypothalamus is an important region for homeostasis, yet it is not included in the Brainnetome atlas.

4.2. Experimental Setup

The experimental design employed to assess the proposed architecture based on a deep learning model is outlined below.

Architecture details: The first layer of the proposed network is designed for multimodal embedding. A total of 645 features are extracted from the input video using FaceNet, and 30 Mel filter bank features are extracted from the input audio, with a window length of 2048 and a skip length of 512. The features are then sequenced (with timestep $=5$ as shown in Section 3.2.2) before being encoded by the second network layer. The Transformer encoders consist of three modality-specific attention blocks, each incorporating a multi-head attention layer with a key dimension of 64. These blocks differ in the number of attention heads and dropout rates: the audio block employs 16 heads with a dropout rate of $0.1$, the video block employs 32 heads with a dropout rate of $0.15$, and the blood pulse block employs 8 heads with a dropout rate of $0.1$. Layer normalization with $\epsilon ={10}^{-6}$ is applied following each attention block. Once the individual modalities are encoded, a late-fusion step unifies the three output features into a single feature representation. This fused representation is then fed into three fully connected layers of sizes 512, 256, and 64, each using ReLU activation and dropout rates of $0.6$ and $0.4$. Finally, a linear output layer generates 17 values to predict the continuous BOLD signal. Figure 2 illustrates the overall architecture.

Hyperparameters: For all evaluated models, the number of training iterations is set to 50, with a batch size of 64. The Adam optimization algorithm is used with ${\beta }_1=0.9$, ${\beta }_2=0.98$, and an initial learning rate of $\alpha ={10}^{-4}$. In our model, the value of the parameter $\gamma$ in the normalization layer is fixed at ${10}^{-6}$.

Implementation: The datasets are split into training, validation, and test sets, maintaining class proportions of $70\%$, $10\%$, and $20\%$ respectively. The split was divided by participants so that the data of each participant belonged to one and only one set (training, validation, or test set) to avoid overfitting. All models were implemented using Google Colab with GPU support, and the preprocessing data with code repository code will be made available upon request.

4.3. Evaluated Models

To evaluate our approach, we conduct a comparative analysis with existing baselines and multimodal architectures from the literature. Our study is the first to utilize a specific combination of different input signals, including behavioral and peripheral physiological data, in order to predict a continuous BOLD signal representing brain activity recorded with fMRI. The following models are selected for comparison with our work due to their relevance to the multimodal prediction task, which involves various signals as well as multimodal feature extraction and fusion techniques. Other similar versions of these models have been applied to other tasks, such as the Emotion Recognition in the Wild Challenge (EmotiW) [45,46] and Educational Video Summarization (EDUVSUM) [47]. To compare our proposal with the selected models, we adapted and reconfigured them to suit our specific task. They are described as follows:

• MULTILAYER PERCEPTRON (MLP): Baseline model that consists of two hidden layers with dropout, and ReLU activation functions applied after each layer. The third layer with the linear activation function and 17 output neurons represents the targets of the model. The dimensionality of the hidden layers is considered as a hyperparameter to be tuned.
• MODEL 1 [27]: This model applies MFCCs and DenseFace to extract acoustic and visual features, respectively. It employs LSTM networks with early fusion to represent the hidden multimodal features and feed-forward layers to predict emotions from seven labels (anger, disgust, fear, happiness, neutral, sad, and surprise). Originally, it was trained on a dataset of short emotional video clips taken from movies and TV shows.
• MODEL 2 [31]: This approach integrates audio, video, and text recorded from video frames with speech and subtitles. It incorporates a CNN network to extract visual features, evaluated using three classical classification backbones (VGGNet, ResNet, and GoogleNet). It also incorporates Bidirectional Encoder Representations from Transformers (BERTs) and spectral analysis to extract features from text and audio, respectively. Finally, a fusion step uses Bidirectional LSTM networks to capture the temporal dependencies and combine the extracted hidden features.
• MODEL 3 [30]: The proposed model combines CNN and LSTM, along with self-attention, to extract acoustic features from speech signals. Simultaneously, a BiLSTM network is employed to extract textual features from transcripts. The extracted features are then fused and fed into a deep fully connected network to predict the probabilities of the four target emotions (sad, excited, neutral, and angry).

4.4. Statistical Test

In this section, we evaluate the stability of the performance improvements using statistical significance testing, comparing key settings at a confidence level of $\alpha =0.05$. The analysis was conducted using the Almost Stochastic Order (ASO) test [48,49], as implemented in the deep significance Python library [50].

This test is commonly used to compare the performance scores of two evaluate machine learning models, without making any assumptions about the distributions of the scores. Given the performance scores of two models, $M_1$ and $M_2$ over multiple runs, the ASO test computes a test statistic ${\epsilon }_{min}$ which quantifies how far model $M_1$ is from being significantly better than model $M_2$, for a predefined significance level $\alpha \in (0,1)$. When the distance ${\epsilon }_{min}=0.0$, it can be concluded that model $M_1$ is stochastically dominant over model $M_2$. If ${\epsilon }_{min}<0.5$, model $M_1$ is said to almost stochastically dominate model $M_2$. Conversely, when ${\epsilon }_{min}=1.0$, model $M_2$ stochastically dominates model $M_1$. Finally, for ${\epsilon }_{min}=0.5$, no stochastic order can be established between $M_1$ and $M_2$.

4.5. Results

The evaluated models were trained using the dataset described in Section 4.1, with the Average MSE as the prediction metric. The performance of the current model is compared to the state of the art, as shown in Figure 3. Additionally, the predictions of blood responses in each region are presented in

Table 2. HHI test set results: average MSE for each region of interest. The bold values highlight the lowest (best) average MSE for that specific region.

Models	MSE per ROI
	lvA	rrA	ldA	rdA	lvD	rVD	ldD	rdD	ldG	rdG	lH	rH	lMA	rMA	lLA	rLA	Hy
CURRENT	0.0321	0.0318	0.0323	0.0322	0.0334	0.0324	0.028	0.0252	0.0334	0.0328	0.0322	0.0286	0.0256	0.0304	0.0304	0.0326	0.0324
MLP	0.0365	0.0356	0.0337	0.0342	0.040	0.0360	0.0294	0.0321	0.0321	0.0338	0.0332	0.0311	0.0306	0.0289	0.0287	0.0337	0.0338
MODEL 1	0.0328	0.0329	0.0329	0.0332	0.0353	0.0337	0.0289	0.0313	0.0338	0.0340	0.0343	0.0297	0.0270	0.0298	0.0296	0.0335	0.0339
MODEL 2	0.0322	0.0320	0.0326	0.0324	0.0337	0.0331	0.0284	0.0309	0.0328	0.0320	0.0313	0.0287	0.0254	0.0293	0.0285	0.0340	0.0333
MODEL 3	0.0331	0.0325	0.0327	0.0326	0.0346	0.0337	0.0286	0.0309	0.0326	0.0325	0.0322	0.0289	0.0254	0.0287	0.0285	0.0347	0.0332

and

Table 3. HRI test set results: average MSE for each region of interest. The bold values highlight the lowest (best) average MSE for that specific region.

Models	MSE per ROI
	lvA	rrA	ldA	rdA	lvD	rVD	ldD	rdD	ldG	rdG	lH	rH	lMA	rMA	lLA	rLA	Hy
CURRENT	0.0342	0.0313	0.0318	0.0298	0.0335	0.0301	0.0286	0.0302	0.0315	0.0348	0.0317	0.0271	0.0290	0.0306	0.0354	0.0318	0.0314
MLP	0.0577	0.0499	0.0458	0.0424	0.0414	0.0387	0.0364	0.0320	0.0341	0.0371	0.0309	0.0519	0.0695	0.0258	0.0304	0.0387	0.0396
MODEL 1	0.0404	0.0361	0.0414	0.0349	0.0350	0.0317	0.0315	0.0310	0.0306	0.0324	0.0304	0.0362	0.0424	0.0242	0.0283	0.0340	0.0355
MODEL 2	0.0365	0.0329	0.0332	0.0302	0.0330	0.0304	0.0293	0.0303	0.0311	0.0322	0.0313	0.0293	0.0373	0.0257	0.0289	0.0325	0.0316
MODEL 3	0.0352	0.0324	0.0322	0.0300	0.0336	0.0305	0.0283	0.0306	0.0318	0.0324	0.0309	0.0291	0.0334	0.0257	0.0282	0.0324	0.0317

for human–human and human–robot interactions, respectively. Furthermore,

Table 4. Results for average MSE across all possible modalities (unimodal, bimodal, and all modalities (AVB) data) for HHI and HRI experiments. The bold values indicate the lowest (best) average MSE for each experiment.

	Modalities				MSE
MODELS	A	V	B		HHI	HRI
AVB	✓	✓	✓		0.0317	0.0314
Bimodal	✓	✓	✗		0.0337	0.0352
	✓	✗	✓		0.0370	0.0328
	✗	✓	✓		0.0339	0.0367
Unimodal	✓	✗	✗		0.0347	0.0338
	✗	✓	✗		0.0426	0.0614
	✗	✗	✓		0.0321	0.0319

displays the performance of our model (AVB) compared to other models based on one and two modalities, following an evaluation protocol similar to that used in [51].

5. Discussion

We investigate three principal aspects of our findings: how the proposed model outperforms previously evaluated models, an ablation study that details the contribution of each modality, and the broader implications for affective neuroscience. In general, the model demonstrates robust predictive performance in regions of the brain related to emotion, highlighting the benefits of integrating multiple signals. However, certain findings emphasize the complexity of predictive modeling in social contexts, indicating that different analytical methods may offer complementary perspectives. We address these elements in the following subsections.

5.1. Models Comparison

According to Figure 3, which illustrates the average MSE across regions of interest for both human and robot conditions, and as confirmed by the ASO test comparisons in Figure 4a,b, the proposed model AVB (column) is almost stochastically dominant across all evaluated models, with $\epsilon <0.5$. This highlights the enhanced predictive performance compared to the other evaluated models, particularly outperforming the MLP, which utilizes a basic fusion approach with fully connected layers. Table 2 and Table 3 demonstrate that, in more than half of the regions investigated, the proposed model achieves the lowest MSE during interactions with both humans and robots, with a higher frequency observed for human interactions compared to those with robots.

The proposed model provided better predictions for the majority of brain regions selected for their involvement in emotion processing, with particularly strong results for the hypothalamus in both conditions (human–human and human–robot). This is particularly significant given the hypothalamus’s role not only in emotions but also in maintaining homeostasis, which refers to the responses of the autonomic and peripheral nervous systems, often considered markers of emotional arousal. Additionally, the proposed model outperformed other models in both conditions within the more anterior insular regions, which are more strongly associated with homeostatic functions compared to the posterior insular regions, which are generally linked to the cognitive or motor dimensions of behavior.

In contrast, the improvement of the proposed model over others that were tested is not clearly evident in the amygdala. Although speculative, the difference between the model’s clear improvements in the hypothalamus and anterior insula and the mixed results in other regions may be related to their distinct roles in emotional processing. The paradigm indicates that we are investigating emotion processing in the context of social interactions, rather than emotions per se.

5.2. Ablation Study

The ablation study results, presented in Table 4, demonstrate that the proposed approach based on three modalities outperformed models employing a combination of two modalities or only one modality. Remarkably, the model relying solely on blood pulse demonstrated performance closely comparable to the model incorporating all three modalities: $0.0321$ vs. $0.0317$ for HHI, and $0.0319$ vs. $0.0314$ for HRI. In contrast, other combinations yielded an MSE above $0.0328$, or $0.033$ when excluding models including the blood pulse.

These findings are clearly illustrated in Figure 4. They include test statistics indicating that the unimodal model (blood pulse—Model B), which is very close to the proposed model AVB in terms of average MSE, was evaluated using the ASO test. This evaluation confirms that model AVB (column) almost stochastically dominates model B (row), with $\epsilon =0037$ and $\epsilon =0.01$ in Figure 4c and Figure 4d, respectively. Although the proposed model demonstrates a superior overall predictive accuracy, model B outperforms all other single- or two-modality models. Indeed, as shown in Figure 4c,d, model B (row) stochastically dominates every other model, with $\epsilon =1.0$. This strongly indicates that blood pulse represents a significant modality for understanding brain responses, extending beyond its conventional association as a source of artifacts (e.g., pulsations in the cerebrospinal fluid).

5.3. Implications for Affective Neuroscience

Our interpretation of the results follows [52], where the authors show a significant correlation between facial expressions of happiness by the interlocutor during conversation and activity in the insular and hypothalamic regions of the recorded participants’ brains. Figure 5, which is adapted from [52], illustrates the localization of these brain areas, which were also used in the current exploration. Therefore, it is known in advance which of these brain areas respond to at least one of the features used as a predictor in the current exploration, namely. the facial emotion expressed by the interlocutor (see results illustrated in Figure 2 of [52]).

Irrespective of the brain area, the obtained MSEs are comparable around $0.03$, implying that, for all models, the approach presented here is able to predict the brain response in regions involved in emotional processing with similar accuracy. An important remark is the absence of a clear effect of the two experimental variables—type of interaction and region of interest—on the prediction accuracy. For example, we could have expected differences in prediction to be similar to the differences of BOLD response in all amygdala regions except the left medial amygdala reported in [52], but we found the opposite: the most reduced MSE (and the larger difference between the two types of interactions) was for the human interlocutor in the left medial amygdalar region. Also, counterintuitively, predictions improved for the robot agent in the hypothalamus while the response was reported as higher in previous experiments [12]. Similar improvements have been seen in other brain regions (e.g., rdA, l and r vD, or even rLA). Together, these differences between the current approach that uses prediction of brain activity and previous approaches that rely on general linear models to calculate the importance of experimental factors in explaining the brain response suggest that these two approaches should be considered complementary. One possibility could be that emotional processes are blunted in human–robot interactions so that, for some but not all processes, they are easier to predict. Explaining this complementarity is beyond the reach of the current study.

The ablation investigations also provide fuel for developing a better understanding of the neurobiology of emotional processing in natural interactions. First, all but one case of ablation yielded an increased MSE, which means that the three types of multimodal information used in the prediction were truly complementary. Also noteworthy is the fact that the unimodal model using the blood pulse B yielded the second-best score, after the full model, confirming that internal physiological states reflect emotional processing to a similar extent as observable behaviors [13]. The next step will be to dissect these effects in individual regions of interest instead of the average across the regions, given the variability highlighted in the previous paragraph.

6. Conclusions

In this study, we introduced a new methodology to predict fMRI responses in brain areas involved in positive and negative emotions. This approach, as well as the datasets used, had not been explored in the previous literature related to the objectives of the present study. To evaluate our method, we adapted existing face-based multimodal emotion recognition architectures from the literature to our task. The proposed multimodal deep learning network is simple in design, but effective, as demonstrated by the comparative results.

In this study, we mainly investigated the predictive power of multimodal conversational signals, including visual information from interlocutors, for predicting emotion processing linked to brain activity of participants. It is important to note that one should not actually expect a link between the interlocutor’s expression and the participants’ own emotional states. Therefore, a null hypothesis could be the following: “The interlocutor’s expression should have no predictive power over the individual’s internal emotional states.” However, it is possible to defend a possible predictive power based on the contagion of emotions, which is a generalization of the role of mirror neurons sharing the same code for first- and third-person representations of internal states in the domain of emotions [53].

Finally, another important question is about the link between the activation of the areas of the brain studied and emotions. The main limitation here concerns the ability to objectively label participants’ emotions, which requires a specific experimentation protocol. To address this issue, we conducted a study to analyze the predictability of positive and negative emotions based on activation of these areas compared to a model predicting emotions from audio. We evaluated two architectures: one using audio input [54] and another using fMRI signals to predict emotions. Emotions were classified as neutral, positive, or negative from text depending on the context of the conversation of the same datasets used in the current work. Surprisingly, we found that fMRI signals predicted emotions better than audio, indicating that they better encode internal emotional states. In order to focus on a specific topic for each contribution, we decided to explore this idea further in a future study, to investigate this possibility in detail, and to construct a deep learning model to decode emotions from fMRI brain signals. As a result, we can integrate this with the proposed model to design an end-to-end encoder–decoder that connects multimodal conversations to fMRI and emotions, serving two objectives: predicting fMRI activity and linking the prediction of regions of interest with emotions.