Design of an Immersive Virtual Reality Framework to Enhance the Sense of Agency Using Affective Computing Technologies

Abstract

Virtual Reality is expanding its use to several fields of application, including health and education. The continuous growth of this technology comes with new challenges related to the ways in which users feel inside these virtual environments. There are various guidelines on ways to enhance users’ virtual experience in terms of immersion or presence. Nonetheless, there is no extensive research on enhancing the sense of agency (SoA), a phenomenon which refers to the self-awareness of initiating, executing, and controlling one’s actions in the world. After reviewing the state of the art of technologies developed in the field of Affective Computing (AC), we propose a framework for designing immersive virtual environments (IVE) to enhance the users’ SoA. The framework defines the flow of interaction between users and the virtual world, as well as the AC technologies required for each interactive component to recognise, interpret and respond coherently within the IVE in order to enhance the SoA.

1. Introduction

As Virtual Reality (VR) expands into multiple fields of application, it is becoming necessary to address its benefits and limitations. Usually, VR is defined in terms of the overall user experience and sense of immersion or presence, but it is less frequent to associate it with the sense of agency (SoA), which is a central aspect of human self-consciousness since it refers to the experience of being in control of your own actions and their consequences.

In the field of Affective Computing (AC), several techniques and tools make it possible for computers to recognise, interpret and express emotions. In fact, our work is based on the assumption that these technologies are useful for further applications, not just emotional management.

This paper reviews the state of the art of the technologies developed in the field of Affective Computing (CA) with the aim of analysing how each of them can be used to improve the SoA of the users in immersive virtual environments (EVI). First, in Section 2, the importance of the sense of agency in our daily lives is explained. Then, in Section 3, we review the main techniques and tools from the field of AC that could enhance people’s experience in immersive virtual environments (IVEs). This can be achieved by using sensations and emotions as driving elements which influence the user’s perception of their interaction with the IVE and especially their sense of agency in the virtual world.

Based on the state of the art, in Section 4, a framework is proposed as a design guide to properly manage the emotional interactive flow between the elements involved in the IVE. The framework shows, for each element of the interaction flow, the CA technologies that allow each element recognition, interpretation and coherent response within the IVE in order to improve the SoA. This framework helps designers to select both the appropriate technologies to effectively convey emotions between the people involved within the virtual world and the techniques that allow for the environment to respond emotionally in a coherent way, thus increasing the SoA.

2. The Relevance of the Sense of Agency in Virtual Reality

The term sense of agency (SoA) refers to the experience of controlling one’s own actions and, through them, those of the outer world [1]. It is important to distinguish between agency (performing an action) and sense of agency (feeling in control of that action). In [2], this phenomenon is explained in terms of intention. This is, to have a high sense of agency, one’s previous intention or expected outcome to an action should match the real result of performing that action. For example, when the intention of a person is to burst a balloon that they are holding, they expect the action to happen in this way (Figure 1): they hold a sharp object in their hand, they move that hand towards the balloon and, when the sharp object reaches the balloon, it bursts. The person knows the outcome because they watch how the balloon bursts (visual feedback), they hear it burst (audio feedback) and they feel its air spreading out of it into their skin (touch feedback). Since the visual, auditory and tactile feedback match the expected outcome, the person has a high sense of agency, i.e., they feel in control of their action.

Research about SoA is relevant because it influences several aspects of our daily life, such as our health, our learning process or simply our perception of the technology that we use.

2.1. Sense of Agency in Health

The sense of controlling the actions of our own body is a fundamental feature for having a healthy perception of our body self. This means that when there is a correct SoA, the person recognises themselves as the agent of their own actions. The sense of agency is altered in psychosis. For example, patients with a brain disorder called schizophrenia believe that their movements and thoughts are being driven by external forces. This happens because their brains cannot predict the consequences of the actions that they perform [3].

Although the SoA has an important impact on daily life, there are few studies about the relationship between SoA and other health circumstances, such as stress, depression or the loss of control in schizophrenia and agoraphobia [4]. In this sense, IVEs can be useful as tools to research about the relationship between SoA and mental health. This is the case of [5], where the authors used virtual hands to study how stress and the body self are related to the SoA. Also, in ref. [6], a virtual environment was used to analyse whether a deficit in agency, similar to the one produced in schizophrenia, influences the sense of presence or the performance in sensory–motor tests. In general, a reduction in the SoA is related to a bad health situation and a decrease in the quality of life [7].

2.2. Sense of Agency in the Learning Process

Agency is inherent to students’ capacity of regulating, controlling and supervising their own learning [8]. Students’ efficiency to regulate their cognitive, affective and behavioural processes while interacting with their learning environment is key for their academic success.

Research such as [9] has used immersive Virtual Reality technologies to demonstrate that an increase in the perceived sense of presence and agency makes the learning process easier. In the field of immersive collaborative learning, there is also research that analyses the sense of agency when two users collaborate to complete a task while they are represented by the same virtual avatar [10]. In this study, researchers observed that users tend to feel certain control over the avatar even when they have little or no real control. These results present a new set of possible applications related to Virtual Reality and collaborative teleworking, where users could share virtual bodies.

2.3. Sense of Agency in Human–Computer Interaction

In terms of Human-Computer-Interaction (HCI), the sense of agency is a crucial feature. It is needed to evaluate the ways in which users experience the interactions with any technology. For this reason, SoA has become a central aspect of research in the HCI field [11]. Research about the sense of agency can benefit from new interaction techniques that are being quickly developed in the field of HCI, such as gestural input, physiological or intelligent interfaces, and assistance methods [11]. Moreover, immersive Virtual Reality is helping researchers study the sense of agency related to the body self. For instance, they analyse the possibility of experiencing the same emotions towards a virtual body inside an IVE and towards a biological body, and the extent of such possibility [12].

2.4. Sense of Agency: Theoretical Models

Researching mechanisms to improve the sense of agency when interacting in an IVE imply knowing the psychological mechanisms that influence that perception. There are two main models to describe the neurocognitive models underlying the SoA [13], the Comparator Model and The Theory of Apparent Mental Causation. The first one suggests that the SoA primarily comes from underlying processes of motor control and it is inferred retrospectively, after the action is performed, based on its external consequences. This suggests that the SoA comes from the comparison between sensory predictions and real sensory feedback. The second one focuses on situational signals and suggests that the SoA is linked to the correspondence between the external events and our intentions.

During the last decades, the perspective that has dominated research about the sense of agency is the one based on the Comparator Model. However, according to [14], the complete system of the SoA may be more complex and imply multiple processes on different cognitive levels. In addition to the comparative mechanism, many other factors contribute to a large degree to the SoA. These factors are, for instance, the selection of actions, the intention, the effort, the emotion, the retrospective inference, the achieved objectives and the social interaction. There are also other studies that suggest factors that do not influence the sense of agency. For example, the results in [15] show that presence and agency are not influenced by the appearance or the type of visualisation of the avatar.

Research conducted in [14] shows a definition of SoA in terms of two layers: the sense of agency of one’s own actions (body agency) and the sense of agency about the external events (external agency) (see Figure 2).

The first layer implies self-consciousness: when one feels that their consciousness controls their body. The second layer is relevant for the interaction with the environment: when one feels in control of external events and that the environment is reacting coherently to their actions. This two-layer definition seems interesting for its application on IVEs, because it relates to the SoA. In fact, it relates not only to the avatar’s behaviour, but also to the reactions from the environment. Thus, it involves multiple senses, such as sight, hearing and touch. This is relevant because research shows that the human sense of agency becomes stronger or weaker depending on the senses involved in the action. In the context of IVEs, the senses of sight and hearing have been extensively implemented. However, the sense of touch can greatly enhance the realism and immersion of the experience, but few environments have been developed taking into account haptic feedback. Haptic feedback can simulate physical interactions within the virtual world, making virtual experiences more tangible and believable ([16,17]). Haptic influence in user experience was also studied in [18], where the authors suggested that haptic feedback on its own generates at least a better sense of presence than visual feedback. Stimulating the sense of touch by interacting with other agents and with virtual content can profoundly impact the user’s psychological and emotional state.

3. State-of-the-Art Technologies for Emotional Management

In this section, we analyse the technologies that allow management of the emotional interactive flow between users and the content of the virtual world. The aim of this analysis is to select the appropriate technologies to design a framework for the development of IVEs that could enhance the sense of agency.

3.1. Affective Computing

Traditionally, in the field of Human–Computer Interaction (HCI), users leave emotions behind to be able to interact with computers in an efficient and rational way [19]. However, recent research considers that emotions have significant relevance for human–computer interactions and also for virtual environments [20].

Affective Computing (AC) was defined in the decade of 1990 and consolidated in 1997, when the first book about this term was published. It was written by professor Rosalind Picard (MIT) [21]. In the beginning, AC was defined as computers’ ability to recognise, to understand and even to have and to express emotions. Some time after that, Picard proposed yet another definition: computational approaches for the deliberate detection and induction of affection [22]. Nowadays, wider definitions can be found, such as the one in [23], where the authors propose that AC is computing that relates to emotions or other affective phenomena that emerge from them or that deliberately influence them.

Taking these definitions into account, to design a user interface based on emotions, the frameworks and toolkits to explore should address the following characteristics:

• Emotion classification: models that allow defining a set of emotions that the system is able to recognise and express.
• Emotion recognition: research of the techniques to be used for the recognition of emotions felt by the users during the interaction with the system.
• Generation of emotional responses: emotional models and technologies that allow for the system to make decisions in terms of selecting emotions to answer the users and of choosing the best way to show this emotional response.
• Emotional expression: definition of the audiovisual techniques to be used by the system to show emotional responses.

3.2. Classification of Emotions

The first step to design an emotional system is to define the set of emotions that it might be able to manage. There are different classifications of emotions, which can be divided mainly into two groups: discrete and continuous. This means that emotions can be described as a defined set of discrete entities or as elements placed along continuous dimensions [24].

Discrete models propose a specific set of emotions. For instance, Ekman [25] categorises six basic emotions: happiness, rage, fear, sadness, surprise and disgust (Figure 3). Other less restrictive models include sets of emotions that are not considered universal due to their social and cultural influence. This is the case of Plutchik’s discrete model [26], which represents the following emotions in terms of a wheel: trust, surprise, joy, fear, disgust, sadness, anticipation and anger. There are other classifications that define sets of 19 [27] or 22 [28] discrete emotions.

On the other hand, dimensional models represent emotions as the combination of several psychological dimensions. For instance, Russell’s circumplex [30] considers each emotion as a lineal combination of two affective dimensions: valence and arousal. The arousal dimension expresses the intensity of the emotion, while the valence dimension quantifies the positive or negative scaling of the emotion in a continuous pleasure–disgust scale.

Later on, Mehrabian and Russell [31] created a new three-dimensional model adding the label of dominance to those of arousal and valence. Dominance values represent the users’ feeling of being in control of the application, and they can range from submissive/without control to dominant/in control and empowered.

3.3. Emotion Recognition

From the technological point of view, there are two main methods to identify users’ emotions: measuring physiological changes and analysing expressive behaviour.

Recognising emotions based on users’ expressive behaviour has the main advantage of not requiring contact with a measuring device or a sensor, because usually this type of recognition is achieved by analysing facial expressions [32] and body gestures [33] by means of computer vision and artificial intelligence techniques or by analysing variations in the tone or speed of speech by audio-signal techniques [34].

Nevertheless, current emotional recognition systems that use physiological signals have increased research interest [35,36]. Although plenty of devices are capable of measuring these physiological signals in the human body [37], the most frequently used devices in the field of Affective Computing are described below.

• Analysis of electrodermal activity (EDA): it is related to the sympathetic nervous system. The human being reacts to behavioural, cognitive and affective phenomena activating sweat glands to prepare the body for action. The sensors measure this secretion of sweat [38].
• Analysis of the electrocardiogram (ECG): ECG is a reliable source of information and it has considerable potential to recognise and predict human emotions such as anger, joy, trust, sadness, anticipation and surprise. More specifically, to detect these emotions, it is required to extract the Heart Rate Variability (HRV) from ECG measures [39].
• Analysis of the electroencephalogram (EEG): EEG is an electrophysiological monitoring method used to register the electrical activity of the brain by the use of electrodes. EEG signals can reveal relevant characteristics of emotional states. Recently, several BCI emotion recognition techniques have been developed based on EEG [40]. Knowing that ECG, EDA and EEG are the main methods to detect emotions without users’ voluntary control over them, some authors prefer EEG-based systems for emotion recognition [41]. This is because ECG measurements experience a large delay between stimuli and emotional response, and EDA-based systems cannot report the valence dimension when used on their own.
• Analysis of the respiratory rate (RR): emotional states can be identified by means of their respiratory pattern. For instance, happiness and other related positive emotions produce significant respiratory variations, which include an increase in the pattern variability and a decrease in the volume of inspiration and breathing time. Positive emotions vary their effects in the respiratory flow depending on how exciting they are; i.e., the more exciting ones increase the respiratory rate. On the other hand, feeling disgust suppress or cease respiration, probably as a natural reaction to avoid inhaling noxious elements [42].

As mentioned in Section 3.2, discrete and dimensional models can be used to classify a set of emotions. Discrete models are widely used when emotion recognition is performed by analysing expressive behaviour. In particular, it is even more common to recognise emotions by facial expressions due to Ekman’s research [43].

When recognising emotions by means of physiological signals, the preferred models are the dimensional ones, that is the models that are based on valence, arousal and dominance (Figure 4). Frequently, several types of physiological signals are mixed. For instance, in the literature, heart rate has been used to measure different levels of valence; electrodermal activity (EDA) and eye tracking have been used to measure arousal. Analysis of facial expressions and head poses, has been used to predict both valence and arousal. An example of the combination of recognition techniques is [44], in which the dimensional model is used to describe how emotional states provoked by affective sounds could be recognised efficiently by estimating dynamics in the autonomic nervous system (ANS). This work also shows that emotional states are modelled with a combination of arousal and valence dimensions, while the dynamics in the ANS are estimated by the standard non-linear analysis of heart rate variability (HRV), which is derived from the electrocardiogram (ECG).

3.4. Generating Emotional Responses

The following aspect of Affective Computing refers to the capacity of machines to answer appropriately to users’ emotions. Once the emotions are recognised, the system needs to decide how to react to those emotions. In the literature, two types of systems for generating emotional responses can be found: those that react in a predefined way for interactive events and those that modify the applications’ content and interactive flow to adapt it to users’ emotions based on their context. In some specific cases, such as in learning process or during a body rehabilitation treatment, adapting to users’ emotional states is more relevant. This is because considering an extra level of difficulty or an inadequate situation may cause complications to the users’ health or learning processes [37].

To adapt the response in relation to the users’ emotional state, the system needs a reasoning mechanism to be able to select the emotion to express based on context. This means that the emotional interaction is generated dynamically, as a result of an internal evaluation process of the events produced by the interaction.

There are emotional theories that sutdy how different emotions manifest in human beings. Some of these models can be implemented in computer systems to simulate human emotional mechanisms.

Four main theoretical perspectives in the study of emotion can be found: the evolutionary expressive perspective, the physiological perspective, the cognitive perspective and the social structures perspective.

Several authors have inspired the development of their emotional models using those perspectives [47].

The aforementioned evolutionary expressive proposal was first developed by Darwin [48] and states that human emotional expressions are determined by their own evolution, and that they are universal and innate.

The second proposal, the physiological one, suggests that body gestures are responsible for the emotions. One of the most relevant theories of this perspective is the James–Lange theory [49], which states that emotions emerge from physiological arousal. For instance, one feels more happy when one smiles, sadder when one cries, and fearful when one flees. Thus, this theory states that, as an answer to different experiences and stimuli, the autonomic nervous system creates physiological responses (muscular tension, tearing, cardio-respiratory acceleration) from which emotions are created. Within the physiological context, there are neurological theories. Research as that by Damasio [50] indicates that people have neuronal patterns that represent changes in the body and in the brain, those changes building emotions [51]. Examining physiological theories, Cannon stated that physiological arousal and emotional experience are produced simultaneously, although in independent ways [52]. In their experiments, Cannon and Bard determined that emotion perception that awakes stimuli originates two phenomena: the conscious experience of emotion and general physiological changes. All of it is generated because the thalamus sends its pulses to the cerebral cortex and the hypothalamus.

The third perspective, the cognitive theory, states that thoughts play an essential role in the creation of emotions. Supporters of this perspective suggest that knowledge is another relevant part when experiencing emotions. In this case, emotions are the result of a cognitive evaluation process of the events in the environment. There are several emotional models that follow this perspective. Some of the most relevant ones in the field of Affective Computing have been developed by Roseman [53], Scherer [54] and OCC [28]. Specifically, the OCC evaluation theory (Figure 5) is one of the most used theories in the field of Affective Computing because their elements correspond with commonly used notions in agent models [55].

Some authors, such as Schachter and Singer [56], merge physiological theories and cognitive ones, stating that the origin of emotions comes from, on the one hand, our interpretation of the peripheral physiological responses in our organism, and on the other hand the cognitive evaluation of the situation that creates those physiological responses. Therefore, this theory suggests that emotions appear due to the cognitive evaluation of events and the body response. The person perceives the physiological changes, acknowledges what is happening in their environment and produces their emotions according to both observations. What determines the intensity of emotion that a person feels is mainly determined by the physiological changes.

Figure 5. Several studies employed the OCC model to generate emotions for embodied characters; image source [57].

Figure 5. Several studies employed the OCC model to generate emotions for embodied characters; image source [57].

Finally, the social constructive perspective proposes the idea that emotions cannot be strictly explained in terms of physiological or cognitive facts. Instead, it states that emotions are social. Therefore, a social level of analysis is necessary to truly understand the nature of emotion.

In the context of HCI, the two most commonly used perspectives to develop computational models of emotion are cognitive and physiological. While physiological theories are mostly used to develop emotional enhancement [58] and emotion recognition systems (Section 3.3), cognitive theories are the most frequently used theories to develop systems that adapt their answers to human emotions. Intelligent systems that are developed based on cognitive theories are known as cognitive agents. NEW TEXT PROPOSAL: According to research in [59] there are many different approaches to building such agents. The main mature frameworks for intelligent agents are based on cognitive architectures, such as Soar [60], ACT-R [61] and BDI (Beliefs, Desires and Intentions) [62]. Compared to Soar and other complex architectures, BDI stands out for its simplicity and versatility. In the case of BDI architectures, those that include affective aspects that influence the cognitive process are known as EBDI. EBDI agents can be classified into two main groups: (1) those that incorporate emotions and (2) those that consider other aspects such as personality, emotional state, empathy, coping mechanisms and emotional regulation [63].

In this section, we reviewed technologies that allow dynamic adaptation of the response of the EVI to the emotional events of the users. Dynamic adaptation is essential to achieve a natural and coherent interaction. This may also apply to the sense of agency, as recent studies recognise that SoA goes beyond being a static phenomenon and may fluctuate as a function of various factors such as user interaction, environmental changes, and interindividual variability ([4,17]).

3.5. Expressing Emotions

Once the system recognises the emotion in a person, it should make a decision about which emotion and through which sensory channel it should reply. To achieve that, the system should offer the necessary tools to express that emotion.

In the case of IVEs, the system tools to express emotions include avatars, with verbal and body language ability, as well as virtual content, such as 3D models and audio, as potential transmitters of emotions. Therefore, emotional responses could be expressed by one of the following elements:

• Non-verbal language: Mehrabian [64] concluded that, in the case of face-to-face communication, the three parts of a message (words, tone of voice and body language) should be coherently supportive of one another. In fact, 7% of our acceptance and empathy for a message emitted face to face depends on our acceptance of the choice of words, 38% depends on the use of voice (tone, volume) and 55% depends on over-gesticulation or facial structure. Because of this, when expressing emotions, non-verbal language has an important impact. Virtual Reality environments allow both non-verbal communication, due to the use of avatars with full-body movements, and verbal communication in real time, although they usually lack a complete system for non-verbal signals [65]. In [66], the authors highlight the lack of interaction paradigms with facial expression and the almost nonexistence of significant control over environmental aspects of non-verbal communication, such as posture, pose and social status.
  Several research works have focused on defining how human beings express emotions with their body and face. Although studies similar to Giddenss’ [67] state that one of the main aspects of non-verbal communication when communicating emotions is facial emotional expressions, body movements are also considered essential to accompany words [68]. To generate believable avatars, it is imperative that an animation motor is developed based on these studies.
  On the one hand, in the area of facial animation, there is diverse research. A lot of these studies are based on [43]. Ekman and Friesen developed FACS which stands for Facial Action Coding System. It is an exhaustive system based on human anatomy that captures all visually differentiable facial movements. FACS describes this facial activity by means of 44 unique actions called Action Units (AUs), which categorise all possible head movements and eye positions.
  On the other hand, the body animation in the field of Affective Computing has been less researched than that of facial expression [69], although some studies were found focused on body positions and the movement of the hands in non-verbal expressiveness [70,71]. Nowadays, most of the work related to body animation is based on data collected by systems of motion capture (mocap). Mocap is a popular technique for representing human motion based on tracking markers corresponding to different regions or joints of the human body [72].
  Nevertheless, this kind of studies can exclusively be used when an avatar has anthropomorphous appearance. In the case of other types of emotional agents with no human shape, the way to provide them with some tools for non-verbal communication could be to modify their shape [73], colour [74], or to apply some changes to the way they move. Following this last approach, there is research that analyses the effect that amplitude, acceleration and duration of movements have on different emotions [75]. Other studies analyse the variation of heart rate [76] and respiratory rate [42] to link each rhythm variation to its corresponding emotion.
• Verbal Language: Dialog Systems and Digital Storytelling: Bickmore and Cassell [77] confirmed that non-verbal channels are important not only for tansmiting more information and complementing the voice channel but also for regulating the conversational flow. Because of this, when the IVE is expressing emotional response, it is important to provide it with mechanisms that can adapt the conversational flow to the emotional states of the interlocutors. Several authors work with Digital Storytelling techniques to adapt the IVE’s narrative flow to the emotions of participants and offer a more natural and adaptive verbal response. For instance, in [78], players’ physiological signals were mapped into valence and arousal, and they were used as interactive input to adapt a video game narrative.
• Audiovisual Mechanisms: Beyond expressing an emotional response through an avatar’s verbal or non-verbal language, the way in which content is shown inside the IVE is also an option to emotionally respond to the user. For instance, augmenting verbal communication by means of adding sounds that are not related to talking, such as special effects and narrative music, can not only offer more information about the content, but also affect the atmosphere and the mood. This happens because sound is capable of achieving emotional engagement, of improving the learning process and of augmenting the overall immersion [79]. In studies such as [78], sounds such as music are not the only additions to the IVE. Instead, visual elements are also included to accompany the avatar, like a flying ball that changes its colour depending on the users’ emotions detected by physiological measures. Results show that participants are in favour of narrative, musical and visual adaptations of video games based on real-time analysis of their emotions.
  In the literature, several emotion-based multimedia databases can be found to elicit emotions. The content to be shown can range from a static image (such as the IAPS database [80], which is labelled according to the affective dimensions of valence, arousal and dominance/control) to a single sound (such as the IADS database [81], which is labelled according to the affective dimensions of valence, arousal and dominance/control) or a video sequence (such as the CAAV database [82], which provides a wide range of standardised stimuli based on two emotional dimensions: valence and arousal). Virtual Reality can be an interesting medium due to its complexity and flexibility, as it offers numerous possibilities to transmit and elicit human emotions. An example of this are the ten scenarios developed in [83], two for each of the five basic emotions (rage, fear, disgust, sadness and joy) that contain design elements mapped in valence and arousal dimensions.

3.6. Methods for User Evaluation

User experience (UX) in Virtual Reality environments can be measured by both subjective and objective methods. Subjective methods are a convenient option to evaluate usability, comfort and satisfaction. However, these methods are more prone to judgement mistakes [84]. Also, the use of subjective methods hardly allows evaluation of performance and system efficiency [85].

Several user evaluation methods related to emotional management in the context of IVEs and more specifically to the sense of agency are described below.

3.6.1. Subjective Measures

For evaluating the perceived emotional state of users based on subjective measures, there are several questionnaires such as PANAS [86] and SAM [87]. PANAS is an international short questionnaire composed of 10 items to evaluate the positive and negative affections. Nevertheless, one of the most commonly used questionnaires is the Self-Assessment Manikin (SAM), Figure 6. It consists of a non-verbal pictorial evaluation technique that evaluates valence, arousal and dominance associated with reaction to a stimulus. SAM has been extensively used in VR applications [88,89,90,91,92].

Other widely used methods for evaluating the subjective experience in VR environments are subjective questionnaires. These are usually composed of several qualitative questions to be answered using Likert scales. These questionnaires evaluate characteristics such as presence, interactivity and, more commonly, immersion [94,95,96].

For the measurement of the sense of agency in IVEs, researchers have developed tools to experimentally measure its components. Explicit measures of the SoA are commonly related to verbal reports in which users are asked to evaluate their SoA during a task or simply to rate whether they felt more or less that they were the agents of the actions [97]. Other researchers use Likert scales as subjective questionnaires. Their questions are commonly related to the psychophysical aspects of agency, such as confidence and feeling in control of a target [5]. There are predefined questionnaires, such as the Sense of Agency Scale [98], which consist of 13 items to measure the positive and negative sense of agency, and the Sense of Agency Rating Scale (SOARS) [99]. Specifically for IVEs, Longo and Haggard [100] developed a set of items to measure the sensation of property and agency related to a virtual hand.

3.6.2. Objective Measures

Several studies highlight the necessity of correlating the results of subjective questionnaires, such as SAM, with physiological measurements [88,89,90,91,92]. In the context of immersive environments, some of these studies use emotion recognition systems [101], heart rate measurement devices [102] or analysis of EEG signals [103], seeking objectivity. In the past, multimodal measurements have been recognised as the foundation of promising research in the field of multimodal data and learning analysis. This is the case of [104] that follows an approach based on multimodal data analysis to achieve a more continuous and objective vision of how commitment is developed in users and how it influences their task execution. In this case, physiological data flows were captured in terms of facial expression sensors, eye tracking and electrodermal activity (EDA). Results were correlated to subjective self-reporting data.

Related to objective measurements for evaluating the SoA, there are two widespread techniques to measure the feeling of implicit control: intentional binding and sensory attenuation. This last approach aims to identify the decrease in the intensity of the effects on a self-initiated action, although it is not applied in the field of HCI. Paradigms of intentional binding refer to the subjective temporal compression between actions and results. They are commonly used as an implicit measure of the SoA, because they are based on the hypothesis that voluntary actions and results are perceived closer in time when there is a high SoA [105]. Thus, intentional binding has been measured, for instance, by using a task for the estimation of time intervals [106]. One of the most commonly used techniques developed by Haggard, Clark and Kalogeras [107] is based on Libet’s clock [108].

These types of measurement techniques are an important challenge for IVE because they require great visual attention to Libet’s clock, while users are exposed to constant visual information and need to perform more complex actions. Kong et al. [109] evaluated the implicit SoA by means of that paradigm, although they represented a static context in VR, allowing for users to focus attention on the clock. In [110], the authors took these objective measures by augmenting the clock stimuli with auditory and tactile signals as an approach for future testing in VR environments.

3.7. Frameworks Related to Immersion and Sense of Agency in Immersive Virtual Environments

Immersive virtual environments (IVE) are computer-generated three-dimensional spaces that integrate a set of technologies necessary to simulate real or fictional scenes that can make people feel present in a virtual world.

In fact, feeling immersed in a virtual world is one of the most important characteristics of IVEs. This aspect depends on both the quality of the IVE and the users’ subjective experience. Regarding the quality of the IVE, several features can influence the feeling of immersion, including field of view (FOV), field of regard (FOR), display size and resolution, stereoscopic imaging quality, lighting realism, and frame and refresh rates [111]. However, the subjective experience is influenced by users’ own previous experiences and emotional states.

In this section, we analyse frameworks that have been proposed to manage emotional interaction between the different IVE elements. Our main goal is to find proposals focused on developing IVEs that improve the SoA. In this review, we do not find any frameworks specifically designed for improving the SoA. Nonetheless, there are interesting frameworks that propose technologies to manage IVE features and other characteristics that do influence the SoA, such as emotions, immersion, feeling of presence, physiological changes and content adaptation.

To begin with, there are proposals of closed-loop interactive systems in VR that address the necessity of not only recognising the users’ emotions, but also adapting the environment to them. This is the case in [112,113], in which a framework composed of two modules was proposed: one to recognise emotions and another to elicit them. Thus, the authors conveyed an emotional loop for IVEs, where the virtual world dynamically changes (expression of emotions) based on users’ perceived emotions (emotion recognition). Also, in order to properly recognise emotions, capturing users’ bio-feedback is suggested, as well as collecting the contextual data of relevant features such as attention and engagement. Moreover, in ref. [114], the authors focused on users’ feelings of presence; more specifically, they computed the error of presence. They measured and modelled users’ state of presence in IVE based on discrete emotion recognition. They included a feature extraction component that proposed using physiological signal analysis to understand users’ emotional state and relating it to their sense of presence. These signals, including heart rate (HR), skin conductance level (SCL) and respiratory rate (RR), were proposed to be extracted from ECG and EDA techniques.

In addition, there are other proposals of closed-loop frameworks related to the evaluation of immersive virtual environments. It is worth noticing that this type of frameworks could also guide the implementation of IVEs. This is the case in [115], where an evaluation framework was proposed to define and assess the different components that may affect immersive experience. The emotional components relate to a three-level response system: visual, auditory and haptic.

Related to the implementation of IVEs, there are also frameworks that include interaction modules to be used when developing VR applications. For instance, in [116], an application framework was proposed for developing VR experiences that require equipment, such as sensors to measure physiological data. Although this framework only uses physiological data to show it onscreen for users to acknowledge their physical state, it integrates modules to record these data in real time so that other functionalities can be developed to make use of those data. Also, in [117], an Affective Computing framework was presented to develop serious games for virtual rehabilitation. It is composed of three modules related to emotion recognition, system decision making and environment adaptation. To recognise users’ emotional states, their facial expressions are recorded in real time and analysed through face mapping based on FACS in terms of seven discrete emotions (anger, contempt, disgust, fear, joy, sadness and surprise). This recognition system is also adaptable to dimensional emotions (valence and arousal) captured by, for instance, the use of EEG. Another example of this type of frameworks is present in [118], which focused on the system’s capability to capture and analyse physiological data. It supports diverse sensors, such as EDA, EEG and RR, as well as camera monitoring, to extract physical reactions.

Other works focused on the necessity of developing IVEs that consider emotion cognitive theories. For instance, in ref. [119], the lack of computational emotional models was addressed, specifically for the use of Virtual Humans. Thus, the authors proposed a framework based on appraisal eliciting emotions, related to the cognitive theories of emotion such as Scherer’s [54].

On the other hand, there are several studies related to enhancing the sense of agency by adapting the virtual environments to the users’ body movements. Specifically, in ref. [97], the authors showed that externally reducing the error between one’s motor intention and the visual feedback promotes the sense of agency when performing continuous body movements. Also, in refs. [120,121], the authors suggested that improving sensorimotor hints enhances implicit SoA. Other studies related to the largely explored Virtual Hand Illusion (VHI). In ref. [122], the authors showed that users perceive the virtual hand as part of their own body and that the sense of agency is dependent on the delay and movement variability between their real and the virtual hand. Also, in ref. [123], it was suggested that the sense of agency is influenced by the users’ capability to efficiently control other virtual artefacts instead of by the virtual hand size. In the case of [124], improving body movements and its related agency was addressed from a rehabilitation point of view, suggesting to transfer learning between virtual exercises and the real world.

4. Proposed Framework to Improve the Sense of Agency in Immersive Virtual Environments

Relevant frameworks are studied in Section 3.7, most of them focused on enhancing the sense of presence [114] and immersion [115] or on managing the emotional interactive flow [112]. However, in this work, we are focused on improving and evaluating the sense of agency in virtual environments.

Some virtual environments employ techniques for emotion recognition: facial expressions [117] and modules to capture and analyse real-time physiological data during VR interactions [114,116]. Others, such as [119], highlight the importance of developing frameworks based on appraisal to manage cognitive models, which may be useful for managing intention when addressing the SoA. Also, other research, such as [97,120,122], suggests that the sense of agency can be improved by adapting the virtual environment to the users’ body movements.

Since the sense of agency has a significant impact on our daily lives, in this section we propose a framework which integrates the key elements of the interactive flow within an IVE. We also address the technologies needed to manage its emotions, focusing on enhancing the user’s sense of agency.

The theoretical basis of this framework relies on the definition of agency described in [14], which states that the SoA is composed of two layers (Figure 2). As explained in Section 2.4, the first layer addresses the sense of agency over one’s own action (body agency), and the second layer presents it over external events (external agency).

Although the literature in the field of psychology does not always draw the line between these two layers, this model is selected because its clear structure is more suited for implementation. Moreover, its design is particularly applicable in the context of immersive virtual environments, because it moves far beyond body agency, and it includes external agency, which is essential for evaluating interaction in an IVE. The two-layer model links the SoA not only with a virtual avatar but also with the coherent response of the environment, involving different senses, such as sight, hearing and touch (Figure 7).

This work is based on already existing Affective Computing technologies that can be used on both layers to implement a complete method that improves the SoA for body and external agencies. In fact, AC offers tools to develop closed-loop interactions to improve the SoA by managing emotions. Specifically, AC provides applicable models that could be used to predict intention, as well as systems to generate coherent emotional responses to that intention. In addition, AC technologies allow the evaluation of the SoA through metrics such as anxiety variations or detection of users’ perceived errors during interaction.

The proposed framework shows the main elements involved in closed-loop interaction in immersive virtual environments: the users and their virtual avatars, the environment itself, and the interaction that could happen between them. This interaction flow is shown in Figure 8. Dashed lines indicate the transmission of sensations or emotions and continuous lines indicate the coupling of two elements, implying direct interaction between them. Each element groups the technologies that we propose in order to manage the necessity of recognising, expressing and generating emotions depending on a given situation.

Each of these interactive elements generates events and responses that could influence the sense of agency of the users involved in the interaction. This framework aims to integrate the technologies related to the AC field that allow interpreting the intention and emotional state of the user, to generate a response from the IVE that fulfills their expectations following the theory of the two-layer model which integrates body and external agencies.

Since the framework shows the general CA technologies that allow each element to recognise, interpret and coherently respond to inquiry for all the elements involved in the interaction flow, the technical implementation of this framework depends on the needs of the EVI. For example, in the case of the avatar’s body expressiveness, the framework proposes technologies such as real-time (RT) facial or body animation systems that the EVI designer has to choose according to the technical and functional requirements of their virtual environment.

The closed-loop interactive flow works as follows: the user is immersed in an IVE and they are interacting with the virtual world through their virtual avatar (represented with a dashed orange line). The system also sends their emotions, feelings, and physiological changes to their avatar in addition to VR controller events. This ensures real-time generation of appropriate avatar behaviour, enhancing body agency. In Figure 8, we propose AC technologies discussed in Section 4 for emotion recognition within the orange circle labelled ER. Through the technologies of the blue circle labelled EE, the avatar is able to generate its corresponding animation and thus transmit the emotional state to the person with whom the user is interacting through the IVE (dashed blue line). When the user interacts directly with the 3D content (green line) or with non-player characters (NPCs), the user’s emotional information is also sent to the IVE. This way, the response to the interactive event can be adapted to the user’s emotional state, thereby improving their external agency. This adapted response is generated from recognition of the user’s emotion and intention, which is sent to the IVE. Then, the IVE employs the technologies described in the pink circle labelled EG, selects the most appropriate emotional response and expresses it by employing the technologies of the EE circle. Ideally, during the entire interactive flow, the SoA is also controlled to adapt IVE behaviour. As seen in the state of the art, evaluating the SoA in real time is a complex task. To achieve this goal, the framework proposes to work with the technologies described in the yellow circle. Finally, the overall user experience and sense of agency are also evaluated. For this purpose, there are several subjective questionnaires that can be easily applied.

5. Discussion and Future Work

After conducting a review of the technologies involved in the area of Affective Computing in Section 3, we concluded that there are mature technologies that can help to improve the sense of agency in immersive virtual environments. However, we detected that there are still several challenges that need to be addressed in the following ways:

• Develop new techniques for adjusting emotion recognition. In the literature, recognition can be achieved by analysing expressive behaviour and measuring physiological changes. The first one has significantly improved in recent years with the advancement of Artificial Intelligence; however, recognizing emotions by analyzing expressive behavior from the data sources required by AI may involve gender and cultural biases. When recognising emotions by means of physiological signals, the preferred models are the dimensional ones. In the literature, we found that most researchers use the two-dimensional model, since there are sensors that can help to predict the level of valence and arousal reached by the user. However, in order to work with agency and the sense of agency, it is important to acknowledge the third dimension, the dominance, although objective measurement of dominance has not been achieved yet. Schachter and Singer [56] established that the origin of emotions comes, on the one hand, from our interpretation of the peripheral physiological responses of the organism, and on the other hand from the cognitive evaluation of the situation that originates those physiological responses. Following this theory, in this work, we propose as a future challenge to develop a system capable of recognising emotions in real time through the combination of physiological metrics and emotional cognitive models. These cognitive models allow modelling users’ objectives, standards and attitudes, as well as launching one emotion or another based on the agency of an event. We believe that the system could use, on the one hand, physiological measurements for situating a user’s emotional state in the correct global space of dimensional models. Then, with cognitive models such as Roseman [53] or OCC [28] which allow us prediction of the emotion in context (influenced by objectives, norms and attitudes), the system could refine emotional recognition. Using those techniques, the system will simulate the Schachter and Singer [56] process which established that the origin of emotions comes, primarily, from physiological responses of the organism, and then from the cognitive evaluation of the situation.
• Improve the external agency of the two-layer model [14] by adapting audio, visual and tactile content to users’ emotions recognised through the combination of physiological metrics and cognitive theories. Adaptation of IVE content has been implemented at the visual level by means of interactive storytelling techniques that employ user events as inputs. However, they rarely introduce physiological metrics as input to the system or use touch as part of system response, which is essential when working with the SoA.
• Improve the body agency of the two-layer model [14] by integrating the physiological metrics on the avatar body animation. The development of techniques that allow adapting the avatar animation based on the user’s physiological metrics is less widespread than the expression of emotions through facial and body animation techniques. As physiological signals may involve external body changes, such as chest movement (from respiratory rate) or skin colour (from temperature), in this work, achieving a high body agency is considered essential.
• Establish objective metrics for the assessment of the SoA in IVEs. In the literature review, several subjective questionnaires have been found for the assessment of the SoA. However, these subjective questionnaires should be accompanied by the analysis of objective metrics for a more unbiased assessment. Techniques based on Libet’s clock [108] measure Intentional binding, but they are not applicable to IVEs, as they are constrained by requiring high visual attention and they do not allow natural interaction with the immersive environment. Therefore, it is necessary to investigate the creation of tools that allow objective measurement of agency.