Reviews of Social Embodiment for Design of Non-Player Characters in Virtual Reality-Based Social Skill Training for Autistic Children

Abstract

The purpose of this paper is to review the scholarly works regarding social embodiment aligned with the design of non-player characters in virtual reality (VR)-based social skill training for autistic children. VR-based social skill training for autistic children has been a naturalistic environment, which allows autistic children themselves to shape socially-appropriate behaviors in real world. To build up the training environment for autistic children, it is necessary to identify how to simulate social components in the training. In particular, designing non-player characters (NPCs) in the training is essential to determining the quality of the simulated social interactions during the training. Through this literature review, this study proposes multiple design themes that underline the nature of social embodiment in which interactions with NPCs in VR-based social skill training take place.

1. Introduction

Autism is a cognitive disorder that can result in deficits to social functioning. As a treatment of this symptom, special training to enhance social skill functioning of children with Asperger syndrome (ASD) has been extensively executed in virtual reality (VR) in autism research [1]. In a VR-based environment, autistic subjects are supposed to build social skills including interpersonal communication, negotiation, and learners’ accountability to malleably face various social situations. In social skill training interventions, to foster autistic children’s social practice, the role of a non-player character as a social robot in an in-situ context during the intervention has been emphasized during the intervention. A NPC is a kind of social robots used to manage learners’ communication practices and thus helps to simulate social contexts similar to those in the real world. Contrary to the importance of NPC as demonstrated in prior studies [2,3], defining how NPCs in VR-based social l skill training need to be applied is still elusive. Moreover, there is no clear overview of which design aspects should be addressed specifically for autistic children. Therefore, the paper aims at investigating design components of social NPCs based on fundamental reviews of social embodiment studies. This paper will briefly glimpse at backgrounds of autism social skill training, as well as the role of NPCs. Then, the paper proposes design features in the development of social NPC, which are underlined by theoretical issues in social embodiment. Finally, the research question of this study is as follows: How can social embodiment explain design features of NPCs in VR-based social skill training for autistic children?

1.1. Social Skill Training in Virtual Reality

Social interactions are key to defining human natures. Interpersonal communication by means of either verbal or nonverbal cues helps to form social relationships among people [1]. In comparison, autistic children have traditionally been perceived as individuals that significantly lack the usual social senses to interact with people in the real world [4]. Some of the seminal research, including the theory of mind (ToM) [5,6] and weal-coherence theory [7,8], suggest that autistic children have difficulty obtaining opportunities to rehearse their social skills. Rather, they are likely to avoid in situ contexts where they are supposed to interpersonally communicate with others in social interactions.

Researchers have proposed interventions aimed at learners’ social skill acquisition [1,9]. In particular, to simulate a naturalistic environment, recent studies have simulated VR-based interventions for social skill training that is likely to allow autistic children to achieve senses of social competencies [10,11]. VR-based social skill training environments had been purposefully proposed as an alternative platform to enhance social interactions for children with ASD. Internet-based virtual communication in the training platform encourages learners to become accustomed to a variety of social manners. Current studies using VR-based interventions highlighted the importance of “learner-centered design” [12,13,14] that facilitates autistic children’s active participation in the social world simulated by VR. To establish an interpersonal and communicative VR environment, many studies adopted a second-life based environment, which can easily manipulate social communication among multiple users. Although many immersive VR devices with head-mounted devices have been released, the second-life platform is still promising due to several reasons. Second-life-based training environments encourage either instructional designers or therapists to easily modify virtual-social settings for social skill training. In contrast to the immersive platforms, which require much development labor, the second-life platform reduces the time to simulate social contexts. In addition, second-life platforms are affordable options that can be accessible, leading to ASD children’s involvements in a virtual world without necessitating their having technology backgrounds.

VR-based training interventions feature several advantages when implementing social skill training for autistic children. First, the sense of being themselves in a VR setting for autistic children increases their engagement. Visually seeing a realistic design in a virtual world provides a similar mood as what they are likely to experience in the real world. Second, VR-based social skill training interventions give an autistic child a new social entity in a virtual world. Avatars, as camouflaged users in a virtual world, create the child’s own identity, helping to reduce social anxiety [12] and then promoting social confidence in interpersonal communication [13]. Lastly, the use of a virtual world allows either psychiatric therapists or instructional designers to readily change or adjust intervention settings in consideration of autistic children’s characteristics. In other words, VR can be versatile in simulating diverse mock-up social contexts for training autistic children [14]. Malleable case design in training provides autistic children an opportunity to experience interpersonal engagement in a variety of social situation.

1.2. Roles of Non-Player Characters in VR

Social awareness is key to boost interpersonal communication in certain social circumstances. Generally, in several studies [14,15], non-player characters (NPC) have been treated as dummy objects that tend to be socially interactive. It is believed to enhance the social awareness of users in simulated social worlds. In early studies related to affective computing, researchers shed light on how to engineer social dynamics to operate NPCs. Those studies did not fully emphasize the social roles of NPCs, but rather automation techniques in order to implement NPCs’ interactive responses. However, recent studies based on social embodiment theories accentuated the necessity of NPCs’ role distributions, particularly in a variety of training applications in VR. In particular, educational researchers have adopted the uses of NPCs to either facilitate or guide social interactions in using training environments simulated by VR. With a scenario-based instructional design, NPCs have multiple roles as social agents: (1) an informer that explains multiple cues to solve a problem-solving task, (2) an emotional companion to maintain high social engagement in the learners, and (3) a task initiator to evoke problematic situations that learners are supposed to encounter in a task.

Furthermore, when it comes to managing social skill training in VR, it is believed that the role of non-player characters is essential. NPCs as social robots in VR can be part of social dynamics that deliberately foster social interaction with autistic children [15]. In the context of VR-based interventions, NPCs are a social agent, which guides autistic children’s social skill practices. The major function of NPCs in the social skill training is to contribute to autistic children’s rehearsals of social skills. During social skill training, NPCs behave as a real human avatar that interacts with autistic children. For instance, NPCs initiate either verbal or nonverbal communication and then build the mood of the given social situations in the training. Through the interplay between NPCs and participants in the training, autistic children acquire opportunities to test their own conversation manners [16], as well as shape their attitudes in proper contexts [17]. Moreover, from a design perspective, NPCs are an operational variable that may govern the task complexity of social skill training for autistic children. For instance, a social skill training intervention may be able to adjust the number of NPCs, as well as the sophistication of social actions, in order to control the task complexity.

1.3. Social Embodiment in Designing a Social Robot

Derived from the fundamental discussion regarding embodied cognition [18,19,20], the notion of embodied cognition has continuously evolved. Researchers are convinced that the interplay between the external world and a person is a key indicator to determine social interactions. They assume that embodied experiences in social communication are synthesized and then transformed [21,22]. As a lens of social embodiment [23], which is a branch of the field of social psychology, it is assumed that body perception of human being in interpersonal communication is likely to be determined by sensory channels. The theories of social embodiment [22,24] offer empirical clues of how to simulate inter-transmission between sensory channels and an intelligent model. For example, social embodiment studies have investigated the way multiple nonverbal clues from human leverage person’s perception and emotional responses. A key theoretical framework highlighting social embodiment is the perceptual symbol system proposed by Barsalou, Niedenthal, Barbey and Ruppert [22]. The theory explains the transaction of concrete and abstract conceptual information with sensory-motor cues. It assumes that a perceptual symbol system underlines a modal representation of conceptual knowledge. In other words, our bodily-sensory cues could be a transmitter to deliver social meanings. In the field of artificial intelligence, simulating interpersonal communication based on the perceptual symbol system has increasingly emerged in association with the development of social robots.

Recent reviews regarding embodiment theory suggests ways in which social perception in human interactions can be replicated by a robot [24,25,26,27]. Researchers believe that human perception in social interactions may influence our cognitive and affective states of a human system [22]. With regard to the transition of socially-perceptual information, Barsalou et al. [22] propose four major issues concerning social embodiment: (1) bodily states influenced by social stimuli, (2) bodily-mimicry implementation, (3) self-produced affective states, and (4) comparability of bodily and cognitive states influencing performance effectiveness. Robot design studies have shown how robot design work can be empirically aligned with the aforementioned principles in prior social embodiment research [25,26]. A few essential elements to be applied for the robot design have been defined: identity, character, stereotypical representations, and global task. Identity is a unique characteristic a robot possesses, which must be differentiated in comparison to others. Character refers to the mental nature exhibiting a differentiated belief system. Stereotypical representations imply social attributes portrayed by explicit social actions. Lastly, global task describes the significance of the social environment design, such that the social task should be a collection of a subset of goals for robots. In other words, the global task should be decomposed by a group of subtasks that the robots perform in which multiple agents appear in a system. It is also connected with entrenched situated conceptualization [22], which is able to contribute to understanding personality and contexts in social interactions. This notion demonstrates the way people can reflect a great deal of social situations that frequently take place.

2. Method

The goal of this paper is to propose design themes for NPCs in VR-based social skill training by using literature reviews. Under this purpose, this paper implemented a systematic literature review [28], which inclusively collected relevant academic articles and synthesized the findings together. Accordingly, this paper sampled a total of forty-two published papers in academic journals and conference proceedings. The databases chosen for this study focused on peer-reviewed journal papers and proceedings regarding social skill training for autistic children, social embodiment, and nonplayer character design as follows:

• PSYINFO: Governed by the American Psychological Association, this database carries out searching for relevant citation sources from a great number of journal articles in psychology and education fields.
• ACM (Association for Computing Machinery): This database is particularly useful for accessing empirical studies demonstrating how computer-based system applications have been exploited.
• ScienceDirect: This database is remarkable, and includes a large amount of scientific and medical research published by Elsevier. It provides free article abstracts with affordable full-text services for peer-reviewed journal articles.
• Google Scholar: This database allows users to find extensive results for relevant literatures. This website filters in many academic sources from publishers and scholarly societies.

Inclusion Criteria

The goal of the search was to extensively analyze full-text scholarly work using several search terms, such as “autism”, “embodied social cognition”, “social embodiment”, and “social skill training”. This study did not limit the date range of the sampled studies. This study emphasized full-text and peer-reviewed scholarly work including high-quality journal articles, conference proceedings, and book chapters. As shown in Figure 1, in this review, an initial screening resulted in 120–130 studies, and we finalized forty-two published studies, which could be essential in addressing the aforementioned concepts. The inclusion criteria of the study are described as follow: (1) relevance of the topic and (2) maintenance of a rigorous research design along with methodology. The criteria also included literature review sources, which stressed conceptual issues about social embodiment. Those papers were specifically used for portraying how social embodiment can be conceptually associated with prior empirical studies.

3. Findings

Derived from reviews of studies regarding social robots and social skill training for autistic children, the study findings suggest two major themes regarding development and design features of NPCs in social skill training as demonstrated in

Table 1. Design factors to simulate a VR-based social skill training.

Major Factors	Design Principles	Underlying Concepts
Activation of bodily-initiated experience	Facial mimicry;	Facial elicitation;
	Simulations of body gestures	Communicative mimicry Online embodiment
Design features of non-player characters in social skill training	Emotional modeling;	Category priming;
	Role specification	Entrenched situated conceptualizations
	Authentic social scenario

.

3.1. Activation of Bodily-Initiated Experiences

3.1.1. Facial Mimicry

Autistic children barely acts on normal facial mimicry [29]. Facial mimicry regards physical reactions that specifically replicate the facial expressions others make. The study by McIntosh [30] reports that autistic participants were likely to fail to carry out the facial mimicry required by autistic children to show their emotional responses externally. Rather, they tend to receive the information from others’ facial expressions. In other words, autistic children mostly face a challenge in that they cannot instantly interpret and react to emotions interpersonally. Oberman et al. [31] implemented an intervention in order to train autistic children’s facial expressions. They sampled a total of 13 autistic children diagnosed with ASD. The intervention is composed of 192 static images of facial expressions. The finding from the study points out that there is a gap of temporal dynamics between participants’ voluntary initiation of facial mimicry and the given materials to be mimicked. The study findings reveal that there are the delayed responses from autistic children, influenced by facial mimicry training. This is called a human-mirror neuron, and is a part of the human brain that particularly controls image replication of external information. Dapretto et al. [32] also conducted experiments to test whether high-functioning autistic participants had any mirror neuron activities in their facial mimicry test. The study resulted in no mirror neuron activation in the inferior frontal gyrus of the human brain in comparison to those of non-autistic participants. Since the findings of subsequent studies have confirmed this, it is believed that facial mimicry is one of the crucial aspects to address in training rudimentary social skills.

With regard to social skill training, NPCs should use multiple kinds of facial mimicry to stimulate emotional responses. High variation of facial mimicry is a gifted ability unique to humans. Facial mimicry has been supportive in giving emotional cues that explain the states of mind. However, owing to the remote condition in VR, autistic children are scarcely exposed to the interactions initiated by facial expressions. In addition, even prior social skill training interventions have mostly not used interactive interventions to train facial mimicry. Otherwise, even if autistic participants had opportunities to experience facial mimicry, this has been limited to receiving facial expression images. Learner-centered activation of facial mimicry to simultaneously communicate with others in VR could effectively address the aforementioned problems in training. For example, collaborative VR, such as Secondlife and Opensimulator, allow users to control facial expressions to interpersonally share emotional states. The heads-up display (HUD) technique in VR platforms encourages learners to initiate different types of facial mimicry in an instantaneous way.

3.1.2. Simulation of Body Gestures

Spontaneous mirroring by neuron activation also influences the initiation of human gestures. Human gestures have been typically defined as all kinetic behavior that delivers either cognitive or emotional information to another being. To generate gestures, humans operate various types of body parts including a group of hand and, arm muscles to signify an external meaning. Mostly, gestures are a supplementary tool used for interpersonal communication [33,34]. Once a person wants to augment the meaning of what he or she is stating, gestures concurrently occur. Specifically, researchers have classified gesture types based on their functionality and intensity of emotion [35,36]. Evidence from early studies [37] proposed that gestures deliver innate emotional signals [36], as well as giving socio-cultural cues to understanding certain circumstances in which interpersonal conversation compensates for verbal impairment [38].

Use of gestures has been a determinant in gauging social functioning disorder of autistic children. According to some studies regarding embodiment [39], it is believed that embodied mimicry usually occurs when beginning with communication. De Marchena and Eigsti [40] noticed that individuals with ASD tended to use fewer gestures when initiating interpersonal communication. In this study, as the researchers sampled 20 adolescents with ASD to observe the frequency of conversational gestures in social interactions in comparison to those of a control group. Through behavioral analyses, the study finding confirmed that gesture count was statistically correlated with the ratings of communicative quality in interactions (r(15) = 0.69, p = 0.005). In addition, to date, various studies have stated that autistic children experienced difficulty in imitating gestures in interpersonal communication [31,32]. The study by Smith and Bryson [39] also elucidated that autistic children struggled with simulating both types of gestures (i.e., socio-communicative and pantomimed actions) compared to non-autistic participants. In response to this need, it is evident that special training to produce gestures by autistic children themselves is beneficial in social communication skill development [41]. Similarly, the study by Ellawadi and Weismer [42] reports evidence that gestures by autistic children are correlated with their behavior regulation and joint attention.

Online embodiment [22,43] and underlying situated cognition [22,43,44,45] are key notions to describe how gestures can be applied in VR-based social skill training for autistic children. Online embodiment is the simulated body schema in virtual environment [22]. Online embodiment demonstrates the importance of sensory and motor responses that ensure autistic children can manipulate their online avatars. In accordance with this assumption, the role of NPCs in social skill training needs to facilitate avatar body movements that can display autistic children’s emotional reactions in different social settings. Technically, for instance, several interventions with body-gesture recognition devices (e.g., Microsoft Kinect) [42,45] have been used for understanding autistic children. Schuller, Marchi, Baron-Cohen, O’Reilly, Robinson, Davies, Golan, Friedenson, Tal and Newman [45] showed a body-tracking based social skill intervention that enabled autistic children to replicate certain gestures in the training.

3.2. Design Features of Non-Player Characters in Social Skill Training

In VR-based social skill training, not only physical mimicry, but also the instructional design of social skill training must be essential in promoting learners’ social skills [46,47]. In social skill training for autistic children, understanding socio-cultural cues in environments are an important component to be acquired [48]. It is a critical measure of to what extent autistic children are able to make use of what they have learned about social cues in different contexts. For example, once autistic children need to resolve arguable issues among other social entities (e.g., NPCs) in the training, they should acutely ponder the underlying reasons for which the argument takes place. Also, autistic children should propose proper alternative solutions in malleable social contexts, as may occur in interpersonal communication. Via this endeavor in the training, they can smoothly enhance cognitive flexibility [12] in how to deal with multifaceted social problems by themselves.

3.2.1. Emotional Modeling of Non-Player Characters

Under social embodiment theories [20,21,22], category priming, is a design idea that explains how to specify the role of NPCs in social skill training for autistic children. Basically, category priming is the cognitive state that describes how human beings are likely to refer relevant stereotype images when evaluating knowledge of another person. Category priming gives a clue of which NPC characteristics can help autistic children to train their category-priming-relevant behaviors via the intervention. Aligned with this issue, some researchers [14] have proposed a goal-based model wherein a NPC reacts to different conversation cases based on a specified emotional model. This system was used to encourage autistic children to practice identification of emotional responses. The model focuses on NPCs’ reciprocation of interactions, which responsively change their actions to proceed in naturalistic social interactions. In addition, to mock an emotionally-realistic NPC, the characteristics of each NPC were compiled using the model of cognitive appraisal theory [49], which considers the variability of social events, actions, and objects in an environment. Another study [50] also suggests a role for NPCs as social companions, which takes into consideration emotional responsiveness, as well as imitation of facial mimicry. NPCs also facilitated autistic children’s facial mimicry based on several types of emotions.

3.2.2. Naturalistic Design of Social Skill Training

Naturalistic intervention design has been progressively adopted in VR-based social skill training for autistic children. Recent interventions in VR [51,52,53] have also been used to enhance the authenticity of a simulated social environment such that it is feasible in the real world. Under social embodiment theory, Niedenthal, Barsalou, Winkielman, Krauth-Gruber and Ric [43] highlight experiential feedback, which encourages a user to experience multifaceted stimuli in different circumstances. The study finding proposed that goal-driven simulation of a social environment could augment the complexity of cognitive tasks. For example, in the intervention, social role playing [53] allows autistic children to smoothly receive verbal and nonverbal information from NPCs. It gives experiential feedback that encourage autistic children to go over various social situations. Wang, Laffey, Xing, Galyen and Stichter [51] designed verbal and nonverbal communication types that are feasible to experience in the real world.

A key for designing a naturalistic intervention is role specification [1,54,55,56] of multiple NPCs in social skill training. Role playing has been a major topic in social skill training for autistic children. It includes trainees’ social perspective-taking [57] skills in accordance with ToM. Role playing offers autistic children a variety of sensory feedback virtually from NPCs. Associated with the entrenched situated conceptualization [21], with regard to issues related to autism research [1,4], it is apparent that children with autism are likely to experience difficulties in addressing other social entities’ emotional responses properly. This indicates that autistic children are responsible for coping with different cases of social issues [57,58] when going over their role-playing activities in their training. In favor of these theoretical frameworks, the role specification also highlights to the importance of designing a variety types of NPCs as social actors. Basically, the nature of a naturalistic intervention in social skill training for children with autism assumes learners’ smooth acquisition of required social manners adaptive to a given social circumstance. It helps learners themselves to flexibly exploit their social behaviors to interact with other social actors in social skill interventions.

3.2.3. Authentic Social Scenario Design

In favor of naturalistic intervention design, social scenarios are deemed to be a narrative, which describes background stories autistic children should identify before resolving a given task. Naturalistic intervention design includes the underlying goal behaviors of children with autism. The goal behavior should not be explicit but be provoked by external social stimuli autistic children experience in the VR. Associated with this principle, social scenarios have been broadly adopted particularly for naturalistic intervention design [51,52,53,57,58], because they encourage autistic children to enthusiastically explore an open-ended virtual world. NPCs have been implemented for (1) evoking socially-feasible issues [13], and (2) giving adaptive feedback to correct inappropriate social manners [12]. The interplay between NPCs and children with ASD in simulated social scenarios provides opportunities, including finding relevant social clues and acting on certain social manners by the children themselves. Authentic social scenarios can be a viable option to enhance learners’ internal motivation in the naturalistic intervention. With VR, storytelling encourages learners to be immersed in a given social simulation. The high quality of authentic social scenarios in the training is believed make it feasible for autistic children to become highly involved in what they virtually experience.

4. Conclusions

In VR-based social skill training for autistic children, using an NPC has been an option to simulate realistic social situations. More than shaping an authentic environment, NPCs have more potential capabilities in terms of being designed to increase learners’ social competency. In this article, I have provided an overview of the design features of NPCs that interact with autistic children in VR-based social skill training. Under reviews of social embodiment, this review paper focuses on major aspects that an autistic child needs to experience for social skill training: (1) facial mimicry, and (2) simulation of body gestures. In addition, the study addresses design themes of NPCs in social skill training: (a) emotional modeling of NPCs, and (b) naturalistic intervention design. Specifically, this review identified the importance of authentic social scenario to leverage the quality of naturalistic intervention design. The review points out the importance of experiential feedback with role specification of NPCs, which should be included for the purposes of naturalistic intervention design.