A Digital Coach to Promote Emotion Regulation Skills

Abstract

There is growing awareness that effective emotion regulation is critical for health, adjustment and wellbeing. Emerging evidence suggests that interventions that promote flexible emotion regulation may have the potential to reduce the incidence and prevalence of mental health problems in specific at-risk populations. The challenge is how best to engage with at risk populations, who may not be actively seeking assistance, to deliver this early intervention approach. One possible solution is via digital technology and development, which has rapidly accelerated in this space. Such rapid growth has, however, occurred at the expense of developing a deep understanding of key elements of successful program design and specific mechanisms that influence health behavior change. This paper presents a detailed description of the design, development and evaluation of an emotion regulation intervention conversational agent (ERICA) who acts as a digital coach. ERICA uses interactive conversation to encourage self-reflection and to support and empower users to learn a range of cognitive emotion regulation strategies including Refocusing, Reappraisal, Planning and Putting into Perspective. A pilot evaluation of ERICA was conducted with 138 university students and confirmed that ERICA provided a feasible and highly usable method for delivering an emotion regulation intervention. The results also indicated that ERICA was able to develop a therapeutic relationship with participants and increase their intent to use a range of cognitive emotion regulation strategies. These findings suggest that ERICA holds potential to be an effective approach for delivering an early intervention to support mental health and wellbeing. ERICA’s dialogue, embedded with interactivity, therapeutic alliance and empathy cues, provide the basis for the development of other psychoeducation interventions.

1. Introduction

Digital interventions that support mental health and wellbeing may offer one solution to the challenge of lowering rates of mental health problems in specific at-risk populations [1]. Fully automated, online or app-based interventions focusing on self-management can reach people in an immediate and cost-effective way [2] and could potentially be provided to at-risk populations through their workplaces, education facilities or health practitioners. Many studies have shown that digital technologies can be used to increase access to mental health services and to provide effective treatment for people with a mental illness. There is, however, still a significant gap in knowledge as to whether digital technologies can be harnessed to assist in supporting mental health and potentially preventing the development of mental illness over time [3].

To address this challenge, there has been a significant increase in the development of mental health and wellbeing digital apps and programs. Many of these programs lack empirical testing [4,5] and little is known about key effective design features. A recent systematic review by Eisenstadt, Liverpool, Infanti, Ciuvat and Carlsson [6] identified published evaluations of 48 different apps designed to promote positive mental health and wellbeing and/or emotion regulation. Results from their meta-analysis demonstrated a small effect size for reducing mental health symptoms (k = 19, Hedges g = −0.24, 95% CI −0.34 to −0.14; p < 0.001) and improving wellbeing (k = 13, g = 0.17, 95% CI 0.05–0.29, p = 0.004), and a medium effect for emotion regulation (k = 6, g = 0.49, 95% CI 0.23–0.74, p < 0.001). The review concluded that the emerging evidence for digital mental health support and wellbeing apps is promising and that emotion regulation may be an important mechanism to include in future digital mental health support interventions.

Despite the perceived potential of digital mental health support interventions, many authors have acknowledged that more work is needed to better understand and maximize their efficacy [7,8]. Numerous systematic reviews have demonstrated digital mental health interventions have high rates of attrition, low rates of adherence [6,9] and limited use of available features [5]. This can directly impact their effectiveness and indicates a need for careful intervention design.

One approach that appears to hold promise in optimiszng engagement with fully automated, digital interventions is the use of a virtual human or embodied conversational agent (ECA) as a digital coach. ECAs are able to emulate social relationships and have been employed as virtual coaches in many health domains [10]. Studies have shown that ECAs can develop trust and build a therapeutic relationship with users [11] and there is also evidence that they can have a positive effect on user engagement and behavior change intent [12,13].

Although the use of ECAs in healthcare is growing rapidly, it is an emerging area and the efficacy of digital agents is not well understood [14]. The recent growth in popularity of conversational interfaces has been facilitated, in part, by AI enablement, with the recently released ChatGPT as an example. Although exciting in its possibilities, the use of such interfaces in the health domain requires rigorous assessment. Given potential ethical concerns [15], it is particularly important to ensure that the dialogues provide evidence-based advice in a way that has been shown to be effective in human–human contexts. This includes investigating the efficacy of interactive delivery approaches which use empathic, relational, social and reflective choices, as we present later, aimed at empowering the user to self-manage.

In this paper, we provide a detailed description of the design and development of a digital mental health support intervention: ERICA—Emotion Regulation Intervention Conversational Agent. The long-term goal of the research group is to develop a conversational agent that could be used to support people following a traumatic injury as this is a population that has an increased risk of developing mental health problems over time [16]. This paper is divided into the following sections. Section 2 outlines background and related work that have underpinned ERICA’s development. This includes related work in the field of conversational agents (CAs) and ECAs. Section 3 describes the initial pilot trial of ERICA including a detailed description of the key components of ERICA’s conversational dialogues and design. Section 4 presents results, which are then discussed in Section 5. Limitations appear in Section 6 and Future work and Conclusion is presented in Section 7.

2. Background and Related Work

2.1. Conversational Agents (CAs)

CAs have been described as text-based dialogue systems designed to engage in conversation-like behavior with a human to provide support with particular tasks [17,18]. In comparison, ECAs additionally have a virtual visual representation, usually human-like, that simulate face-to-face interactions and incorporate a range of non-verbal behaviors to enhance communication [18]. Both CAs and ECAs are said to provide a more natural medium through which individuals can engage with technology and authors have begun to acknowledge that these agents might provide unique possibilities to engage, build self-awareness, and teach self-management strategies in a range of healthcare domains [10].

The use of CAs and ECAs in health and wellbeing applications is an emerging but fast-growing field of practice. A recent systematic review of mental health chatbots described 12 studies, 6 of which used a CA and 6 used an ECA [4]. The authors of the review concluded that there is currently insufficient evidence to determine whether mental health chatbots (both CAs and ECAs) can produce clinically significant improvements in mental health and wellbeing and called for more research in this area.

In addition to these findings, it is also noted that there are currently no specific design guidelines for ECA development [7] and little is known about appropriate ECA design [19]. It is therefore important to look at recent studies of ECAs to uncover key mechanisms by which ECAs appear to promote user engagement and increase user intention to change behavior. Two mechanisms that have been identified include having an interactive and empathetic approach and establishing a therapeutic relationship with the user [12,20]. These mechanisms are examined in greater detail below.

2.2. Interactive and Empathic Approach

In the field of mental health, a study by Fitzpatrick, Darcy and Vierhile [21] compared two digital mental health interventions, both based on cognitive behavioral therapy (CBT) for people suffering from depression and anxiety disorders (N = 70). One intervention was delivered via a text-based CA called Woebot, while the other was provided as an educational e-book resource. Results of the study indicated that participants who interacted with Woebot demonstrated a significant reduction in their symptoms of depression as measured by the PHQ-9 (F = 6.47; p = 0.01), while those in the information control group did not. Participants who interacted with Woebot were also found to have high levels of engagement, with most individuals using the bot nearly every day. The study authors were unable to track the extent to which the individuals in the control group engaged with the e-book, but they did identify that half the control group reported reading the e-book at least once. These findings indicate that CAs appear to be an engaging way to deliver CBT.

In comparison to CAs, ECAs can be equipped with a range of non-verbal behaviors, for example, head nods and eye gaze, which can increase the connection between ECAs and users [22]. In a study by Bickmore, Gruber and Picard [22], these features have been found to lead to greater likeability and increased trust. In addition, in a study conducted by Lisetti, Amini, Yasavur and Rishe [20], an animated empathetic character, On-Demand Virtual Counselor (ODVIC), was used to deliver a brief motivational intervention aimed at reducing alcohol consumption. This study compared the usability, user experience and the user’s intention to continue to use the intervention delivered via three different delivery mediums. The first was via a CA, the second was via an empathetic ECA and the third was via a non-empathetic ECA. Results from this study indicated that users reported a 30% greater intention to use the intervention that was delivered by the ECA over the one delivered by the text-based CA system. They also found that the empathic ECA was rated significantly higher than a text-only system on several measures of usability; however, the text-only CA was perceived to be more useful than the non-empathetic ECA. These results advocate the importance of ECAs being designed with empathetic attributes.

2.3. Therapeutic Relationship

The therapeutic relationship between a healthcare professional and a patient is often referred to as a therapeutic or working alliance and is considered a key element in successful treatment outcomes [23,24]. Studies of internet-delivered therapies have found evidence of the ability of software programs to form a therapeutic alliance with the user [12] and have found a similarly positive relationship between the therapeutic alliance and clinical outcomes [24]. A systematic review conducted by Flückiger, Del Re, Wampold and Horvath [24] found a correlation (r = 0.275; k = 23) between the therapeutic alliance and clinical outcomes for internet-based psychotherapy. The authors noted that this correlation coefficient was very similar to that obtained in correlations of face-to-face therapeutic alliance and outcomes, suggesting that fostering a therapeutic alliance is an important design element of digital interventions.

The most well-known conceptualization of the therapeutic alliance was proposed by Bordin [25] and suggests the therapeutic alliance is based on three core components. These components are an agreement on therapy goals, a mutual agreement on the tasks that need to be completed to achieve the therapy goals and the development of a bond between the therapist and the patient. Most research on a digital therapeutic alliance has focused on the “bond” element of this alliance. Studies of ECAs have shown that ECAs that adopt human relational behaviors such as humor, empathy and small talk increase a user’s perception of a therapeutic alliance [12,20] and also significantly increase a user’s desire to continue to work with the agent [26].

In more recent work by Abdulrahman and Richards [12], the ECA’s ability to develop the “mutual agreement” elements of the therapeutic alliance were examined. This study investigated whether an ECA delivering stress management advice to students in a single session could increase perceived therapeutic alliance by providing tailored explanations. Their study confirmed that an ECA could build a therapeutic alliance and trusting relationship with users by engaging in positive emotional communication and/or developing a mutual understanding through appropriate explanation. The authors of this study noted that both these factors impacted the students’ intention to change their study behaviors.

The above findings indicate that ECAs with empathic and social dialogue, mutual understanding and explanation can play a role in developing the therapeutic alliance within a single intervention session and can assist in increasing user engagement with and effectiveness of digital health interventions. These characteristics may be particularly important for early mental health intervention approaches when people are more likely to be in the precontemplation stage of behavior change and thus may have lower motivation to engage with such programs.

2.4. Emotion Regulation

It is well recognised that effective emotion regulation is critical for health and wellbeing [27]. At its core, emotion regulation involves the ability to initiate, inhibit and regulate one’s emotions in different situations [8]. There is a growing body of evidence suggesting that poor selection and deployment of emotion regulation strategies contributes to vulnerability and maintenance of a wide range of mental illnesses [28,29,30]. This suggests a need for the development of interventions that assist people to develop a range of emotion regulation skills that they can flexibly draw on when under stress [8,30].

Emotion regulation is recognized as being affected by activities in biological, social, and behavioral domains as well as conscious and unconscious cognitive processes [31]. This broad range of factors means that there are potentially numerous targets for interventions which aim to increase a person’s emotion regulation capacity and has indeed resulted in a wide range of intervention approaches described in the literature. Intervention approaches that target emotion regulation include Acceptance and Commitment Therapy (ACT), Dialectic Behavior Therapy (DBT), and Mindfulness. Many of the emerging digital mental health and wellbeing apps and programs offer a combination of these intervention approaches and require users to engage on multiple occasions over a period of several weeks or months [32]. In designing this intervention, we have chosen to concentrate primarily on a single intervention approach (cognitive emotion regulation) and to develop a highly focused, time-limited intervention, much akin to a digital micro intervention as described by Baumel and colleagues [33]. In doing so, we aim to gain a deeper understanding of the efficacy of ECA delivery for this specific intervention.

It is noted that the selection of cognitive emotion regulation as the target for this intervention was influenced by health outcome research in the domain of injury recovery. Predictors of poor recovery following injury include catastrophising, blame and perceived injustice [34]. These are all cognitive processes that lead to increased negative emotions. Our intervention thus aims to assist individuals to self-reflect on the emotion regulation strategies that they are currently using and assist them to develop a broader range of emotion regulation strategies to draw on in their recovery.

2.5. Promoting Mental Health and Wellbeing

Promoting mental health and wellbeing in at-risk populations can be difficult because individuals within these populations may not be aware that they are engaging in behaviors that place them at risk of poor health. The Transtheoretical Model (TTM) of behavior change [35] proposes that behavior change involves a person progressing through several different stages of change, and that at any one time, 80% of the population will be in the precontemplation (not thinking about change) or contemplation (considering change) stages of change [35]. Noting this, early intervention programs for at-risk populations need to focus on increasing a person’s knowledge of why change may be helpful and on self reflection of current behaviors. Doing so may thus move people into the preparation and then action stages of change.

To help address the pressing need for interventions that can potentially prevent the development of mental illness in at risk populations and to progress knowledge in the field of ECA design and deployment, we have developed and piloted ERICA, a digital coach. Drawing on evidence from the health literature, our intervention targets cognitive emotion regulation and the initial pilot study examines whether an ECA can successfully deliver a highly targeted cognitive emotion regulation psychoeducation intervention. To validate our design, the pilot study aims to address the following research questions:

• RQ1—What is the feasibility and usability of an ECA to deliver a cognitive emotion regulation psychoeducation intervention (ERICA)?
• RQ2—Do participants perceive that they have developed a therapeutic relationship with ERICA?
• RQ3—Does interacting with ERICA increase a participant’s repertoire of cognitive emotion regulation strategies?

3. Materials and Methods

3.1. Study Design

Study participants were randomly allocated to interact with ERICA via one of three different emotion regulation dialogues, each dialogue describing different cognitive emotion regulations strategies. In addition to measuring the usability of ERICA, we aimed to determine whether ERICA’s dialogues promoted the perception of a therapeutic alliance with users and increased their intention to implement cognitive emotion regulation strategies. The study was approved by Macquarie University’s Human Research Ethics Committee (reference number 20211046833620) and the study was conducted online from October 2021 to January 2022. Data were collected anonymously using the Qualtrics research platform.

3.2. Recruitment

Participants were undergraduate psychology students, recruited via an online research participation portal at the host university. This cohort was selected both for its accessibility and because they are recognized as a group that is at risk of psychological distress and poor health [36]. Participants were screened using the Kessler Psychological Distress Scale (K10) [37] and were excluded from participating in this study if they scored as having moderate to high levels of distress. The K10 is a well-validated, clinical measure of psychological symptoms and was used as a screening tool because ERICA was designed as an intervention for at-risk populations rather than individuals currently experiencing a significant mental health problem. Excluded students were advised to seek more comprehensive support through campus and health services. Participants received half an hour of course credit for their participation and participation was voluntary. A total of 289 students initially expressed an interest to participate in the pilot study. Screening of participants with the K10 resulted in 155 participants being eligible to enrol. Of these participants, 15 did not progress to interact with ERICA. Two students interacted with ERICA but did not go on to complete the post-intervention outcome measures.

3.3. Materials

The following subsections present the design and development of the ERICA prototype including Agent Characteristics, User Interaction and Agent Dialogues. ERICA was created in Adobe Fuse, animated with the online service mixamo.com and implemented in the Unity3D Game Engine with the SALSA lip-synching plugin using an in-house ECA dialogue generator in which the Action Interpreter and Dialogue Generator process dialogue files.

3.3.1. Agent Characteristics

Research has shown that the appearance of a conversational agent can influence attitudes and behaviors adopted by users [38]. ERICA’s development was guided by research that had examined gender and aesthetic preferences of ECAs that had been developed in the health and wellbeing domain. In particular, a study by Richards, Alsharbi and Abdulrahman [38] exploring university student preferences for the aesthetics of a virtual support advisor was reviewed. This work found that most students preferred a character of the same age or older than themselves and, of the students who cared about the character’s gender, most females preferred a female support person and males were more spilt in their gender preference. This led the researchers to recommend the use of a female embodied conversational agent if only one agent was available. Furthermore, Schmid Mast, Hall and Roter [39] in their studies of participant perceptions during a virtual medical visit found that a female sex role was congruent to a caring communication style that led to users being quicker to talk and to converse with more emotional statements. Based on these findings, we have chosen to use a female-embodied conversational agent for this study.

We also considered recent work examining the importance of realism in the appearance of an ECA. A study conducted by Thaler, Schlögl and Groth [40] compared four different visual representations of ECAs, each with differing levels of realism. The results of this study indicated that increased perceived humanness correlated with increased perceived eeriness. This appears to indicate that highly realistic agents may not be essential for digital health interventions.

For the aesthetic design of ERICA, we drew from the work of Parmar, Ólafsson, Utami and Bickmore [41], who observed that ECAs in professional attire and within professional environments were perceived to be more credible, trustworthy and likeable, with users being more likely to act on the agent’s advice. To design a professional background, we modified an environment developed for a previous study at our institution for a similar study cohort employing the modelling software Blender. Using an iterative process between the primary researchers, we sought to develop a backdrop representing an informal healthcare consulting environment. ERICA was seated facing the user with an open posture to promote a friendly conversation. The final design (see Figure 1) included an abstract painting with green colouring as research has shown that green colours can promote tranquillity and may help put people at ease in a new environment [42].

3.3.2. User Interaction

ERICA interacted with participants by speaking (using Australian accented text-to-speech voice, Microsoft Catherine) and displaying her conversational dialogue via text on the screen. This duplication was used because studies have shown that the text presentation of educational materials has been found to improve learning over speech alone [43]. Participants interacted with ERICA via a constrained format dialogue with a decision tree system of pre-set options. This design aimed to ensure patient safety and mitigation of harmful, incorrect, or invalid ECA responses due to natural language processing errors [44]. Constraining the user input allows the agent to have a more accurate understanding of the user’s communicative intent in health contexts [22] and reduces the risk of the agent escalating the participant’s distress [45].

Some authors have expressed a concern that interactions with embodied conversation agents can present risks to participants if the technology does not adequately address a reported scenario. Authors have suggested that administrators of these systems need to provide sufficient information to users regarding the scope of use and limitations regarding the use of conversational agents [46]. In our study, participants were informed prior to enrolment in the study that ERICA had not been designed to replace face-to-face therapy and were reminded periodically throughout the study that were chatting with an automated system.

3.3.3. Agent Dialogue

ERICA was designed with the goal of assisting users to learn cognitive emotion regulation strategies to manage negative emotions, specifically those that are common following traumatic injury such as blame, rumination and catastrophising. The conversational dialogues were based on the nine discrete cognitions identified by Garnefski and Kraaij [47] that someone might initiate following a challenging or stressful life event. The nine cognitions include: Self Blame, Blaming Others, Catastrophising, Rumination (worry), Positive Reappraisal, Positive Refocusing, Refocus on Planning, Acceptance, and Putting into Perspective. ERICA’s conversational dialogues paired cognitive emotion regulation strategies that increase negative emotions (Self Blame/Blaming Others, Catastrophising, Rumination) with those that can help to decrease the intensity of negative emotions (Positive Reappraisal, Positive Refocusing, Refocus on Planning, Acceptance and Putting into Perspective); the conversations also incorporated psychoeducation about the detrimental effects that strong and enduring negative emotions could have on mental and physical health. ERICA’s dialogues were constructed with a specific focus on populations at risk of developing mental health problems but not currently experiencing a disorder. The interactive dialogues were scripted with a focus on increasing a person’s knowledge of why change may be helpful. The dialogues also encouraged users to self-reflect on the way they currently manage strong negative emotions with a goal of increasing participants’ self-awareness of mood states.

ERICA’s dialogues were designed to convey empathy through relational cues embedded in the verbal content delivered by the agent. Relational cues have been found to further a human–agent working alliance [48,49]. ERICA’s dialogues included social content, e.g., greetings and farewells, the expression of emotions, e.g., “I am glad/sorry to hear that”, the expression of empathy, e.g., “that must be hard”, politeness and humour. Relational elements also included communication of hope for improvement, motivational statements to promote continued engagement and checking clients understanding and satisfaction with process. In the dialogue design process, we exercised caution regarding reciprocal self-disclosure when designing the empathic verbal content as researchers have highlighted that conversations which attempt to empathize by having the agent describe similar lived experiences have the potential to fracture a working alliance because individuals are aware that digital agents do not have lived experiences [17].

Table 1. Dialogue Example.

Participant	Dialogue	Cue/Behavior
ERICA	I am very happy to meet you and hope you’ll find our time together worthwhile. How has your day been?	Relational/Social Dialogue
User Options	It’s been great (1) It’s been good (2) It hasn’t been that great (3)	Relational
ERICA	That’s good to hear Sorry, to hear it hasn’t been that great (2,3)	Empathic
ERICA	Worrying or rumination becomes a problem when we have difficulty disengaging from these behaviors. The repetitive negative thinking which underpins worry and rumination often has a snowball and spiral-like quality. In this spiral, your feelings of anxiety and depression often increase, which tends to make people think that their thoughts deserve more and more attention. Pretty soon, a person can be living within their mind rather than functioning within the world around them. How often do you find yourself excessively worrying or ruminating?	Education /Reflection
User Options	I am a constant worrier and/or ruminator, I find it very difficult to disengage once I have a thought in my head. (1) I ruminate probably more than I should. I am often very critical of myself and worry about other’s opinions of my decisions. (1) I think I worry and/or ruminate at an average level. I can get stressed now and then but I am able to break a thought spiral when I am feeling overwhelmed. (2) I don’t worry or ruminate very often; people generally describe me as laid back. (3)	Reflection
ERICA’s Response	That must be stressful. Hopefully I can give you some strategies today to help you manage your worrying and rumination more effectively. That’s good to hear. I will be talking about strategies to manage excessive worrying and ruminating so perhaps you can see whether they are the same ones that you use? Lucky you, I am going to be discussing some strategies to help manage excessive worrying and rumination. I wonder if these are the same ones that you use?	Empathic

below provides an excerpt from one of ERICA’s dialogues demonstrating these cues.

ERICA’s dialogues also focussed on promoting self-management and self-efficacy by providing users with knowledge and practice examples to build confidence in their ability to implement strategies to regulate emotions. ERICA’s dialogues incorporated cognitive therapy techniques and activities that focused on developing skills in the areas of planning, problem solving, positive refocusing and reappraisal. Examples and activities suggested by ERICA were guided by seminal works on cognitive therapy including “Cognitive therapy techniques: a practitioner’s guide” [50] and via feedback from professionals in the field of psychology and computing. Figure 2 provides an overview of the structure of the three different conversational dialogues developed for ERICA. It is noted that the three dialogues followed the same structure and were designed to be completed within 15 min. Short intervention timeframes have been recommended in the literature, as studies of digital mental health intervention user behavior highlight that only a small portion of users engage with programs for a long period [51]. In response to this, authors have recommended that there is potentially greater benefit in developing shorter, highly focused interventions which reduce the burden on users [33].

3.4. Outcome Measures

Cognitive Emotion Regulation Questionnaire (CERQ)-SF [47]: The CERQ—SF is an 18-item self-report measure which asks participants to rate how often they use specific cognitive emotion regulation strategies. Ratings range from 1, which is described as “almost never”, to 5, which is described as “almost always”. The measure has been shown to have acceptable internal consistency with Cronbach’s alphas ranging from 0.67 to 0.81 [47].

Session Rating Scale (SRS) [52]: The SRS is a brief measure of therapeutic alliance with four items: Relationship, Goals and Topics, Approach, Overall. A sliding scale from 0 to 10 is used to answer the questions. Psychometric testing of the measure has identified a Cronbach’s alpha of 0.88 and a test–retest reliability of 0.64 [52]. The measure has also been found to have a moderate correlation with the Working Alliance Inventory r = 0.63 [53].

System Usability Scale (SUS) [54]: The SUS is a 10-item questionnaire which assess the usability of digital technology. Recent confirmatory factor analysis identified that the total sum score of the SUS appears to be a valid and interpretable measure to assess the usability of internet-based interventions [55].

General Self Efficacy Scale (GSE) [56]: The GSE is a 10-item psychometric scale that is designed to assess optimistic self-beliefs to cope with a variety of difficult demands in life. Participants respond to each statement on a Likert scale from 1 (never) to 5 (always). The psychometric properties for this scale include a Cronbach’s alpha ranging from 0.76 to 0.90, with the majority of studies in the high 0.80s.

It is noted that the results from the CERQ -SF and General Self Efficacy Scale are not discussed in this paper as they will be discussed in another paper that focuses on ERICA’s psychotherapeutic outcomes.

3.5. Procedure

The study was completed online with informed consent being collected prior to engagement in the study. At commencement, the participants were asked to complete the Kessler Psychological Distress Scale (K10) [37]. The participants who were found to be experiencing moderate to high levels of psychological distress (score > 25) were advised to withdraw from the study. Upon withdrawal, the participants were directed to a page which listed accessible support services and resources.

Participants who scored below 25 on the K10 moved on to complete the baseline measures which included the CERQ-SF [47] and the GSE [56]. Following completion of the baseline measures, the participants were randomly assigned to interact with ERICA via one of three different emotion regulation skill conversations: Refocusing/Planning, Reappraisal, and Putting into Perspective. Study participants were provided with examples and practice activities aligned to their specific skill conversation.

After interacting with ERICA, the participants repeated the baseline measures, the SUS [54] and the SRS [52] and a set of questions exploring their interaction experience. The complete study took approximately 30 min to complete, with participants spending an average time interacting with ERICA of just under 15 min (SD 9:55 min) with a range from 5 s to 1 h and 13 min.

3.6. Data Collection and Analysis

Data were collected using the Qualtrics online research survey software, and all statistical analyses were performed using SPSS V 28.0.1.0 with confidence intervals set at 95%. We used descriptive statistics (Mean and SD) to examine the interval scale data from the System Usability Scale (SUS) and the Session Rating Scale (SRS). To determine whether there were any differences in the SRS ratings across the three different conversation dialogue groups, we conducted a one-way ANOVA of the mean rating for each scale item. To assess user experience, we also collected qualitative data via surveys, and a conventional qualitative content analysis approach was adopted, allowing for themes or categories to emerge from the data [57]. An ad hoc analysis was also conducted to investigate the association between therapeutic alliance and intention to use cognitive emotion regulation strategies.

4. Results

4.1. Study Participants

The study sample consisted of 138 university students with a mean age of 21.7 years (SD 6.747) and a range of 18–58 years (see

Table 2. Sample Demographics.

Sample	N	Mean Age/SD	Female	Male
Refocusing	47 (34%)	21.40 (5.625)	30 (63.8%)	17 (36.2%)
Reappraisal	46 (33%)	22.02 (8.076)	29 (63.0%)	17 (37.0%)
Putting into Perspective	45 (33%)	21.67 (6.530)	24 (55.3%)	21 (46.7%)
Total	138 (100%)	21.7 (6.747)	83 (60.1%)	55 (39.9%)

). Most participants were female (60%) and were in their first year of study. The participants reported diverse cultural backgrounds (see Figure 3 with the majority, 56.5%, identifying as Oceanic (including Australian). A total of 65.9% of participants indicated that they had not studied emotion regulation previously, while 10.2% indicated that they had studied emotion regulation (the remainder were unsure if they had studied emotion regulation).

4.2. System Usability Scale

The SUS asks individuals to rate usability items across five response options ranging from “strongly disagree” to “strongly agree”. Details of the average scores on each of the SUS items is provided in

Table 3. System Usability Scale Results.

Question	Average	Standard Deviation
I think that I would like to use this system frequently	3.09	1.16
I found the system unnecessarily complex	1.67	0.85
I thought the system was easy to use	4.68	0.81
I think that I would need the support of a technical person to be able to use this system	1.45	0.92
I found the various functions within this system were well integrated	4.17	0.90
I thought there was too much inconsistency in the system	1.63	0.71
I imagine that most people would learn to use this system very quickly	4.51	0.82
I found the system very cumbersome to use	2.11	1.19
I felt very confident using the system	4.41	0.90
I needed to learn a lot of things before I could get going with this system	1.67	1.03
Total Score	80.69	11.87

. A single study participant identified that they were not able to interact with ERICA (via qualitative feedback provided in the survey), while two people identified that they had experienced technical difficulties while interacting with ERICA (e.g., lagging).

4.3. Therapeutic Alliance—Session Rating Scale (SRS)

Results from the Session Rating Scale are provided in

Table 4. Session Rating Scale Results.

Dimension	Scored 9 or 10	Scored 5–8	Scored below 5	Missing
Relationship	53 (38.5%)	71 (51.5%)	13 (9.5%)	1
Goals and Topics	29 (21%)	99 (72%)	10 (7.2%)	0
Approach	38 (27.5%)	79 (57.2%)	21 (15.2%)	0
Overall	46 (33.3%)	75 (54.3%)	16 (11.5%)	1

. Participants rate four separate components of alliance on a slider scale from 0 through to 10. For example, the Relationship scale rates the session with the slider on the left labelled “I did not feel heard, understood or respected” and labelled on the right with “I felt heard, understood and respected”. Participant scores ranged from 0 through to 10 on all subscales of the Session Rating Scale except the scale which asked participants to rate whether they had felt that they were able to work on and talk about what they wanted to talk about. On this subscale, scores ranged from 2 to 10. One third of participants rated ERICA overall as a 9 or 10, while just over 10% rated ERICA as below 5. Results from a one-way ANOVA comparing the mean of the Overall Score across the three different conversation groups demonstrated that there were no significant differences in the SRS Overall score across the three different conversational dialogue groups (p = 0.97).

4.4. Qualitative Feedback

To help identify factors for improving the ERICA program, we asked for qualitative feedback from participants who had rated ERICA as not being a good fit for them (item 3 of the SRS, N = 43) and from those had identified that something was missing in the session (item 4 of the SRS, N = 31). It was decided that these two items would best facilitate the provision of constructive feedback from study participants.

Table 5. Themes from Qualitative Questions.

Key Theme	Frequency (N = 74)	Example
Not enough response options	6	“The answers I could select from were limited and not accurate to how I would respond in person”
Preference for human interaction	8	“I found the information very useful. I didn’t like the robot feeling it makes me sad. I prefer human interaction. I felt if I like her, they might stop having a real psychologist or counsellor but I actually found it very useful”
Robotic Voice	2	“The robotic voice made it hard for me to take serious advice and tips”
No new information	3	“I have done a reasonable amount of stress inoculation; most of ERICA’s touch points were known to me already”
Technical issues	2	“The delivery was extremely difficult to keep track of with lagging and delays”
Lack of Empathy	3	“It felt a little impersonal and cold”
Therapy not needed/wanted	2	“I’d rather just not do therapy in general”
Lack of personalisation	5	“It only provided generalized strategies; I prefer to speak to someone who would be able to provide more specific strategies which I could fit into my lifestyle more effectively”
Implementation of strategies	3	“The approach was a good fit for me; however, it is hard to implement such strategies in every given situation”

summarizes key themes identified from the qualitative responses.

4.5. Intention to Use Cognitive Emotion Regulation Strategies

Participants were also asked to identify whether they had an intention to try any of the cognitive emotion regulation strategies described by ERICA.

Table 6. Intention to Implement Strategies.

Strategy	Putting into Perspective N = 45		Refocusing N = 47		Reappraisal N = 46		Total N = 138
Referenced within Conversation	Yes	No	Yes	No	Yes	No
Event Continuum	24			7		13	44 (32%)
Challenging your thoughts	19			7	20		46 (33%)
Writing down your thoughts	24		12		9		45 (33%)
Setting aside 30 min a day		2	21			2	25 (18%)
Acceptance		4		5	28		37 (27%)
Purposeful distraction		3	11			7	21 (15%)
Constructing Alternatives		7		3	13		23 (17%)
Planning		13	23			7	43 (31%)
None	3		4		4		11 (8%)

below highlights strategies that the participants selected and the conversations in which these strategies were described.

Using the event continuum, challenging your thoughts and writing down your thoughts were the three strategies that were most commonly endorsed by study participants. It is noted that the results for those participants who engaged in the Positive Refocusing conversation with ERICA are likely to be affected by a coding error whereby the strategy of refocusing described within ERICA’s conversation was incorrectly labelled within the follow-up survey as “Purposeful distraction”. It is also noted that the “challenge your thoughts” strategy was highly endorsed by participants who listened to the Reappraisal conversation. The “challenge your thoughts” strategy was not specifically described in in the reappraisal conversation but this strategy is very similar to the “constructing alternatives” one and may have been viewed as one and the same by study participants.

4.6. Ad Hoc Analysis

The ad hoc analysis was conducted to examine whether there was any association between participants’ rating of the therapeutic alliance and their intention to use cognitive emotion regulation strategies. The analysis involved dichotomising participants into those who reported intending to try a strategy and those who did not intend to try using a cognitive emotion regulation strategy. The mean of the Overall Approach scale on the SRS was then compared for the two groups. A significant difference in the mean SRS Overall Approach score was found to be 7.33 vs. 4.45, respectively (p < 0.001).

5. Discussion

As can be seen from Table 2, we enrolled 138 students over a short, three-month recruitment period, highlighting the demand and feasibility of the ECA-delivered intervention. In addition, only 3 of the 138 participants indicated that they accessed and used ERICA, and the average participant rating of ERICA on the system usability scale was 80.6938. This rating is considered to reflect high usability and is comparable to scores found in other studies that have examined CA and ECA ease of use. These include a study by Balsa and colleagues who reported an average score 73.75 in a study examining the usability of an ECA to support older people with type 2 diabetes [58] and a score of 81.82 for a study which used an ECA to support the delivery of Dialectical Behavioral Therapy in a blended therapeutic program (where therapy was provided face-to-face and via a digital app) [59]. Taken together, these results support the assertion that delivering an emotion regulation psychoeducation intervention via ERICA is feasible and easy to use (RQ1).

In answering RQ2, one third of study participants rated their therapeutic alliance with ERICA as either a 9 or 10 out of 10 on the Session Rating Scale (for their overall experience), in line with the score expected to be obtained with a human therapist. Only 11.5% of participants scored ERICA below 5. This finding is encouraging as participants interacted with ERICA for only a single session, for an average of just under 15 min. The findings also corroborate the findings of other studies which have demonstrated that ECAs can develop a therapeutic alliance with a user [12,22]. We further comment that the SRS has been designed for face-to-face human-to-human sessions in which the therapist asks the patient at the end of the session to provide a score [52]. This context is likely to result in a predominance of high scores as the patient is likely to feel inclined not to hurt the therapist’s feelings or cause an issue. In this study, the SRS was delivered anonymously online after interacting with ERICA. The SRS findings do, however, need to be treated with caution as it is questionable that ERICA’s dialogues (in their current form) facilitate the development of an agreement on therapy goals, which is one of the three key elements of the therapeutic alliance as proposed by Bordin [25]. A number of authors have begun to question whether current measures of therapeutic alliance, which are based on face-to-face therapy sessions, adequately reflect the unique context of a therapeutic alliance in fully automated digital interventions [12,23,60].

Qualitative feedback collected in this study indicated that three participants found ERICA’s voice robotic and that this contributed to their low ratings of ERICA on the SRS scale. In a recent study investigating the impact of ECA voice types on likeability and user–agent relationships, users were reported to prefer a human voice over text-to-speech synthesis, but this preference did not appear to interfere with their user–agent relationship or interaction outcomes [61]. TTS voices continue to improve, but if outcomes are negatively impacted in future studies, we can record a human voice to speak ERICA’S dialogue, though this makes adaptive dialogue difficult.

RmQ3 of this study aimed to examine whether ERICA was able to increase a person’s repertoire of cognitive emotion regulation strategies. Our results showed that 127 out of 138 study participants identified that they intended to try to implement at least one of the cognitive emotion regulation strategies described by ERICA with only 11 (8%) of the study participants indicating that they did not intend to try any of them. Interestingly, we conducted an ad hoc analysis comparing the mean of the overall approach subscale on the SRS for those who intended to try at least one cognitive emotion regulation strategy with those who did not and found a significant difference in the means which were 7.10 vs. 4.45, respectively. With higher SRS scores indicating greater therapeutic alliance, this would seem to confirm that intervention efficacy in this study was associated with greater therapeutic alliance.

6. Limitations

While the above findings are promising, the absence of a control group in this study limits the ability to draw strong conclusions about the clinical effectiveness of the ERICA program. Future work needs to be completed to confirm that increased intention to use cognitive emotion regulation strategies translated into behavior change and were due to the intervention and not to a placebo or time effects. Further testing also needs to be conducted to determine whether any changes are enduring. In addition, testing on the target population of trauma injury, while not appropriate for a pilot study, may produce different results.

An additional limitation of this study relates to our use of a female ECA. Although our decision was based on previous work indicating females prefer interacting with female ECAs and males were more evenly split in their preferences, this is likely to perpetuate current gender stereotypes whereby female ECAs are predominantly used in helping and assistance roles. As the long-term goal is to develop ERICA for a population that has experienced traumatic injury, it may be more appropriate to offer individuals the choice of a male or female ECA to interact with.

It must also be noted that in this initial study, we did not collect feedback from those who rated ERICA as a 9 or 10 out of 10. This information is important in helping to develop our understanding of ERICA’s design features that participants found engaging and effective and will be collected in planned future work.

7. Future Work and Conclusions

Qualitative data collected in this study indicate that there are significant opportunities to improve and refine the ERICA program. A consistent theme that emerged was that scripted response options were inadequate for capturing the breadth of user thoughts, feelings and behaviors. If a user is unable to locate a response that adequately reflects their beliefs or feelings, then this is likely to lead to frustration and disengagement. These findings suggest formative research with the target population should be prioritized as a next step for ERICA’s ongoing development [62].

The long-term goal for the research group is to develop a digital mental health support intervention for people who have sustained a traumatic injury and are navigating the personal injury compensation process. We have now gained the ethics approval to commence a coproduction study to refine ERICA with two key informant groups. The first is the experienced claims consultants who support injured people making compensation claims, and the second is the people who have finalized a significant claim. Collaboration with these two groups will assist in developing a breadth of response options that are meaningful for users and further refinement of the agent design and dialogues to ensure that they convey empathy, are tailored, relevant and engaging for this population. In addition, in the future, if ERICA is modified for mobile phone delivery, redesign will explore the use of the guidelines for development of mHealth applications [63].

In the field of illness prevention, there is growing interest in whether the development of emotion regulation skills can help protect an individual from developing a mental illness. Recent published protocols highlight that many countries are turning to digital technologies to deliver emotion regulation interventions. Stemming from this interest is a need to understand key mechanisms in the digital environment that can contribute to a person being able to learn and effectively deploy emotion regulation skills after a stressor. Our results show that an ECA can be used to feasibly and acceptably deliver an emotion regulation psychoeducation intervention. Furthermore, ERICA was able to establish a therapeutic relationship with the majority of study participants. This is also the first ECA study that we are aware of that provides preliminary evidence that is short and highly targeted. ECA-delivered intervention can be both engaging and have the potential to generate behavior change intent in the use of cognitive emotion regulation strategies to manage enduring negative emotions.