Next Article in Journal
Deep Transfer Learning for Image Classification of Phosphorus Nutrition States in Individual Maize Leaves
Next Article in Special Issue
iBVP Dataset: RGB-Thermal rPPG Dataset with High Resolution Signal Quality Labels
Previous Article in Journal
Evaluation Method of IP Geolocation Database Based on City Delay Characteristics
Previous Article in Special Issue
Language-Led Visual Grounding and Future Possibilities
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Electronics
Volume 13
Issue 1
10.3390/electronics13010014
electronics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Virtual Agents in DTx: Focusing on Usability and Therapeutic Effectiveness

by
Harim Jeong
1,
Joo Hun Yoo
2,
Michelle Goh
1 and
Hayeon Song
1,*
1
Department of Interaction Science, Sungkyunkwan University, Seoul 03063, Republic of Korea
2
Department of Artificial Intelligence, Sungkyunkwan University, Suwon 16419, Republic of Korea
*
Author to whom correspondence should be addressed.
Electronics 2024, 13(1), 14; https://doi.org/10.3390/electronics13010014
Submission received: 17 November 2023 / Revised: 8 December 2023 / Accepted: 14 December 2023 / Published: 19 December 2023
(This article belongs to the Special Issue Future Trends and Challenges in Human-Computer Interaction)
Browse Figures
Versions Notes

Abstract

:
In the emergent field of digital therapeutics (DTx), this study examines the impact of virtual agent design on usability and therapeutic outcomes. Emphasizing the virtual agent’s role, our research highlights a marked therapeutic effect tied to the DTx’s developed parameters. Continuous usage, influenced by perceived usefulness, user attitudes, and intrinsic enjoyment, emerges as a crucial determinant for desired outcomes. The study finds anthropomorphism and agent likeability as pivotal factors in enhancing user experience and promoting sustained DTx use. Although focusing on mental health, particularly depression, the implications suggest varied results across DTx types. Given these insights, our findings advocate for a deeper exploration into agent-centric DTx designs, particularly in mental health applications. The nuances of user engagement with these therapeutic tools, especially in treating conditions like depression, demonstrate a diverse range of effects and underscore the importance of personalized approaches in digital therapeutics. This study’s outcomes not only shed light on the significant role of virtual agents but also call for continuous innovation and research in this evolving domain.

1. Introduction

The application scope of agents, which has enhanced user engagement and provided a better user experience on computers or mobile devices, has steadily expanded [1]. Although some opinions once believed that virtual agents were less effective compared to robots assisting humans, advances in artificial intelligence have transformed this perception. Virtual agents have achieved significant outcomes when integrated into services [2]. Such evolved virtual agents have been expanded across various domains, one of which is digital therapeutics (DTx) [3]. The benefits of utilizing agents have been in reducing human resources traditionally used in retail customer service, especially in building better system engagement with more social interaction [4].
Particularly in DTx, where long-term use and digital therapeutic alliance, which can be referred to as the relationship between DTx and user, are crucial, the importance of agents is accentuated, given the fact that agents can boost digital alliance [5]. Furthermore, in the prevalent digital therapeutics field of mental healthcare, enhancing sociability to foster agent-user relationships and emotional support added a secondary benefit [6]. Numerous prior studies have thus provided insights into the design and role of agents in the digital healthcare field. However, while these agents offer advantages, the tangible impact of their design on therapeutic outcomes, especially over extended periods, was understudied. This gap was especially significant given the difference between standard digital healthcare products and digital therapeutic devices where therapeutic efficacy is paramount. Specifically, the high dropout rate, which can directly influence therapeutic outcomes in the DTx realm, necessitated research into whether improved usability, facilitated by these agents, could mitigate this challenge.
Despite the emphasized importance of agents in digital therapeutics and healthcare, there was a dearth of in-depth studies reflecting the agent’s genuine therapeutic effects and the characteristic necessity for long-term use. The primary aim of digital therapeutics is behavior change, necessitating sustained usage [7]. However, most studies had focused on the initial impact of agents, leaving their long-term impact unexplored and a subject for future research. Moreover, in DTx, where genuine therapeutic outcomes are paramount and valued over mere placebo effects [8], the scarcity of research into the actual therapeutic effects of agents during the therapy process became even more pronounced.
Against this backdrop, our study sought to explore the causality between the design of agents in digital therapeutic devices and their actual influence on behavior change for prolonged usage. The study adopted an exploratory approach to investigate the impact of virtual agents within DTx. Our primary focus was not on hypothesis testing but on gaining insights and enhancing our understanding of the role of agents in DTx. The research was designed to explore the correlations and implications of agent design on user engagement and therapeutic outcomes in a DTx setting, particularly for depression treatment. Through a month-long experiment, we aimed to gather longitudinal data that captured the evolving dynamics between the virtual agents, users, and therapeutic results. This research specifically focused on identifying which characteristics of a virtual agent are correlated with enhanced usability and therapeutic effectiveness in a digital therapeutic device for depression treatment. The study aimed to understand how these inherent attributes of the agent contribute to achieving desired outcomes in the context of treating depression.
The longitudinal data collected during this period was analyzed using a mixed model to understand the significance of variables over time. Furthermore, we employed stepwise regression to identify the most influential variables and agent-related factors that significantly contributed to long-term usage. We anticipated that these methodological approaches would provide deeper insights into the genuine usability between agents and users and the therapeutic efficacy of agents in DTx. Ultimately, our findings aimed to inform and guide future developers of digital therapeutics in designing more effective agents tailored to their therapeutic objectives.

2. Digital Therapeutics for Mental Health

The digital healthcare market continues to expand, driven by the benefits of delivering effective treatments at a more affordable cost than traditional medical services and better accessibility [7]. As the technology evolved, a new market, officially termed ‘Digital Therapeutics’ (DTx), has emerged, identified by their products that have undergone Food and Drug Administration (FDA) clinical trials to validate their efficacy [9]. Unlike conventional digital healthcare services, DTx values usability and emphasizes proven therapeutic efficacy, requiring clinical validation [10]. As the significance of DTx continues to grow, an interesting observation is its profound impact on a particular sector: mental healthcare. A recent study highlighted that most DTx available in the market today primarily focus on mental health [11]. The increase in such DTx applications is believed to be largely influenced by the global pandemic, which has highlighted the pressing need for effective mental health management solutions [12].
When aiming to treat mental healthcare, especially through digital interventions rather than pharmaceuticals, Cognitive Behavioral Therapy (CBT) is frequently employed [13]. CBT assists patients in understanding their thoughts, emotions, and behaviors and is a technique used for treating various psychological disorders such as insomnia, anxiety, and depression [14]. Depression, in particular, is a prevalent condition where CBT, especially the efficacy proposed by Beck, is well-recognized [15]. Another non-pharmaceutical intervention for depression is Mindfulness Meditation. Unlike traditional meditation, mindfulness has a specific program and guidance for provision [16]. Furthermore, with its clear outcome in stress relief, Mindfulness Meditation is actively used for mental illnesses, including depression [17].
Given the growing importance of mental health management and the potential of digital therapeutics, digital intervention methods like CBT have become increasingly significant [18]. Especially for conditions like depression, where continuous monitoring and timely interventions can make a marked difference, DTx can provide an added advantage [19]. In the context of depression treatment within digital healthcare, approaches like mindfulness and CBT are frequently employed due to their proven efficacy [20,21].
Incorporating these methods in our study not only ensures a stable therapeutic effect but also aligns with the most active DTx approaches, enabling us to offer a more direct perspective on the current state of the art. Therefore, this research deems it appropriate to focus on DTx agent design, specifically targeting mental health and, more precisely, depression treatment. Providing CBT and mindfulness as therapeutic content within the therapeutic agent is a fitting strategy for our research objectives.

3. Virtual Agent in Digital Therapeutics

3.1. Agent Effect

The form of virtual agents and their potential have evolved due to advancements in computer technology, leading to enhanced graphics and more lifelike embodied forms [22]. Additionally, the evolution of natural language processing (NLP) allows these agents to engage in seamless conversations [23]. As a result, there is an increasing expectation for the broader application of virtual agents. In the realm of digital therapeutics, where trust and patient engagement are of paramount importance, the presence of virtual agents has become indispensable [24]. In light of these developments, harnessing the capabilities of agents in DTx is not just beneficial—it is imperative for the future of patient care.
When a virtual agent takes an embodied form, it enhances both spatial and social presence compared to agents that assist in just voice or text format, effectively improving the user experience [25]. Moreover, these agents are capable of non-verbal communication, offering a feeling comparable to face-to-face conversations. This provides a more organic experience for users [26]. Given these enhanced experiences, such agents are finding applications in digital therapeutics, especially in treatments for conditions like autism, where there is a pronounced need to address the lack of social interactions. Additionally, since embodied agents possess an actual form, there is an added potential to adjust their appearance according to the desired objective, heightening expectations in terms of usability [27]. Furthermore, employing agents can enhance user experience, therefore increasing the acceptability of therapeutic interventions [28].
In DTx for mental health, conversational agents are believed to inquire about one’s daily mood and provide cognitive behavior therapy, potentially resulting in more effective treatment outcomes [29]. For instance, Woebot, designed for depression, utilizes NLP technology to offer a conversational agent. It has been reported that its effectiveness in treating symptoms of depression and anxiety was notably higher compared to control groups without the conversational agent [30]. Furthermore, in cases related to mental health, patients or users often need to share private information about themselves. It is suggested that they felt more comfortable and less burdened sharing personal stories with an agent compared to a real person [31]. Moreover, forming rapport, which traditionally refers to the therapeutic relationship between a doctor and a patient, is crucial in mental health. Given the absence of a real person in a digital format, it is posited that having an agent can be advantageous in establishing this necessary relationship [32]. In mental health scenarios, various interactions are essential through user engagement. Experts suggest that when an agent is embodied, it can provide a wider range of interactions, therefore enhancing therapeutic outcomes [33].
Based on the aforementioned studies, it is evident that virtual agents hold significant promise in enhancing the experience and outcomes in digital therapeutics. Through advanced NLP technologies and embodied interactions, these agents not only facilitate more meaningful engagements but also bridge the gap in personal connections often lacking in digital platforms. In our research, we anticipate that integrating such sophisticated virtual agents will pave the way for a more holistic, patient-centric approach in DTx, especially in the realm of mental health. The integration of advanced technology with the interactive capabilities of agents has the potential to redefine how patients experience digital care.

3.2. Agent Design

To maximize the efficacy of agents, it is essential to design them appropriately, considering the objectives of human-agent interaction. When the agent is embodied, understanding the impact of its appearance is especially essential to ensure a beneficial user experience [34]. One of the most frequently mentioned elements related to agent design is anthropomorphism, which refers to the attribution of human characteristics or behaviors to non-human entities or objects. When agents interact with users, their human-like qualities become prominent, and it is possible to enhance trust and expect higher user engagement [35].
However, one study compared humanoid and zoomorphic (animal-like) agents’ effect in a mental health context. The findings revealed that users had a more enjoyable experience with the zoomorphic agents, leading to a higher intention for continued use [36]. From these results, it becomes evident that the significance of anthropomorphism does not necessarily imply that the agent must take on a human form. Instead, what aligns more with the traditional concept is that agents should behave in a human-like manner. Therefore, in designing an agent suitable for our experiment, we felt it was fitting to incorporate elements where the agent speaks like a human while adopting a zoomorphic appearance.
Moreover, a study employing embodied conversational agents for health assessments found no significant difference based on the agent’s gender [37]. In light of this, we considered it appropriate for the agent’s voice to resemble that of a child, which is gender-neutral and hard to discern. Furthermore, according to one study, while the intricacy of an agent’s design did not significantly impact motivation, the effect of the agent was maximized when its design resembled the user [38]. However, users tend to select agents that might not necessarily be the most beneficial for them [39].
Based on these considerations, our research aim is not to determine which agent design is the most effective but rather to identify the specific characteristics of agents that support the long-term use of digital therapeutics and contribute to therapeutic outcomes. Consequently, it seemed appropriate not to offer a choice of agents for our study.

4. Implementation

4.1. Therapeutic Program

Drawing from the insights of the studies reviewed earlier, our experiment endeavors to design a digital therapeutics (DTx) mobile application specifically targeted at treating depression. We incorporated an embodied virtual agent within this application to assess its impact on the treatment process. With the assistance of clinical psychologists, we devised a therapeutic program that, over a month, aids users in identifying their emotions and the associated thoughts, pinpointing any cognitive distortions. We further crafted content rooted in cognitive behavior therapy (CBT) principles to assist users in discerning the core beliefs potentially leading to these distorted thoughts. This content guides users in rectifying misaligned automatic thoughts and core beliefs, equipping them with functional and adaptive cognitive patterns that create less emotional distress and more helpful behaviors.
The program starts with fundamental breathing exercises, setting the groundwork for deeper meditation practices. As users transition to relaxation meditation and body scan exercises, they are also introduced to guided meditation content specifically aligned with the principles of CBT. Over a month, users will delve into 24 CBT-focused sessions and 17 meditation modules. Throughout this journey, the virtual agent offers consistent guidance—introducing activities, monitoring users’ progress, and providing timely feedback. Once users complete the cycle, the program encourages them to repeat the sessions from the beginning.

4.2. Agent Implementation

Drawing insights from previous research and understanding user preferences, we chose a zoomorphic agent design inspired by the Pembroke Welsh Corgi, specifically that of a dog. In the case of the Pembroke Welsh Corgi, it was judged that it would be familiar to people as a livestock herding dog since ancient times, and its cute appearance and small body shape made it suitable for the design of the agent in the smartphone. This agent was crafted in 3D using Maya 2024.01 by Autodesk (San Francisco, CA, USA), a decision driven by the software’s advanced capabilities and user-friendly interface for complex animations. To bring the agent to life and ensure dynamic interactions, we integrated specific animations using the same software.
In the application, users can interact with the agent via typing or voice. The voice interactions leverage state-of-the-art Speech-to-Text (STT) technology, ensuring seamless and natural communication. We utilized Unity 2021.3.18f1 for the overall system development, given its compatibility and efficiency in integrating various elements of the application. For a visual representation, illustrations of the embodied virtual agent, as well as screen samples of the therapeutic content, can be seen in the subsequent Figure 1.

5. Methodology

5.1. Experiment Setting

To validate the effectiveness of the developed DTx and agent, we conducted an experiment involving 70 participants at Sungkyunkwan University, all of whom reported feeling symptoms of depression over the past two weeks. Participants were provided with the specially developed application and encouraged to use it freely over a month, with the recommendation of accessing it at least once every two days. Any technical issues, such as questions about application usage or networking connection errors, were swiftly resolved by our technical support team, ensuring uninterrupted participation and use of the application throughout the study.
Participants were required to make three visits over the month: an initial visit at the commencement of the experiment, a second visit two weeks after that, and a final visit at the month’s conclusion. Given the therapeutic nature of the DTx and to monitor changes in depression-related symptoms, participants were asked to complete the Patient Health Questionnaire (PHQ-9), often used to assess the degree of depression, comprised of nine questions, during each visit. The Institutional Review Board of Sungkyunkwan University approved and oversaw this study for safety and ethical considerations.

5.2. Questionnaire

Our study’s primary objective was to investigate how the virtual agent affects the usability and therapeutic outcomes of digital therapeutics (DTx). Beyond utilizing the previously mentioned Personal Health Questionnaire (PHQ-9), which assesses users’ depression levels [40], we also conducted surveys to gauge participants’ perceptions of the agent and the application’s usability.
First, we utilized a Godspeed measurement, specifically developed for human-robot interaction, to understand users’ views on the agent [41]. Though this study employed a virtual agent instead of a robot, the chosen measurement tool has been frequently applied in scenarios of interaction with virtual agents, making its use justifiable in this context [42].
As for the usability survey, considering that DTx is a new technology and applicable to healthcare, we incorporated the technology acceptance model (TAM) and added the perceived enjoyment variable [43]. Numerous studies have highlighted the significant influence of ‘perceived enjoyment’ on the intent for continued use. Given the expectation that interactions with the agent should be enjoyable, we deemed this variable pertinent [44].
Given the crucial role of credibility in DTx, we used the source credibility measurement to administer another survey. This aimed to evaluate the credibility of therapeutic content offered through interactions with the agent [45].
Although participants completed these questionnaires during all three visits, the source credibility measurement was conducted only during the second and third visits. We made this decision not only because a user’s perception of credibility often evolves with continued use of the application [46] but also because it was deemed impractical for participants to experience the content during their first visit fully.
To summarize, for the interaction with the agent, we assessed variables like anthropomorphism (ANTH), likeability (LIKE), animacy (ANI), perceived intelligence (PI), and perceived safety (PS). These were assessed through five questions derived from the tools developed by Bartneck [47]. For the technology acceptance model, we measured perceived usefulness (PU), perceived ease of use (PEOU), perceived enjoyment (PENJ), attitude (ATT), and intention to use (ITU) or continuous use (CU) after the first visit. These domains were constructed based on prior research and assessed through three distinct questions. Finally, for source credibility, the domains of competence (COMP), goodwill (GOOD), and trustworthiness (TRUST) were measured. The questions for these domains, consisting of five for COMP and six each for GOOD and TRUST, were developed by McCroskey & Teven [48].
To answer these queries, this study will gather data via experiments, and the subsequent analysis will employ mixed model and stepwise regression methodologies using the lme4 package in R Version 4.3.0.

6. Results

A total of 70 participants were initially enrolled in this study, consisting of 38 females and 32 males, aligning with prior research indicating a higher prevalence of depression among females [49]. The participants were predominantly university students with an average age of 23 years. However, during the month-long study period encompassing three scheduled visits, 4 participants withdrew. Specifically, one participant expressed an inability to continue due to personal reasons, and three were no-shows on the second visit. Consequently, data from the first visit involved all 70 participants, but the second and final visits included only 66 participants. It is important to note that the analysis was conducted using data exclusively from the 66 participants who consistently engaged with the application throughout the study. The dropout of participants is believed to be primarily influenced by their academic commitments, a consideration inherent to the university student population from which the sample was drawn. This attrition rate and its potential impacts on the study outcomes have been carefully considered in our analysis.
Preliminary validation was essential to confirm the DTx’s therapeutic effects before analyzing the agent’s influence. Initial PHQ-9 measurements indicated a depression level with a mean of 10.166 (SD = 4.334). After a month of using the application, the mean dropped to 5.742 (SD = 3.935). A t-test was conducted to determine the significance of this change, yielding a p = 4.381 ×  10 − 12 (t = 8.464). In addition to this, an effect size was measured using Cohen’s d, which resulted in a value of 1.04186. This Cohen’s d value, being greater than 1.0, indicates a significantly large effect size. Typically, a Cohen’s d value of 0.2 is considered small, 0.5 medium, and 0.8 large [50]. Therefore, our result suggests a substantial therapeutic impact of the DTx. The detailed results are in Table 1, and the boxplot of participants’ PHQ-9 scores can be found in Figure 2. With this manipulation, the study further analyzes how participants’ perceptions of the agent’s unaltered characteristics influence the therapeutic effect.
Before the main analysis, we ensured the survey’s data reliability by performing a Cronbach’s alpha test. All items exhibited a reliability coefficient above 0.70, indicating acceptable internal consistency. However, a confirmatory factor analysis (CFA) was still conducted to ensure that each item effectively represented its intended construct. Items with standardized factor loadings below 0.65 were identified and removed to improve the analysis’s accuracy. Specifically, the discarded items were: one from PEOU, ANTH, LIKE; two from ANI, PI, PS; three from GOOD, and two from TRUST.

6.1. Mixed Model Analysis

The primary focus of this research was to determine the effect of variables associated with the agent and its usability on the duration of digital therapeutics (DTx) device usage over time. Given that our data were longitudinal, spanning multiple points in time, we adopted a mixed model analysis. This approach allowed us to measure both ‘Fixed effects’, which remain consistent across the entire sample, and ‘Random effects’, which account for variations within specific entities or groups [51]. Our goal was to discern how the overall DTx experience influenced the most pivotal determinant of therapeutic efficacy: sustained usage.
The random effect per participant yielded a SD of 0.2542. This indicates a minimal variance in continued usage perspectives among participants. The intercept in the fixed effect registered a p-value of 0.943833 (t = 0.071), pointing to a minor inter-participant variation. Such results reinforce the feasibility of analyzing the interplay between the variables measured. Notably, significant effects were evident from PU with a p-value of 7.56 ×  10 − 11 (t = 7.266), ATT with a p-value of 0.001 (t = 3.201), and PENJ with a p-value of 0.019 (t = 2.361). The respective estimates for these variables were 0.71304, 0.458635, and 0.38615. This underscores that shifts in PU have the most pronounced influence on variations in CU. Detailed results from the mixed model analysis can be referenced in Table 2.
In this section, we conducted a mixed model analysis to examine the primary factors in our research model, which include the agent, technology acceptance, and source credibility, all within the context of sustained usage of DTx. Our analysis revealed significant fixed effect variables, particularly PU, PENJ, and ATT. These findings underscore the pivotal role of these variables in maximizing the sustained usage of digital therapeutics.

6.2. Stepwise Regression

Following the mixed model analysis, we confirmed the significance of PU, ATT, and PENJ in determining the prolonged use of DTx. Our next focus was to intensively study the agent’s impact, our research’s core subject. Using stepwise regression, we aimed to pinpoint the specific agent attributes influencing each of the main variables (PU, ATT, and PENJ). The data for this examination was averaged over the three visits, representing a month of user interaction.
Before delving into the results, it is essential to understand the Akaike Information Criterion (AIC). AIC is a measure used to compare and select models. In general, a model with a lower AIC is preferable as it suggests a better fit to the data while penalizing for model complexity [52]. Therefore, through our stepwise regression process, our goal was to achieve a model with a minimized AIC value.

6.2.1. PU (Perceived Usefulness)

The starting AIC value stood at −69.2. Through successive removal of variables ANI, LIKE, and PS, only ANTH and PI remained, lowering the AIC to −73.85. Among these, ANTH was the only significant influencer of PU, with p = 0.00414 (t = 0.3117).

6.2.2. PENJ (Perceived Enjoyment)

Starting with an AIC value of −92.78, the model was simplified by removing variables ANI, PI, and PS, preserving ANTH and LIKE, resulting in an AIC of −98.74. Both variables significantly influenced PENJ, with ANTH having a p-value of p = 0.000644 (t = 3.592) and LIKE recording p = 0.011103 (t = 2.617).

6.2.3. ATT (Attitude)

Beginning with an AIC of −86.1, the model streamlined by eliminating variables PS, ANI, and PI, leaving just ANTH and LIKE. This adjustment led to an AIC of −90.28. Both variables proved influential; however, LIKE had a slightly stronger impact with p = 0.0149 (t = 2.505). This analysis results also can be found in Table 3 below.
Through our stepwise regression analysis, we thoroughly investigated the influence of agent attributes, such as ANTH and LIKE, on our primary variables: PU, ATT, and PENJ. This systematic approach enabled us to identify the specific agent-related factors that significantly affect these critical dimensions of technology acceptance. Our research findings offer valuable guidance for agent design within DTx. When striving to enhance users’ perceptions of usefulness, enjoyment, and attitude towards DTx, our study provides actionable insights into the essential agent attributes that should be taken into consideration. This knowledge will allow designers to make informed decisions on optimizing agent characteristics, ultimately leading to improved user experiences.

7. Discussion

Our study ventured into the relatively unexplored field of digital therapeutics (DTx), emphasizing the influence of a virtual agent on usability and therapeutic outcomes—a synergy suggested by prior research [6]. Through a month-long longitudinal study, we provided concrete data on how the agent’s design features correlate with therapeutic efficacy and usability. Before delving into the specific impacts of these design characteristics, we validated the effectiveness of our application as a DTx by observing changes in the PHQ-9 scores. This initial success not only confirmed the therapeutic potential of our application but also established the reliability of the subsequent usability and agent-related data analyzed.
In accordance with the existing literature emphasizing the importance of usability in DTx [7], our study specifically identified which aspects of usability substantially influence therapeutic effectiveness. Through mixed model analysis, we substantiated that perceived usefulness, user attitudes—encompassing both positive and negative attitudes towards the application—and the perceived enjoyment experienced by users are critical for the therapeutic impact. These findings align with our literature review, highlighting the importance of these variables in enhancing both usability and effectiveness in DTx. Future endeavors in this domain should, therefore, consider these variables as pivotal in strengthening the dual aims of usability and therapeutic efficacy.
Evaluating the impact of agent design elements on these identified key components of usability, we found anthropomorphism to be a particularly significant influencer, especially in amplifying perceived usefulness. Both anthropomorphism and the agent’s likeability played a role in cultivating a favorable attitude toward the application. The sensation of anthropomorphism had a notably strong impact on users’ enjoyment, with likeability also making a significant contribution. These insights emphasize that for DTx to reach its full therapeutic potential, sustained usage, supported by perceived usefulness, positive attitude, and user enjoyment, is essential. The roles of anthropomorphism and likeability are crucial in strengthening these foundational elements.
Although previous proposals have suggested the integration of agents in DTx [3], research detailing how these agents influence usability and, ultimately, therapeutic effectiveness has been limited. This study, therefore, holds significant value in proposing a concrete path through which agent design impacts these outcomes. Going beyond mere suggestions for the inclusion of agents, our research provides specific insights into which design aspects can have particular effects, especially within the context of DTx. We believe the value of our study lies in its ability to offer concrete guidance on how agents should be designed within DTx environments, informed by our experimental data.
In this study, we identified key variables such as anthropomorphism and likeability, which are crucial for agent design in DTx to enhance its effectiveness. Although the study spanned a month and involved participants using the application in their daily lives, it is important to acknowledge the limitation of not being able to control all external factors. However, the analysis yielded clear correlations with appropriate effect sizes and p-value validations, supporting the findings within the scope of our research objectives. Despite these limitations, the results provide valuable insights for future DTx applications. Further research is needed to pinpoint specific design elements that effectively augment these qualities. Although our research primarily focused on mental health, particularly depression, we acknowledge the potential for varying results across different types of DTx. Given the burgeoning field of digital therapeutics, continued research in diverse applications is imperative.
In conclusion, our study not only underscores the importance of agent design in enhancing DTx’s usability and effectiveness but also provides a foundational blueprint for incorporating these elements into future DTx solutions. By utilizing longitudinal data in a domain where evidence of long-term usage is limited, our findings pave the way for more comprehensive and actionable approaches in the evolving landscape of digital therapeutics.

8. Conclusions

Our exploration into the realm of digital therapeutics (DTx) has shed light on the significant interplay between virtual agent attributes and the therapeutic outcomes and usability of DTx. The findings confirm the centrality of continuous usage for therapeutic results, with usability being a prime predictor. Moreover, attributes like anthropomorphism and likeability crucially influence user attitudes and perceived enjoyment, further driving sustained DTx use. These insights not only underscore the importance of agent design in future DTx interventions but also emphasize the need for continued research in this rapidly evolving domain. As we advance, pinpointing specific design elements that enhance agent attributes and expanding research across diverse health domains in DTx will be pivotal for crafting more effective digital therapeutic solutions.

Author Contributions

Conceptualization, H.J., J.H.Y. and M.G.; Writing—original draft, H.J., J.H.Y. and M.G.; Supervision, H.S.; Project administration, H.J. and H.S. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2020-0-00990, Platform Development and Proof of High Trust and Low Latency Processing for Heterogeneous·Atypical·Large Scaled Data in 5G-IoT Environment) and Technology Innovation Program (or Industrial Strategic Technology Development Program-Source Technology Development and Commercialization of digital therapeutics) (20014967, Development of Digital Therapeutics for Depression from COVID-19) funded By the Ministry of Trade, Industry & Energy (OTIE Sejong City, Republic of Korea).

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to data privacy.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Abbreviations

The following abbreviations are used in this manuscript:
DTxDigital Therapeutics
CBTCognitive Behavioral Therapy
NLPNatural Language Processing

References

  1. Oertel, C.; Castellano, G.; Chetouani, M.; Nasir, J.; Obaid, M.; Pelachaud, C.; Peters, C. Engagement in human-agent interaction: An overview. Front. Robot. AI 2020, 7, 92. [Google Scholar] [CrossRef]
  2. Thellman, S.; Silvervarg, A.; Gulz, A.; Ziemke, T. Physical vs. virtual agent embodiment and effects on social interaction. In Proceedings of the Intelligent Virtual Agents: 16th International Conference, IVA 2016, Los Angeles, CA, USA, 20–23 September 2016; Proceedings 16. Springer: Cham, Switzerland, 2016; pp. 412–415. [Google Scholar]
  3. op den Akker, H.; Cabrita, M.; Pnevmatikakis, A. Digital Therapeutics: Virtual Coaching Powered by Artificial Intelligence on Real-World Data. Front. Comput. Sci. 2021, 3, 750428. [Google Scholar] [CrossRef]
  4. Verhagen, T.; Van Nes, J.; Feldberg, F.; Van Dolen, W. Virtual customer service agents: Using social presence and personalization to shape online service encounters. J. Comput.-Mediat. Commun. 2014, 19, 529–545. [Google Scholar] [CrossRef]
  5. Tong, F.; Lederman, R.; D’Alfonso, S.; Berry, K.; Bucci, S. Digital therapeutic alliance with fully automated mental health smartphone apps: A narrative review. Front. Psychiatry 2022, 13, 819623. [Google Scholar] [CrossRef] [PubMed]
  6. Wang, Q.; Peng, S.; Zha, Z.; Han, X.; Deng, C.; Hu, L.; Hu, P. Enhancing the conversational agent with an emotional support system for mental health digital therapeutics. Front. Psychiatry 2023, 14, 1148534. [Google Scholar] [CrossRef] [PubMed]
  7. Dang, A.; Arora, D.; Rane, P. Role of digital therapeutics and the changing future of healthcare. J. Fam. Med. Prim. Care 2020, 9, 2207. [Google Scholar] [CrossRef] [PubMed]
  8. Lutz, J.; Offidani, E.; Taraboanta, L.; Lakhan, S.E.; Campellone, T.R. Appropriate controls for digital therapeutic clinical trials: A narrative review of control conditions in clinical trials of digital therapeutics (DTx) deploying psychosocial, cognitive, or behavioral content. Front. Digit. Health 2022, 4, 823977. [Google Scholar] [CrossRef]
  9. Yoo, J.H.; Jeong, H.; Chung, T.M. Cutting-Edge Technologies for Digital Therapeutics: A Review and Architecture Proposals for Future Directions. Appl. Sci. 2023, 13, 6929. [Google Scholar] [CrossRef]
  10. Fürstenau, D.; Gersch, M.; Schreiter, S. Digital Therapeutics (DTx). Bus. Inf. Syst. Eng. 2023, 65, 349–360. [Google Scholar] [CrossRef]
  11. Hong, J.S.; Wasden, C.; Han, D.H. Introduction of digital therapeutics. Comput. Methods Programs Biomed. 2021, 209, 106319. [Google Scholar] [CrossRef]
  12. Dang, A.; Dang, D.; Rane, P. The expanding role of digital therapeutics in the post-COVID-19 era. Open Covid J. 2021, 1, 32–37. [Google Scholar] [CrossRef]
  13. Grant, A.; Townend, M.; Mulhern, R.; Short, N. Cognitive Behavioural Therapy in Mental Health Care; Sage: Newcastle, UK, 2010. [Google Scholar]
  14. Fenn, K.; Byrne, M. The key principles of cognitive behavioural therapy. InnovAiT 2013, 6, 579–585. [Google Scholar] [CrossRef]
  15. Hollon, S.D.; Beck, A.T. Cognitive and cognitive-behavioral therapies. Bergin Garfield’s Handb. Psychother. Behav. Chang. 2013, 6, 393–442. [Google Scholar]
  16. Eberth, J.; Sedlmeier, P. The effects of mindfulness meditation: A meta-analysis. Mindfulness 2012, 3, 174–189. [Google Scholar] [CrossRef]
  17. Shonin, E.; Van Gordon, W. The mechanisms of mindfulness in the treatment of mental illness and addiction. Int. J. Ment. Health Addict. 2016, 14, 844–849. [Google Scholar] [CrossRef]
  18. Shah, A.; Kraemer, K.R.; Won, C.R.; Black, S.; Hasenbein, W. Developing digital intervention games for mental disorders: A review. Games Health J. 2018, 7, 213–224. [Google Scholar] [CrossRef]
  19. Mantani, A.; Kato, T.; Furukawa, T.A.; Horikoshi, M.; Imai, H.; Hiroe, T.; Chino, B.; Funayama, T.; Yonemoto, N.; Zhou, Q.; et al. Smartphone cognitive behavioral therapy as an adjunct to pharmacotherapy for refractory depression: Randomized controlled trial. J. Med. Internet Res. 2017, 19, e373. [Google Scholar] [CrossRef]
  20. Taylor, H.; Strauss, C.; Cavanagh, K. Can a little bit of mindfulness do you good? A systematic review and meta-analyses of unguided mindfulness-based self-help interventions. Clin. Psychol. Rev. 2021, 89, 102078. [Google Scholar] [CrossRef]
  21. Cheng, P.; Luik, A.I.; Fellman-Couture, C.; Peterson, E.; Joseph, C.L.; Tallent, G.; Tran, K.M.; Ahmedani, B.K.; Roehrs, T.; Roth, T.; et al. Efficacy of digital CBT for insomnia to reduce depression across demographic groups: A randomized trial. Psychol. Med. 2019, 49, 491–500. [Google Scholar] [CrossRef]
  22. Baldassarri, S.; Cerezo, E.; Seron, F.J. Maxine: A platform for embodied animated agents. Comput. Graph. 2008, 32, 430–437. [Google Scholar] [CrossRef]
  23. de Cock, C.; Milne-Ives, M.; van Velthoven, M.H.; Alturkistani, A.; Lam, C.; Meinert, E. Effectiveness of conversational agents (virtual assistants) in health care: Protocol for a systematic review. JMIR Res. Protoc. 2020, 9, e16934. [Google Scholar] [CrossRef] [PubMed]
  24. Philip, P.; Dupuy, L.; Auriacombe, M.; Serre, F.; de Sevin, E.; Sauteraud, A.; Micoulaud-Franchi, J.A. Trust and acceptance of a virtual psychiatric interview between embodied conversational agents and outpatients. NPJ Digit. Med. 2020, 3, 2. [Google Scholar] [CrossRef] [PubMed]
  25. Schmidt, S.; Bruder, G.; Steinicke, F. Effects of virtual agent and object representation on experiencing exhibited artifacts. Comput. Graph. 2019, 83, 1–10. [Google Scholar] [CrossRef]
  26. Provoost, S.; Lau, H.M.; Ruwaard, J.; Riper, H. Embodied conversational agents in clinical psychology: A scoping review. J. Med. Internet Res. 2017, 19, e151. [Google Scholar] [CrossRef] [PubMed]
  27. Bonfert, M.; Zargham, N.; Saade, F.; Porzel, R.; Malaka, R. An Evaluation of Visual Embodiment for Voice Assistants on Smart Displays. In Proceedings of the 3rd Conference on Conversational User Interfaces, Bilbao, Spain, 27–29 July 2021; pp. 1–11. [Google Scholar]
  28. Burton, C.; Szentagotai Tatar, A.; McKinstry, B.; Matheson, C.; Matu, S.; Moldovan, R.; Macnab, M.; Farrow, E.; David, D.; Pagliari, C.; et al. Pilot randomised controlled trial of Help4Mood, an embodied virtual agent-based system to support treatment of depression. J. Telemed. Telecare 2016, 22, 348–355. [Google Scholar] [CrossRef] [PubMed]
  29. D’Alfonso, S. AI in mental health. Curr. Opin. Psychol. 2020, 36, 112–117. [Google Scholar] [CrossRef]
  30. Fitzpatrick, K.K.; Darcy, A.; Vierhile, M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): A randomized controlled trial. JMIR Ment. Health 2017, 4, e7785. [Google Scholar] [CrossRef]
  31. Yokotani, K.; Takagi, G.; Wakashima, K. Advantages of virtual agents over clinical psychologists during comprehensive mental health interviews using a mixed methods design. Comput. Hum. Behav. 2018, 85, 135–145. [Google Scholar] [CrossRef]
  32. Luxton, D.D. An introduction to artificial intelligence in behavioral and mental health care. In Artificial Intelligence in Behavioral and Mental Health Care; Elsevier: Amsterdam, The Netherlands, 2016; pp. 1–26. [Google Scholar]
  33. Suganuma, S.; Sakamoto, D.; Shimoyama, H. An embodied conversational agent for unguided internet-based cognitive behavior therapy in preventative mental health: Feasibility and acceptability pilot trial. JMIR Ment. Health 2018, 5, e10454. [Google Scholar] [CrossRef]
  34. Groom, V.; Nass, C.; Chen, T.; Nielsen, A.; Scarborough, J.K.; Robles, E. Evaluating the effects of behavioral realism in embodied agents. Int. J. Hum.-Comput. Stud. 2009, 67, 842–849. [Google Scholar] [CrossRef]
  35. Epley, N.; Waytz, A.; Cacioppo, J.T. On seeing human: A three-factor theory of anthropomorphism. Psychol. Rev. 2007, 114, 864. [Google Scholar] [CrossRef] [PubMed]
  36. Jeong, H.; Yoo, J.H.; Song, H. Virtual Agents with Augmented Reality in Digital Healthcare. In Proceedings of the 2022 13th International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Republic of Korea, 19–21 October 2022; IEEE: New York, NY, USA, 2022; pp. 2016–2021. [Google Scholar]
  37. Ter Stal, S.; Broekhuis, M.; van Velsen, L.; Hermens, H.; Tabak, M. Embodied conversational agent appearance for health assessment of older adults: Explorative study. JMIR Hum. Factors 2020, 7, e19987. [Google Scholar] [CrossRef] [PubMed]
  38. Baylor, A.L. Promoting motivation with virtual agents and avatars: Role of visual presence and appearance. Philos. Trans. R. Soc. Biol. Sci. 2009, 364, 3559–3565. [Google Scholar] [CrossRef] [PubMed]
  39. Seeger, A.M.; Pfeiffer, J.; Heinzl, A. Texting with humanlike conversational agents: Designing for anthropomorphism. J. Assoc. Inf. Syst. 2021, 22, 8. [Google Scholar] [CrossRef]
  40. Torous, J.; Staples, P.; Shanahan, M.; Lin, C.; Peck, P.; Keshavan, M.; Onnela, J.P. Utilizing a personal smartphone custom app to assess the patient health questionnaire-9 (PHQ-9) depressive symptoms in patients with major depressive disorder. JMIR Ment. Health 2015, 2, e3889. [Google Scholar] [CrossRef] [PubMed]
  41. Bartneck, C.; Cochrane, T.; Nokes, R.; Chase, G.; Chen, X.; Cochrane, T.; Mitrovic, A.; O’Sullivan, A.; Wang, W.; Adams, B. Godspeed Questionnaire Series: Translations and Usage. In International Handbook of Behavioral Health Assessment; Krägeloh, C.U., Medvedev, O.N., Alyami, M., Eds.; Springer: Cham, Switzerland, 2023; pp. 1–35. [Google Scholar]
  42. Schneider, S.; Kummert, F. Comparing the effects of social robots and virtual agents on exercising motivation. In Proceedings of the Social Robotics: 10th International Conference, ICSR 2018, Qingdao, China, 28–30 November 2018; Proceedings 10. Springer: Cham, Switzerland, 2018; pp. 451–461. [Google Scholar]
  43. Isernia, S.; Pagliari, C.; Jonsdottir, J.; Castiglioni, C.; Gindri, P.; Gramigna, C.; Palumbo, G.; Salza, M.; Molteni, F.; Baglio, F.; et al. Efficiency and patient-reported outcome measures from clinic to home: The human empowerment aging and disability program for digital-health rehabilitation. Front. Neurol. 2019, 10, 1206. [Google Scholar] [CrossRef] [PubMed]
  44. Perski, O.; Short, C.E. Acceptability of digital health interventions: Embracing the complexity. Transl. Behav. Med. 2021, 11, 1473–1480. [Google Scholar] [CrossRef] [PubMed]
  45. Lundgren, A.S.; Lindberg, J.; Carlsson, E. “Within the hour” and “wherever you are”: Exploring the promises of digital healthcare apps. J. Digit. Soc. Res. 2021, 3, 32–59. [Google Scholar] [CrossRef]
  46. Chang, Y.S.; Zhang, Y.; Gwizdka, J. The effects of information source and eHealth literacy on consumer health information credibility evaluation behavior. Comput. Hum. Behav. 2021, 115, 106629. [Google Scholar] [CrossRef]
  47. Bartneck, C.; Kulić, D.; Croft, E.; Zoghbi, S. Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int. J. Soc. Robot. 2009, 1, 71–81. [Google Scholar] [CrossRef]
  48. McCroskey, J.C.; Teven, J.J. Goodwill: A reexamination of the construct and its measurement. Commun. Monogr. 1999, 66, 90–103. [Google Scholar] [CrossRef]
  49. Albert, P.R. Why is depression more prevalent in women? J. Psychiatry Neurosci. 2015, 40, 219–221. [Google Scholar] [CrossRef] [PubMed]
  50. Kyaw, B.M.; Saxena, N.; Posadzki, P.; Vseteckova, J.; Nikolaou, C.K.; George, P.P.; Divakar, U.; Masiello, I.; Kononowicz, A.A.; Zary, N.; et al. Virtual reality for health professions education: Systematic review and meta-analysis by the digital health education collaboration. J. Med. Internet Res. 2019, 21, e12959. [Google Scholar] [CrossRef] [PubMed]
  51. Sorensen, D.; Kennedy, B. Analysis of selection experiments using mixed model methodology. J. Anim. Sci. 1986, 63, 245–258. [Google Scholar] [CrossRef]
  52. Yamashita, T.; Yamashita, K.; Kamimura, R. A stepwise AIC method for variable selection in linear regression. Commun. Stat.-Theory Methods 2007, 36, 2395–2403. [Google Scholar] [CrossRef]
Electronics 13 00014 g001
Figure 1. Screenshot of the developed DTx application interface.
Figure 1. Screenshot of the developed DTx application interface.
Electronics 13 00014 g001
Electronics 13 00014 g002
Figure 2. Visualization of changes in PHQ-9 scores over visits using a box plot.
Figure 2. Visualization of changes in PHQ-9 scores over visits using a box plot.
Electronics 13 00014 g002
Table 1. Comparison of PHQ-9 scores across visits using t-test.
Table 1. Comparison of PHQ-9 scores across visits using t-test.
PHQ-9Paired t-Test
MeanSdDifferencetp-Valuetp-Value
1st Visit10.1664.334
2nd Visit7.5304.272−2.6365.6403.9 × 10 − 7
3rd Visit5.7423.935−1.787 4.2068.1 × 10 − 5
Table 2. Mixed Model Results for Random and Fixed Effects.
Table 2. Mixed Model Results for Random and Fixed Effects.
Groups VarianceStd.Dev.
Random EffectsSubjectID(Intercept)0.06460.2542
Residual 0.23830.4882
Groups EstimateStd. Errordft valuePr (>|t|)
Fixed Effects
(Intercept)0.2221970.61702776.6635950.3600.719755
PU0.4429040.123171116.9980733.5960.000475 ***
PEOU−0.0447080.08063485.641294−0.5540.580710
PENJ0.2742630.116143115.2448072.3610.019884 *
ATT0.4586350.143274116.9660133.2010.001763 **
COM0.0448500.077301113.8413210.5800.562925
GOOD−0.0035060.09404596.229423−0.0370.970341
TRUST−0.0267900.104166107.199793−0.2570.797530
ANTH−0.0405610.082591116.917753−0.4910.624272
LIKE−0.2843250.163218100.589169−1.7420.084567
ANI0.0770890.090734115.2841780.8500.397302
PI−0.0319540.089334112.556949−0.3580.721246
PS0.0567540.10957789.9920230.5180.605772
time3−0.0894530.09059859.719444−0.9870.327449
Statistical Significance: 0 “***” 0.001 “**” 0.01 “*”.
Table 3. Stepwise Regression Results.
Table 3. Stepwise Regression Results.
EstimateStd. ErrorPr (>|t|)
PU ∼ ANTH0.31170.10480.00414 **
PU ∼ PI0.23030.14130.10823
PENJ ∼ ANTH0.318290.088620.000644 ***
PENJ ∼ LIKE0.417000.159370.011103 *
ATT ∼ ANTH0.208610.093770.0297 *
ATT ∼ LIKE0.422430.168630.0149 *
Statistical Significance: 0 “***” 0.001 “**” 0.01 “*”.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Jeong, H.; Yoo, J.H.; Goh, M.; Song, H. Virtual Agents in DTx: Focusing on Usability and Therapeutic Effectiveness. Electronics 2024, 13, 14. https://doi.org/10.3390/electronics13010014

AMA Style

Jeong H, Yoo JH, Goh M, Song H. Virtual Agents in DTx: Focusing on Usability and Therapeutic Effectiveness. Electronics. 2024; 13(1):14. https://doi.org/10.3390/electronics13010014

Chicago/Turabian Style

Jeong, Harim, Joo Hun Yoo, Michelle Goh, and Hayeon Song. 2024. "Virtual Agents in DTx: Focusing on Usability and Therapeutic Effectiveness" Electronics 13, no. 1: 14. https://doi.org/10.3390/electronics13010014

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop