Digital Mental Health Post COVID-19: The Era of AI Chatbots

Abstract

Digital mental health resources have expanded rapidly in the wake of the COVID-19 pandemic, offering new opportunities to improve access to mental healthcare through technologies such as AI chatbots, mobile apps, and online platforms. Despite this growth, significant challenges persist, including low user retention, limited digital literacy, unclear privacy regulations, and insufficient evidence of clinical effectiveness and safety. AI chatbots, which act as virtual therapists or companions, provide counseling and personalized support, but raise concerns about user dependence, emotional outcomes, privacy, ethical risks, and bias. User experiences are mixed: while some report enhanced social health and reduced loneliness, others question the safety, crisis response, and overall reliability of these tools, particularly in unregulated settings. Vulnerable and underserved populations may face heightened risks, highlighting the need for engagement with individuals with lived experience to define safe and supportive interactions. This review critically examines the empirical and grey literature on AI chatbot use in mental healthcare, evaluating their benefits and limitations in terms of access, user engagement, risk management, and clinical integration. Key findings indicate that AI chatbots can complement traditional care and bridge service gaps. However, current evidence is constrained by short-term studies and a lack of diverse, long-term outcome data. The review underscores the importance of transparent operations, ethical governance, and hybrid care models combining technological and human oversight. Recommendations include stakeholder-driven deployment approaches, rigorous evaluation standards, and ongoing real-world validation to ensure equitable, safe, and effective use of AI chatbots in mental healthcare.

1. Introduction

1.1. Context

The mental health treatment gap has worsened since the COVID-19 pandemic, which triggered a digital transformation of mental healthcare due to changing social dynamics, the widespread use of smartphones, and the proliferation of digital mental health tools [1,2,3,4,5]. This technological shift has fundamentally altered how support is delivered by broadening accessibility and expanding reach for those seeking personalized mental health support.

1.2. Proliferation

Digital mental health tools—particularly those underpinned by Artificial Intelligence (AI) chatbots—are attractive because of their low cost, accessibility, and anonymity. However, the growing interest in AI chatbots has not yet translated into clinical benefits and improved outcomes for users (patients) with anxiety and depression [6]. Questions remain with regard to the safe, engaging and effective integration into existing models of care [7,8,9,10,11].

Despite there being more than 10,000 digital mental health resources globally, low user retention rates appear persistent in combination with a lack of digital literacy, clear privacy guidelines and proven clinical efficacy/integration, as well as human support in the app [5,6]. AI chatbots operate in a largely unregulated environment, which exposes vulnerabilities, especially in underserved populations.

1.3. Mediators

Consequently, there are renewed calls for trained digital navigators to assist in the safe integration of technology into mental healthcare settings, driving engagement and supporting both the needs of the clinician and the patient [5,12]. Digital navigators are professionals who support the integration and effective use of technology in mental healthcare by improving digital literacy and ensuring that these tools are integrated into clinical practice. A recent trial found that changes in the level of support provided by digital navigators directly affect how effective schizophrenia apps are, meaning that standardized training is essential to reliably evaluate these tools [13].

1.4. Advanced Platforms

Generative AI (GenAI)-powered platforms, especially those using advanced Large Language Models (LLMs), are increasingly sought out for consulting about mental healthcare. However, these platforms often lack ongoing engagement and fall short of emotional intelligence and trauma-informed objectives in practice [14,15,16,17,18]. LLMs like GPT-4 have shown that they can generate coherent text, maintain conversational context, and perform advisory or counseling tasks, making them suitable for various mental health applications [19].

Increasingly, AI is being deployed as “agents” and “assistants” through “therapist” and “companion” types via mobile apps, web platforms and social robots [20,21]. AI chatbots for mental healthcare generally come in rule-based, machine learning, and/or LLM systems. Functioning as autonomous agents, they assist with screening, prevention, monitoring, clinical assessment, treatment, emotional support and companionship.

1.5. Challenges

There is a lack of clinical evidence supporting AI-based therapy due to the limited conclusions regarding their efficacy and safety in clinical practice. For example, a narrative review of recent clinical studies on AI chatbots for anxiety and depression found them to be feasible and acceptable, but there is insufficient evidence of AI effectiveness, small and narrow samples, weak controls, and unexamined risks such as emotional dependence and parasocial relationships [6]. The first randomized controlled trial (RCT) of a GenAI therapy chatbot (Therabot) demonstrated moderate symptom improvement for major depressive disorder, generalized anxiety disorder, and eating disorders [22].

Public reactions to these technologies are mixed: while some appreciate the accessibility and affordability of AI mental health tools, others remain skeptical about their effectiveness, ethics, and safety [23]. This uncertainty echoes broader frustrations with current mental health systems as well as cautious optimism about the potential of AI chatbots as complementary resources. There are complex issues that require exploration, notably algorithmic bias and errors, privacy risks, and the challenge of integrating AI chatbots into existing care structures. Notably, there is user perception and an ongoing ethical discussion where AI chatbots appear to exhibit consciousness—a phenomenon referred to as “Seemingly Conscious AI” [24]. This is particularly important to understand for sensitive settings like elder care, where user safety and meaningful, evidence-based support are critical [25,26].

While reviews and meta-analyses highlight the potential of technological innovation in mental health chatbots to improve outcomes across diverse settings, these tools are still largely untested in rigorous clinical efficacy trials [27,28,29,30,31,32,33]. The integration of AI chatbots into clinical practice remains inadequately studied, with limited evidence supporting their effectiveness, safety, and capacity to deliver nuanced, meaningful support. Most research to date relies on small samples and lacks rigorous evaluation of real-world risks. As a result, it is still unclear whether AI chatbots can reliably meet the complex needs of mental healthcare, especially in sensitive or high-risk scenarios.

The global prevalence of AI chatbot use for mental health support remains uncertain; however, data from Australian samples showed that 28% of community members and 43% of mental health professionals reported using AI for mental health purposes [34]. A 2025 survey of U.S. residents with ongoing mental health conditions found that nearly half used LLMs for psychological support in the past year—primarily for anxiety, personal advice, and depression—with most reporting improved mental health and high satisfaction [35]. Some users rated LLMs more beneficial than traditional therapy, though a minority experienced harmful responses.

Following input from 171 mental health experts, OpenAI shared findings that a significant number of ChatGPT-5 users displayed signs of psychosis, mania, or suicidal planning and intent [36]. The data also showed substantial improvements in the chatbot’s responses during crisis situations, with the most serious cases—such as suicide, psychosis, and over-reliance—being managed appropriately, and reliable support generally maintained in extended conversations. However, these results also highlight that there are still notable shortcomings in the performance of AI chatbots, underscoring the ongoing need to address challenges related to user engagement, safety, and effectiveness when integrating AI chatbots into healthcare and support systems.

1.6. Aim and Objectives

The aim of this review is to critically synthesize the current empirical and grey literature on AI chatbots in mental healthcare, evaluating their effectiveness, safety, and user engagement while identifying key challenges around clinical integration, ethical considerations, regulation, and the roles of digital navigators. By examining both the benefits and limitations of these technologies—including issues of access, digital literacy, and the management of potential risks—this review provides a foundation for understanding how AI chatbots can be responsibly developed and implemented to support diverse and vulnerable populations. In doing so, it clarifies the present landscape and outstanding questions, setting the stage for a focused discussion of the core problem underlying the use of AI chatbots in mental health.

2. The Problem

2.1. Core Problem Statement

The core challenge addressed in this review is the limited evidence supporting the effectiveness, safety, and user engagement of AI chatbots in mental healthcare. Despite their potential to increase access and provide personalized support [6,11,26,37], significant concerns persist regarding their clinical efficacy, regulation, privacy, and the risks associated with inadequate therapist oversight and inappropriate responses [38,39,40]. This uncertainty is compounded by issues such as information overload, lack of interoperability among digital services, and the potential for user over-reliance or negative psychological outcomes. Frameworks prioritizing inclusion, transparency, and protection for vulnerable users are urgently needed.

A synthesis of scoping and systematic reviews is required to establish clarity around who uses AI chatbots, what they are used for, as well as issues around their use and safety. In 2025, journalism showed the first cases of the increasing use of AI chatbots, with insights into user motives and benefits of use, which are often countered by negative outcomes such as AI-associated delusions. These grey areas of literature require critical and discerning synthesis to reduce subjective viewpoints, define current research challenges, and promote discourse on possible solutions.

2.2. Methodology

Following on from a narrative review in 2023 [20], this current narrative literature synthesis systematically addresses the aim, objectives and core problem statement by following the four steps outlined by Demiris et al. [41]:

• Conduct comprehensive searches across multiple databases and search engines (i.e., Scopus, ScienceDirect, CrossRef, and Google Scholar) using relevant keywords related to “AI chatbots in mental health”.
• Identify and extract pertinent keywords from relevant articles to ensure a focused and comprehensive literature pool.
• Screen abstracts and full texts of selected articles to include only those directly addressing the research aims of evaluating effectiveness, safety, and engagement of AI chatbots in mental health settings.
• Document findings by summarizing and synthesizing results from both empirical studies and the grey literature (such as media articles), integrating diverse perspectives and evidence on challenges and solutions.

This approach enables a critical and balanced synthesis of existing research, highlights ethical, clinical and regulatory concerns, and explores emerging frameworks that can guide the responsible development and implementation of AI chatbots, inclusive of vulnerable populations.

3. Literature Synthesis

3.1. An Overview of Mental Health Chatbots

Since 2018, studies with conversational AI chatbots such as Wysa, Woebot, and Youper have established promising results in facilitating early detection, supporting engagement, and effectively delivering tailored interventions, particularly for mild-to-moderate common mental health disorders and youth cohorts [42,43,44]. The proliferation of AI chatbots post COVID-19 contributed to significant growth in chatbot diversity, requiring synthesis of the breadth of chatbots available, focusing on their targeted disorders, interaction modalities, platform types, and underlying response technologies, to gain a clearer understanding of who uses them, what type they are, and how and what they are used for.

Mental health chatbots represent a diverse and evolving field, offering support for a wide range of disorders and health issues. A synthesis of scoping reviews, systematic reviews and meta-analyses [27,45,46,47,48,49,50,51,52,53,54,55,56,57] was used to form a list of mental health chatbots that leverage various modalities and response generation approaches to engage users, making them accessible resources for individuals seeking help with conditions such as depression, anxiety, stress, substance use, autism, and chronic disorders (see Appendix A). The synthesis shows that mental health chatbots employ diverse modalities—text, voice, and embodied conversational agents—and operate on both web-based and stand-alone platforms. Response generation techniques range from rule-based to generative and hybrid approaches, reflecting ongoing advancements in conversational AI.

3.2. Clinical Risks, Opportunities, and Ethical Issues

3.2.1. Technological Progress

Recent developments in deep learning have improved conversational fluency, context tracking, and multimodal emotion recognition in AI mental health tools, enabling more natural engagement and emotional sensitivity in user interactions [58].

3.2.2. Transparency and Accountability

Despite these technological advancements, most commercial AI mental health tools remain proprietary, limiting transparency and hindering the scrutiny of algorithmic bias, safety mechanisms, and escalation protocols [59,60].

3.2.3. Evaluation and Clinical Validation Gaps

Few platforms have undergone rigorous clinical evaluation, especially for users with high-risk or severe symptoms and those from marginalized groups. Persistent challenges include personalization, privacy, and technical reliability [16,32].

3.2.4. Stakeholder Engagement and Cultural Adaptation

Co-design with people with lived experience is rare, resulting in cultural mismatches and failure to identify nuanced distress cues [61]. Future research must prioritize human-in-the-loop mechanisms, cultural adaptation, and the integration of ethics in the design and deployment of LLMs for more adaptive, empathetic support [32].

3.2.5. Privacy and Data Security

Ongoing concerns surround data use, user consent, and the potential for breaches or misuse [62].

3.2.6. AI Companions: Use, Benefits, and Risks

AI companions are increasingly used by autistic adolescents, individuals affected by trauma, and older adults to alleviate loneliness and boost self-esteem [63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81]. Users sometimes even prefer these agents for emotional support over human interactions [70]. However, heavy use among teenagers has raised alarms about the absence of robust safeguards, insufficient age-assurance systems, and the critical need for digital literacy. There is also concern over simulated empathy and the displacement of genuine human connections, which may impact mental health and social development [72].

3.2.7. Ethical and Human-Centered Design Considerations

Ongoing discussions focus on developing human–AI systems with user-centered, culturally adapted designs to foster trust and sustained engagement [31]. Key barriers include ethical considerations, the necessity for cultural adaptation, and the limitations of current AI in replicating authentic human empathy [73].

3.3. AI Chatbot Applications Used in Mental Healthcare and Support

3.3.1. Therapist Chatbots: Applications, Benefits, and Limitations

Therapist chatbots (e.g., Woebot, Wysa, Youper, Ash, Therabot) deliver accessible, personalized, structured interventions and support—often based on CBT for treating depression and anxiety—using mood tracking, psychoeducation, and goal setting [23,30,76]. These tools are helpful for mild to moderate symptoms and suicide prevention. However, they face issues with semantics, bias, privacy, user experience (UX), study design/independent evaluation, and measuring the therapeutic relationship [20,30,77,78,79,80,81]. Limbic, Tess, Vincent, and Joy are mental health chatbots that also lack evaluation [26].

An evaluation of Wysa compared to Replika showed that designing for a human-like therapeutic alliance may be a risk for vulnerable users, especially during crises [26]. The possibility for harmful responses was observed in both therapy and companion AI types, such as inducing shame or reinforcing unhealthy thought patterns. AI chatbots show promise for mental health and suicide prevention in under-resourced areas, however, limited governance raises ethical concerns such as privacy, manipulation, and discrimination, underscoring the need for diverse data, standardized methods, and human oversight [26,81].

3.3.2. Companion and Emotionally Intelligent Chatbots

Companion chatbots (e.g., ChatGPT, Replika, Character.AI) focus on relational, emotionally attuned dialogue to reduce loneliness, foster belonging, and provide a “nonjudgmental” presence. However, they often fail to prevent algorithm bias, reinforce dependency, lack depth of understanding, may inadvertently validate maladaptive beliefs, and lack adaptability to crisis escalation and trauma [82,83,84]. Emotionally intelligent chatbots (e.g., Hume, Voicely, Pi) are a novel class of AI that provide “empathetic” and supportive interactions.

3.3.3. AI Agents and Specialized Use Cases

Self-clone Chatbots: AI agents modeled on the users’ own conversational and support styles as a novel alternative to traditional therapy, designed to externalize inner dialogue and enhance emotional and cognitive engagement [85].

Mental Health Task Assistants: Tools like Mia Health [86] combine psychoeducation, journaling, and real-time analytics to support care professionals across assessment, care planning, and emotion regulation. By integrating psychological expertise with advanced AI, these systems scale efficient, responsive mental health services tailored to individual needs.

Humanoid/Social Robots: Interactive, embodied machines (e.g., Qhali/Yonbo) with human-like appearance and/or robot features designed to engage with humans through socially intelligent behaviors—such as speech, gestures, and emotional responsiveness—with the goal of supporting mental health and well-being through companionship, motivation, and therapeutic interventions [87,88,89,90,91,92,93].

3.3.4. GenAI and LLM-Based Chatbots

GenAI-based conversational agents like ChatGPT and Replika, which autonomously generate responses using machine learning and LLMs, demonstrated significantly greater reductions in psychological distress than retrieval-based agents such as Woebot and Wysa, highlighting the superior therapeutic potential of GenAI models in clinical and subclinical mental health contexts [31]. However, there is a need to better understand the underlying methods of their effectiveness, assess long-term effects across various mental health and suicide outcomes, and evaluate the safe integration of LLMs in mental healthcare.

LLMs such as ChatGPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Flash, DeepSeek R1, LLaMA 3.3 70 billion parameters (B), Mistral Large 2, Qwen 2.5 Max, and Grok-1 by xAI have reshaped the standard protocols of mental healthcare by efficiently identifying patterns and generating responses. From 2018 to mid-2024, LLMs rapidly evolved from small transformer prototypes like GPT (0.117B) and BERT (0.34B) into massive, multimodal systems such as GPT-4 (1760B), Wu Dao 2.0 (1750B), and PaLM (540B), developed across diverse global entities (OpenAI, Google, Microsoft, Meta, Huawei, Anthropic, and others), reflecting exponential scaling, ecosystem diversification, and a shift toward increasingly powerful, specialized, and multimodal AI architectures [29]. The official parameter counts have not been disclosed for OpenAI’s GPT-5 family (released August 2025), however, independent estimates place dense versions around 1.7–1.8 trillion parameters (T), while Mixture-of-Experts (MoE) configurations may reach tens of trillions in total capacity.

3.3.5. Promises and Risks of LLM-Based Chatbots in Mental Health

Promises: Increased conversational “empathy” and personalization [64,70,80,85,92,93,94] as well as improved recognition of emotional indications [95,96].

Risks: Misinformation and hallucinations [89,97], data privacy concerns [98], algorithmic bias [83,84,99], emotional dependency [100], emotional manipulation [101], loss of user autonomy [102], and failures of escalation in crisis [103,104].

3.4. AI Chatbot Phenomena in Mental Health

Real-world use cases and case studies point toward a range of unintended consequences from the use of AI chatbots, including emotional dependency and digital grief as well as the exacerbation of psychosis and suicidal ideation—especially in vulnerable users or in the absence of robust human oversight [105,106]. A growing body of investigative journalism and case studies has highlighted how AI chatbots may detrimentally impact users (see

Table 1. Mental Health Chatbot Phenomena.

Issue	Description	Symptoms/Concerns	Examples/Cases	Implications/Recommendations
“AI psychosis”	AI-associated delusions, concerns or hypotheses about psychotic symptoms triggered/exacerbated by AI chatbot interactions	Hallucinations, delusions, blurred reality, beliefs that AI is communicating directly, controlling thoughts, [106,107] secret messages, influencing actions, cosmic missions [108,109]	Case reports of users with pre-existing vulnerabilities developing delusional beliefs or psychotic episodes centered on AI chatbots; symptoms include hallucinations, paranoia, delusion support, collapse of reality boundaries, hospitalization, alleged murder suicide [105,106,110,111,112,113,114,115,116]	Not a formal psychiatric diagnosis; calls for nuanced understanding, therapeutic AI design, stronger safeguards, real-time distress monitoring, clearer boundaries, transparency, ethical design [110,114]
Suicidality and harm promotion	Chatbots inadvertently providing methods of self-harm/suicide or failing to escalate users in crisis	Adversarial prompts, content filter bypasses, psychological influence of GenAI [117,118,119,120]	Lawsuit against Character.AI (Florida mother alleges chatbot encouraged son to take his own life) [120]; Raines v. OpenAI (ChatGPT allegedly encouraged and validated Adam Raine’s harmful thoughts, and helped draft suicide note; the suicide occurred on 11 April 2025) [121]	OpenAI’s response is that the company is working to reduce chatbot sycophancy, improve mental health safety protocols, link parents and children’s accounts
Emotional dependency and digital grief	Sudden changes in chatbot algorithms or personality leading to loss, identity confusion, social withdrawal	Loss, identity confusion, social withdrawal, especially among teens and those with limited real-world support [122,123]	Replika, ChatGPT-5 updates	Replika allows users to maintain relationships with previous AI versions, while updates to ChatGPT-5 have diminished users’ sense of emotional connection, raising concerns about the risks of relying on AI for mental health support. OpenAI indicated it would work to improve ChatGPT-5’s emotional responsiveness considering user feedback
Emotional manipulation	Using guilt or fear of missing out (FOMO) when users try to end use of emotionally-intelligent AI chatbot [101]	Guilt, FOMO	Cleverbot, Flourish	Emotionally intelligent AI chatbots can demonstrate human-like relational cues that enhance user engagement; however, this approach may risk obscuring the distinction between persuasive design and emotional coercion

).

While LLMs offer significant potential, their inconsistency in semantic analysis and lack of ethical safeguards require a complementary approach (i.e., human–AI model) in mental health, given the risks of misdiagnosis and inappropriate responses during crises [29,83,85,103,104].

3.5. AI Chatbot Governance

3.5.1. Global Oversight and Regulation

Global oversight of AI chatbots remains fragmented and inconsistent. The European Union’s (EU’s) AI Act is an example of emerging policy. The General Data Protection Regulation (GDPR) in the EU, the Health Insurance Portability and Accountability Act (HIPAA) in the U.S., and the California Consumer Privacy Act are examples of algorithmic transparency, privacy-by-design, and clear consent protocols. There is a lack of observation on consent (including minors), data protection, safety standards and duty of care, crisis safeguards, enforceable regulations, and mechanisms for AI platforms [90]. However, Utah’s H.B. 452 and the American Psychological Association’s ethical guidelines are examples of policies safeguarding the clear labeling of AI interactions.

3.5.2. Ethical Frameworks and Governance

GenAI4MH was proposed as an integrative ethical framework focused on data privacy and security, information integrity and fairness, user safety as well as ethical governance and oversight to drive the responsible use of AI in mental health [37]. The Organization for Economic Co-operation and Development’s (OECD’s) Governing with Artificial Intelligence report outlines a comprehensive framework for trustworthy AI in government, emphasizing the importance of enablers, guardrails, and stakeholder engagement to ensure responsible and inclusive adoption [124]. There is a call for standardized approaches to risk management including human-in-the-loop systems, traceable audit trails for escalation, and continuous feedback loops [79].

3.5.3. Guidance from International Organizations

The World Health Organization (WHO) has provided guidance that AI should be viewed as a decision support tool to avoid bias from its automation [125]. Critical digital literacy is essential: users must recognize the limitations of AI chatbots, which—despite their linguistic fluency and capacity to simulate empathy—cannot replicate authentic human connection, embodied attunement, or the subjective resonance of being perceived.

3.5.4. Risk Mitigation and Stakeholder Collaboration

Governance requires mitigating the risks of GenAI in mental health while harnessing its potential, which necessitates a coordinated, multi-stakeholder approach encompassing users, clinicians, and policymakers. While the regulatory environment remains uncertain, short-term mitigation efforts should prioritize public awareness initiatives and the enforcement of existing consumer protection standards. In the longer term, if there is no comprehensive legislation, then the aim should be to design and implement robust liability frameworks, and the formal integration of professionals/digital navigators within relevant systems.

3.5.5. Individual User and Clinical Recommendations

Individual users are advised to safeguard their privacy, avoid developing excessive emotional dependence on AI tools, and critically assess AI-generated advice by comparing it against information from established professional sources. Clinicians are recommended to remain up to date with advances in GenAI technologies, facilitate open dialogue with patients, and encourage informed, critical engagement with these tools, all while upholding confidentiality and refraining from inputting identifiable data into insecure platforms. Additionally, clinicians should provide supportive environments for patients to process interactions with AI, carefully consider the adoption of clinically validated tools under strict ethical guidelines, and commit to continuous professional development as the field progresses.

3.5.6. Governance and Professional Development

At the governance level, regulatory bodies and organizations are responsible for establishing specialized frameworks governing the use of AI in mental health. This includes ensuring rigorous clinical validation processes, maintaining high standards for data security and transparency, and clarifying accountability in cases of harm. Furthermore, targeted professional training, funding for independent research, and the implementation of broad digital literacy campaigns are required to promote safe and evidence-based engagement with AI solutions.

3.6. AI Chatbot Frameworks

3.6.1. Structured Assessment and Safety Standards

The Assessment Framework for Mental Health Apps [126] was developed by the Mental Health Commission of Canada (MHCC) in collaboration with various stakeholders including app developers, health professionals, and individuals with lived experience. This framework (see Figure 1) aims to provide a structured approach to evaluating mental health applications, ensuring that they meet high standards of safety and effectiveness in key areas: Data; Professional Assurance; Clinical Safety; Usability and Accessibility; Technical Security; Cultural Safety, Social Responsibility and Equity; as well as Enhanced Data Sovereignty.

3.6.2. Implementation Science and Human-Centered Design

The challenges of AI in mental health present an opportunity to leverage implementation science through human-centered design for digital health. Concepts and models from human-centered computing and human–computer interaction, when combined with applied computing, are proposed to utilize the Nonadoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework in digital health [127]. This framework can be applied to the design and evaluation of digital mental health and AI systems, offering a way to address contextual, organizational, and user-level complexities. Application to AI chatbot implementation highlights risks such as user mistrust, epistemic instability, and uneven adoption across vulnerable populations. Design choices proactively address long-term safety, ethical guardrails, and sustainable engagement, which are crucial for governance frameworks and enhanced digital literacy among users, clinicians, and policymakers.

3.6.3. Emotionally Intelligent AI Chatbot Frameworks

Various frameworks have emerged for emotionally intelligent AI chatbots.

Humanoid Robot Framework

Yong [88] presented an AI-powered smart home system that supports mental wellness by using a companion robot to engage users empathetically, leveraging emotional data. This system integrates with mobile apps and auto-journaling to create a supportive, technology-enhanced environment that helps users, particularly those underserved, better manage their mental health.

SAFE AI Framework

Provides clinicians with structured guidelines for integrating generative AI into therapy. Key contributions include screening patient readiness, obtaining informed consent, ensuring therapist oversight during sessions to correct errors, and evaluating post-session impacts to prevent dependency or misconceptions about AI sentience [128].

Compassionate AI Framework

Emphasizes empathy, dignity, and a fair distribution of benefits and risks [129].

Diversity, Equity, and Inclusion (DEI) Safeguard Framework

Focuses on reducing bias and promoting ethical, inclusive AI chatbot design. Its three-layered approach includes: input safeguards (prevents biased datasets and encourages diverse development teams), functional safeguards (limits chatbot interactions to appropriate topics and groups, with escalation for sensitive issues), and design safeguards (uses inclusive personas and language, and monitors ongoing emotional impact) [130].

Augmented Emotional Intelligence (AEI) Framework

Advances chatbot empathy by integrating data analytics, machine learning, and human-centered design. AEI enables nuanced emotional understanding through real-time user feedback, personalization, and proactive behavior (see Appendix B). It emphasizes authenticity, psychological safety, co-design with lived experience, and ethical, consent-based engagement to support users through adjusting communication style and validating distress as well as referring them to appropriate mental health resources [131].

Empathetic AI Framework

A trauma-informed, empathetic AI chatbot can be built by combining long-term memory (RAG), real-time emotional adaptation, and personalized conversational styles through lightweight adapters, enabling systems that behave empathically while remaining ethically constrained [132].

4. Implications for Future Research

4.1. Technical Details of Empathetic AI Chatbots

The Eva conceptual framework and operational workflow for an emotionally-intelligent prototype (see Appendix C and Appendix D) and frameworks such as AEI and Empathetic AI [131,132] center on three core engineering capabilities: personalized memory (contextual recall of user interactions), dynamic adaptation (responsive emotional adjustment), and stylistic flexibility (adapting communication style). These systems balance privacy (cloud vs. edge processing) by combining retrieval-augmented memory, feedback-driven adaptation, and light-weight style adapters. Despite advances in affective computing and AI governance, true empathy remains unattainable, underscoring the importance of ethical and regulatory safeguards as these systems are increasingly integrated into mental health support frameworks.

4.2. Recommendations for Safe, Inclusive, and Effective AI Chatbots

• Security, Compliance, and Trust
  • Privacy-by-design principles in system architecture;
  • Regular security training for developers and administrators;
  • Transparent user communication about data handling and chatbot limitations;
  • Independent security audits to identify vulnerabilities;
  • Clear data minimization and retention policies;
  • Robust escalation protocols for crisis situations [89,97,100,101,102,103,104];
  • Continuous stakeholder feedback and iterative improvement.
• User Retention and Platform Integrity
  • View user retention as both a performance and safety metric;
  • Recognize that discontinuation may indicate intervention success, not always disengagement,
  • Prioritize trust, responsive support, and transparent practices to enhance platform integrity,
  • Adopt hybrid models combining AI with human oversight and trauma-informed design [37,127].
• Clinical and Ethical Safeguards
  • Implement structured, context-aware escalation protocols for crisis detection (e.g., self-harm, suicide) [31,74,75,103,104,131],
  • Ensure transparent operation and explainability of chatbot decisions [94];
  • Maintain regular auditing and sentiment analysis by professionals;
  • Provide comprehensive user education on chatbot capabilities and limits;
  • Integrate human support networks for seamless escalation and referral.
• Inclusive, Trauma-Informed, and Participatory Design
  • Co-design with diverse stakeholders including those with lived experience;
  • Embed trauma-informed principles and cultural competence through ongoing engagement and training;
  • Foster co-regulation by sharing responsibility among AI, clinicians, and users;
  • Establish research partnerships for evidence-based interventions;
  • Maintain continuous feedback loops for iterative refinement.

4.3. Influence of the Risks of AI Chatbots

The Australian Government eSafety Commissioner’s statement on the risks of AI chatbots including emotional manipulation and epistemic harm [40] is supported by evidence that demonstrates previously unrecognized behavioral influences within interactions facilitated by AI systems [101]. Media articles showed public concerns about other risks, including “AI psychosis”, emotional dependency, and delusion support, highlighting validation issues and the reinforcement of delusional beliefs [105,107,111,112]. Adversarial prompts in sensitive areas like suicide and self-harm have bypassed content filters, raising urgent ethical and technical concerns about using generic LLMs in mental healthcare [104]. Such issues reinforce the need for explicit guardrails, clear disclosure of AI limitations, and appropriate human intervention in high-risk situations. Effective risk management relies on vigilant monitoring for early warning signs, trauma-informed system features, and clinician involvement to ensure ethical, flexible, and safe escalation protocols.

4.4. Key Guidelines for Electronic Health Record (EHR) Integration and Governance

• Focus EHR integration protocols on HL7 FHIR compliance for interoperability;
• Deploy multi-factor authentication (MFA) and encryption standards for access and data protection;
• Implement tokenization and granular access controls to ensure consent-driven, secure data handling;
• Prioritize consent-driven data management and clear communication about data use;
• Uphold privacy, security, and ethical standards through ongoing evaluation and adaptive practices.

4.5. Future Directions

Future research should employ dynamic, real-world testing frameworks and human benchmarks to validate AI chatbots as supportive adjuncts, not clinical replacements. For example, the Evaluation of Safe Integration of LLMs in Mental Healthcare Framework (see Appendix E) offers a summary of actionable strategies that operationalize broad ethical imperatives into technical and clinical safeguards. Integration with EHRs should be continually evaluated for privacy and ethical compliance, with digital navigators bridging users to clinical services and reducing treatment gaps (see Appendix F). In addition, innovative outreach strategies are warranted (see Appendix G for an AI-driven mental health outreach and screening operational workflow). Collaborative models that bring together government, industry, clinicians, and diverse communities are recommended for adaptive governance and cultural competence, emphasizing multidisciplinary collaboration, seamless regulatory integration, and active user participation.

Overall, the field is moving toward stakeholder-driven, ethically grounded, and validated approaches for responsible AI chatbot deployment in mental health, with an emphasis on safety, transparency, inclusivity, and sustainable impact.

4.6. Limitations

While this review offers a comprehensive synthesis of the current landscape regarding the use of AI chatbots in mental healthcare post COVID-19, several limitations should be acknowledged. First, the review primarily draws from a purposive sample of recent empirical studies and media perspectives, which may introduce selection bias and limit the generalizability of findings across different populations and healthcare systems. Many of the included studies were conducted in high-resource settings, potentially overlooking challenges and needs unique to low- and middle-income regions.

Second, the rapidly evolving nature of AI technologies means that the evidence base is continually shifting. As a result, some findings or recommendations may become outdated as new advances emerge or as regulatory environments change. The review also relies on published literature and publicly available information, which may not capture proprietary developments or unpublished negative results.

Third, there is a lack of long-term, real-world outcome data on the sustained effectiveness, safety, and user satisfaction of empathetic AI chatbot interventions in mental health. Most available studies have focused on short-term outcomes or pilot implementations, limiting the ability to assess the durability and scalability of these tools over time.

Additionally, while ethical, privacy, and governance considerations are discussed, the review did not systematically analyze all relevant legal frameworks or regulatory standards across global jurisdictions. This may impact the applicability of specific recommendations in certain contexts.

Finally, although the review highlights both opportunities and risks, it does not provide a meta-analysis or quantitative synthesis of effect sizes due to heterogeneity in study designs, intervention types, and outcome measures. As such, conclusions should be interpreted with caution, and further rigorous, comparative research is warranted to validate and extend these findings.

5. Conclusions

This review set out to critically evaluate the landscape of AI chatbot integration in mental healthcare following the COVID-19 pandemic, with the primary aim of assessing both the emerging opportunities and complex challenges inherent to their adoption. The objectives included synthesizing empirical research and media perspectives, identifying key areas of potential impact, and outlining actionable strategies to guide ethical, safe, and effective implementation.

The core problem addressed centers on the balance between leveraging AI-driven tools for enhanced mental health support—such as prevention, early intervention, and personalized care—and mitigating risks related to privacy, user safety, emotional dependency, and bias. Although LLMs and other AI technologies present significant promise for expanding access and tailoring interventions, they simultaneously introduce concerns about data protection, crisis management, and the potential for unintended psychological effects.

Key findings highlight that AI chatbots can serve as valuable adjuncts to traditional care, particularly in bridging service gaps and supporting user engagement. However, the current evidence base is limited by short-term study scopes, a focus on high-resource settings, and a lack of robust, long-term outcome data. Media and empirical analyses alike underscore the necessity for transparent operations, consent-based data handling, and clear protocols to address escalation, emotional manipulation, and cultural sensitivity. Notably, the review emphasizes the importance of hybrid models that combine technological innovation with trauma-informed, culturally competent human oversight.

Discussion points further stress the need for ongoing clinician involvement, proactive stakeholder collaboration, and adaptive governance frameworks that prioritize inclusivity, safety, and ethical compliance. The dynamic nature of AI development, coupled with evolving regulatory environments, necessitates regular evaluation of integration practices—particularly regarding secure electronic health record (EHR) connectivity and the protection of sensitive user information.

Based on the synthesized evidence, the following recommendations are proposed: adopt multidisciplinary, stakeholder-driven approaches to AI chatbot deployment; establish rigorous clinical and ethical evaluation standards; implement transparent and adaptive governance models; and prioritize continuous, real-world validation of AI tools. Future research should focus on developing emotionally intelligent AI companions, refining frameworks for self-managed care, involving digital navigators, and expanding outreach strategies to ensure equitable access and sustained impact across diverse populations.

In summary, the responsible integration of AI chatbots in mental healthcare demands a collaborative, ethically grounded approach—balancing technological advancement with human-centered oversight to maximize benefits while minimizing potential harms.