Reengineering eADVICE for Long Waitlists: A Tale of Two Systems and Conditions

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The document outlines the development, implementation, and evaluation of the eADVICE system, designed to address extended outpatient waiting times in pediatrics healthcare, focusing on incontinence and then adapted also to pediatric sleep issues.

The presented system seems to be relevant and has the potencial to improve childrens' quality of life. Main points are the use of ECA the Embodied Conversational Agent. The document is clear and well-structured which makes easy to follow the path they present. The authors also analysed user experience which is quite important to measure acceptance and design future improvements.

 

The authors seem well aware of the limitations of the study, as they well document, in relation to the problems of specificity (it is hard to preview and adapt to all cases) and also about the dialogs since now there is a world of new AI tools that they can adapt, the best known is probably chatgpt but among those some will provide a proper conversation if taking in account, and preventing, the risk of errors and false information.  Important to notice is the perspective and willingess of the authors to look for opportunities in managing waiting lists for other pathologies and to expand their work.

Author Response

Review Comments

The document outlines the development, implementation, and evaluation of the eADVICE system, designed to address extended outpatient waiting times in pediatrics healthcare, focusing on incontinence and then adapted also to pediatric sleep issues.

The presented system seems to be relevant and has the potencial to improve childrens' quality of life. Main points are the use of ECA the Embodied Conversational Agent. The document is clear and well-structured which makes easy to follow the path they present. The authors also analysed user experience which is quite important to measure acceptance and design future improvements.

 

The authors seem well aware of the limitations of the study, as they well document, in relation to the problems of specificity (it is hard to preview and adapt to all cases) and also about the dialogs since now there is a world of new AI tools that they can adapt, the best known is probably chatgpt but among those some will provide a proper conversation if taking in account, and preventing, the risk of errors and false information.  Important to notice is the perspective and willingess of the authors to look for opportunities in managing waiting lists for other pathologies and to expand their work.

 

RESPONSE: Thank you for your review. We have discussed further our limitations and future directions.

Reviewer 2 Report

Comments and Suggestions for Authors

This scientific paper presents the development and evaluation of eADVICE (electronic Advice and Diagnosis Via the Internet following Computerised Evaluation), an eHealth system designed to address long outpatient waiting times in pediatric healthcare. The paper focuses on two implementations of eADVICE: one for pediatric incontinence and another for pediatric sleep disorders.

The eADVICE system consists of three main components: an interactive website for data collection, algorithms for treatment recommendation, and an Embodied Conversational Agent (ECA) for treatment delivery. For pediatric incontinence, a randomized controlled trial (RCT) with 239 participants was conducted, showing significant improvements in daytime and nighttime continence for the intervention group compared to the control group. User experience and working alliance with the ECA (Dr. Evie) were also evaluated. For pediatric sleep disorders, the system was developed based on lessons learned from the incontinence implementation and introduced a new ECA (SAM) with enhanced dialogue capabilities, receiving positive preliminary feedback from healthcare professionals and families. Additionally, the paper proposes a generalizable framework for developing eADVICE systems for various pediatric conditions.

Despite these strengths, the study has several critical limitations. The evaluation of eADVICE for sleep disorders lacks a formal assessment, relying only on preliminary feedback, which limits the robustness of the findings. The RCT for incontinence, while rigorous, had a small sample size and short duration of six months, which are insufficient to draw definitive conclusions about long-term efficacy. Furthermore, the use of handcrafted dialogues, although ensuring accuracy, may limit the system's scalability and personalization compared to more advanced AI-driven approaches.

A significant shortcoming of the paper is its superficial state-of-the-art analysis. The analysis includes limited literature and references some seemingly random projects, weakening the theoretical foundation of the study. While the eADVICE system shows promise in addressing long outpatient waiting times in pediatric healthcare, these limitations highlight the need for more comprehensive and rigorous evaluation. Future research should aim to validate the system's effectiveness across diverse conditions and explore more advanced AI-driven approaches for improved scalability and personalization.

Author Response

REVIEWER: This scientific paper presents the development and evaluation of eADVICE (electronic Advice and Diagnosis Via the Internet following Computerised Evaluation), an eHealth system designed to address long outpatient waiting times in pediatric healthcare. The paper focuses on two implementations of eADVICE: one for pediatric incontinence and another for pediatric sleep disorders.

The eADVICE system consists of three main components: an interactive website for data collection, algorithms for treatment recommendation, and an Embodied Conversational Agent (ECA) for treatment delivery. For pediatric incontinence, a randomized controlled trial (RCT) with 239 participants was conducted, showing significant improvements in daytime and nighttime continence for the intervention group compared to the control group. User experience and working alliance with the ECA (Dr. Evie) were also evaluated. For pediatric sleep disorders, the system was developed based on lessons learned from the incontinence implementation and introduced a new ECA (SAM) with enhanced dialogue capabilities, receiving positive preliminary feedback from healthcare professionals and families. Additionally, the paper proposes a generalizable framework for developing eADVICE systems for various pediatric conditions.

Despite these strengths, the study has several critical limitations.

 

RESPONSE: Thank you for your comments.  We fully agree with the limitations you identified. In our submitted version we sought to express those in the Limitation and Future work section as identified below.

REVIEWER: The evaluation of eADVICE for sleep disorders lacks a formal assessment, relying only on preliminary feedback, which limits the robustness of the findings.

RESPONSE: eADVICE sleep is only in its pilot stage. Formal assessment is not appropriate at this stage. As we state in the paper “The key limitation of our proposed generalised framework is that we have not yet run a study with eADVICE-sleep. To address this issue, we recently obtained funding to commence the evaluation of eADVICE-Sleep in the context of children with neurodisability. This necessitates further work with specialists and stakeholders to adapt eADVICE-Sleep to this specific cohort of children and their families. The lessons learned from adapting from continence to sleep, as described in this paper, will be valuable for this new project, and we anticipate for other health conditions too.”

REVIEWER: The RCT for incontinence, while rigorous, had a small sample size and short duration of six months, which are insufficient to draw definitive conclusions about long-term efficacy.

RESPONSE:  We had already expressed the following limitation in the paper “Limitations concerning the RCT with eADVICE-Continence include study numbers, the short six-month duration of the trial, and limited power to detect significant effects for sites with shorter waiting times.” On reflection of your comment, we have revised as follows:

“When compared with most trials reported in the literature involving children and adolescents [95, 96], our RCT which recruited 239 paediatric patients and captured data over an 18 month period with 6-month waitlisted control, 6 month treatment and followup 6 months later, can be considered a large paedatric study over an extended period. Nevertheless, limitations concerning the RCT with eADVICE-Continence include study numbers, the short six-month duration of the trial, and limited power to detect significant effects for sites with shorter waiting times. As a further limitation, is the complexity of conducting a trial in children.”

REVIEWER:  Furthermore, the use of handcrafted dialogues, although ensuring accuracy, may limit the system's scalability and personalization compared to more advanced AI-driven approaches.

RESPONSE:  The purpose of dialogues designed by senior experienced clinicians ensure the accuracy of the health information provided, which is very important for paediatric health conditions. As we stated in the paper “As a key goal of eADVICE is to provide evidence-based treatment and guidance while patients are awaiting a specialist appointment, it is essential that the content is fully reviewed by domain experts rather than generated by AI.“  We have updated the following paragraph also with further clarification.

RESPONSE: “The use of handcrafted dialogues may be seen as both a strength and a limitation. The dialogue templates presented were designed to support automation. Our state-based dialogue engine allowed us to use parameters to adapt the ECA’s responses based on the patient’s responses, particularly their beliefs and goals. However, it was not possible to ensure that the templates would produce well-structured, grammatically correct sentences and thus, for the sake of usability, comprehensibility and acceptability, it was necessary to provide handcrafted wording. We anticipate the advances in Gen-erativeAI will provide an imminent solution to this issue. With the increasing use of generative AI and tools such as ChatGPT, we are seeing the deployment of chatbots in many areas, including health. With this deployment are many concerns regarding the accuracy of the advice being provided and also the resultant sharing of sensitive health data used by the Large Language Model (LLM) to grow its knowledge and database. As a key goal of eADVICE is to provide evidence-based treatment and guidance while pa-tients are awaiting a specialist appointment, it is essential that the content is fully re-viewed by domain experts rather than generated by AI. In the future, we might consider the use of an LLM to tailor the expression used in the dialogues to the individual but identifying such preferences remains open research questions [97, 98]. As a step in this direction, in two safer contexts, we have used an LLM to modify dialogues to embue relational cues to further increase working alliance [99] concerning health and well-being advice delivered by a digital cognitive coach, physical coach and dietician [100, 101] and to embue personality cues in non-player characters who offer different ethical perspectives [102] in a serious game for ethics training of cybersecurity professionals [103]. “

RESPONSE: Also, in the conclusion, our paper states “In contrast to the increasing use of artificial intelligence in medicine where the machine’s algorithms make decisions and generate natural language conversations, the eADVICE framework captures the knowledge of the domain expert, including their ability to accurately diagnose and empathically discuss appropriate treatments with patients. With the current excitement, and fear around ChatGPT and large language models, the advantage of our system is that it is embedded in a health care system and not likely to be influenced by false information that Chat GPT can't guarantee. It is endorsed by health professional and is therefore safe.”

RESPONSE: To clarify the nature of our implementation approach and why it was taken, we have also added: “The dialogue templates presented were designed to support automation. Our state-based dialogue engine allowed us to use parameters to adapt the ECA’s responses based on the patient’s responses, particularly their beliefs and goals. However, it was not possible to ensure that the templates would produce well-structured, grammatically correct sentences and thus, for the sake of usability, comprehensibility and acceptability, it was necessary to provide handcrafted wording. We anticipate the advances in GenerativeAI will provide an imminent solution to this issue.”

REVIEWER: A significant shortcoming of the paper is its superficial state-of-the-art analysis. The analysis includes limited literature and references some seemingly random projects, weakening the theoretical foundation of the study.

RESPONSE: We have rewritten the literature review.  We have added and introduction and two new sections to the literature review 2.2. Conversational Agents, Health and Children” and “2.3 Working Alliance to Address Adherence”.

REVIEWER: While the eADVICE system shows promise in addressing long outpatient waiting times in pediatric healthcare, these limitations highlight the need for more comprehensive and rigorous evaluation. Future research should aim to validate the system's effectiveness across diverse conditions and explore more advanced AI-driven approaches for improved scalability and personalization.

RESPONSE: This is also our aim but this needs to be done in a stepwise manner and with great care. We have added to the paper as the final sentence “When current GenerativeAI technologies significantly improve their ability to guaran-tee accuracy and not to hallucinate and also are able to ensure patient data remains secure, we anticipate improved scalability and personalization, and to potentially to adapt to new health conditions following the framework outlined in this paper.”