Development of a Chatbot for Pregnant Women on a Posyandu Application in Indonesia: From Qualitative Approach to Decision Tree Method

No

Reviewer 1 Comments

Author Response

Page/Line Number

1

Why you choose Decision Tree only? Are all question can be answered with Y/N only? Justify.

Thank you for the valuable comment.

We use Decision Trees only because the chatbot built in this research is still initiating and underdeveloped and aims to help users to predict their condition and support decision-making. According to previous studies, a decision tree is widely used in the health sector significantly to help make clinical decisions by maximizing effectiveness and minimizing harm. Among the many classification algorithms, decision trees are the most suitable because they are simpler to understand and interpret than association rules or logistic regression. Decision trees also require a simple data preparation stage and can handle categorical data (Li et al., 2019)

Besides, as the development of this chatbot has just begun, there is still limited information in the knowledge base that has just been explored through this qualitative research, so we use a simple question method, Yes or No question where in the final decision is to chat to the midwife directly. It is also suitable with the function of decision trees as the simple and most appropriate modeling for interventions in brief and relevant conditions. In contrast, for conditions that take a long time, it is less flexible to apply (Haji Ali Afzali and Karnon, 2014).

In the future, there will be further research to evaluate and carry out further development of the chatbot.

Page 2,

Line 84-99

Page 6,

Line 195-207

Page 17,

Line 512-517

2

Data are too small, Why not you have more midwives and pregnant women?

Thank you in advance for the valuable suggestion. According to this question, we would like to answer that we have conducted interviews in a qualitative manner through focus group discussions (FGD). According to the literature, the number of participants per focus group ranged from 3 to 21, with a median of 10 participants (O.Nyumba et al., 2018).

In this case, at the 2nd FGD, we felt the answer was constantly repeated and that the need to use chatbots revolved around the same thing that had been explained repeatedly. In addition, the criteria for the area of the puskesmas used were only a few puskesmas that had received exposure to information on the iPosyandu application where the Community Health Workers (CHWs) in that puskesmas had actively used the iPosyandu application, so it was limited to find respondents according to the criteria. In this case, the author states that the data has reached a saturation point, so we suffice for 22 informants involved.

Page 4,

Line 138-140

3

How this system differs from iposyandu? any comparison based on the results like accuracy, time, cost, etc.

Thank you for the valuable comment. The chatbot developed in this study is part of the iPosyandu application developed earlier. The iPosyandu application itself is an integrated service post based on mHealth that focuses on recording and reporting Community Health Workers (CHWs) during service to become a bridge of information between the community and the government (Rinawan et al., 2021, 2022). While the development of the chatbot feature in the iPosyandu application is an additional feature to optimize mHealth-based posyandu services so that it can be more interactive and can reach pregnant women's questions in monitoring their health.

Page 2,

Line 57-64

Line 67-69

4

When you said decision tree is accurate in classification and prediction, do you know about overfitting?

Thank you for the worth comment. We have already improved this secion. According to

previous research, the challenge of the decision tree is the problem of overfitting. Overfitting usually occurs when the tree has too many nodes relative to the amount of training data available (Rokach dan Maimon, 2014). To avoid this, this research did not use machine learning, but it used forward chaining or an expert data-based system, which works with a set of known facts and applies rules to generate new facts whose premise matches the known facts to achieve the goal previously determined. Forward chaining is also a solution to some problems naturally, starting with information gathering (Al-Ajlan, 2015).

Therefore, this study was conducted to collect information to strengthen the knowledge base by exploring the needs of patients and health workers and reviewing literature or evidence whose content is then validated by experts.

Page 17

Line 48-500

Page 6,

Line 216-227

No

Reviewer 2 Comments

Author Response

Page/Line Number

Major points

1

Overall, the content of the manuscript can be understood by the readers. There are noteworthy efforts to explain the background and make the manuscript self-contained. However, numerous expressions would need polishing. There are so many expressions that I would recommend the involvement of a professional scientific writer; MDPI offers such a service if needed. There are small oddities, such as "the need to use a semi-automated chatbot can be applied" (it is strange for a 'need' to be 'applied'), or "the extended version is intended to optimize family services focusing on pregnant women currently under development" (the sentence suggests that the pregnant women are under development, instead of the app), "was extended on its features", "tree is one of the tools of consideration", "is needed to ensure that decisions with high-quality evidence" (reads as a fragment; the sentence cannot end here), "The theme references chatbot content or answers prepared as chatbot-based AI conversation materials" (missing verb?), "that auto-212 mated chatbots should be for general education, while for specific complaints, it is recom-213 mended for non-automated chats with midwives." (the sentence had no beginning), and so on. 
Line 77, there is a "However" but it is not in logical opposition to any clause.
These are just some of many examples throughout the manuscript.

Dear respected reviewer, thank you for your meaningful suggestion. Following your recommendations, we have improved the manuscript and polished the writing of this manuscript. We reread and reviewed the English grammar many times, which is still not understandable and ambiguous. We also provide experts to proofread and improve sentence structure to make it suitable and easy for readers to understand. I hope this improvement aligns with the reviewer's expectations so that it is more appropriate and becomes a good reference for evidence readers.

2

Please clarify the research contribution. There is an attempt (lines 114-118) but it would be clearer if we knew (i) what is the one research gap addressed, (ii) how is this gap addressed overall and (iii) through specific contributions. 

Thank you very much for your meaningful suggestion.

(i)   The research gap in this paper include:

1.    Fully-automated chatbots are not mature enough to diagnose a patient's condition or replace the judgment of a healthcare professional, thus this research conducted to develop semi-automated chatbots

2.    Recently, studies examining semi-automated chatbots mainly for handling pregnant women's health are still limited.

3.    A recent systematic review shows that the decision tree is one of the tools of consideration in assisting the patient-provider in decision-making. However, external validation is needed to ensure that decisions with high-quality evidence

(ii)  How is this gap addressed overall and

1.    For the first research gap, this research purpose to develop semi-automated chatbot, as a combination between chat to bot (automatic) and chat to midwife (non-autmatic). It is expected to be an alternative to prevent errors in providing diagnosis, optimizing remote consultation and strengthen clinical decision among pregnant women's health

2.    The development of this semi-automated chatbot applying decision tree method which is carried out through a qualitative research approach, it is expected to  produce an evidence-based chatbot that can be a reference for future research

3.    The chatbot for pregnant women using a decision tree method in this research was built by involving information on user needs and health experts as one of the strategies to support the quality.

(iii)       Specific contributions for the research:

The development of semi-automated chatbots through this research was carried out through qualitative research with a user-centered approach to explore user needs. Then the researchers collected the chatbot content material (Knowledge base) according to the qualitative results, literature, and expert recommendations to develop the decision tree method. This decision tree method is applied in the development of semi-automatic chatbots. Therefore, this development can increase the accuracy of clinical decisions among pregnant women assisting their early detection of health problem. Besides, the decision tree method can strengthen the semi-autmated chatbot content

Page 3,

Line 102-109

Page 3,

112-114

Page 2,

Line 95-96

Page 3,

Line 108-112

Page 3,

114-118

Page 2,

97-99

Page 3,

Line 108-118

3

Please summarize the eligibility criteria, then the exclusion criteria. Currently it's a mixed bag in lines 131-144, including notions that don't belong to either one (e.g., 'unwillingness to participate' is not an exclusion criterion). Eligibility/exclusion criteria define who you are looking for; after that, they are invited to participate, and some of them may refuse. They are not 'ineligible' (or they would never have been contacted), they refused to participate (it's a different category).

In "Excluded midwives as informants were midwives who were known to have not carried out their duties for more than the last month", please clarify: they weren't good employees, or they were only recently hired in their positions?

In Table 2, please clarify 'parity'.

What does it mean that "Puskesmas had actively utilized" the app? I am asking for the criterion here (e.g., a sense of threshold and the logic for its determination).

Dear respected reviewer, thank you for the valuable comment. According to your suggestion, we have corrected and summarized the eligibility and exclusion criteria sections. We have grouped this section according to eligibility criteria, then proceed with excluded participants.

For excluded midwives who were known to have not carried out their duties for more than the last month, means that midwives are not actively working and do not follow the development of socialization and training for the iPosyandu application.

In terms of parity in table 2, means that parity is one of the factors that related to women's health and quality of life. The relevant literature often uses the terms “primiparous” versus “multiparous” or “low parity” versus “high parity” to refer to women who have given birth once versus those who have done so multiple times. Women with high parity have been shown to be less satisfied with life and have a poorer quality of life when compared to women with low parity (Alzboon dan Vural, 2021).

Puskesmas had actively utilized the app means health workers and CHWs who worked there had received dissemination and training on the use and were active in using the iPosyandu app

Page 4, line 137-145

Page 4

Line146-148

Page 5

Line 155-159

4

Lines 366-375, the authors explained how they developed the chatbot. The explanation is much too short to be clear, and it should not be lumped under Results, which is primarily devoted to the qualitative analysis. I suggest a separate section (e.g., 4. Development of the Chatbot) which explains this development in greater details (e.g., using a flowchart).

Thank you in advance for this worth suggestion. We have improved and made separate section between ‘Results’ and ‘Development of the Chatbot’.

We also generate the explanation of chatbot development through the description and the flowchart figure.

Page 11-21

Line 360-374

5

"Data Availability Statement": it is applicable. The authors have collected data from people, so it is a valid question to ask how it would be shared. Presumably, anonymized data could be uploaded to a repository with a permanent identifier (e.g., Open Science Framework at osf.io). The app itself is considered data and would typically be open source, as a public research product.

Thank you for your suggestion. We considered to upload our anonymized qualitative data to the website of Open Science Framework at osf.io

Page 18,

543-544

Minor Points

i

Affiliations are usually to departments / institutions. Not to programs of enrollment such as 'Master Program of Midwifery' or 'Informatics Engineering Study Program' (refer to the department/institution administering it instead).

Thank you for your comment. We have corrected the affiliations from master program of Midwifery to Master of Midwifery Study Program

Page 1, Line 8

ii

Instead of "The decision tree method was developed from", consider "The decision tree method was applied on". The authors did not develop the method of decision trees; they applied that method. Instead of "One is through" consider "One such effort involves".

Thank you very much for your suggestion, we have made correction to use this statement ‘One such effort involves’

Page 2, line 55

iii

"internet users grew significantly from 8.9% in 2018 45 to 73.7% in 2019-2020": within a single year you jumped from 9% of internet users to 74%? Please confirm.

Thank you very much for your comment. We apologize in advance for any mistake in the information presented. We have corrected and clarified this information in our manuscript. Based on a survey conducted by the Indonesian Internet User Service Association (APJII), there was an increase in internet users by 8.9% or equivalent to 25.5 million users from 2018 to 2019. Furthermre, in 2016-2020, according to the Central Statistics Agency (BPS), stated that the current survey of households accessing the internet continues to experience a fairly high increase from 66.2 to 74.55. This is due to the ease of accessing the internet through several media such as wifi, facilities at the office or school, even mobile phones.

In Indonesia, mobile health utilization has increased as internet users grew significantly from by 8.9% or equivalent to 25.5 million users, from 2018 to 2019in 2018 to 73.7% in 2019-2020 [5]. Central Statistics Agency (BPS), stated that the current survey of households accessing the internet continues to experience a fairly high increase from 66.2 to 74.55 in Indonesia from 2016 to 2020

Page 1, line 44-46

iv

I believe that each table should have a unique number. "Table 2. 1" should just be "Table 2", "Table 2.2" should be "Table 3", and so on. Please check the formatting guidelines for the journal. If I am wrong, then ignore this suggestion

Thank you for your suggestion, we have checked again to the manuscript guide that the correct format should be ‘Table 1’ and continue to ‘Table 2’, ‘Table 3’ and so on. We have already improved this section.

Page 4

Line 151-152

v

I understand that "A Focus group discussion through semi-structured face-to-face interviews was conducted". However, I do not understand how it relates to Figure 2. The interview guide for semi-structured interviews normally states how you start the interview, and how you go onto new questions depending on the initial answers. Figure 2 does not show a semi-structured interview conducted by a human, it shows an interview (for Midwives) mixed with a usability study (of people using an automatic chatbot).

Thank you for the very valuable suggestion. Based on your recommendation, we have corrected the writing in the interview guide section. We stated that the interview guideline to explore user needs in utilizing chatbots is guided by the chatbot function according to previous research by Car, et al. 2020. Then, we corrected the writing according to your suggestion regarding the interview step, starting from exploring the familiarity of informants in using chatbots. Continued by exploring according to the 5 chatbot functions that the interview guide is available on the appendix. In addition, Figure 2 shows to readers an overview of the flow of consultation structure using semi-automated chatbots.

Page 5,

Line 167-178

vi

For readers not familiar with qualitative analyses, consider explaining the nuance between codings, categories, sub-themes, and major themes.

Thank you for your suggestion. We have improved this section based on your recommendation to give more explanation about the nuance between codings, categories, sub-themes, and major themes.

After researchers read the transcription results repeatedly to understand the content clearly and identified significant statements from the transcript to find codes, followed by formulating meanings of each substantial piece of information. It called as coding process. The codes were then grouped into categories to find major findings. It was discussed by the authors to check for the correctness of these processes and consistency. After that, the codes and categories were used to develop sub-themes. Those sub-themes were then grouped into themes that represent the main findings.

Page 5-6,

Line 183-195

vii

Tables 4.1 and 4.2 may be more appropriate as appendices / supplementary online material. In their current location, they are bloating the paper.

Thank you for your suggestion. We have made chance to move the table 4 and 5 to appendix B and C

Page 20,

Line 554-556

viii

I do not understand the 'acknowledgements' statement. Permission to conduct research sounds like an institutional review board, and it would be under "Institutional Review Board Statement". If a project does not need institutional review board, then it does not need a form of permission.

Thank you for essential comment. We have improved this section according to your suggestion to delete form of permission from acknowledgments.

Page 18,

Line 562-563

No

Reviewer 2 Comments (round 2)

Author Response

Page/Line Number

1

I commend the authors for their extensive revisions and their clear response letter. The many improvements made to this manuscript will ultimately benefit the readers. I note the need for very minor fixes (e.g., appendices should be after the references rather than before), but these can typically be conducted at the copy-editing stage by MDPI personnel together with a brief grammar check.

Dear respected reviewer, thank you for your valuable suggestion. We have revised the section of appendices to move after reference.

Page 23-33

Line 753-761

2

I am still not sure about what the 'acknowledgements' are saying, since the specific involvement of the Purwakarta Regency Office is unclear to me. If they're just paying salaries and providing offices, that'd be their normal role and they would not be acknowledged for that. An acknowledgement has to go above and beyond what is supposed to be provided, and the nature of this support should be stated. However, this is a minor note and I'll let MDPI personnel decide on how to handle it. 

Thank you for your meaningful comment on the acknowledgment section. According to your suggestion, we tried to improve and revise this section to be more explicit and more details.

Page 18

Line 549-556

3

The conceptual map in Figure 5 is clearer with the new display. The explanations (categories, sub-themes, etc.) should also be embedded as a legend in the figure (i.e. part of the figure should show the meaning of each color). I believe that the direction of each edge (i.e., arrow) should be reversed too, since the researchers started with blue then moved onto pink then onto orange etc.

Thank you for this worthwhile comment. According to your suggestion, we tried to make corrections to the direction of each edge (i.e., arrow) and adding the meaning of symbol as well as the color inside the figure 6.

We also improved the method section by providing more explanation of the thematic process and adding the illustrative figure of the conceptual map.

Page 8,

Line 250

Page 6

Line 195-210