Remote Moderated Usability Testing of a Mobile Phone App for Remote Monitoring of Pregnant Women at High Risk of Preeclampsia in Karachi, Pakistan

Round 1

Reviewer 1 Report

The article presents the result of the usability testing of the mobile app. While usability testing is the main focus of the paper, it suggested that the authors provide a section on the Raabta app. In this section, the authors could provide some background on the app, the architecture of the app, the features and functionalities. It is advisable to show some screenshots of the application. For example, it is good to know how the user would use the app to enter blood pressure, frequency of entering the data, who would monitor these data etc. This paper evaluates the usability of the Raabta, an mobile that monitors pregnant women with high risk for preeclampsia to monitor their blood pressure and symptoms at home. The usability testing was done remotely with 14 participants using Zoom. The data collection techniques are think aloud and task-completion and the data analysis technique is thematic analysis.  

The strength of this paper is the methodology. While the methodology is typical to usability testing but to do it remotely is something that author can exploit. The weakness of the paper is that the authors seem to want to report the findings of the usability testing without giving sufficient background on the Raabta app, culturally design mobile app, cultural UI/UX, remote usability testing etc.  

Literature review was not present in the paper. It is suggested that the authors include some literature on the some of the areas mentioned in the study such as cultural UI/UX, remote usability etc and also existing work in these areas.  

The strength of this paper is the methodology. Proper methodology was used in the usability testing but it was explained sufficiently. Nonetheless, it is suggested that the authors include diagram to depict the flow of the research activities.  

For data collection, the participants were group into 4 groups. Two of the groups have 5 participants and the other two groups have 2 participants. The participants in the groups should be balanced meaning that the two groups with 2 participants should have 4-5 members.  

The authors used thematic analysis but it is unclear how the thematic analysis was conducted. What would be the source of data? Interview? remarks or memo from researchers? video? What QDA software was used for this? Using a QDA software, the authors can show the codes that leads to the emergence of themes.  

The presentation of the results can be improved by including the direct quotations from the participant for each theme. For example for clarity of instructions, messaging and terminology theme. The authors can select some quotations from the participants that support this theme.  

The results are common for any usability testing. I think the authors can improve this by discussing the result looking from the angle of culture and remote usability testing.  

It is suggested that for the header for section 3.2 until 3.5, the authors can put the numbering of the theme to indicate that this section is about the theme. For example for Theme 1, authors can write Theme 1: Name of the theme.    

The limitation of the study has been identified. This paper can be improved further if the authors can be addressed some of the limitations. One limitation that can be addressed is number of participants in each group. If the authors can address this limitation, then perhaps the manuscript can be published in the journal.   

The language and the writings are good. Appropriate and understandable.  

Overall, this paper is interesting given the context and the nature of how the usability testing was conducted. The findings of the usability testing are as expected. Nonetheless, while the focus is on usability testing, some focus should be given on culturally designed mobile app and the fact that the testing is done remotely. The authors can include these in the discussion of the results.  

Author Response

Title: Remote Moderated Usability Testing of a Mobile Phone App for Remote Monitoring of Pregnant Women at High-risk for Preeclampsia in Karachi, Pakistan

 

Response to Reviewer 1 Comments
1. Summary
Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding revisions/corrections highlighted/in track changes in the re-submitted files.
2. Questions for General Evaluation	Reviewer’s Evaluation	Response and Revisions
Does the introduction provide sufficient background and include all relevant references?	Yes/Can be improved/Must be improved/Not applicable
Are all the cited references relevant to the research?	Yes/Can be improved/Must be improved/Not applicable
Is the research design appropriate?	Yes/Can be improved/Must be improved/Not applicable
Are the methods adequately described?	Yes/Can be improved/Must be improved/Not applicable
Are the results clearly presented?	Yes/Can be improved/Must be improved/Not applicable
Are the conclusions supported by the results?	Yes/Can be improved/Must be improved/Not applicable
3. Point-by-point response to Comments and Suggestions for Authors   Comment 1: The article presents the result of the usability testing of the mobile app. While usability testing is the main focus of the paper, it suggested that the authors provide a section on the Raabta app. In this section, the authors could provide some background on the app, the architecture of the app, the features and functionalities. It is advisable to show some screenshots of the application. For example, it is good to know how the user would use the app to enter blood pressure, frequency of entering the data, who would monitor these data etc. This paper evaluates the usability of the Raabta, an mobile that monitors pregnant women with high risk for preeclampsia to monitor their blood pressure and symptoms at home. The usability testing was done remotely with 14 participants using Zoom. The data collection techniques are think aloud and task-completion and the data analysis technique is thematic analysis.   Response: Thank you for the comment. We have now added more information on the Raabta app including background, features and functionalities, frequency of monitoring etc. We have also provided the figure 1 to show some screenshots of Rabbta application. The edits have been made under 2.2 Raabta app heading on pg no. 3 and 4.   Comment 2: The strength of this paper is the methodology. While the methodology is typical to usability testing but to do it remotely is something that author can exploit. The weakness of the paper is that the authors seem to want to report the findings of the usability testing without giving sufficient background on the Raabta app, culturally design mobile app, cultural UI/UX, remote usability testing etc.   Response: Thank you for the comment. We have now incorporated literature on remote usability testing in the introduction section on pages 2-3 and have discussed the rationale for employing remote usability testing in the methods section on page 3. We would like to emphasize that remote usability testing has become the default approach to advance technology evaluation and research. As technology continues to evolve, remote usability testing is expected to gain greater recognition in the literature. The findings of this study will contribute to the growing body of evidence on remote moderated usability testing in resource-limited environments. Additionally, as mentioned earlier, we have provided more comprehensive information about the Raabta app, including its background, features, functionalities, and monitoring frequency. Figure 1 has also been included to showcase screenshots of the Raabta application. These edits have been made under the "2.2 Raabta App" section on pages 3 and 4.     Comment 3: Literature review was not present in the paper. It is suggested that the authors include some literature on the some of the areas mentioned in the study such as cultural UI/UX, remote usability etc and also existing work in these areas.   Response: Thank you for your comment. As per your suggestion, we have now incorporated literature on remote usability testing in the introduction section and included relevant literature on culturally appropriate technology and user engagement in the discussion section.   Comment 4: The strength of this paper is the methodology. Proper methodology was used in the usability testing but it was explained sufficiently. Nonetheless, it is suggested that the authors include diagram to depict the flow of the research activities.   Response: Thank you for the suggestion. We have now included figure 2 to depict the flow of the research activities.   Comment 5: For data collection, the participants were group into 4 groups. Two of the groups have 5 participants and the other two groups have 2 participants. The participants in the groups should be balanced meaning that the two groups with 2 participants should have 4-5 members.   Response: Thank you for the comment. This has been addressed in the limitation section of the manuscript on pg no. 11.   Comment 6: The authors used thematic analysis but it is unclear how the thematic analysis was conducted. What would be the source of data? Interview? remarks or memo from researchers? video? What QDA software was used for this? Using a QDA software, the authors can show the codes that leads to the emergence of themes.   Response: Thank you for bringing this to our attention. We have made revisions to the content under the 'Data Analysis' heading to include details about the data sources and to clarify that NVivo software played a primary role in conducting the thematic analysis.   Comment 7: The presentation of the results can be improved by including the direct quotations from the participant for each theme. For example for clarity of instructions, messaging and terminology theme. The authors can select some quotations from the participants that support this theme.   Response: Thank you for your comment. Initially, we included quotations from the data; however, we made a deliberate decision to remove them later. The reason for this adjustment was that the quotations primarily pertained to the specifics of the screens and the user interface, which could be challenging for our audience to grasp. Instead, in the results section, we opted to offer a summarized version of participants' statements verbatim, aiming to enhance overall readability and comprehension.   Comment 8: The results are common for any usability testing. I think the authors can improve this by discussing the result looking from the angle of culture and remote usability testing.   Response: Thank you for highlighting this. we have now incorporated literature on remote usability testing in the introduction section and included relevant literature on culturally appropriate technology and user engagement in the discussion section.     Comment 9: It is suggested that for the header for section 3.2 until 3.5, the authors can put the numbering of the theme to indicate that this section is about the theme. For example for Theme 1, authors can write Theme 1: Name of the theme.     Response: Thank you for the suggestion. We have now used numbering for the themes.   Comment 10: The limitation of the study has been identified. This paper can be improved further if the authors can be addressed some of the limitations. One limitation that can be addressed is number of participants in each group. If the authors can address this limitation, then perhaps the manuscript can be published in the journal.    Response: Thank you for the suggestion. We have carefully read your suggestion regarding the number of participants in each group. Unfortunately, it's not feasible for us to conduct additional interviews as we have already concluded the study and made modifications to the program based on the results. We would like to mention here that we were able to achieve data saturation in those two groups which had a smaller number of participants. In addition, we have engaged with providers, caregivers, and various other groups in a broader context to ensure that the findings of our study align with and resonate with all targeted user groups.   Comment 11: Overall, this paper is interesting given the context and the nature of how the usability testing was conducted. The findings of the usability testing are as expected. Nonetheless, while the focus is on usability testing, some focus should be given on culturally designed mobile app and the fact that the testing is done remotely. The authors can include these in the discussion of the results.   Response: Thank you for highlighting this. we have now incorporated literature on remote usability testing in the introduction section and included relevant literature on culturally appropriate technology and user engagement in the discussion section

 

 

 

 

 

Reviewer 2 Report

The aim of the paper is to test the usability of a smartphone app among pregnant women at high-risk for preeclampsia. The study context is Pakistan having low-income areas and low sosioeconomic status among its large polulation. Referring to those, the topic of this study is important.

However, several corrections need to done to make this paper more scientific sound. Starting from the introduction, authors do not justify the benefits of smarthome apps. There's only a one sentence (lines 44-47) where you refer to 6 references (one is old from 2005 and three from the same author), but you don't open upp what do you mean by telemonitoring interventions, as you call them. A lot of research has done in this field, you should update your literature and write much more of it. In the Discussion you have new references (22-25), some of those could be used already here in the introduction.

Check also the other old references (from 2003, 2000, 2001) and update those. You also refer 4 of your own publications.

You have opened upp the Raabta app in the introduction, and again in chapter 2.2. Sure, it's important, but how to describe it more briefly. Instead, you should write more about usability testing in general - you refer to Nielsen in line 75, but it comes not until in Sample 3.1. I would open up Jakob Nielsen's usability heuristics, if you used those - this remains unclear.

Materials and methods:

You refer here to Nielsen Norman approach (ref. 17), so again, open that up. Interviews were undertaken via Zoom and on site? In line 92 you write Zoom, later in 2.4. you clarify this, you have the research analyst person in Zoom, and the patient and on-line facilitator in the same room.

What are NICE criteria or guidelines, open up.

Results:

I did not count how may times you mention "users". Here the content of the findings are important. This is a qualitative study, respondents' direct quotes as examples, would enliven the text.

Discussion:

Refer more to those references you have used in the Introduction. The aim is that your results are discussing with the earlier studies. E.g., in lines 317-318 you mention the methodologies you used, put references there. You have earlier references of your own of the Raabta app, refer them here.

You start the Discussion " A formative usability testing was carried out...". Yes, this is a formative study, but have you mentioned it before? Does you readers know what is a formative study?

 

Related to result section, language could be more fluent.

Author Response

Response to Reviewer 2 Comments
1. Summary
Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding revisions/corrections highlighted/in track changes in the re-submitted files.
2. Questions for General Evaluation	Reviewer’s Evaluation		Response and Revisions
Does the introduction provide sufficient background and include all relevant references?	Yes/Can be improved/Must be improved/Not applicable
Are all the cited references relevant to the research?		Yes/Can be improved/Must be improved/Not applicable
Is the research design appropriate?		Yes/Can be improved/Must be improved/Not applicable
Are the methods adequately described?		Yes/Can be improved/Must be improved/Not applicable
Are the results clearly presented?		Yes/Can be improved/Must be improved/Not applicable
Are the conclusions supported by the results?		Yes/Can be improved/Must be improved/Not applicable

3. Point-by-point response to Comments and Suggestions for Authors

 

Comment 1: The aim of the paper is to test the usability of a smartphone app among pregnant women at high-risk for preeclampsia. The study context is Pakistan having low-income areas and low sosioeconomic status among its large polulation. Referring to those, the topic of this study is important. However, several corrections need to done to make this paper more scientific sound. Starting from the introduction, authors do not justify the benefits of smarthome apps. There's only a one sentence (lines 44-47) where you refer to 6 references (one is old from 2005 and three from the same author), but you don't open upp what do you mean by telemonitoring interventions, as you call them. A lot of research has done in this field, you should update your literature and write much more of it. In the Discussion you have new references (22-25), some of those could be used already here in the introduction.

Response: Thank you for your comment. We have incorporated the definition of telemonitoring and included references of our research group's papers. While we acknowledge that some of these references are dated, they remain pivotal within the field, which is why we have retained them.

 

Comment 2: Check also the other old references (from 2003, 2000, 2001) and update those. You also refer 4 of your own publications.

Response: As mentioned earlier, we have included references of our research group's papers. While we acknowledge that some of the references in introduction section are dated, they remain pivotal within the field, which is why we have retained them.

 

Comment 3: You have opened upp the Raabta app in the introduction, and again in chapter 2.2. Sure, it's important, but how to describe it more briefly. Instead, you should write more about usability testing in general - you refer to Nielsen in line 75, but it comes not until in Sample 3.1. I would open up Jakob Nielsen's usability heuristics, if you used those - this remains unclear.

Response: Thank you for the comment. We have now added more information on the Raabta app including background, features and functionalities, frequency of monitoring etc. We have also provided the figure 1 to show some screenshots of Rabbta application. The edits have been made under 2.2 Raabta app heading on pg no. 3 and 4. We have also added literature around remote usability testing under the introduction section on pg no. 3.

 

 

Comment 4: Materials and methods:You refer here to Nielsen Norman approach (ref. 17), so again, open that up. Interviews were undertaken via Zoom and on site? In line 92 you write Zoom, later in 2.4. you clarify this, you have the research analyst person in Zoom, and the patient and on-line facilitator in the same room.

Response: Thank you for the comment. We have now provided a description on the remote moderated usability testing approach. Yes, the remote moderated usability testing was conducted remotely via Zoom, as a  suitable local research analyst with expertise in usability testing was not available for this study. An on-site facilitator was hired for patient recruitment, logistics and physical tasks. We have now provided Figure 2 to depict the flow of research activities.

 

 

Comment 5: What are NICE criteria or guidelines, open up.

Response: Table 1 on page number 5 provides details on NICE guidelines.

 

Comment 6: Results: I did not count how may times you mention "users". Here the content of the findings are important. This is a qualitative study, respondents' direct quotes as examples, would enliven the text.

Response: Thank you for your comment. Initially, we included quotations from the data; however, we made a deliberate decision to remove them later. The reason for this adjustment was that the quotations primarily pertained to the specifics of the screens and the user interface, which could be challenging for our audience to grasp. Instead, in the results section, we opted to offer a summarized version of participants' statements verbatim, aiming to enhance overall readability and comprehension.

 

 

Comment 7: Discussion: Refer more to those references you have used in the Introduction. The aim is that your results are discussing with the earlier studies. E.g., in lines 317-318 you mention the methodologies you used, put references there. You have earlier references of your own of the Raabta app, refer them here.

Response: Thank you for highlighting this. I have now included references for methodologies as well as for the Raabta app.

 

Comment 8: You start the Discussion " A formative usability testing was carried out...". Yes, this is a formative study, but have you mentioned it before? Does you readers know what is a formative study?

Response: I have now mentioned in the introduction as well that it is a formative study. The scientific community would know the meaning of the formative study.

 

 

 

Round 2

Reviewer 1 Report

The following are the comments for the authors to consider.

Include a system flow to see clearly how a user would use the Raabta app. First login -> Select Blood Pressure etc.

There is less richness in the results considering that the data comes from the think aloud technique, observation notes etc. Richness can be enhanced by including quotations from the think aloud or on-site facilitator's or research analyst's notes or memo. 

It is interesting to know in the discussion if there are challenges in doing remote usability testing and if there is a possibility that the challenges could impact the findings. 

The literature review was added but only discussing telemonitoring as a tool. The paper is about remote usability testing for a remote monitoring app. Suggest including any remote usability testing that has been done before by other researchers. May be remote usability testing in mobile app in other field and how was the results like?

Since in the results and discussion mentioned culturally aligned user interface/user experience suggest including some literature on this as well. Authors can discuss previous works that have been done in the context of health or in Pakistan perhaps.

 

There is no caption for Figure 1. 

Figure 1 is too small. Suggest to break the figures into multiple figures. Ex. Figure 1: Raabta Dashboard Figure 2: Blood Pressure Reading, etc. The authors can have 2-3 screens in one figure. 

The figure needs to be mentioned in the paragraph. 

 

Figure 2 is too small. Suggest to redo this figure.

Figure 2 has no caption

Mention Figure 2 in the text

Author Response

Response to Reviewer 1 Comments
1. Summary
Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding revisions/corrections highlighted/in track changes in the re-submitted files.
2. Questions for General Evaluation	Reviewer’s Evaluation	Response and Revisions
Does the introduction provide sufficient background and include all relevant references?	Yes/Can be improved/Must be improved/Not applicable
Are all the cited references relevant to the research?	Yes/Can be improved/Must be improved/Not applicable
Is the research design appropriate?	Yes/Can be improved/Must be improved/Not applicable
Are the methods adequately described?	Yes/Can be improved/Must be improved/Not applicable
Are the results clearly presented?	Yes/Can be improved/Must be improved/Not applicable
Are the conclusions supported by the results?	Yes/Can be improved/Must be improved/Not applicable
3. Point-by-point response to Comments and Suggestions for Authors   Comment 1: Include a system flow to see clearly how a user would use the Raabta app. First login -> Select Blood Pressure etc. Response: Thank you for the excellent suggestion. I have now included appendix A under Raabta app heading on page no. 4. The guide provides an overview of how to use Raabta program. This guide will be used to train women during enrollment. Additionally, we have developed a short, animated video in Urdu language that will be used to train women on the use of the Raabta program.   Comment 2: There is less richness in the results considering that the data comes from the think aloud technique, observation notes etc. Richness can be enhanced by including quotations from the think aloud or on-site facilitator's or research analyst's notes or memo.  Response: Thank you for the comment. I understand your concern. I have now added a few relevant quotations from patients in results section.   Comment 3: It is interesting to know in the discussion if there are challenges in doing remote usability testing and if there is a possibility that the challenges could impact the findings.  Response: Thank you for the suggestion. I have now highlighted this aspect under the limitation section, where I am talking about the limitations of remote usability technique. Comment 4: The literature review was added but only discussing telemonitoring as a tool. The paper is about remote usability testing for a remote monitoring app. Suggest including any remote usability testing that has been done before by other researchers. May be remote usability testing in mobile app in other field and how was the results like? Response: Thank you for the comment. We have mentioned that the remote usability testing technique has been used for conducting usability test with older adults during covid-19. There is limited evidence on the use on remote usability testing in Pakistan and other resource limited setting. Our study will contribute to the body of growing evidence concerning remote moderated usability testing in resource-limited environments. This has been added on pg no. 3 under introduction heading.   Comment 5: Since in the results and discussion mentioned culturally aligned user interface/user experience suggest including some literature on this as well. Authors can discuss previous works that have been done in the context of health or in Pakistan perhaps.  Response: Thank you for the comment. I have now added a study on pg 12 understand discussion heading to highlight the significance of culturally aligned user interface.   Comment 6: There is no caption for Figure 1. Figure 1 is too small. Suggest to break the figures into multiple figures. Ex. Figure 1: Raabta Dashboard Figure 2: Blood Pressure Reading, etc. The authors can have 2-3 screens in one figure. The figure needs to be mentioned in the paragraph.  Response: Thank you for the comment. I have now cited figure 1 within the text under Raabta app heading on page no. 5. I have also worked on the figure and improved it for visibility. I am sure journal will take care of its visibility as well. I also feel that the readers will get the sense of the app through appendix A which I have added.   Comment 7: Figure 2 is too small. Suggest to redo this figure. Figure 2 has no caption. Mention Figure 2 in the text Response: Thank you for the comment. I have modified figure 2, added caption, and cited the figure in the text.

 

Reviewer 2 Report

You have made several good corrections. I still do not agree that reference 4 (Meystre et al.2005. The Current State of Telemonitoring: A Comment on the Literature) is that important that you cannot update it to much more previous study/studies. You have 17 references now in your introduction - it's not much. However, you have totally new references in your Discussion. Refer to them already here in the Introduction.

Also, references of usability testing (15, 16) are old, like I wrote earlier. I do not agree that they are that pivotal and any previous studies of usability testing exits.

Because figure 1 is not in english, I cannot understand it. Is it necessary? Instead figure 2 is pretty good.

You must have a reference to NICE criteria.

Check the language in line 272

Author Response

Response to Reviewer 2 Comments
1. Summary
Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding revisions/corrections highlighted/in track changes in the re-submitted files.
2. Questions for General Evaluation	Reviewer’s Evaluation		Response and Revisions
Does the introduction provide sufficient background and include all relevant references?	Yes/Can be improved/Must be improved/Not applicable
Are all the cited references relevant to the research?		Yes/Can be improved/Must be improved/Not applicable
Is the research design appropriate?		Yes/Can be improved/Must be improved/Not applicable
Are the methods adequately described?		Yes/Can be improved/Must be improved/Not applicable
Are the results clearly presented?		Yes/Can be improved/Must be improved/Not applicable
Are the conclusions supported by the results?		Yes/Can be improved/Must be improved/Not applicable

3. Point-by-point response to Comments and Suggestions for Authors

 

Comment 1: You have made several good corrections. I still do not agree that reference 4 (Meystre et al.2005. The Current State of Telemonitoring: A Comment on the Literature) is that important that you cannot update it to much more previous study/studies. You have 17 references now in your introduction - it's not much. However, you have totally new references in your Discussion. Refer to them already here in the Introduction.

Response: Thank you for highlighting this. I have now added relevant references in the introduction section.

 

Comment 2: Also, references of usability testing (15, 16) are old, like I wrote earlier. I do not agree that they are that pivotal and any previous studies of usability testing exits.

Response: I have now added new references as well on the remote usability technique.

 

Comment 3: Because figure 1 is not in English, I cannot understand it. Is it necessary? Instead, figure 2 is pretty good.

Response: Thank you for the comment. I understand your concern regarding the app screens. I have kept it for reviewer 1.

 

Comment 4: You must have a reference to NICE criteria.

Response: I have added a reference for NICE guidelines.

 

Comment 5: Check the language in line 272

Response: I have corrected the language error on line 272.