Development of a T2D App for Elderly Users: Participatory Design Study via Heuristic Evaluation and Usability Testing

Comments 1: The background of the paper should include detailed research and introduction of similar products or competing products. There is indeed no such product or software for such patients on the market, or it already exists but has shortcomings. If the product already exists on the market or the product is very mature, there is no research value.

Response 1:

Thank you for pointing this out. We agree with the reviewer's comments regarding the need for a detailed background that includes research on existing products and a critical evaluation of their effectiveness, especially in relation to elderly users. Therefore, we have made significant revisions to the manuscript to address these concerns. We have expanded the background to incorporate a more comprehensive analysis of existing diabetes management apps, highlighting their limitations and how they fail to meet the specific needs of elderly users. These changes are intended to better substantiate the necessity and novelty of our study. The following points detail the specific revisions made in response to the reviewer's comments:

l  Enhancing the Necessity of the Study

In response to the editor’s concerns about the study's background, we have clearly identified the major issues faced by current diabetes management applications, particularly their inadequacies in meeting the needs of elderly users. Through this revision, we reviewed relevant literature and discussed in detail the challenges of traditional face-to-face medical practices in diabetes management, including deficiencies in preventive systems, unequal distribution of medical resources, and difficulties patients face in self-management. These challenges not only highlight the urgency of finding more effective solutions but also strongly argue for the potential role of digital health technologies (DHT) in improving the efficiency of diabetes management. The revised content is located on page 2, lines 53-61, highlighted in yellow.

By further analyzing specific data and examples provided in the literature, we pointed out that current diabetes management tools often overlook the unique physiological and cognitive needs of elderly users. This finding not only increases the necessity of our study but also adds theoretical and practical value to the design of more targeted and effective management applications. The revised content is located on page 4, lines 113-127, highlighted in yellow.

l  Introduction of similar products

In response to the reviewer’s suggestions, we expanded the content of the paper to include an in-depth analysis of the top five diabetes management applications in the Chinese market. This analysis includes download statistics, release dates, recent updates, user ratings, and feature descriptions, providing a comprehensive overview of the current market landscape, summarized in Table 1. We specifically highlighted the shortcomings of these products in meeting the specific needs of elderly users. This part of the content is located on page 2, lines 66-74; page3, Table1, highlighted in yellow.

Furthermore, to more comprehensively address the reviewer’s comments, we added a detailed analysis of the main features and interface interactions of these diabetes management applications. This section identifies the key functional focuses of the apps, such as data recording, health education, and information analysis, and points out significant issues like information overload, insufficient interactivity, and poor information visualization and readability. The analysis of these specific problems underscores the necessity of our study, as existing products have failed to fully consider the operational habits and cognitive characteristics of elderly users. This content has been added to page 2, lines 75-77; lines 82-92; highlighted in yellow.

Through the updates to the original manuscript, we first provided a broader overview of the current market situation, summarizing the general performance of existing products and their responsiveness to the needs of elderly users. We then moved on to a detailed analysis of product features and user interfaces, identifying specific shortcomings and areas for improvement. This structure not only helps readers fully understand the background and motivation of the study but also clearly demonstrates the study’s focus and urgency for practical application.

Comments 2: In addition, it is suggested that some objective physiological equipment can be used for measurement research to reflect more realistic data.

Response 2: Thank you very much for your valuable suggestion. We fully understand the importance of physiological data in research, and we recognize the unique value these data can bring in reflecting users' real physiological responses. However, the primary focus of this study is on the usability and user experience of diabetes management applications, particularly in terms of the daily interactions of elderly users. Therefore, we chose to concentrate on heuristic evaluation and the System Usability Scale (SUS) to explore subjective user experiences.

While we acknowledge that incorporating physiological measurements could provide more comprehensive data, including this aspect in our current study might have broadened the scope beyond our intended research objectives and resource limitations. Thus, to maintain the focus of our study, we did not include these measurements in this research phase.

We greatly appreciate your suggestion and plan to consider the inclusion of physiological data in future studies to further enrich our findings and enhance the depth of our research. Once again, thank you for your support and invaluable feedback.

4. Response to Comments on the Quality of English Language

Point 1: Proposed rejection

Response 1: Thank you very much for your kind and constructive comments regarding the language and grammar of our manuscript. In response to your feedback, we have extensively revised the entire document to enhance language usage and correct all grammatical errors. Additionally, the manuscript has been thoroughly polished and reviewed by a native English speaker to ensure clarity and accuracy. In the revised version, all changes have been marked, and sentences modified by the native speaker are highlighted in bold red text to facilitate your review.

5. Additional clarifications

Our instructions are all included in point-by-point with no additional instructions, thanks again for your suggestions

Comments 1: However, the specific issues elderly users face when using diabetes management applications, such as visual, auditory, and cognitive impairments, should be more clearly stated. The current version mentions problems but lacks concrete data or examples; incorporating specific challenges would enhance persuasiveness.

Response1:

Thank you for pointing this out. We agree with this comment. Therefore, we have incorporated precise statistical data from the "53rd Statistical Report on Internet Development in China" regarding elderly people's use of technology. Moreover, we have detailed the prevalence of sensory impairments such as hearing loss and visual impairments, which are common among the elderly population, providing specific statistics. These changes not only directly address your suggestion for clearer statements but also support our arguments with concrete and quantifiable data, thereby enhancing the persuasiveness of our paper. We have further elucidated how these sensory and cognitive impairments directly affect the usability of diabetes management applications for elderly users. By citing studies that provide detailed prevalence rates and descriptions of these impairments, our revisions explicitly address the auditory, visual, and cognitive barriers elderly users face when interacting with technology. This revision can be found on page 3, lines 105-112; page 4, lines 113-127, highlighted in yellow.

Comments 2: The research methods section needs to provide more detailed information about the criteria for selecting experts and participants, as well as the specific steps of the evaluation and testing, to ensure transparency and credibility.

Response 2:

Thank you for pointing this out. I agree with this comment. Therefore, I have expanded the research methods section to include more detailed information about the criteria for selecting both experts and participants, as well as the specific steps involved in the evaluation and testing.

• For expert selection, I have provided detailed criteria, including dual-domain expertise in health informatics and usability design, extensive practical experience, relevant academic qualifications such as PhDs in Medical Informatics, and significant medical practice experience. This selection process ensures that the experts are highly qualified to identify and address usability issues in healthcare applications. These revisions can be found in the manuscript on page 7, lines 230-239, highlighted in green.
• For participant selection, I have clarified the inclusion and exclusion criteria to ensure the appropriateness of the study subjects. Specifically, 33 elderly patients with Type 2 diabetes were recruited, and 28 completed the study. These criteria were rigorously applied to ensure that the participants accurately represented the target user group. These revisions can be found in the manuscript on page 8-9, lines 268-307; Table 5, highlighted in green.
• Regarding the evaluation and testing process, I have elaborated on the specific steps, including task execution, issue documentation, and the completion and analysis of the System Usability Scale (SUS). These revisions can be found in the manuscript on page 10, lines 309-343.

Comments 3: The paper should also offer more detailed information about the 126 usability issues and 179 heuristic violations mentioned, including how these problems were classified and analyzed.

Response 3: Thank you for pointing this out. I agree with this comment. Therefore, I have provided more detailed information in the results section regarding the 126 usability issues and 179 heuristic violations identified during the heuristic evaluation. I have classified these issues according to Nielsen's heuristic principles and provided specific examples of common problems encountered by elderly users. Additionally, I have included a comprehensive analysis of how these issues were distributed across different modules of the application, with a particular focus on areas such as medication management, diet management, and medical consultation, which were most affected. Severity ratings were also provided to highlight the impact of these issues on the user experience. These revisions can be found in the manuscript on page10-12, lines 347-380; Table 7; Figure3-4, highlighted in blue.

Comments 4: Although the SUS score is 68.75, it is necessary to explain the significance of this score and compare it with other studies to help readers better understand the research results.

Response 4:

Thank you for pointing this out. I agree with this comment. Therefore, I have expanded the explanation of the SUS score and provided comparisons with other studies in the revised manuscript.

As part of the revision, I also recruited additional participants and conducted further testing with both elderly patients and experts. This expanded testing resulted in an updated SUS score. The new results reflect broader input from a larger group of users, providing a more comprehensive assessment of the system's usability. I have detailed the statistical analysis of these updated SUS scores for both the patient and expert groups, highlighting how the results evolved with the inclusion of more participants.

Additionally, I compared these findings with SUS scores from other diabetes management applications, such as 'Diabetes Manager,' 'Glucose Buddy Pro (GB+),' 'BlueStar Diabetes,' and 'mDiab,' to contextualize the performance of our system. This comparison provides a clearer understanding of where our system stands relative to existing solutions, particularly in terms of usability and user satisfaction. These revisions can be found in the manuscript on page 16-18, lines 542-631; Table5; Table8-10, highlighted in blue.

Comments 5: Finally, the paper should clearly state the improvement measures and their effects, as well as the future research directions. This will help readers fully understand the study's contributions and potential applications.

Response 5: Thank you for pointing this out. We agree with this comment. Therefore, we have thoroughly revised the manuscript to clearly state the improvement measures implemented based on our findings, as well as their effects. We have also outlined future research directions to provide a comprehensive understanding of the study's contributions and potential applications.

In the revised manuscript, we detailed the specific usability issues identified during the heuristic evaluation and how these issues informed the improvement measures we implemented. We highlighted changes such as enhancing system feedback, simplifying the interface design, and improving user control and freedom. These revisions aim to address the challenges elderly users face, such as difficulties in understanding complex terminology and interacting with inconsistent interface elements.

Furthermore, in the discussion, we elaborated on the effectiveness of the implemented improvements, discussing how these changes have enhanced the usability of the application for elderly users. We also provided a clear outline of future research directions, including the development of a senior mode for the "SugarShift" application, which will be tailored specifically for users over 60 years old. This future work aims to further refine the application's interface and functionality based on ongoing feedback and iterative testing with both users and experts. These revisions can be found in the manuscript on page 19, lines 698-719; page 20, lines 737-759, highlighted in red.

4. Response to Comments on the Quality of English Language

Point 1: You should have find some native speaker to modify the whole paper;(

Response 1: Thank you very much for your kind and constructive comments regarding the language and grammar of our manuscript. In response to your feedback, we have extensively revised the entire document to enhance language usage and correct all grammatical errors. Additionally, the manuscript has been thoroughly polished and reviewed by a native English speaker to ensure clarity and accuracy. In the revised version, all changes have been marked, and sentences modified by the native speaker are highlighted in bold red text to facilitate your review.

5. Additional clarifications

Our instructions are all included in point-by-point with no additional instructions, thanks again for your suggestions

Comments 1: on one side, providing more users the experiment with this app at least in the order of 15.

Response 1: Thank you very much for your insightful suggestion. I fully agree with your comment. In response, I have reconducted the usability test and expanded the sample size to include more participants. Specifically, we recruited 33 elderly patients diagnosed with Type 2 diabetes, strictly adhering to the inclusion criteria. Of these, 28 participants completed the entire testing process. We have carefully documented all participants' background information and, through a rigorous testing process, collected their feedback on the app’s usability and performance. This modification significantly enhances the representativeness and credibility of our study findings. The detailed information regarding participant demographics, testing procedures, and the collection of usability feedback and scores has been incorporated into the revised manuscript on page 8-9, lines 268-307; Table 5, highlighted in green.

Comments 2: trying to use inferential statistics (with relative tests of goddess of fit and hypothesis testing) or some other correlation measures, like for example in the following paper (to be cited): Designing human-centric software artifacts with future users: a case study, Human-centric Computing and Information Sciences, Volume 10, Issue 11 December 2020 doi: 10.1186/s13673-020-0213-6.

Response 2: Agree. I have, accordingly, revised the manuscript to incorporate additional statistical methods to enhance the robustness of the analysis. Specifically, I calculated the median, mean, standard deviation, t-statistic, and p-value to assess the central tendency and dispersion of SUS scores, and to determine the statistical significance of differences between the scores and a neutral value. This detailed statistical analysis allowed us to evaluate the system's overall usability and to identify consistencies and variations across different user groups, thereby providing a solid foundation for subsequent optimization efforts.

In addition, I have also applied inferential statistical methodologies, as suggested, including Cosine Similarity, Jaccard Similarity Coefficient, and Simple Matching Coefficient (SMC), to analyze the alignment between the expert and patient groups' evaluations. These measures were inspired by the referenced paper (citation [55]), and they provided a more nuanced understanding of the correlations between the evaluations, highlighting both areas of consensus and divergence. This approach aligns our study with established practices in human-centric software evaluation.

These revisions can be found in the manuscript on page 16-18, lines 543-631; Table5; Table8-10, highlighted in blue.

4. Response to Comments on the Quality of English Language

Point 1:

Response 1:

5. Additional clarifications

Our instructions are all included in point-by-point with no additional instructions, thanks again for your suggestions

Comments 1:   Highlight specific gaps in the existing research and provide a concise statement of the main findings and recommendations in Abstract.

·         Highlight specific gaps in the existing research

Challenges in Medical Practice: The research highlights the limitations of traditional face-to-face medical practices in diabetes management, such as insufficient preventive measures, unequal distribution of medical resources, and inadequate self-management. These issues indicate that current medical practices fail to fully meet the needs of diabetes patients, particularly in terms of continuous monitoring and personalized treatment. The revised content is located on page 2, lines 53-56, highlighted in yellow.

Shortcomings in the App Market: Although there are numerous medical applications available on the market, most of them receive only moderate to good user ratings. These applications are often not designed for specific user groups and lack features that focus on the actual needs of patients. The revised content is located on page 2, lines 70-74, highlighted in yellow.

Interface and Interactivity Issues: Through an analysis of the literature and existing similar products on the market, it was identified that current diabetes management apps commonly employ flat design. However, user feedback indicates problems such as information overload, insufficient interactivity, and poor information visualization and readability. These interface and interactivity issues may hinder users from effectively using these apps, thereby impacting the efficiency and effectiveness of diabetes management. The revised content is located on page 2, lines 82-85, highlighted in yellow.

·         Abstract Update

We have made targeted revisions to the abstract to ensure it accurately and comprehensively reflects the research findings and related recommendations. In the revised abstract, we emphasized the necessity of the study and highlighted a key issue with existing diabetes management apps—their frequent neglect of the specific needs of elderly users. The abstract briefly outlines the research method, which involves participatory design directly engaging elderly patients in the development of the SugarShift app to better meet this user group’s needs. Specific issues identified through heuristic evaluation and usability testing, as well as how these issues impact the app’s acceptance and usage behavior, are summarized. Finally, the abstract concludes with a discussion of the optimization strategies developed based on these findings and provides specific improvement recommendations. The revised content is located on page 1, lines 10-26, highlighted in red.

Comments 2:  Include the most recent systematic reviews to strengthen the argument by highlighting the gaps in current diabetes management. Emphasize how emerging technologies like AI and diabetes (looking through retina) are being leveraged to improve diabetes care.

Response 2: Thank you for pointing this out. I agree with this comment. Therefore, I have revised the manuscript to more clearly present the latest system evaluation and highlight the application of digital health technologies, such as artificial intelligence, in diabetes management. I have elaborated on how digital health technologies, particularly AI, can significantly reduce these barriers, enhancing treatment precision and efficiency, while also driving the shift toward personalized and predictive diabetes management. Through this revision, I have reviewed the literature on systematic reviews and thoroughly discussed the challenges faced by traditional face-to-face medical practices in diabetes management, such as the inadequacy of prevention systems, the unequal distribution of medical resources, and the difficulties patients face in self-management. These issues not only underscore the urgency of finding more effective solutions but also strongly argue for the potential role of digital health technologies (DHT) in improving the efficiency of diabetes management. The revised content is located on page 2, lines 53-61, highlighted in yellow.

In the discussion section, I have integrated advanced technologies that may be applied in future development of diabetes management models for the elderly and explored the potential benefits these technologies could bring to both patients and healthcare teams. The revised content is located on page 20-21, lines 737-759, highlighted in red.

Comments 3:   In Methods section, explain why the particular heuristics were chosen and how they were applied.

Response 3: Agree. I have, accordingly, revised the Methods section to provide a detailed explanation of why the particular heuristics were chosen and how they were applied in this study. The heuristic evaluation method, proposed by Jakob Nielsen and Molich in 1989, was selected because it is one of the most authoritative and widely used approaches in user experience research. It allows for the rapid identification of usability issues by experts who assess real systems or prototypes against established usability guidelines. This method was particularly suited to our study because it facilitates the detection of design flaws that might affect elderly users, a group with specific usability needs.

In applying these heuristics, experts analyzed usage scenarios and tasks to categorize usability issues into distinct heuristic categories, such as system feedback, user cognition, interface consistency, and error prevention. Severity ratings were assigned to these issues, which provided diverse insights into how the system could be improved. This structured approach ensured that the evaluation was comprehensive and directly addressed the unique challenges faced by elderly users of diabetes management applications.

These revisions can be found in the manuscript on page 4-5, lines 152-177, highlighted in green

Comments 4:   Include more detailed statistical analyses of the heuristic evaluation and usability testing results.

Response 4: Thank you for pointing this out. I agree with this comment. Therefore, I have added a more comprehensive set of statistical methods and analyses in the revised manuscript to provide a thorough examination of both the heuristic evaluation and usability testing results.

To enhance the demographic analysis of participants, I have included detailed descriptive statistics, such as mean, standard deviation, and frequency distributions, which clarify the characteristics of the study population. This demographic information provides context for the usability testing results and allows for a better understanding of the variability within the participant group. These revisions can be found in the manuscript on page 8-9, lines 268-307; Table 5, highlighted in green.

For the heuristic evaluation, I have expanded the statistical reporting by detailing the 126 usability issues and 179 heuristic violations, with a breakdown across different usability principles. I have also calculated severity scores and the frequency of violations within each heuristic category. Additionally, I have included comparisons of these results across various modules of the application, as shown in Figures 3 and 4, to identify the areas that need the most improvement. These revisions can be found in the manuscript on page10-12, lines 348-380; Table 7; Figure3-4, highlighted in blue.

Regarding the usability testing, I have implemented a detailed statistical analysis of the System Usability Scale (SUS) scores, including median, mean, standard deviation, t-statistic, and p-value. This analysis offers a nuanced understanding of how the system performed across different user groups. These revisions can be found in the manuscript on page 16-17, lines 543-602; Table8-9, highlighted in blue.

Moreover, I have introduced inferential statistical methods to analyze the consistency and reliability of the evaluations between the expert and patient groups. By calculating metrics such as Cosine Similarity, Jaccard Similarity Coefficient, and Simple Matching Coefficient (SMC), I have provided a more in-depth comparison of how these two groups perceive the system's usability. These revisions can be found in the manuscript on page 18, lines 603-631; Table10, highlighted in blue.

Comments 5:    How many experts have you invited to conduct a heuristic evaluation? Six experts “Lines 14, 121” or four experts “Lines 130, 139”?

Response 5: Thank you for pointing this out. I agree with this comment. Therefore, I have clarified the distinction between the two groups of experts mentioned in the manuscript to avoid any confusion.

In the revised manuscript, it is emphasized that a total of six experts were involved in the heuristic evaluation process. These six experts include four with PhDs in Medical Informatics, who possess substantial experience in health informatics and usability design. Additionally, a seasoned user experience (UX) design expert and an endocrinology professional were included to ensure a comprehensive and well-rounded evaluation. This diverse team was carefully selected to guarantee that the usability issues identified within the application were thoroughly assessed from multiple perspectives, particularly considering the complexities of the healthcare sector. These revisions can be found in the manuscript on page 6-7, lines 230-239, highlighted in green.

The confusion may have arisen from the separate mention of another group comprising eight members, which was specifically formed to develop and validate the task scenarios used in the usability testing phase. This group consisted of two diabetes experts, two UX design specialists, and four elderly diabetic patients, who collaborated to create realistic and relevant scenarios for the system's evaluation. This team’s role was distinct from the six experts involved in the heuristic evaluation, focusing instead on the development of scenarios that reflect real-life use cases. These revisions can be found in the manuscript on page 5, lines 199-214, highlighted in green.

This distinction should clarify the roles and contributions of the different expert groups within the study.

Comments 6: Add labels for the x-axis in Figures 3 and 5, and legends. Add labels for the x and y-axes in Figures 4.

Response 6: Thank you for your valuable feedback. I have made the requested changes to Figures 3 and 5 by adding labels for the x-axis and legends to ensure clarity. Additionally, in this revision, I have reclassified the results of the heuristic evaluation, which led to the removal of Figure 4. The data previously represented in Figure 4 has now been replaced with Table 7, which provides a more detailed and organized presentation of the information. This change can be found in the updated manuscript. These revisions can be found in the manuscript on page12, highlighted in blue.

Comments 7: Bullet points are used to list items or points in a clear and concise manner. Do not use bullet points for empty lines or for explanations that precede the bullet points. Ex: “Lines: 311, 318, 320, 326, 328, 332, 334, 345, 347, 378, 382, 386, 413”

Response 7: Thank you for your insightful feedback. I have made the necessary revisions to remove the inappropriate use of bullet points and ensured that they are used only for listing items or points in a clear and concise manner.

4. Response to Comments on the Quality of English Language

Point 1: Paper has several typographical and grammatical errors: Ex: "who typically achieve a higher detection rate" should be "who typically achieves a higher detection rate", ….

Response 1: Thank you very much for your kind and constructive comments regarding the language and grammar of our manuscript. In response to your feedback, we have extensively revised the entire document to enhance language usage and correct all grammatical errors. Additionally, the manuscript has been thoroughly polished and reviewed by a native English speaker to ensure clarity and accuracy. In the revised version, all changes have been marked, and sentences modified by the native speaker are highlighted in bold red text to facilitate your review.

5. Additional clarifications

Our instructions are all included in point-by-point with no additional instructions, thanks again for your suggestions