Investigating and Measuring Usability in Wearable Systems: A Structured Methodology and Related Protocol
Round 1
Reviewer 1 Report
This research evaluates and measures the usability of wearable systems by offering the reader the methodological bases and the related tools. Considering the fruitful discussion, the literature review could be extended further, and the manuscript requires proofreading. What’s more, the author can use tree charts to make the work more intuitive. The further comments are as follows:
- In the Introduction, please add the last paragraph to explain briefly the following sections of the paper.
- Please review these studies and the latest ones in more detail like their findings, research contexts, and results. This review should convenience the reader of the noteworthy gap that can be addressed by the present study.
- Please add the section number for “Final Usability Test Report” (line 402).
- Please provide the reader with some examples of nonparametric statistical analysis (line 407).
- Please further clarify lines 368 and 369 in “3.2.3. Undressing the System”. For instance, please mention what the obvious modifications related to taking off the system are and why they are noteworthy to consider.
- Please add a brief sentence to concisely explain the sub-sections of sections 2, 3, and 3.2.
- It would be great if you could summarize the guidance using tree charts, as it makes the work more intuitive.
- The author is encouraged to state the entire phrase before using any abbreviations (example: line 341).
- The current submission requires noteworthy proofreading, as several incomplete, and unclear sentences hinder demonstrating the importance of the work. I highlighted some concerns about the English language in the file attached.
- For other corrections, please refer to the attached file.
- Please highlight the corrections for my comments in the revised file to speed up the review process.
Comments for author File: 
Comments.pdf
Author Response
Dear reviewer.
first of all I really appreciate the extensive review and the constructive comments that drove the revision and integration of the paper.
The revised version follows point by point your suggestions
Thank you very much.
Reviewer 2 Report
The topic is interesting, but the material, methods and results are inadequately argued and described.
The current version of the article does not allow me to assess the original contribution of the study, lacking some definitions and references, an in-depth analysis of the available literature on the topic with the identification and characterisation of its gaps, a justification for the proposed protocol steps and tools, a discussion of limitations of the study and future research.
As a consequence, the first recommendation is to explain the original contribution of the paper, demonstrating it into the paper. Currently, there is only one literature review sentence (lines 56-8); it is not sufficient to motivate the study and to evaluate the original contribution.
Specific Comments:
1) Introduction: The author states that wearables have been introduced in the last decade. This should be verified and justified. Some authors, for example, published a literature review on (smart) wearable systems in 2012: Chan, M., Estève, D., Fourniols, J.-Y., Escriba, C., Campo, E. (2012) “Smart wearable systems: Current status and future challenges”. Artificial Intelligence in Medicine 56 (3), 137-156.
2) Introduction/Material and methods: A definition of usability and the identification of its relevant aspects are necessary, also in relation to the available literature. See for example the standard ISO 9241-210:2019 “Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems”.
3) Introduction/Material and methods: As mentioned above, the review of the available literature is not sufficient. The author states that only few studies have faced the usability aspects in a complete setting, citing 6 articles. Dozens of articles can be found in the literature, so the author should explain the meaning of “complete setting” in order to justify the sentence. Furthermore, the literature should be characterised and analysed to show its limitations and thus demonstrate whether and how the proposed study overcomes them.
4) Material and methods: The almost complete lack of bibliographical references could suggest that the entire section is an original contribution by the author. Has no one previously identified the objectives of usability testing, proposed protocols that require defining target users, performing a task analysis, preparing and using analysis tools, performing experiments, analysing and interpreting results either in general or specifically for wearables? Similar questions can be asked for all the paragraphs in the section.
5) Results: The author proposes some (completely new?) questionnaires as tools to investigate aspects of usability. Many issues should be argued and justified; for example: Why does he propose questionnaires? (see ISO/TR 16982:2002 “Ergonomics of human-system interaction - Usability methods supporting human-centred design”). Why did he choose those specific items to investigate? Were groups of potential users involved in the selection and definition of the items? And what is the definition of those items? How are the results of the questionnaires analysed and interpreted?
6) Results: It would also be interesting to show an application and validation of the protocol and its tools.
7) Discussion: This section contains some conclusions but lacks an actual discussion in which to describe whether, how and why the study represents an advancement of the available literature. Furthermore, the limitations of the study and possible further research should be stated and described.
8) Language and typing: Currently, both British English and American English are used, but it is preferable to opt for one. Some sentences are incomplete or ambiguous due to punctuation (e.g., lines 23-24; lines 112-114; lines 247-248). Furthermore, several typing errors are present (e.g., line 207; line 259; line 442). Finally, more linguistic and/or aesthetic (e.g., uppercase/lowercase) uniformity would be preferable (check for example the bulleted and numbered lists).
9) References: [1] This self-citation seems forced.
Author Response
Dear reviewer
I really appreciated you constructive comments and suggestions.
Thank you very much for the extensive review.
I tried to follow point-by-point your recommendations.
About point 3 I'd like to comment that the original contribution is in the integrated approach here proposed, in the identification of a common set of measurement tools and questionnaires to build a reference framework for usability assessment of wearable systems that is usually considered as a whole and done through with few questions. Instead I tried to divide the assessment in the three phases of the usability (wearing, use, taking off). The original contribution also consists in the step-by-step guideline for the usability protocol preparation and execution up to the statistical methods for data analysis. In this way we have a comprehensive methodology.
About point 6: the presentation of a case study is set as a limitation fo the study. I have applied this method in the research but the integration in the text of the application would have extend the text too much (+10/12 pages). So I prefer to prepare a further paper describing the application and the outcomes.
Thank you again for your contribution to paper improvement.
Reviewer 3 Report
Comments for author File: Comments.pdf
Author Response
Dear reviewer,
I thank you very much for the numerous comments and suggestions that allowed me for a significant improvement of the paper.
I have followed all the points you have raised in the revised version.
Round 2
Reviewer 1 Report
The comments in the previous revision are mostly addressed. There are some comments to address as follows:
1- Proofreading is still required. Some parts are unclear like lines 232 to 239. What's more, there are several grammatical errors.
2- Please add numbers for equations in lines 379 and 384 and explains their parameters in the text.
3- Please explain the subsections of 3.2.
4- You perhaps maintain the present gap(s) in the literature in the introduction and move the studies in the introduction to a section named Related works. It helps concise the introduction.
Author Response
Dear reviewer,
first of all thank you for your appreciation of the revised paper.
To complete the suggested corrections:
1- I revised the text to improve clarity of some concepts and the grammatical/orthographical errors.
2- I added numbers for equations and explained their parameters in the text.
3- I added the explanation of the subsections of 3.2.
4- I accepted your suggestion to concise the introduction moving the literature analysis into the section named Related works.
Best regards
Giuseppe Andreoni
Reviewer 2 Report
I have appreciated the reviews made to satisfy the reviewers’ comments. However I suggest improving two previously reported issues:
5) Results: The author proposes some questionnaires as tools to investigate aspects of usability. Many issues should be argued and justified; for example: Why does he propose questionnaires? Why did he choose those specific items to investigate? And what is the definition of those items?
8) Language and typing: Currently, both British English and American English are used, but it is preferable to opt for one (e.g., “generalize” or “categorize” vs. “analyse” or “behaviour”). Some typing errors are present (e.g., line 70 method -> methods; line 478 situation -> situations).
Author Response
Dear reviewer
thank you very much for the constructive criticism and the suggestions,
To complete the revision I better addressed your comments:
About point no. 5) Results, I added the explanation of the definition of the questionnaires and of the other metrics.
Finally I revised the paper to correct grammatical and orthographical errors, and improving the clarity in some parts.
Best regards
Giuseppe Andreoni
Reviewer 3 Report
Minor English revision would be recommended. Ex. line 26: instead of "identified and prepares" for "identified and prepared".
Author Response
Dear reviewer
thank you for the comments and suggestions.
In this last version I tried to improve the clarity of some parts and I revised the paper to correct grammar and orthographical errors.
Best regards
Giuseppe Andreoni
