Review Reports

Observation

How it was attended

Title

- “Through case studies?” Maybe “two”, instead?

The title was modified to specify that two case studies are presented:

“Implementing gamification for blind and autistic people with tangible interfaces, extended reality, and universal design for learning: Through two case studies” (Line 2 – 4).

Introduction

- What’s the research goal? How the results could be useful and used? The introduction section does not make this clear.

Following text has been added to specify the target and how the results can be used:

“With this, we intend to demonstrate through our case studies the convergence between gamification mechanics in applications for people with disabilities (blind and autistic people) using technologies such as TUI, VR and UDL approach. Both case studies can help future developers in building their applications for disabilities mentioned above and preliminary results can provide ideas for improvements or considerations in building their applications” (Line 64 – 69).

Theoretical background

- Heading numbering is wrong (there are more “2.2” headings).

Number headings have been corrected.

- 87-93. Extended reality. A short description of the differences among VR, AR and MR would be appreciated.

Added an explanation of VR, AR and MR to make their differences clearer (Line 100 – 104).

- 99. Why is it so important to point out the scope “for children” at this point? Please explain (otherwise, delete it).

It has been decided to omit the term "for children" to refer only to the difference of the TUI with elements that are only physical (109 – 111).

- 106. Strange reference format. Elsewhere, couples of references appear as [XX]-[XX]. Please make it uniform. Moreover, revise punctuations.

References have been unified, in the case of references that appear as follows [XX]-[XX] are due to the citation of three or more sources

- 114. What’s SAD? Never explained. What’s CAST? Never explained.

SAD referred to UDL, this error has been modified, on the other hand, CAST definition has been added (Center for Applied Special Technology) (Line 126)

- 127-135. Why the order is different against that shown in Figure 1.

It has been decided to omit figure one on the recommendation of the editor as there may be CopyRight problems. However, the mechanisms of the UDL have been ordered according to the order of the picture (Line 136 – 144).

Methods

- 137. “The application…” Which one? What are we talking about?

The wording has been improved by referring to both case studies. “To develop both case studies, we used the MEEXU² methodology (MEthodology, EXtended reality, Universal design for Learning, User centered design), a software methodology to build Extended Reality applications and Tangible User Interfaces that contemplates the use of User Centered Design and Universal Design for Learning (See Figure 1).” (Line 146 - 147)

- 156 Figure 3. The scheme is wrong. The graphic formalism used pushes to think that once answered to the second question, there is a choice between Dynamics and Mechanics. But this is not true.

The graph has been modified with the following clarification: “Which gamification techniques are the most suitable for the user according to their characteristics?” by this we mean the choice of mechanics and dynamics according to the characteristics of the user (Figure 2).

- 196. The caption of Figure 4 must report that this scheme refers just to the two cases studies described in the paper.

The wording of Figure 3 has been improved by indicating that it refers to the two case studies described in the paper (Line 205).

Study case

- Heading numbering is wrong (there are more “4.1.2” headings, etc.).

Numbers within the cases have been corrected.

- 198. “Study case” should be “study cases”

Section 4 to Study cases has been reworded as follows (Line 207)

- 200. “Each of them” must be “Each participant” (otherwise “them” seems to refer to the groups).

The wording has been modified to indicate that we are talking about each participant as suggested (209)

- 206 and 212. Talking about “video games” (212) dealing with “blind people” (206) seems quite exotic, doesn’t it?

It has been decided to change the term video games for application “the first case study proposes an application to teach currencies called “Learning with pesos” whose objective is to teach the Mexican currency” (240 – 242)

- 212. Learning with peso must become “Learning with peso”. Inverted commas are required.

Commas have been modified when we make reference to the application “Aprendiendo en Pesos”

- 214-219. Text structure must be revised.

Text structure has been revised and improved

- 228. “are displayed”. Again, we are dealing with blind people; therefore, I would avoid some terms…

Text has been modified to read as follows: “shows the rewards interface where the achievements and their description are displayed, in case of blind people these achievements are described through audio”. Although the main user is blind people, with the use of the UDL we intend to reach as many people as possible, for this reason we use objects, images, text and audio. Hopefully this new wording will help to improve how these elements are accessed (Line 262 – 264)

- 247. Table 1, as it is, is useless. The two columns contain more or less the same text.

To further specify how these challenges are to be met, it has been added that the corresponding tangible objects must be used. In this way the use of such items is encouraged

- 264. Measuring the success/importance/consistency/soundness… or, in a word, the value of an experiment just counting on a simple usability evaluation is belittling and scarcely meaningful.

We have decided to specify that the results are preliminary and added the following text “This shows that the usability is acceptable for the users who participated in this preliminary evaluation, so they were satisfied and comfortable with its use”

- 271. The topic is more articulated than described here. And, to be precise, what Nielsen says is that the “magic number” of testers to get the most of the usability problems is three to five. Not just three…

It has been decided to follow your suggestion and we rewrite the text as follows: “Three blind people between 30- and 45-years old belonging to the National System for the Integral Development of Families, Aguascalientes (DIF) were recruited, this num-ber is within Nielsen's recommended range for getting the most out of usability issues, which recommends three to five participants, since a higher number does not provide much additional information” (Line 336 309).

- 284-291. At least, the difference between low and high functioning ASD should be cited.

Difference between low and high functioning ASD added as recommended: “Commonly, it is classified as high-functioning Autism to those with autism spectrum disorder without intellectual disability [56] and as low-functioning Autistic those who are unable to follow instructions, who do not let anyone touch them; they may be aggressive or show unreasonable behavioral reactions that are difficult to explain [57], [58]” (

- 307. I am not sure that imposing time limits to ASD people is a good idea…

This limit has been set to restart the activity from the starting point, the time is not displayed at any time during its interaction with the environment.

The following has been modified in the text: “To complete the activity children with ASD have a time limit of three minutes, if they do not reach the indicated destination in that time the activity is restarted. This time is not shown at the time of using the game, it is shown at the end of the activity to know how long it took to complete it”

- 351-352. “A person” (351)… “their actions” (352)… it is not clear how many testers were involved.

Currently the evaluation was performed with only one person, for this reason it has been decided to orient the evaluation as a preliminary evaluation, informing that the use of more individuals is necessary to confirm what is stated here. “It is observed that the use of VR attracted the attention of the child with ASD, so it can be a great tool to subject them to different virtual scenarios and improve their interactions with the real world in decision making. Finally, it is important to mention that since this is a preliminary evaluation, tests with more individuals are necessary to confirm what is stated here”

- 357-368. There is no mention of the metrics used to evaluate the performance, value, etc., of the second case study.

The objective of this evaluation was specified: “The objective of this evaluation was to observe its behavior with the virtual reality technology, its interaction with the environment and how well it was able to complete the misión”

- 369-389. This section describes tools and methods used in the two case studies. Why is it here? Usually, tools and methods are described before the activities adopting them…

The order of this section has been modified, it now appears before showing both case studies to first show the tools with which they were developed.

Conclusions

- 417 “it was demonstrated…” To me, it is inappropriate talking about demonstration of something without real, quantitative comparisons to different/alternative scenarios.

The wording has been modified to indicate that these are preliminary results and that a larger sample is needed:

“preliminary results showed that the use of gamification favored the motivation of people evaluated (blind and people with autism) to continue using the applications presented here. however, it is necessary to consider a larger sample in the future to compare whether these results are maintained” (448 – 451)

- 422 “Users with ASD found that…” How many were they? Again, please make it clear throughout the paper.

The wording has been modified as follows to indicate that it was only one user with ASD in a pretest:

“it was found that motivation of user with ASD who participated in preliminary evaluation increased when obtained many stars while completing the activity on their second attempt” (Line 455 – 457)

- 427-433. Usability and satisfaction are different. For sure, there is a link between them, but you cannot write that “satisfaction was achieved by those who used it” just because usability evaluation had good outcomes. Usability is only one of the components that contribute to the users’ satisfaction.

Text has been modified to clarify that we are talking about the usability of the application: “Usability evaluations of the "Learning in Pesos" application were satisfactory with an average of 91.7, an acceptable score according to System Usability Scale [53]. This shows that usability of the application is acceptable, so users were satisfied and comfortable with its use”  (Line 460 – 463).

Throughout. The paper needs a deep English check. Sometimes sentence construction is enigmatic; other times it seems that the authors miss the differences among the punctuation marks (“,”, “;”, “:”, etc.). Sometimes there is a comma where it should not; other times the comma is missing where it should be…

A revision of the English language has been carried out to improve writing errors and punctuation between citations.

Observation

How it was attended

Since the testing and evaluation are based on an exceptionally small sample size, the authors should describe the research results as the preliminary findings of a pilot study (the first app was tested on only three blind persons and the second on only one person with ASD).

The information on the evaluations has been modified to refer to them as preliminary results as they were attended, and we also mention the need to test with a larger number of participants to corroborate the results obtained

They must not generalise the findings with statements such as 

·         “it was demonstrated how the use of gamification benefits the motivation of people with disabilities (blind people and people with autism) to continue using the applications presented here ” (lines 417, 418).

Text of lines 415-419 has been modified to specify that these results are based on preliminary evaluations: “preliminary results showed that the use of gamification favored the motivation of the people evaluated (blind and people with autism) to continue using the applications presented here. however, it is necessary to consider a larger sample in the future to compare whether these results are maintained

·         “Blind people agreed that the use….”       (line 420)

Line 452 was modified as follows: Blind people who were part of the preliminary evaluations shown in this paper agreed that the use of audio to specify

·          “Likewise users with ASD found …..”        (line 422

Line 455 was modified as follows: it was found that motivation of user with ASD who participated in preliminary evaluation increased when obtained many stars while completing the activity on their second attempt

The study has not demonstrated any of them, and a significantly larger sample is required to assert them with some statistical assurance. They should incorporate a section discussing the study's limitations.

Results presented here are preliminary, which has been specified throughout the document, we emphasize that as future work we intend to conduct evaluations with a larger number of users in order to confirm the preliminary results obtained (Line 473-475).

Similarly, although the authors did address the importance of studying and analysing the user characteristics before the development process of any digital solution, they must detail how this was taken into account when developing the two aforementioned applications.

These tools have been chosen thanks to multiple projects that have shown favorable results with the target user, for this reason, it has been decided to follow their recommendation and add the following sentences:

·          It was decided to use tangible interfaces with the objective of taking advantage of the main means of interaction of blind people, their hands, in addition, through TUI, innovative educational and cognitive interventions can be maintained to listen and stimulate the narratives of visually impaired people [22], Additionally, multiple projects to demonstrate favorable results to improve the learning of blind people in areas such as concepts of shadows, programming, music, Braille, among others [50]–[53]. (Linea 243 - 248).

·          It was decided to use VR because it can bring many advantages to the treatment of ASD symptomatology [8], In addition, Several studies have shown significant improvements in various cognitive indices, such as task learning, attention, executive functioning, daily living skills, so it can be successfully used as an educational tool for children with ASD [64]–[66] (Linea 336-340).

Observation

How it was attended

Title

- Please choose between “learning through two case studies.” and “learning: two case studies.”. “learning: Through two case studies” does not sound.

It has been decided to change the title to: “Implementing gamification for blind and autistic people with tangible interfaces, extended reality, and universal design for learning: Two case studies”

Methods

- 165 Figure 2. My old comment is still valid (The scheme is wrong. The graphic formalism used pushes to think that once answered to the second question, there is a choice between Dynamics and Mechanics. But this is not true.

The question in said graph was modified as follows: “Which gamification dynamics and mechanics are the most suitable for the user according to their characteristics?” In this way we emphasize the selection of dynamics and mechanics considering the characteristics of the user

- 176-177. The order of items in the text is reversed against what appears in Figure 2.  In Figure 2, we have Dynamics and Mechanics. In the text Mechanics appears before Dynamics…

Order of dynamics and mechanics was ordered in the indicated text (Line 172 – 186).

Study cases

- 300. “Usability evaluation” here… “Evaluation” at line 398. There is no need to have them different. Call both “Evaluation” or, better, delete subheadings 4.2.1 and 4.3.1. They are useless.

It has been decided to call points 4.2.1 and 4.3.1 only as evaluation.

- 305. You refer to three people. It is meaningless saying “between 30 and 45”. They are just three; so, write their three ages.

Text was modified as follows: “Three blind people of 31, 44 and 45 years old belonging to the Sistema Nacional para el Desarrollo Integral de la Familia (National System for the Integral Development of the Family), Aguascalientes (DIF) were recruited” (Line 306- 308).

- 401-404. The new text does not answer to my comment (There is no mention of the metrics used to evaluate the performance, value, etc., of the second case study).

Evaluation with the child with autism was qualitative in nature as it was based on observation, opinions and experiences (Linea 404 -407).

- 403. Again, we have a single ASD tester. Then, why there is “In addition, their actions”?

text was modified as follows to clarify that section “Parameters for this evaluation were qualitative in nature, as they were based on observation, opinions, and experiences of the child with autism, as well as his actions within the virtual environment were monitored by our development team through a computer.” (Linea 404 -407).

Observation

How it was attended

Thank you for taking my comments into account. However, I would have preferred a section on the study's limitations.

Due to the nature of the evaluation, which was of a preliminary type, it has been decided not to add a separate section on the limitations of the study; however, within evaluations we mention that due to this, future evaluations with a larger number of individuals are necessary to confirm the results shown here. This can be seen in the conclusions section.

I strongly recommend the following changes to the abstract:

"The present study investigates the use of gamification to foster commitment and engagement among users with disabilities. Two case studies demonstrating the application of gamification are provided. The first is the development of an app to teach blind persons Mexican currency, and the second one is the creation of an app to aid individuals with an autism spectrum disorder in navigating their environment. The study reveals that Universal Design for Learning principles can be used indirectly to adjust apps for users with impairments and highlights the utility of gamification in enticing target users to utilise the software consistently. This study provides preliminary evaluations for both case studies, which were undertaken with relatively small samples. The first case study revealed that three blind individuals who took part in the review scored an average of 91.7 on the System Usability Scale. At the same time, the second case study involving the observation of a single individual with ASD also revealed that utilising the designed app improved performance. Despite the limited sample size, the findings suggest that gamification may effectively encourage and generate commitment among the users with disabilities."

It has been decided to follow the reviewer's suggestion to improve the abstract of the article (Line 25 – 37).