Evaluating the Potential of Immersive Virtual Reality-Based Serious Games Interventions for Autism: A Pocket Guide Evaluation Framework

Round 1

Reviewer 1 Report

The paper suggests an Assessment Framework that comprises crucial methodological standards for assessing the effectiveness and efficiency of Serious game therapies for people with autism that are based on Immersive Virtual Reality. The pocket guide, an integral part of the proposal, is conceived as a practical and accessible tool that facilitates the application of the methodological criteria established in the Assessment Framework.

Suggestions to improve the paper proposal:

If the authors' proposal focuses on proposing a guide to establish an evaluation framework for projects of this type, one would expect an argument about the methodologies used in immersive virtual reality based on serious games and why existing ones do not adapt for individuals with autism.

The proposed guide is very concise and offers a general presentation of six criteria. The information is wordy. It would be appropriate to present, for each criterion, proposals on how to evaluate them quantitatively. 

Present an example of application and development with details of the proposal.

No significant grammatical errors were detected.

Author Response

Response to Reviewer

We express our appreciation to the reviewer for his/her positive feedback on our work and for providing valuable, competent, and insightful suggestions that have greatly enhanced the quality of our manuscript. In the subsequent sections, the reviewer comments are presented in black, and our responses are consistently introduced with an "R" in red.

Reviewer #1

The paper suggests an Assessment Framework that comprises crucial methodological standards for assessing the effectiveness and efficiency of Serious game therapies for people with autism that are based on Immersive Virtual Reality. The pocket guide, an integral part of the proposal, is conceived as a practical and accessible tool that facilitates the application of the methodological criteria established in the Assessment Framework.

Suggestions to improve the paper proposal:

1. If the authors' proposal focuses on proposing a guide to establish an evaluation framework for projects of this type, one would expect an argument about the methodologies used in immersive virtual reality based on serious games and why existing ones do not adapt for individuals with autism.

R: Thank you. We understand the reviewer's perspective; however, this argument was not explicitly addressed in the manuscript for two primary reasons:

• Firstly, the methodologies proposed in the literature to evaluate immersive virtual reality interventions based on serious games are highly customized and divergent from the focus of our study (e.g., Feng et al., 2020; Adhikari et al., 2021).
• Secondly, to our knowledge, there are no existing studies proposing methodologies or frameworks to evaluate the effects of immersive virtual reality interventions based on serious games on the abilities of typically developing individuals or clinical populations other than autism.

Anyway, we added a new section (Background) describing the methodology behind the systematic review of Peretti et al., 2023 from which the framework proposed in this study was partially derived.

The systematic review was informed by the Kitchenham review process (2007) and answered the following research question: How are interventions implemented with Serious Games based on Immersive Virtual Reality for individuals with autism evaluated?

The systematic review of Peretti et al. (2023) stems precisely from two reasons:

• There are no reviews, meta-analyses, systematic reviews, or scoping reviews that answer the research question.
• an in-depth analysis of meta-analysis, reviews, systematic reviews, and scoping reviews (reported in the Background section of Peretti et al., 2023) showed the absence of a framework to evaluate immersive virtual reality-based serious games for autism.

Hopefully, the paper is now clearer.

Referred References:

-Adhikari, R., Kydonaki, C., Lawrie, J., O'Reilly, M., Ballantyne, B., Whitehorn, J., & Paterson, R. (2021). A mixed-methods feasibility study to assess the acceptability and applicability of immersive virtual reality sepsis game as an adjunct to nursing education. Nurse education today, 103, 104944.

-Feng, Z., González, V. A., Mutch, C., Amor, R., Rahouti, A., Baghouz, A., ... & Cabrera-Guerrero, G. (2020). Towards a customizable immersive virtual reality serious game for earthquake emergency training. Advanced Engineering Informatics, 46, 101134.

-Peretti, S.; Caruso, F.; Pino, M.C.; Di Mascio, T. A Systematic Review to Know How Interventions 695 Realized with Immersive Virtual Reality-Based Serious Games for Individuals with Autism 696 are Evaluated. In Proceedings of the 5th Biannual Conference of the Italian SIGCHI Chapter: 697 Crossing HCI and AI, CHItaly 2023, 2023, pp. 1–14. https://doi.org/10.1145/3605390.3605396.

2. The proposed guide is very concise and offers a general presentation of six criteria. The information is wordy. It would be appropriate to present, for each criterion, proposals on how to evaluate them quantitatively. 

R: Thank you. It is very difficult to provide a quantitative or qualitative evaluation standard for each of the six methodological criteria and sub-criteria as these can vary depending on different variables ranging from the research question to the sample size (for instance, as suggested by Journal Article Reporting Standards-JARS- APA, 2008). For this reason, we have provided general, evidence-based guidelines for each methodological criterion and sub-criterion. 

However, following this suggestion and the sequent suggestion provided by the Reviewer, we have added a new subsection entitled “Utilizing the Pocket Guide Evaluation Framework: An Example of its Application”, where we showed how to address each of the six methodological criteria. We constructed a specific example with a specific research question and related research hypothesis so that we could show quantitatively and qualitatively how to address each criterion and sub-criterion.

Referred References:

APA Publications (2008). Reporting standards for research in psychology: Why do we need them? what might they be? The American Psychologist 63, 839).

3. Present an example of application and development with details of the proposal.

R: Thank You. In the wake of the Reviewer's valuable suggestion, we have added an example (the subsection: Utilizing the Pocket Guide Evaluation Framework: An Example of its Application) showing how to address each of the six methodological criteria and sub-criteria. We constructed a specific example with a specific research question and related research hypothesis so that we could show quantitatively or qualitatively how to address each criterion and sub-criterion. Thanks to this suggestion, we believe the proposed evaluation framework is clearer and more powerful.

Author Response File: Author Response.pdf

Reviewer 2 Report

I appreciate the authors for identifying the methodological issues for immersive games for autism. A few notes are below:

1. Abstract must be re-written to highlight the problem (i.e., minimal research on efficacy and effectiveness) and how so.

2. A proper articulation of RQs and a proper literature review are needed as standalone sections. There are various evaluation frameworks on this subject.

3. Methodology: It states that a systematic literature review was performed. What was the framework? If it is a literature review, PRISMA should be articulated. Section 2 needs to be expanded to cover the details of what studies were covered and excluded, along with the methodological issues. The transition from Sections 1 and 2 is awkward. 

4. To indicate the systematic review's outcome, a new section (perhaps section 3) should identify methodological issues related to the evaluation criteria. Table 1 of methodological issues and sub-issues is useful in this new section.

5. Table 2 is missing. 

6. Since Table 3 concerns the evaluation framework, it should be related to the methodological issues identified in the previous section. Individual descriptions in this table need to be articulated in the actual text. Tables and figures are meant to summarize information. Table 3 should be part of the Results section.

7. I appreciate the pocket guide to the evaluation framework. However, this should be a new section for Research and Practitioner Implications, where authors thoroughly discuss the study results and how to use the framework. 

8. A proper conclusion must summarize the review, issues, future directions, and hopes for the evaluation tool (intended uses). 

9. The references are quite dated. It looks like the studies analyzed stopped in early 2022. If that is the case, this needs to be articulated fully in the Methodology section.

10. Consider having an Appendix section to articulate all acronyms. 

Overall, the manuscript needs to be expanded to detail how the authors conducted the systematic review. The authors arrived directly at this evaluation framework. 

Minor editing is required.

Author Response

Response to Reviewer

We express our appreciation to the reviewer for his/her positive feedback on our work and for providing valuable, competent, and insightful suggestions that have greatly enhanced the quality of our manuscript. In the subsequent sections, the reviewer comments are presented in black, and our responses are consistently introduced with an "R" in red.

Reviewer #2

I appreciate the authors for identifying the methodological issues for immersive games for autism. A few notes are below:

R: Thank you.

1. Abstract must be re-written to highlight the problem (i.e., minimal research on efficacy and effectiveness) and how so.

R: Thank you. We have rewritten the abstract following the reviewer's suggestions. Hopefully, the abstract is now clearer.

2. 2. a A proper articulation of RQs and a proper literature review are needed as standalone sections. b There are various evaluation frameworks on this subject.

R: Thank you.

 a We highly value the Reviewer's keen observation; thus, we added a new section (Background) describing the methodology behind the systematic review of Peretti et al., 2023 from which the framework proposed in this study was partially derived.  The systematic review was informed by the Kitchenham review process (2007) and answered the following research question: How are interventions implemented with Serious Games based on Immersive Virtual Reality for individuals with autism evaluated?

b This systematic review stemmed precisely from two reasons:

• There are no reviews, meta-analyses, systematic reviews, or scoping reviews that answer this research question.
• an in-depth analysis of meta-analysis, reviews, systematic reviews, and scoping reviews (reported in the Background section of Peretti et al., 2023) showed the absence of a framework to evaluate immersive virtual reality-based serious games for autism.

The main outcome of the systematic review of Peretti et al. (2023) is that there are several lacks and methodological issues at the base of the evaluation process to evaluate the efficacy and effectiveness of immersive virtual reality-based serious games for autism.

Combining this evidence with that emerging from recent studies on this topic (e.g., Tan et al., 2022; Minissi et al., 2023; Alvarado et al., 2023; Parkinson et al., 2023; Lorenzo et al., 2023; Carneiro et al., 2024) led toward the collaborative development of the evaluation framework proposed in this study.

Hopefully, the paper is now clearer.

Referred References:

-Alvarado, Y., Guerrero, R., & Serón, F. (2023). Inclusive Learning through Immersive Virtual Reality and Semantic Embodied Conversational Agent: A case study in children with autism. Journal of Computer Science and Technology, 23(2), e09-e09.

-Carneiro, T., Carvalho, A., Frota, S., & Filipe, M. G. (2024, February). Serious Games for Developing Social Skills in Children and Adolescents with Autism Spectrum Disorder: A Systematic Review. In Healthcare (Vol. 12, No. 5, p. 508). MDPI.

- Kitchenham, B. Guidelines for performing systematic literature reviews in software engineering 816 - version 2.3. Technical Report EBSE-2007-01, Keele, UK, Keele University, 2007.

-Minissi, M. E., Landini, G. A. R., Maddalon, L., Torres, S. C., Giglioli, I. A. C., Sirera, M., ... & Alcañiz, M. (2023, August). Virtual reality-based serious games to improve motor learning in children with Autism Spectrum Disorder: An Exploratory Study. In 2023 IEEE 11th International Conference on Serious Games and Applications for Health (SeGAH) (pp. 1-6). IEEE.

- Lorenzo, G. G., Newbutt, N. N., & Lorenzo-Lledo, A. A. (2023). Design concepts for solution and solid-state emitters-a modern viewpoint on classical and non-classical approaches. Education and information technologies.

- Mesa-Gresa, P., Gil-Gómez, H., Lozano-Quilis, J. A., & Gil-Gómez, J. A. (2018). Effectiveness of virtual reality for children and adolescents with autism spectrum disorder: an evidence-based systematic review. Sensors, 18(8), 2486.

-Parkinson, S., Schumann, S., Taylor, A., Fenton, C., Kearney, G., Garside, M., & Johnston, D. (2023). SoundFields: A Virtual Reality Home-Based Intervention for Auditory Hypersensitivity Experienced by Autistic Children. Applied Sciences, 13(11), 6783.

-Peretti, S.; Caruso, F.; Pino, M.C.; Di Mascio, T. A Systematic Review to Know How Interventions 695 Realized with Immersive Virtual Reality-Based Serious Games for Individuals with Autism 696 are Evaluated. In Proceedings of the 5th Biannual Conference of the Italian SIGCHI Chapter: 697 Crossing HCI and AI, CHItaly 2023, 2023, pp. 1–14. https://doi.org/10.1145/3605390.3605396.

-Tan, Q. P., Huang, L., Xu, D., Cen, Y., & Cao, Q. (2022). Serious game for VR road crossing in special needs education. Electronics, 11(16), 2568.

3.Methodology: It states that a systematic literature review was performed. What was the framework? If it is a literature review, PRISMA should be articulated. Section 2 needs to be expanded to cover the details of what studies were covered and excluded, along with the methodological issues. The transition from Sections 1 and 2 is awkward. 

Thank you, the Reviewer is right. We have followed his/her suggestion by adding a new section (Background) containing a description of the systematic review of Peretti et al., 2023. This new section specifically outlines the entire review and paper selection process. Through this process, one can deduce the limitations of the methodologies to evaluate immersive virtual reality based on serious game interventions for autism and the absence of clear methodological guidelines or frameworks.

Hopefully, with the addition of the Background section and all related revisions in the manuscript, the transition between the oldest Section 1 and Section 2 will now be clearer and more consistent.

Referred References:

Kitchenham, B. Guidelines for performing systematic literature reviews in software engineering 816 - version 2.3. Technical Report EBSE-2007-01, Keele, UK, Keele University, 2007.

Peretti, S.; Caruso, F.; Pino, M.C.; Di Mascio, T. A Systematic Review to Know How Interventions 695 Realized with Immersive Virtual Reality-Based Serious Games for Individuals with Autism 696 are Evaluated. In Proceedings of the 5th Biannual Conference of the Italian SIGCHI Chapter: 697 Crossing HCI and AI, CHItaly 2023, 2023, pp. 1–14. https://doi.org/10.1145/3605390.3605396.

1. To indicate the systematic review's outcome, a new section (perhaps section 3) should identify methodological issues related to the evaluation criteria. Table 1 of methodological issues and sub-issues is useful in this new section.

R: Thank you. We followed the Reviewer’s suggestions and restructured the manuscript. In the revised manuscript, we have added a subsection of the Background section containing the results of the systematic review (i.e., methodological issues) and the oldest Table 1, accompanied by another more analytic table on the results (i.e., Table 4).

2. Table 2 is missing. 

R: Thank you. We have corrected the mistake.

3. Since Table 3 concerns the evaluation framework, it should be related to the methodological issues identified in the previous section. Individual descriptions in this table need to be articulated in the actual text. Tables and figures are meant to summarize information. Table 3 should be part of the Results section.

R: Thank you. We have followed the valuable suggestions provided by the reviewer by removing the old Table 3, which contained the methodological criteria. Subsequently, we have described these methodological criteria and sub-criteria, which represent the results of the present study, in a new section entitled “Results and Research Implications”.

7. I appreciate the pocket guide to the evaluation framework. However, this should be a new section for Research and Practitioner Implications, where authors thoroughly discuss the study results and how to use the framework. 

R: Thank you. We have entitled the subsection in the revised manuscript: Research and Practitioner Implications: A Pocket Guide Evaluation Framework. Furthermore, to facilitate understanding of the practical advantages of the pocket guide, we have presented an example (a research study) where we have shown how to hypothetically address each of the six methodological criteria. We constructed a specific example with a specific research question and related research hypothesis so that we could show quantitatively and qualitatively how to address each criterion and sub-criterion.

8. A proper conclusion must summarize the review, issues, future directions, and hopes for the evaluation tool (intended uses). 

R: Thank you. In the revised manuscript, we have modified the conclusion, considering the reviewer’s suggestions.

9. The references are quite dated. It looks like the studies analyzed stopped in early 2022. If that is the case, this needs to be articulated fully in the Methodology section.

R: Thank you. We followed the reviewer's valuable suggestion and updated the manuscript with more recent references.

Regarding the articles reviewed in the systematic review of Peretti et al. (2023), we have added a new section (Background) where the entire process of reviewing and selecting papers is explained. Specifically, Peretti et al. (2023) reviewed research articles published between January 2009 and July 2021; including all the research articles published after 2009 guarantee that results deal with the current generation of Immersive Virtual Reality technology, in agreement with other reviews in the same field (Mesa Gresa et al., 2018).

It is important to note, however, that the evaluation framework present in this study is only partially derived from the methodological issues identified in Peretti et al. Indeed, the awareness of the methodological issues that emerged from systematic review prompted us to make a greater effort by combining such evidence with what is actually known in the literature on the subject (e.g., Tan et al., 2022; Minissi et al., 2023; Alvarado et alò., 2023; Parkinson et al., 2023; Lorenzo et al., 2023; Carneiro et al., 2024) to transform these methodological issues into criteria that can be addressed systematically, thus enabling controlled evaluation.

Thus, the result is the identification of six methodological criteria and their sub-criteria emerging as an overall solution to the issues that compromise the evaluation process for evaluating the efficacy and effectiveness of Immersive Virtual Reality -based Serious Games interventions for ASD.

Referred References:

-Alvarado, Y., Guerrero, R., & Serón, F. (2023). Inclusive Learning through Immersive Virtual Reality and Semantic Embodied Conversational Agent: A case study in children with autism. Journal of Computer Science and Technology, 23(2), e09-e09.

-Carneiro, T., Carvalho, A., Frota, S., & Filipe, M. G. (2024, February). Serious Games for Developing Social Skills in Children and Adolescents with Autism Spectrum Disorder: A Systematic Review. In Healthcare (Vol. 12, No. 5, p. 508). MDPI.

-Minissi, M. E., Landini, G. A. R., Maddalon, L., Torres, S. C., Giglioli, I. A. C., Sirera, M., ... & Alcañiz, M. (2023, August). Virtual reality-based serious games to improve motor learning in children with Autism Spectrum Disorder: An Exploratory Study. In 2023 IEEE 11th International Conference on Serious Games and Applications for Health (SeGAH) (pp. 1-6). IEEE.

- Lorenzo, G. G., Newbutt, N. N., & Lorenzo-Lledo, A. A. (2023). Design concepts for solution and solid-state emitters-a modern viewpoint on classical and non-classical approaches. Education and information technologies.

- Mesa-Gresa, P., Gil-Gómez, H., Lozano-Quilis, J. A., & Gil-Gómez, J. A. (2018). Effectiveness of virtual reality for children and adolescents with autism spectrum disorder: an evidence-based systematic review. Sensors, 18(8), 2486.

-Parkinson, S., Schumann, S., Taylor, A., Fenton, C., Kearney, G., Garside, M., & Johnston, D. (2023). SoundFields: A Virtual Reality Home-Based Intervention for Auditory Hypersensitivity Experienced by Autistic Children. Applied Sciences, 13(11), 6783.

-Peretti, S.; Caruso, F.; Pino, M.C.; Di Mascio, T. A Systematic Review to Know How Interventions 695 Realized with Immersive Virtual Reality-Based Serious Games for Individuals with Autism 696 are Evaluated. In Proceedings of the 5th Biannual Conference of the Italian SIGCHI Chapter: 697 Crossing HCI and AI, CHItaly 2023, 2023, pp. 1–14. https://doi.org/10.1145/3605390.3605396.

-Tan, Q. P., Huang, L., Xu, D., Cen, Y., & Cao, Q. (2022). Serious game for VR road crossing in special needs education. Electronics, 11(16), 2568.

10. Consider having an Appendix section to articulate all acronyms. 

R: Thank you. We added in the revised manuscript an Appendix with an explanation of all acronyms. 

Overall, the manuscript needs to be expanded to detail how the authors conducted the systematic review. The authors arrived directly at this evaluation framework. 

R: Thank you. We highly value the Reviewer's keen suggestion, and we added a detailed section on the method underlying the systematic review and the results obtained that led to the creation of the evaluation framework presented in this study. We hope that in the revised manuscript the transition between the various sections is now clearer and more consistent.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

I appreciate the authors for incorporating extensive improvements into the paper. In my opinion, the manuscript can be accepted for publication in this journal.

Author Response

Dear Reviewer,

Thank you very much for your new feedback. 

Reviewer 2 Report

Thank you for submitting the revised manuscript. I appreciate the detailed revision letter addressing the reviewers' comments.

One thing to consider is adding a limitation related to systematic reviews of the literature in the Methodology section related to heterogeneity, publication bias, missing data, and conflicting results. 

Author Response

Dear Reviewer,

Thank you very much for your new feedback.

Regarding the suggestion: One thing to consider is adding a limitation related to systematic reviews of the literature in the Methodology section related to heterogeneity, publication bias, missing data, and conflicting results. 

We followed the Reviewer's suggestion and we have added a "Limitations" section in the manuscript at the end of the research method of Systematic review. We hope that the manuscript will be improved.