Effectiveness of “Escape Room” Educational Technology in Nurses’ Education: A Systematic Review

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Editor,

 

I appreciate the opportunity to review the manuscript titled "Effectiveness of 'escape room' educational technology in nurses’ education: A systematic review". The authors conducted a literature review aiming to "determine the effectiveness of escape room games on the training of nurses and nursing students in the international context".

 

I was intrigued by the premise of the manuscript, given its focus on a new pedagogical approach in nursing education (escape rooms), and the conduction of a systematic review following Joanna Briggs Institute’s recommendations.

 

While I applaud the authors’ efforts and the manuscript produced, there are several main issues that require comprehensive revision of the manuscript before any consideration on my behalf can be made regarding its scientific soundness and merit.

1. On page 2, lines 59-63, the authors reference a narrative review on the topic, published in 2022 by Reinkemeyer et al., claiming that the authors only included studies published in English as a major limitation. While this is true, this can be applied to most systematic reviews given that English is the predominant language for scientific publication. This means that although the studies are published in English, they may have been conducted anywhere in the world (which undermines the authors’ rationale for a new review). Nevertheless, I believe the authors should clearly identify the methodological gaps of the review produced by Reinkemeyer et al., starting from the fact that this previous review is narrative in nature – not a systematic review.

2. On pages 2 and 3, while claiming to have followed the Joanna Briggs Institute’s recommendations for systematic reviews, the authors did not present their effectiveness question using the PICO mnemonic. I searched throughout the entire manuscript for a description of this systematic review’s Outcomes (in other words, the effectiveness of ER on which outcomes exactly?). To my dismay, the authors seem to have opted for an "everything counts" approach to their study outcomes, which can be read from their take on page 2: "The main research results were the improvement of knowledge in different areas, the evaluation of satisfaction with the educational activity, and the improvement of attitudes or competences of the individuals in the population studied". Are you focusing on the effectiveness of ER on nursing students’ communication skills? On their knowledge of infection control? Including every potential outcome is not the basis for a structured and sound systematic review of effectiveness.

3. The authors claim that "the high clinical heterogeneity found did not allow a meta-analysis to be performed". While I am unsure of what the authors mean with “clinical” heterogeneity in a review focused on nursing education, the high heterogeneity is also a testament to my previous concern – there are no clear inclusion and exclusion criteria for each dimension of the PICO mnemonic (P – undergraduate or postgraduate nursing students? Or both?; I – Physical or digital ER will be included? Or both approaches? The game will explore what topic?; C – any comparator?; O – What outcomes? Knowledge on a specific topic? Attitudes? Teamwork? Leadership skills? Communication?).

4. While the authors provide some rationale for the methodological quality assessment threshold used during the critical appraisal of potential studies, the final results for each of the included studies are not reported. This is a limitation that must be revised by the authors to sustain some of the major choices made.

 

I believe the authors must revise these four critical issues before any further in-depth consideration may be given to the manuscript.

Comments on the Quality of English Language

English will require minor syntax and grammar revisions. 

Author Response

Thanks to the reviewers. These contributions have, undoubtedly, helped to improve the manuscript. We have made the changes in red to make it easier for the editor and reviewers. We hope the changes have been satisfactory. We send our best regards on behalf of all the authors.

The replies to reviewers’ comments for each of the points made are described below with red font.

Reviewer 1:

I appreciate the opportunity to review the manuscript titled "Effectiveness of 'escape room' educational technology in nurses’ education: A systematic review". The authors conducted a literature review aiming to "determine the effectiveness of escape room games on the training of nurses and nursing students in the international context".

I was intrigued by the premise of the manuscript, given its focus on a new pedagogical approach in nursing education (escape rooms), and the conduction of a systematic review following Joanna Briggs Institute’s recommendations.

 We welcome the reviewer comments. Thank you.

While I applaud the authors’ efforts and the manuscript produced, there are several main issues that require comprehensive revision of the manuscript before any consideration on my behalf can be made regarding its scientific soundness and merit.

1. On page 2, lines 59-63, the authors reference a narrative review on the topic, published in 2022 by Reinkemeyer et al., claiming that the authors only included studies published in English as a major limitation. While this is true, this can be applied to most systematic reviews given that English is the predominant language for scientific publication. This means that although the studies are published in English, they may have been conducted anywhere in the world (which undermines the authors’ rationale for a new review). Nevertheless, I believe the authors should clearly identify the methodological gaps of the review produced by Reinkemeyer et al., starting from the fact that this previous review is narrative in nature – not a systematic review.

Regarding this comment, we agreed with the reviewer that the scientific production in health sciences is in English; however, there is the possibility of finding studies in other languages relevant to the field of nursing, such as Spanish and Portuguese. We also agree that this limitation should not be labeled as a “major limitation” so we have refined this statement in the manuscript. Regarding the small number of different contexts or countries in which studies have been developed that have addressed the use of escape room as an educational methodology, this matter is pointed out in the discussion (page 3, lines 283-285).

On the other hand, we have included the limitations suggested by the reviewer on the design and narrative analysis of the results of the review published by Reinkemeyer et al. The study by Reinkemeyer et al. conducted a narrative approach to the results on group dynamics, educational outcomes such as teamwork/communication, knowledge gains, and confidence and satisfaction; in addition, they have addressed the theoretical basis of escape rooms and barriers such as the time required to prepare and conduct escape rooms, costs, and participant stress. However, it did not perform statistical data extraction to report on the effectiveness of escape rooms (page 2, lines 63-66).

2. On pages 2 and 3, while claiming to have followed the Joanna Briggs Institute’s recommendations for systematic reviews, the authors did not present their effectiveness question using the PICO mnemonic. I searched throughout the entire manuscript for a description of this systematic review’s Outcomes (in other words, the effectiveness of ER on which outcomes exactly?). To my dismay, the authors seem to have opted for an "everything counts" approach to their study outcomes, which can be read from their take on page 2: "The main research results were the improvement of knowledge in different areas, the evaluation of satisfaction with the educational activity, and the improvement of attitudes or competences of the individuals in the population studied". Are you focusing on the effectiveness of ER on nursing students’ communication skills? On their knowledge of infection control? Including every potential outcome is not the basis for a structured and sound systematic review of effectiveness.

To define the focus of the review, we have implemented the structured question with PICO format in the methodology with the following text: “As this is a review on effectiveness of an intervention, the research question has been shaped following the structure: Population (P), Intervention (I), Comparison (C) and Outcomes (O); so that the P: Nurses and nursing students, I: Escape rooms, C: Other educational methods and O: Improvements in acquired knowledge and satisfaction with the training received”. This aspect is supported by the inclusion of a new reference (13. Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol. 2018,10;18(1):5. https://doi.org/10.1186/s12874-017-0468-4.)

3. The authors claim that "the high clinical heterogeneity found did not allow a meta-analysis to be performed". While I am unsure of what the authors mean with “clinical” heterogeneity in a review focused on nursing education, the high heterogeneity is also a testament to my previous concern – there are no clear inclusion and exclusion criteria for each dimension of the PICO mnemonic (P – undergraduate or postgraduate nursing students? Or both?; I – Physical or digital ER will be included? Or both approaches? The game will explore what topic?; C – any comparator?; O – What outcomes? Knowledge on a specific topic? Attitudes? Teamwork? Leadership skills? Communication?).

We believe that the response to the previous comment responds to the reviewer's doubts regarding the PICO structure and the focus of the research question with respect to the inclusion criteria.

Regarding clinical heterogeneity, the authors wanted to point out that in the literature different escape room games have been used, measuring with different scales; although these studies have focused on the analysis of the results on the knowledge acquired in nurses and nursing students, it is not possible to identify a homogeneous outcome, which has favored the variability of the findings and prevents them from being meta-analyzed jointly. The explanation in this regard is provided in the discussion, when we explain the limitations of the review. The research started from the premise of being able to find greater homogeneity in relation to the available studies, in order to study the effectiveness with the use of the escape room technique in the field of nursing education through intervention designs; but unfortunately, the results indicate the need to continue deepening in this aspect in order to develop research to compare the results with this educational technique in the field of nursing, as can be seen in the conclusions of the review. However, and for a better understanding for readers we will change “clinical heterogeneity” by “context heterogeneity.”

4. While the authors provide some rationale for the methodological quality assessment threshold used during the critical appraisal of potential studies, the final results for each of the included studies are not reported. This is a limitation that must be revised by the authors to sustain some of the major choices made.

We appreciate the reviewer's comments. This information is available for the included studies and we have added it as Supplementary Table 1. Excluded studies and the reasons for their exclusion are included in Supplementary Table 2.

I believe the authors must revise these four critical issues before any further in-depth consideration may be given to the manuscript.

We are pleased that the implementations made are satisfactory for the improvement of the manuscript and can be favorably considered for publication in Nursing Reports.

 

Comments on the Quality of English Language.

English will require minor syntax and grammar revisions. 

The quality of the English has been revised according to the suggestions of the two reviewers. ThanK you.

Reviewer 2 Report

Comments and Suggestions for Authors

Abstract line 17 change Escape room's plural to Escape room games. - change throughout manuscript when escape room is followed by the plural 'games'.  If only stating escape rooms without 'games' after, leave 'escape rooms'.

Change line 18-19 to 'introducing a team of players into a physical or digital space in search of clues to answer puzzles, riddles or enigmas and ultimately solve a mystery or problem.

Line 30 - delete 'y' that appears in the last sentence, this appears to be an error, maybe meant to be 'in' or 'in nursing education.'

Introduction 

Line 46 change 'being used to improve the academic learning performance' to 'with the aim of improving the academic learning performance'

Line 53-54 change to 'digital space in search of clues to complete puzzles, riddles or enigmas, with the aim of solving a mystery or a problem.'

Line 64 remove 'therefore'  and start with A new systematic review of this topic ...was proposed...

Material and Methods

Line 92 change to 'were included'

Line 99 change to 'Gray literature was not included'

Line 117 change to "Finally, the following information was extracted from the studies:

The conclusion of Chen article 'to help them have a good game'. Is there something missing from the end of this sentence? Help them have a good game, does this refer to the ER game? It's a little confusing the way it is worded.

Hsu article - missing the 't' on the end of 'that' in the conclusion sentence study reported that the proposed course was innovative...

This may be just formatting that the journal can do but if possible, in this chart move the table a bit so that the word "interprofessional' isn't cut off and on the next line just appears 'al' for each of the Themes and Learning Topics categories in the table. 

Hursman article - change the Conclusions to:

This activity lays the groundwork for collaborative telehealth nursing that students will be exposed to in their future career. Results show the activity helped to build collaboration among team members, including those not in the same physical space. It also showed that virtual ER can be an effective activity to increase.....

Rodriguez - not sure ' Several Mental Illness' is meant to be 'severe mental illness'?

Wettergren - there is an "I' - was this meant to say change in attitudes toward interprofessional collaboration?

Moore and Campbell, need a space between effective and engaging interprofessional learning activity.  Also 'learning' is missing the 'g' at the end

Discussion:

Formatting needs to move this over on the page so the end of the sentences aren't all cut off.

Move up the third paragraph to combine with the second paragraph as the second paragraph is only 2 sentences. Same with 4th and 5th paragraph, they are the same topic so combine these paragraphs. Start a new paragraph with line 193 - Although GAMEX is not a specific instrument for ER, we consider...

Add to the paragraph Finally, it should be noted that studies have only been identified from 5 countries...as this is only one sentence and doesn't make a whole paragraph on it's own. Add at least 2 more sentences. There may be some evidence of why studies have only been conducted in these countries etc.

Suggest making Limitations its own section

Conclusion

Last sentence, remove 'y' and change to 'in nursing education'. This may be leftover from translation?

Suggest adding to the conclusion something about the fact that only a few countries have these types of articles and maybe another sentence or two on the future you see in this type of addition to traditional education.

I would also be curious to know the impact of ER on 100% online education programs - this may be outside of the scope of your review, but many programs especially graduate studies are now 100% online and gamification can be a great addition to keep students actively engaged and learning in a new creative way.

 

 

Comments on the Quality of English Language

Quality of the English was good - a few minor edits/suggestions in the above box.

Author Response

Thanks to the reviewers. These contributions have, undoubtedly, helped to improve the manuscript. We have made the changes in red to make it easier for the editor and reviewers. We hope the changes have been satisfactory. We send our best regards on behalf of all the authors.

The replies to reviewers’ comments for each of the points made are described below with red font.

Reviewer 2:

Abstract line 17 change Escape room's plural to Escape room games. - change throughout manuscript when escape room is followed by the plural 'games'.  If only stating escape rooms without 'games' after, leave 'escape rooms'.

Thank you. We have reviewed the whole manuscript and are implementing this recommendation.

Change line 18-19 to 'introducing a team of players into a physical or digital space in search of clues to answer puzzles, riddles or enigmas and ultimately solve a mystery or problem.

Thank you for your kind comments, which improve the language of our manuscript.

Line 30 - delete 'y' that appears in the last sentence, this appears to be an error, maybe meant to be 'in' or 'in nursing education.'

Sorry. It was a mistake from the translation.

Introduction 

Line 46 change 'being used to improve the academic learning performance' to 'with the aim of improving the academic learning performance'

Line 53-54 change to 'digital space in search of clues to complete puzzles, riddles or enigmas, with the aim of solving a mystery or a problem.'

Line 64 remove 'therefore'  and start with A new systematic review of this topic ...was proposed...

All proposal changes have been made. Thank you

Material and Methods

Line 92 change to 'were included'

Line 99 change to 'Gray literature was not included'

Line 117 change to "Finally, the following information was extracted from the studies:

All proposal changes have been made. Thank you.

The conclusion of Chen article 'to help them have a good game'. Is there something missing from the end of this sentence? Help them have a good game, does this refer to the ER game? It's a little confusing the way it is worded.

This was Chen's conclusion and has been transcribed textually as it appears in his study. We believe that he was not referring exclusively to ER but wanted to communicate the idea that the escape room, in addition to improving the attitude towards learning, can help students to have a good time. Therefore, we have not made any changes.

Hsu article - missing the 't' on the end of 'that' in the conclusion sentence study reported that the proposed course was innovative...

This may be just formatting that the journal can do but if possible, in this chart move the table a bit so that the word "interprofessional' isn't cut off and on the next line just appears 'al' for each of the Themes and Learning Topics categories in the table. 

Done.

Hursman article - change the Conclusions to:

This activity lays the groundwork for collaborative telehealth nursing that students will be exposed to in their future career. Results show the activity helped to build collaboration among team members, including those not in the same physical space. It also showed that virtual ER can be an effective activity to increase.....

Rodriguez - not sure ' Several Mental Illness' is meant to be 'severe mental illness'?

Wettergren - there is an "I' - was this meant to say change in attitudes toward interprofessional collaboration?

Moore and Campbell, need a space between effective and engaging interprofessional learning activity.  Also 'learning' is missing the 'g' at the end

Sorry. It was a mistake from the translation.

Discussion:

Formatting needs to move this over on the page so the end of the sentences aren't all cut off.

Move up the third paragraph to combine with the second paragraph as the second paragraph is only 2 sentences. Same with 4th and 5th paragraph, they are the same topic so combine these paragraphs. Start a new paragraph with line 193 - Although GAMEX is not a specific instrument for ER, we consider...

Thank you. We have made the suggested changes.

Add to the paragraph Finally, it should be noted that studies have only been identified from 5 countries...as this is only one sentence and doesn't make a whole paragraph on it's own. Add at least 2 more sentences. There may be some evidence of why studies have only been conducted in these countries etc.

Thanks for the suggestion. We have not found any possible explanation in the literature consulted. We therefore believe that this is an aspect that should be studied in the future. (we have reflected it in the text.)

Suggest making Limitations its own section

We have added a subsection for limitations

Conclusion

Last sentence, remove 'y' and change to 'in nursing education'. This may be leftover from translation?

Suggest adding to the conclusion something about the fact that only a few countries have these types of articles and maybe another sentence or two on the future you see in this type of addition to traditional education.

I would also be curious to know the impact of ER on 100% online education programs - this may be outside of the scope of your review, but many programs especially graduate studies are now 100% online and gamification can be a great addition to keep students actively engaged and learning in a new creative way.

Thank you for your comments. We have rewritten the conclusion, trying to include the aspects that you have rightly suggested. We believe that now the conclusion is better and more complete.

 

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Dear authors,

I commend the dedication and effort you've invested in refining the manuscript. Nonetheless, there are lingering methodological concerns that remain unresolved despite your revisions.

My primary apprehension revolves around the lack of clarity regarding the synthesis of effectiveness. Your PICO mnemonic requires a more structured approach, and I'll elucidate areas needing a clearer direction:

- Participants (P): Including both nursing students and nurses may skew your assessment, given the significant disparities in professional development and experience levels outlined in Patricia Benner's model. Focusing on one group would provide a more unbiased evaluation. If you opt to include both, clarify this rationale in the methods section.

- Intervention (I): The inclusion of both physical and digital Escape Rooms requires a precise definition. Were both physical and digital interventions accepted in this review? How did you differentiate Escape Rooms from similar activities like a classroom scavenger hunt? Clarifying these distinctions is essential for ensuring the integrity and specificity of your review's findings.

- Comparators (C): Were all study comparators eligible for inclusion in your review?

- Outcomes (O): The crux of my concern lies here. Your review lacks a clear definition of primary and secondary outcomes.

In your registered protocol, you state the outcomes as "Improves the effectiveness of interventions; improves satisfaction, and improves attitudes in the population studied." This formulation is problematic for two reasons: firstly, outcomes should be neutral rather than directed as "improves," and secondly, "improve the effectiveness of interventions" lacks clarity.

In the manuscript, the outcomes are described differently, focusing on "improvement of knowledge in different areas, evaluation of satisfaction with the educational activity, and improvement of attitudes or competencies of individuals in the population studied."

Your primary outcomes must be unequivocally articulated to readers. In my previous review, I provided a concise list for clarity: Is the synthesis focusing on participants' knowledge on a specific topic? Attitudes towards a specific topic? Teamwork? Leadership skills? Interprofessional communication?

Currently, your review aggregates data concerning participants' knowledge across various domains, encompassing topics like infection control, post-maternal care, and pressure injury prevention. However, is it appropriate to pool data from studies on such disparate topics? Will Escape Rooms be beneficial for improving participants' knowledge, regardless of the topic or field?

Furthermore, it's crucial to ensure that all primary outcomes listed are addressed in every included study. Regrettably, some studies, such as Foltz-Ramos et al., 2021, do not cover all main outcomes, raising concerns about their inclusion in your review. It's imperative to clearly differentiate between primary/main outcomes and secondary outcomes; while the former must be represented in all studies, the latter are not required for eligibility. 

I recommend consulting JBI's manual entry on Outcome Definition (link: https://jbi-global-wiki.refined.site/space/MANUAL/355828161/4.2.4.4+Outcomes) and revisiting your manuscript and PROSPERO protocol to address these issues thoroughly.

Comments on the Quality of English Language

Minor syntax and grammar issues that can be addressed if the manuscript is accepted.

Author Response

Thank you for your time and effort. Please find attached a file with the reviewer's feedback. Best regards

Author Response File: Author Response.pdf