Game on for Climate Action: Big Game Delivers Engaging STEM Learning

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Congratulations to the authors for the work done, I have a couple of comments that I hope will help you improve your study:

The main problem with the article is the lack of discussion and conclusions. I think it is necessary to have a discussion that clearly indicates what this study contributes in relation to the current scientific literature, and in the conclusions section, I think it is necessary to go deeper into them. It is also necessary to discuss the limitations of the study.

Regards

Author Response

The authors sincerely thank the Reviewer for their diligent work. Based on the Reviewer's comments and suggestions, the article has been extensively revised. The overall structure, analysis of references, and quotations have been reworked where necessary to highlight the core ideas that the work is based on and contributes to. Additionally, the reference list has been thoroughly revised. Detailed responses to the reviewer's comments are provided below. Please note that all revisions and additions are marked in red in the current version of the article.

Congratulations to the authors for the work done, I have a couple of comments that I hope will help you improve your study:

1. The main problem with the article is the lack of discussion and conclusions.

Response: Thank you for your feedback regarding the discussion and conclusions presented in the article. We appreciate your perspective and recognize the importance of these sections in providing a comprehensive understanding of the research findings.

We regret for not having provided these sections before (Discussion in particular), because initially we put more accent on the results’ delivery. But of course, as every work, containing the research element, we have to provide a wider perspective for the findings’ contribution and eventual impact in the field. Therefore, in its actual form, we have added the section. Having accentuated the core arguments the Discussion is based on, we have provided deeper insights into the references present and open issues contained within. Thus, we hope to have highlighted the contributions of the work more substantially. 

In particular, the article in its actual form incorporates discussion containing the analysis of the BIG GAME approach, highlights its interdisciplinary nature and effectiveness in developing critical skills such as analytical thinking, communication, and collaboration. This analysis serves to contextualize the findings within the broader educational landscape, emphasizing the project's strengths and areas for improvement. By referencing existing literature and various findings from empirical studies, in particular, open issues contained within,  we aim to situate our findings within the ongoing discourse on innovative teaching methodologies, thereby contributing to the understanding of how game-based learning can address challenges in STEM education.

2. I think it is necessary to have a discussion that clearly indicates what this study contributes in relation to the current scientific literature, and in the conclusions section, I think it is necessary to go deeper into them. It is also necessary to discuss the limitations of the study.

Response. Thank you for your valuable feedback. We appreciate your suggestions regarding the need for clearer articulation of our study's contributions and a more comprehensive discussion of conclusions and limitations. In response, we have revised our manuscript to include a dedicated section in the discussion that explicitly outlines our study's unique contributions to the current scientific literature. We emphasize how our findings build upon and extend previous research, offering novel insights and advancements in the field.

Additionally, we have expanded the conclusions section to provide a deeper analysis of the implications of our findings, highlighting their significance and potential applications. We have also added a detailed discussion of the study's limitations, addressing potential constraints and suggesting areas for future research to overcome these challenges. We believe these revisions address the concerns raised and enhance the manuscript's clarity and impact.

 

 

Reviewer 2 Report

Comments and Suggestions for Authors

Dear Authors

 

It is a pleasure for me to review your article. I would like to suggest some areas for improvement.

The abstract: You should be clearer about the research, providing some clues about the results, as well as numerical data about the participants, the period and the number of games.

In your theoretical framework you should talk about or conceptualise serious games. I miss mentions of authors such as Abt (1970), who coined the term, as well as more direct references to other research with serious games. Add current research that has worked with them, this will help you both now and later in your results and conclusions to establish relationships.

 

Tab 1 is very unattractive, can you improve it? I would change stars to a graphic representation of a star.

The letters in figure 4 are difficult to read because of the size, they should be more visible.

Be careful when showing people's faces in the images, maybe you should pixelate some faces or check if you had permission to show them.

Is BIG GAME a methodology? You should argue this. Personally, it is not clear to me that it is a methodology, but rather a way of applying serious games. And in particular the title of a proposal or project.

 

Finally, you should expand on your conclusions, relate them to previous research, limitations you encountered in conducting your study and future lines of research.

 

Thank you

 

Author Response

The authors sincerely thank the Reviewer for their diligent work. Based on the Reviewer's comments and suggestions, the article has been extensively revised. The overall structure, analysis of references, and quotations have been reworked where necessary to highlight the core ideas that the work is based on and contributes to. Additionally, the reference list has been thoroughly revised. Detailed responses to the reviewer's comments are provided below. Please note that all revisions and additions are marked in red in the current version of the article.

 

Dear Authors

 It is a pleasure for me to review your article. I would like to suggest some areas for improvement.

1. The abstract: You should be clearer about the research, providing some clues about the results, as well as numerical data about the participants, the period and the number of games.

Response. The authors thank the Reviewer for the feedback. We have made revisions to address your comments and ensure clarity in our research presentation, providing the related details. The updated version of the abstract is provided below:

A decline in European students' interest in STEM subjects, exacerbated by the COVID-19 pandemic's disruption of education, has raised concerns about the continent's future workforce. To address this challenge, this study investigates the efficacy of the BIG GAME project methodology, a cooperative story-driven digital game approach designed to engage secondary students in collaborative environmental problem-solving.

Implemented across Romania, Italy, Estonia, and Finland, the six-month study employed a mixed-methods design involving 62 teachers and 239 students in ten distinct game missions focused on environmental challenges. Quantitative results indicated significant improvements in students' transversal skills (teamwork, communication, critical thinking), as reported by over 75% of teachers. Qualitative data emphasized the value of structured peer review in developing students' reflective practices.

The research underscores the importance of teacher facilitation in game-based learning and problem-based simulation and highlights the potential of such methodologies to boost student engagement and environmental awareness. These findings suggest that the BIG GAME project approach could be a valuable tool for revitalizing STEM education and cultivating future-ready citizens.

2. In your theoretical framework you should talk about or conceptualise serious games. I miss mentions of authors such as Abt (1970), who coined the term, as well as more direct references to other research with serious games. Add current research that has worked with them, this will help you both now and later in your results and conclusions to establish relationships.

Response. Thank you for your valuable feedback regarding this aspect. We regret not having provided a more accurate exposition of the theoretical context before, but exclusively because we were primarily focused on delivering the experimental results. But, surely, and we absolutely agree with the Reviewer, that providing a more sound background for the actual work is important. Therefore, we have updated the theoretical framework to include a discussion of serious games, emphasizing the relevant recent studies and research on serious games to show their evolution and application in our study.

Undoubtedly, these additions strengthen our theoretical basis and help connect our findings to the broader field of serious games. We believe these changes address the issues raised and enhance our manuscript.

Following another Reviewer's suggestions, we have extended and deepened the exposition of ideas and open issues that arose in the referenced literature in general. This has provided a more solid foundation for the current work while underlining its contribution to the field. 

3. Tab 1 is very unattractive, can you improve it? I would change stars to a graphic representation of a star.

Response. We thank the Reviewer for the comment. The suggestion has been taken into account.

4. The letters in figure 4 are difficult to read because of the size, they should be more visible.

Response. We thank the reviewer for the comment, the Figure has been updated.

5. Be careful when showing people's faces in the images, maybe you should pixelate some faces or check if you had permission to show them.

Response: Thank you for bringing this to our attention. We have reviewed all images in the manuscript to ensure compliance with privacy and ethical standards. Faces have been blurred where necessary, and we have confirmed that we have the appropriate permissions to use all images included in our study.

We appreciate your diligence in highlighting this important issue, and we are committed to maintaining the highest standards of privacy and consent in our research.

6. Is BIG GAME a methodology? You should argue this. Personally, it is not clear to me that it is a methodology, but rather a way of applying serious games. And in particular the title of a proposal or project.

Response. We thank the Reviewer for this insightful reflection, and apologize for not having been explicit enough in providing a distinct exposition regarding the issue. Big Game is the acronym of the title of the project “Immersive and multidisciplinary STEM learning methodology through a Cooperative Story-Driven Digital Game”.

The proposed methodology for this project is a problem-based simulation game in which teams of students role-play as elite squads of experts tasked with tackling urgent environmental issues.

The idea was to use and apply the principles of Cooperative Story-Driven Digital Game to engage students in STEM Education through the solutions of fictional missions based on real environmental issues.

The objectives were:

1. Promoting interest and excellence in science, technology, engineering, and mathematics (STEM) approach
   to education through multidisciplinary learning and problem-solving related to environmental context in the form of a digital learning game.
2. Supporting the versatile accumulation of digital skills of both teachers and 11-16-year-old students.
3. Raising awareness of environmental and climate change issues and engaging in the fight against climate change.

The corresponding clarifications have been introduce throughout the text.

7. Finally, you should expand on your conclusions, relate them to previous research, limitations you encountered in conducting your study and future lines of research.

Response. Thank you for your insightful comments on our manuscript. We appreciate your suggestion to enhance the conclusions section to provide a more complete understanding of our study's implications.

To address this, we have revised the conclusions to include a more thorough analysis of our results in relation to existing research. We have also detailed the challenges we faced during the study and proposed directions for future investigation. Our revised conclusions offer an in-depth examination of the BIG GAME approach, emphasizing its interdisciplinary benefits and its role in fostering critical skills such as analytical thinking, communication, and collaboration. This provides a broader context for our findings and underscores the innovative nature of our methodology.

By connecting our work with previous studies and highlighting open issues, we aim to contribute to the broader discussion on educational strategies, particularly in the context of game-based learning. We hope these revisions effectively highlight our study's contributions and address your concerns.

Thank you again for your feedback, which has been invaluable in strengthening our manuscript.

 

Reviewer 3 Report

Comments and Suggestions for Authors

The BIG GAME project employed a gamified approach to STEM learning by focusing on environmental problem-solving and collaborative learning. The project aimed to enhance student engagement, STEM proficiency, and awareness of global challenges. This multidisciplinary approach underscores the potential of serious gaming.

The Big Game's layout is very well-organized and engaging.

While I believe the presented work is highly relevant, I suggest considering revisions to the following aspects:

·        The abstract should be revised to provide a clearer explanation of the research method.

·        The claims made in the Abstract, specifically "Given the declining interest in science education, intensified by the disruptions caused by COVID-19" and in the Introduction "In light of the declining interest in science subjects," require stronger evidence. A thorough review of the cited reference (in introduction) did not yield sufficient scientific support for these assertions. Direct quotes or additional references could be used to substantiate these claims.

·        While the authors have outlined the project's objectives, they have not clearly articulated the specific research questions. A precise identification of these questions is essential to guide the research and enable a more rigorous evaluation of the findings.

·        The section titled "Methodology" (Section 2) contains content that, in my opinion, would be more suitable for a "Theoretical Framework" section.

·        The theoretical framework could be enriched by incorporating peer assessment to provide a more comprehensive understanding of decision-making within the game environment and its features.

·        The authors refer to "conventional teaching methods." I suggest they clarify what these are.

·        Instead of the broad statement “Additionally, it integrates active learning scenarios, resource planning, spatial skills, immersion…” (line 123), I propose a more specific phrasing: 'Additionally, storytelling can be designed to incorporate active learning elements'. This revised statement acknowledges that while storytelling can integrate these aspects, it does not necessarily encompass all of them in every instance.".

·        I recommend restructuring the paper for enhanced clarity and coherence. Specifically, I propose creating dedicated sections for 'Method' and 'Results'. The current Sections 3 and 4 can be incorporated into the 'Method' section, as they are part of the research process. However, prior to this, a 'Research Design' section should clearly outline the study design. Additionally, a 'Participants' section should provide detailed information about participants, including demographics, inclusion/exclusion criteria, and recruitment procedures. A 'Data Collection Instruments' section should describe the tools used for data collection, such as surveys (at the moment we do not know the questions in the survey). It's also important to discuss the validity and the possibly of the reliability of these instruments.

·        The claim made in section 3.2.1 is not backed up by any evidence.

·        The figure numbering needs to be checked. There is inconsistency between the references in the text and the figures.

·        The graphs should have labelled axes. For example, in Figure 9, what do the vertical and horizontal axes represent?

·        There is no mention of Figure 9 within the text.

·        The reference list formatting requires attention. References with more than two authors should list all authors, not use 'et al.'. Additionally, the reference style does not adhere to MDPI standards. For instance, reference of line 877 should be formatted as

Shaby, N.; Staus, N.; Dierking, L.D.; Falk, J.H. Pathways of interest and participation: How STEM-interested youth navigate a learning ecosystem. Sci. Educ. 2021, 105, 628–652, doi:10.1002/SCE.21621.

I recommend using an automated referencing system to streamline the formatting process. If you are using Mendeley, you can go to View, then Citation Style, and select the Multidisciplinary Digital Publishing Institute style. If it's not available by default, click on 'More Styles' and install it.

Author Response

The authors sincerely thank the Reviewer for their diligent work. Based on the Reviewer's comments and suggestions, the article has been extensively revised. The overall structure, analysis of references, and quotations have been reworked where necessary to highlight the core ideas that the work is based on and contributes to. Additionally, the reference list has been thoroughly revised. Detailed responses to the reviewer's comments are provided below. Please note that all revisions and additions are marked in red in the current version of the article.

 

1. The BIG GAME project employed a gamified approach to STEM learning by focusing on environmental problem-solving and collaborative learning. The project aimed to enhance student engagement, STEM proficiency, and awareness of global challenges. This multidisciplinary approach underscores the potential of serious gaming.

Response. We thank the Reviewer for having summarized the project presented. It served us to understand that the basic ideas have been perceived correctly.

2. The Big Game's layout is very well-organized and engaging.

 

Response. The authors are grateful for the complement.

While I believe the presented work is highly relevant, I suggest considering revisions to the following aspects:

3. The abstract should be revised to provide a clearer explanation of the research method.

 

Response. Thank you for your feedback. We appreciate you pointing out the need for a clearer explanation of the research method in the abstract.

We will revise the abstract to provide a more detailed overview of the research design, including:

• A specific mention of the mixed-methods approach (quantitative and qualitative data)
• A brief description of the data collection methods (survey, game scenarios)
• A clear statement about the sample size and participant demographics (62 teachers from four countries)

We believe these additions will enhance the clarity and comprehensiveness of the abstract.

 

The revised version of the abstract is the following:

 

A decline in European students' interest in STEM subjects, exacerbated by the COVID-19 pandemic's disruption of education, has raised concerns about the continent's future workforce. To address this challenge, this study investigates the efficacy of the BIG GAME project methodology, a cooperative story-driven digital game approach designed to engage secondary students in collaborative environmental problem-solving.

Implemented across Romania, Italy, Estonia, and Finland, the six-month study employed a mixed-methods design involving 62 teachers and 239 in ten distinct game missions focused on environmental challenges. Quantitative results indicated significant improvements in students' transversal skills (teamwork, communication, critical thinking), as reported by over 75% of teachers. Qualitative data emphasized the value of structured peer review in developing students' reflective practices.

The research underscores the importance of teacher facilitation in game-based learning and problem-based simulation and highlights the potential of such methodologies to boost student engagement and environmental awareness. These findings suggest that the BIG GAME project approach could be a valuable tool for revitalizing STEM education and cultivating future-ready citizens.

The claims made in the Abstract, specifically "Given the declining interest in science education, intensified by the disruptions caused by COVID-19" and in the Introduction "In light of the declining interest in science subjects," require stronger evidence. A thorough review of the cited reference (in introduction) did not yield sufficient scientific support for these assertions. Direct quotes or additional references could be used to substantiate these claims.

Response. Authors thank the Reviewer for pointing out the need for more substantial evidence to support our claims regarding the declining interest in science education, particularly concerning the impact of COVID-19. The corresponding additional references and direct quotes to support these statements have been included and quoted.

These quotes can be integrated into the Abstract and Introduction to provide stronger evidence for the claims about declining interest in science education due to the impacts of COVID-19.

4. While the authors have outlined the project's objectives, they have not clearly articulated the specific research questions. A precise identification of these questions is essential to guide the research and enable a more rigorous evaluation of the findings.

• Response: Thank you for your feedback regarding the need for a clear articulation of research questions. After a thorough reflection, we agree that specifying these questions is crucial for guiding the research and ensuring a rigorous evaluation of our findings. We have refined and clarified our research questions as follows from the following perspectives: 1) Impact of Game-Based Learning on Environmental Awareness; 2) Student Preferences and Challenges; 3) Teacher perspectives. In particular, the corresponding research questions were formulated as follows:
• How does participation in the BIG GAME project influence students' awareness and understanding of environmental issues such as climate change, deforestation, and pollution?

• What specific activities within the BIG GAME project do students find most and least engaging in developing their STEM skills, and what challenges do they encounter during the project phases?
• How do teachers perceive the effectiveness of the BIG GAME project in enhancing students' learning experiences and fostering environmental awareness?

These questions have been designed to comprehensively address the project's objectives, and we believe they will facilitate a more structured and rigorous evaluation of our findings.

 

5. The section titled "Methodology" (Section 2) contains content that, in my opinion, would be more suitable for a "Theoretical Framework" section.

Response. We thank the Reviewer for the insightful comment. Following the recommendations of the 2 Reviewer as well, the overall structure of the article has been re-organized. The section Theoretical framework has been added and encapsulated all the relevant subsections outlined by the Reviewer.

6. The theoretical framework could be enriched by incorporating peer assessment to provide a more comprehensive understanding of decision-making within the game environment and its features.

 

Response. We thank the Reviewer for a valuable insight. In fact, in-class per-reviewing was one of the core components in the final decision making regarding the solutions to the proposed to the expert team. There is no decision making process inside the digital game environment. Once the mission is delivered, overall decision making process takes place outside the game environment, in the classroom. Anyway, we have added a paragraph in the section 4.2.4 Assessment procedures in the classroom, explaining the importance of the peer assessment. 

7. The authors refer to "conventional teaching methods." I suggest they clarify what these are.

Response: We thank the reviewer for their valuable comment highlighting the need for clarification regarding "conventional teaching methods." In the context of the work, authors refer to these as traditional, teacher-centred approaches that primarily rely on lecturing, textbook instruction, and standardized assessments. These methods often prioritize knowledge acquisition through memorization over higher-order thinking skills like critical analysis and problem-solving.

The corresponding clarifications have been introduced in the text.

8. Instead of the broad statement “Additionally, it integrates active learning scenarios, resource planning, spatial skills, immersion…” (line 123), I propose a more specific phrasing: 'Additionally, storytelling can be designed to incorporate active learning elements'. This revised statement acknowledges that while storytelling can integrate these aspects, it does not necessarily encompass all of them in every instance.".

Response:  We thank the reviewer for the insightful comment. The suggestion has been introduced. This refinement significantly enhances the precision and clarity of our statement. We agree that this change strengthens the overall quality of the text.

9. I recommend restructuring the paper for enhanced clarity and coherence. Specifically, I propose creating dedicated sections for 'Method' and 'Results'. The current Sections 3 and 4 can be incorporated into the 'Method' section, as they are part of the research process. However, prior to this, a 'Research Design' section should clearly outline the study design. Additionally, a 'Participants' section should provide detailed information about participants, including demographics, inclusion/exclusion criteria, and recruitment procedures. A 'Data Collection Instruments' section should describe the tools used for data collection, such as surveys (at the moment we do not know the questions in the survey). It's also important to discuss the validity and the possibly of the reliability of these instruments.

Response. We thank the Reviewer for such a constructive suggestion. In fact, we agree that organized in the way proposed, the article gains more clarity and coherence. Therefore, all the proposed sections were introduced. While elaborating on the distribution of the content, we have considered it reasonable to incorporate the old Exploratory research section into the Research design section due to its organizational character for the subsequent experimental phases of the project.

 Following the suggestion, we have re-organized the article structure and overall presentation.

10. The claim made in section 3.2.1 is not backed up by any evidence.

Response. We thank the Reviewer for the comment. The data in question a coming from the teachers’ responses explored during the exploratory phase, therefore representing exclusively the subjective points of view based  on their personal experience on country situation. However, this section has been revised to clarify the origin of the data.   

11. The figure numbering needs to be checked. There is inconsistency between the references in the text and the figures.

Response: We appreciate the reviewer for pointing out the inconsistency in figure numbering.

12. The graphs should have labelled axes. For example, in Figure 9, what do the vertical and horizontal axes represent?

Response. We thank the Reviewer for the comment. In fact, Figure 9 is not informative due to missing labelling. The vertical axes (Figs. 9-11) represent the percentage of the respondents who chose the corresponding option, whereas the horizontal axis basically lists the options themselves. In the case of Figure 9, the corresponding corrections have been made for both axes. In the rest of the figures, for the horizontal axes being self-representative, only the vertical axes have been corrected. 

13. There is no mention of Figure 9 within the text.

Response.  We thank the Reviewer for having noticed the oversight. The reference was implicit, stating: «as shown in the Figures below», followed by the Figure. But we have changed the reference into “… as shown in Figure 9 below”.

14. The reference list formatting requires attention. References with more than two authors should list all authors, not use 'et al.'. Additionally, the reference style does not adhere to MDPI standards. For instance, reference of line 877 should be formatted as

Shaby, N.; Staus, N.; Dierking, L.D.; Falk, J.H. Pathways of interest and participation: How STEM-interested youth navigate a learning ecosystem. Sci. Educ. 2021, 105, 628–652, doi:10.1002/SCE.21621.

Response. We thank the Reviewer for pointing out the issues with the reference list formatting. We have revised the references to ensure that all authors are listed for entries with more than two authors, in accordance with MDPI standards. We have also corrected the formatting throughout the reference list, using your example as a guide to ensure accuracy and consistency.

15. I recommend using an automated referencing system to streamline the formatting process. If you are using Mendeley, you can go to View, then Citation Style, and select the Multidisciplinary Digital Publishing Institute style. If it's not available by default, click on 'More Styles' and install it.

Response. The authors are thankful for your suggestion regarding the use of an automated referencing system. We currently use EndNote for managing our references and have ensured that it is set to format citations according to the Multidisciplinary Digital Publishing Institute (MDPI) style. We the reference list was thoroughly reviewed to ensure full compliance with MDPI standards using this tool. We appreciate your recommendation.

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

The authors have extensively reviewed the paper, justifying all decisions. However, there are still aspects that I recommend addressing:

·        Line 128: The sentence is incomplete; the authors forgot to integrate the review.

·        Figure 10 is not easy to read. In my opinion, it would be more intuitive to display what is currently in the subtitles on the x-axis, as they did for the other graphs.

·        The authors must review the bibliographic references again. For example, the expression 'et al.' still appears in the reference list (lines 1067, 1080, 1087, 1089, etc.). In my previous report, I gave an example of how the reference on line 1067 should be formatted.

Instead of the following:

Shaby, N.; Staus, N.; Dierking, L.D.; Falk, J.H. Pathways of interest and participation: How STEM-interested youth navigate a learning ecosystem. Sci. Educ. 2021, 105, 628–652, doi:10.1002/SCE.21621.

It erroneously appears as:

Shaby, N., et al., Pathways of interest and participation: How STEM‐interested youth navigate a learning ecosystem. Science education, 2021. 105(4): p. 628-652.

Author Response

The authors have extensively reviewed the paper, justifying all decisions. However, there are still aspects that I recommend addressing:

• Line 128: The sentence is incomplete; the authors forgot to integrate the review.

Response: Authors thank the reviewer for the suggestion. The sentence is referred to the methodology developed and piloted during the project. Therefore any references should be connected to the official project report prepared by the project partnership. We have provided the direct link to the report Handbook and Toolkit on Digital Storytelling Approach in STEM. All the project reports are available under Creative Common license at https://big-game.eu-track.eu/results.php.

• Figure 10 is not easy to read. In my opinion, it would be more intuitive to display what is currently in the subtitles on the x-axis, as they did for the other graphs.

Response. Authors thank the reviewer for the valuable insight. In fact, it would be reasonable to preserve the labelling throughout the figures. Therefore we have rearranged the Figure in such a way, that it follows the same labelling.

• The authors must review the bibliographic references again. For example, the expression 'et al.' still appears in the reference list (lines 1067, 1080, 1087, 1089, etc.). In my previous report, I gave an example of how the reference on line 1067 should be formatted.

Instead of the following:

Shaby, N.; Staus, N.; Dierking, L.D.; Falk, J.H. Pathways of interest and participation: How STEM-interested youth navigate a learning ecosystem. Sci. Educ. 2021, 105, 628–652, doi:10.1002/SCE.21621.

It erroneously appears as:

Shaby, N., et al., Pathways of interest and participation: How STEM‐interested youth navigate a learning ecosystem. Science education, 2021. 105(4): p. 628-652.

Response. We apologize for not having resolved the issue before. Now, with the appropriate referencing and output style checked, the issue no longer persists.