Game Jams as Valuable Tools for the Development of 21^st-Century Skills

Round 1

Reviewer 1 Report

Shorter paragraphs in the theoretical framework, 
State explicitly the objectives (general and specific) of the research.
Dates on which the surveys were conducted. 
How were the answers coded in Atlas.ti? 
The order of the results should be maintained as explained in the methodology.
4.1.2. Communication
4.1.3. Collaboration
4.1.4. Critical thinking
Creativity 
...
In the conclusions, I would add a more detailed summary according to the objectives of the research. I would add at least 2 more paragraphs highlighting the most relevant information.

Author Response

We appreciate that the reviewer’s comments. The followings are our point-by-point responses:

-State explicitly the objectives of the research.
More information has been added.
-Dates on which the surveys were conducted. 
More information has been added.
-How were the answers coded in Atlas.ti? 
More information has been added.

-The order of the results should be maintained as explained in the methodology.
Done

-In the conclusions, I would add a more detailed summary according to the objectives of the research. I would add at least 2 more paragraphs highlighting the most relevant information.
More information has been added.

Thank you very much for reviewing our manuscript

Reviewer 2 Report

Dear authors,

I read your article with great interest because it examines an interesting topic: Game Jams as Valuable Tools for the Development of 21st Century Skills

The article has a good structure. However, in the introduction the authors did not present the gaps in the literature and the novelty element. The usefulness of the analysis must be presented much more clearly. The authors ask 3 research questions.

Also, the literature analysis requires some adjustments and improvements regarding the fluency and coherence of the analyzed theory. The subchapters of the theory end abruptly. The reader is left in suspense and without understanding the essence of the information. Here, I recommend reviewing the literature review approach even if the theoretical mechanisms seem clear.

Research methodology needs improvement. I recommend that you better explain the sample included in the analysis and explain with arguments whether the sample is representative. The authors write in the abstract and then in the paper that they applied questionnaires and interviews. Here's what you wrote "Data was compiled through direct observation, open-ended questionnaires, and interviews and was subject to thematic analysis in order to construct new knowledge on a previously underexplored topic." From the explanations it is not understood how many questionnaires were applied and how many interviews. Moreover, I recommend that you present the items included in the research tools and separate the results obtained.

The analysis of the results is treated very generally, with brief explanations. I recommend providing research data. Also, can the results obtained be analyzed and confirmed by the results in the literature?

What are the limits of research? What about future research directions?

That's all! Good luck! 

Author Response

We appreciate the reviewer’s comments. The followings are our point-by-point responses:

 in the introduction the authors did not present the gaps in the literature and the novelty element. 
More information has been added.

Research methodology needs improvement. 
More information has been added.

can the results obtained be analyzed and confirmed by the results in the literature?What are the limits of research? What about future research directions?
More information has been added.

Thank you very much for reviewing our manuscript

Reviewer 3 Report

This manuscript describes the contribution of game jams to the development of 21^st century skills, based on the observations gathered during 3 years of work organizing and analyzing such events. Moreover, the authors provide some recommendations on how to use game jams to foster 21^st century skills.

Game jams are becoming increasingly popular, but are not yet widespread, neither in formal nor in informal education. However, they provide interesting opportunities for fostering a series of transversal competencies necessary for long-life learning and successful job-market integration in the 21^st century; namely, critical thinking, collaboration, creativity and communication. As such, the paper spots an innovative strategy that many educators might want to consider.

Moreover, data collection is based on multiple data sources, combining researchers’ and user’s perceptions, gathered through observation and questionnaires, respectively.

However, the paper fails in the fundamental, which is adequately addressing the proposed research questions. As for the first question (how do game jams contribute to the development of 21^st century skills), the authors provide a theoretical proposal, but (almost) no empirical evidence that these contributions are realized. Indeed, they anticipate that “Due to the constructivist nature of the learning that takes place during game jams, it is not possible to use tangible evidence to prove the change occurred in participant skills. Self-evaluation does not always prove that learning took place, as situations in which participants inform picking up a new skill may not be clear” (lines 243-245). Firstly it’s at least debatable that such as unreliable source of data could be used as origin of any evidence, as introduced in the methodology. Secondly, that evidence is given a subsidiary role, suggest that this should not be presented as an output of the research.

Besides, sometimes it is not clear where the case study ends and starts the literature review. For example, in table 3 language barriers are identified as an individual challenge, compromising collaboration and communication skills, but language seemed not to be an issue here (l.428 – l.429). Please clarify that.

Likewise, questions 2 and 3 are not directly addressed by research, as it would be the case if they followed and experimental or quasi-experimental approach, by comparing different formats and its effects on learning.

All this does not discredit the valuable recommendations, based on a three-years experience on a so-far poorly described area. But it would be more honest to word it as objectives, and no research questions, and to frame them as “prospective utility of game jams”, “suggested format and resources”…, explicitly acknowledging that these are mostly based on the experience gained by the researchers.

Moreover, I suggest revising the order of the paragraphs, as the order of the information might not be the most obvious to the reader. For example, talking about the advantages and types of game jams before defining what they are (lines 24-64). Likewise, all along the text there are information fragments that would fit more naturally in other sections (for example, lines 178 and ff., which provide a general definition of game jams and their desirable features).

Author Response

We appreciate the reviewer’s comments. The followings are our point-by-point responses:

-State explicitly the objectives  of the research.
More information has been added.

-Dates on which the surveys were conducted. 
More information has been added.

-How were the answers coded in Atlas.ti? 
More information has been added.

-The order of the results should be maintained as explained in the methodology.
Done.

-In the conclusions, I would add a more detailed summary according to the objectives of the research.
More information has been added.

Thank you very much for reviewing our manuscript

Round 2

Reviewer 2 Report

Dear authors,

Thank you so much for the improved version of the article!

I read through all the additions you made. Thus, we found that the article was not improved according to the recommendations. In addition, the research questions have been turned into objectives, although they are also formulated as questions. That is, they have not been changed at all. In conclusion, does the paper have research questions or research objectives? I think you need to clarify that. I also insist that the previous recommendations be resolved.

All the best! 

Author Response

Dear reviewer,

Thank you very much for your valuable input. We have tried to address all your concerns to the best of our ability, and we hereby proceed to clarify each of the latest changes made.

1. “The article has a good structure. However, in the introduction the authors did not present the gaps in the literature and the novelty element”.

We believe that the following modified excerpt now clearly states that there is a large knowledge gap regarding the use of game jams as a pedagogical method to develop 21^st century skills. There is only 1 research paper on the topic except for our work. Considering this, it is not possible to discuss any further the research gap, as the body of literature on the matter is very limited. We consider that our work is highly relevant as it contributes to a very unexplored research area (novelty).

“Current game jam literature mainly focuses on case studies in which students develop video games from a recreational perspective (e.g. [1, 5, 11]) but does not discuss the potential of game jams as valuable tools to develop 21st century skills. We know very little about the motivation of game jam organizers or how game jamming can be an appropriate teaching method for soft skills, such as communication and collaboration, except for what we can infer from the overall theme of a jam [16]. Despite Aurava et al. [10] already confirmed that teachers think that game jamming can be a teaching method for soft skills, and with game jams already being increasingly popular in the education sector, there is still a lack of qualitative research identifying the specificities of each type of events [16].

The present work thus addresses a large knowledge gap by focusing on game jams directed towards enhancing 21st century skills and provides unique theoretical and practical perspectives about game jams as a pedagogical method, including recommendations on how to organize such events, reduce their typical duration, and employ them as an extra-curricular activity.”

2. “The usefulness of the analysis must be presented much more clearly. The authors ask 3 research questions.”

We believe that the practical application of the outcomes of this work is now clearly stated (see previous excerpt). The three research questions have now been formulated as objectives of this paper.

3. Also, the literature analysis requires some adjustments and improvements regarding the fluency and coherence of the analyzed theory. The subchapters of the theory end abruptly. The reader is left in suspense and without understanding the essence of the information. Here, I recommend reviewing the literature review approach even if the theoretical mechanisms seem clear.

The Background section has been reviewed in depth and reorganized; new references were added. We believe that all the important information is now properly connected through a clear, cohesive, and logical train of thought.

4. Research methodology needs improvement. I recommend that you better explain the sample included in the analysis and explain with arguments whether the sample is representative. The authors write in the abstract and then in the paper that they applied questionnaires and interviews. Here's what you wrote "Data was compiled through direct observation, open-ended questionnaires, and interviews and was subject to thematic analysis in order to construct new knowledge on a previously underexplored topic." From the explanations it is not understood how many questionnaires were applied and how many interviews. Moreover, I recommend that you present the items included in the research tools and separate the results obtained.

The number of questionnaires and interviews are included in Table 1, together with a description of each of the events. Please refer to the following excerpt for a description of the participants:

“A total amount of 90 students aged 12-16 joined the events, together with 6 teachers, 6 game designers, and 3 researchers that acted as tutors. The figure of the tutor was represented by experts in various disciplines, such as video games, programming, art, or sustainability, that supported the teams with their advice or helped them with problem-solving during the sessions.”

Given the non-empirical and qualitative nature of this work, which is one of the first attempts to study game jams as a pedagogical tool, during the design of the game jam events, sample representativeness was not taken into consideration. Therefore, sample representativeness cannot be justified, nor was it considered relevant for the purposes of this work.

As stated in the section Materials and Methods, the events were organized following the recommendations, results, and experiences of other authors, such as: Aurava et al., Juergen et al., or Tenório et al. Furthermore, the qualitative analysis of the data was performed in line with other similar studies (see Aurava et al.). The results are not presented according to the data source, but to the key discussion points that emerged from this methodology and the hermeneutic approach, which is the qualitative information of interest in this work. These two key perspectives (organizer and jammer) are represented and separated in the discussion of results, as illustrated in the following excerpts:

• Organizer perspective:

“Systematic and critical thinking processes also arose during the video game testing phase. During this phase, group members need to review their design proposal, strategically adjust what does not work (bugs, glitches), and add necessary features (music, characters). The multidisciplinary environment contributes to the jammers' compromise towards task completion, while the diversity of participants stimulates new ways of thinking regarding certain topics, the management of programming activities and the video game development timeline, and bug handling, which are components of critical thinking.”

• Jammer perspective:

“Through the post-event survey, participants reported having positive experiences during each phase [1] and, apart from learning new concepts, they mentioned that the game jam motivated them to develop skills such as critical thinking: “testing allowed me to try different ways to test the game and avoid bugs” (Participant, Event 2), as reported in Aurava et al. [10].”

 

5. The analysis of the results is treated very generally, with brief explanations. I recommend providing research data. Also, can the results obtained be analyzed and confirmed by the results in the literature?

Given the lack of qualitative research identifying the specificities of each type of game jam event (Gorm Lai et al.), we opted for contributing to this line of work. We did not carry out a quantitative analysis and, therefore, we can not provide quantitative data. We provide for each sub-section (where applicable): an interpretation from the perspective of the organizers, examples of the perspective of the jammers, and a brief literature compilation of results that complete and support our observations. Several references were added. As an example, the continuation of the previous excerpt:

“According to Bezanilla et al. [57], learning how to think critically can be an individual experience, and six categories of skills support the general development of critical thinking skills. These include (1) Analyzing/Organizing; (2) Reasoning/Arguing; (3) Questioning/Asking oneself; (4) Evaluating; (5) Taking a position/Taking decisions; and (6) Acting/Compromising. Almerich et al. [58] mention that technology use and the subsequently developed skills effectively favor the development of critical thinking. Co-designing is a collective and transformative action aimed at generating new forms of activity with meaningful objectives to participants [59] and leads to thinking critically, for instance, when testing video games to find bugs. Lastly, Papert [60] states that, from a constructivist perspective, game design promotes critical thinking through the design of physical objects.”

6. What are the limits of research? What about future research directions?

The limitations of the present study were included in the last paragraph of the discussion. We have now created a sub-section for the limitations to facilitate their location in the text (see sub-section 4.4; line).

Future research directions are included at the end of the Conclusions:

“Participants indicated that a more detailed skill selection in jammers could have benefited the teams. Thus, the authors suggest studying in-depth the different ways in which the skills and knowledge of jammers can be identified in order to create more effective and diverse working groups with different talents and skills.

Finally, new studies should examine the scope and development of the 4C skills, as game jams offer great opportunities through their various activities. Another area of future research includes the exploration of the experiences of game designers and teachers, in addition to that of the students, and how these actors navigate issues such as introducing concepts, methodologies, etc. into game jams.”

Thank you for your time.

Kind regards

Round 3

Reviewer 2 Report

Dear authors,
Thanks a lot for the detailed explanations.
I wish you all the best!