Leveraging Gamification in ICT Education: Examining Gender Differences and Learning Outcomes in Programming Courses

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear authors, congratulations on your manuscript. It is well organised and the subject matter is current within the different pedagogical methods. I recommend you the following reference on gamification and gender which could provide significant information for your study https://revistaprismasocial.es/article/view/3745.

 

The research questions are well formulated and coherent.

 

The methodology is thoroughly explained and with possibilities for replication by other researchers.

 

The results are abundant and revealing.

 

The discussion is quite extensive if we bear in mind that the conclusions are also too long. I think that the conclusions should be more concrete and offer clear results on the main ideas detected.

The limitations of the study and the prospective of the study are missing. Finally, in the summary I would include the instrument used for the evaluation that has been used.

Author Response

Comment 1:I recommend you the following reference on gamification and gender which could provide significant information for your study https://revistaprismasocial.es/article/view/3745.    
Response 1: The following sentence has been added: "In [ref], it is revealed that implementing badge-based gamification in higher education improved content acquisition and increased the perception of gender equity among students, with a more notable increase in participation and perceived learning among women than men."

Comment 2:I think that the conclusions should be more concrete and offer clear results on the main ideas detected.    
Response 2: The writing in the conclusions was improved by focusing on: key results, clarity, practical implications, limitations, and future research.

Comment 3:The limitations of the study and the prospective of the study are missing.    
Response 3: Two paragraphs were added, one on limitations: "This study has several limitations. First, the sample size and duration of the experiment may not be sufficient to generalize the results to other populations or educational contexts. Second, the study was conducted at a university in Chile, which limits the generalizability of the findings to other cultural and educational settings. Furthermore, measuring variables such as motivation and perceived usefulness was based on self-reported surveys, which could introduce response biases." And another on future perspectives: "For future research, it is recommended to explore various gamification configurations and their impact on other demographic groups. It is crucial to develop personalized gamification strategies that consider gender differences and other individual factors to optimize educational outcomes. Furthermore, longitudinal studies could provide a more comprehensive view of the long-term impact of gamification on student learning and motivation. These additional investigations may help refine gamification techniques and make them more inclusive and effective for all students."

Comment 4:Finally, in the summary I would include the instrument used for the evaluation that has been used.    
Response 4: The requested addition to the abstract was made: "Learning gains were assessed through pre- and post-tests, motivational factors were measured via surveys, and usability perceptions were evaluated using the Technology Acceptance Model (TAM) survey."

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Comments in the attached file.

Comments for author File: Comments.pdf

Comments on the Quality of English Language

Some little typos are found along the manuscript. Please, check it carefully and correct them. There is a missing reference at page 5.

Author Response

Comment 1:Regarding the use of gamification in education, a recent study of Oliveira, Hamari et al. conducted a systematic literature review of studies with good statistical analysis to understand the used tools and how to design an efficient gamified activity. Moreover, some studies, analyzed the gamified activities in different sectors, such as: learning language (Wulantari et al.), science education (Kalogiannakis et al., Chiarello et al.), sustainability (Di Paolo et al.), computer science (Toccafondi et al.), and so on. A lot is done in the last years and a more updated literature might help.    
Response 1: Current references from 2023 and 2024 have been added. We were unable to access the last proposed reference. The main changes were applied in the sections "Use of gamification in education," "Gamification and Computer Programming," and "Differences in learning programming using gamification by gender."

Comment 2:Moreover, the research questions are separated from the literature, and it is not easily understandable how they
fill the gap in the current literature."    "
Response 2: Adjustments were made in the literature to ensure coherence. Changes were introduced in the sections ""Difficulty in learning programming"" and ""Differences in learning programming using gamification by gender.""

Comment 3:While the tools used into the course are well defined, the differences between the control and experimental (or treatment) groups are not clear. I understand that in the gamified group authors used the PBL techniques, it is not clear how students get points and what they gain. Moreover, authors described what Scores and Leaderboards are (including the relevant literature that, to my opinion, is better if it is described in the introduction), but not how they are used, visualized (publicly or privately) and how the possibility of repeated attempts is taken into consideration to calculate the final rank (more attempts, more points? Or it is an average?)    
Response 3: The explanation of the techniques used was moved to the "Gamification" section. Additionally, all the requested details about the use of techniques, differences between groups, score accumulation, and visualization methods, among others, were added to the "Software Tool" section.

Comment 4:Both groups used EducaPlay, a game-based learning tool (which is different from a gamification tool), to learn programming. Basically, both groups used a GBL tool which activate most of the interested variables studied by authors (motivation, enjoyment). This point is not well assessed by the authors, and it might be a cofounder factor in the analysis. Specifically, authors do not assess previously if students are gamers, how much they play games and some variables of interest to control for.    
Response 4: The focus of the study is on gamification techniques. We agree with the reviewer that the tool, by functioning as a game, may influence the results. Despite this, it is important to consider that both groups (control and experimental) used the same tool, with the difference being that the experimental group included gamified aspects, while the control group did not. These study limitations have been added to the conclusions. A section on limitations has also been added to the discussion.

Comment 5:More details are needed to understand the process: the timing of the activities (periods, months, days), whether the surveys were administrated simultaneously to both groups. Moreover, I’m wondering why the first survey is not administered before “moment 2”.    
Response 5: The requested explanations were provided in the "Process" section.

Comment 6:Considering the questionnaire, if I understood correctly, the questions were not the same before and after the end of the activities. Does it mean that also between the control and the experimental group the questions are not the same? Moreover, the problems that each student must solve in each attempt are always extracted randomly. These two parts are described together, and it was not so understandable to me. Please, explain better.    
Response 6: Clarifications were provided in the "Instruments" section.

Comment 7:I did not find a part describing if there are differences between students in the two groups. Is the result driven by the experimental setting or individual characteristics? Why not include individual characteristics in the pre-test questionnaire to see if they are statistically equivalent?    Response 7: This aspect has been added as a study limitation.

Comment 8:Moreover, I’m wondering if the two groups began the course with the same level of knowledge. There is no evidence about the use of pre test (a part using is as a covariate in the ANCOVA analysis). For example, you can take a Diff-in-Diff analysis given the fact you a have a pre-post test survey.    
Response 8: An ANCOVA test was conducted using pretest data as a covariate. This allows us to control for initial differences between groups and adjust the post-intervention results, enabling a more accurate comparison between groups. The results have been added.

Comment 9:I’m also wondering if having around 60 subjects gives enough statistical power for the analysis.    
Response 9: This observation has been addressed in the study's limitations.

Comment 10:Although the experiment is already done and the data collected, if authors do not have the possibility to collect more data, I believe they must assess the limitation of the study and take into consideration all the variables and the similarity/differences between groups.    
Response 10: It has been considered in the study's limitations.

Comment 11:To my opinion, the paper needs to be improved to be published. Moreover, the statements of the authors need to be smoothed given the limitations, which might be assessed in the conclusion of the study.    
Response 11: Phrases have been added and/or modified to provide nuance. Additionally, the study's limitations have been included, addressing the aspects mentioned above.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

I thank the authors for the opportunity to read this interesting study. Overall, the study has potential but, the purpose of the study in relation to past findings, study's contributions, etc. are unclear. The author(s) would need to provide a detailed literature review and enhance the method and discussion. I have provided my comments below, which I hope you find useful.

 

Overall comments: Strengths and weaknesses of the paper and suggestions for revision.

Strength of paper:
the strength of this paper lies on its timely information particularly the issue of gender differences in learning via gamification.

Weaknesses of the paper are:
1. The introduction section needs improving. It is predominantly focused on general gamification issues rather than the proper focus of the article indicated in both the title and the research questions. Gender issues should be significantly deepened in the introduction.

2. Breathe of literature could show citations for 2024 in the introduction section since this is missing. Gamification issues are developing rapidly. Much of the literature cited is now out of date, while new items are missing.

Containing significant work in the field: There are important works that have been left out particularly for 2024. The paper can be enhanced with below suggested literature of 2023-2024:

Calles-Esteban, F., Hellín, C. J., Tayebi, A., Liu, H., López-Benítez, M., & Gómez, J. (2024). Influence of Gamification on the Commitment of the Students of a Programming Course: A Case Study. Applied Sciences, 14(8), 3475. https://doi.org/10.3390/app14083475

 

Seal, D. B., & Bag, S. (2023). Effects of Gamified Learning on Academic Achievements: Does Gender Matter?. In Digital Technologies for Smart Business, Economics and Education: Towards a Promising Future (pp. 225-242). Cham: Springer International Publishing.

 

Duda, E., Anacka, H., Kowal, J., Nowakowska, I., Obracht-Prondzyńska, H., Geirbo, H. C., ... & Zawieska, J. (2023). Encouraging Pro-environmental Behaviour Through an Educational Mobile Application: Preliminary Insights from Early Adopters. International Journal of Pedagogy, Innovation and New Technologies, 10(1), 64-78. https://doi.org/10.5604/01.3001.0053.9400

 

3. The research sample is very small, which makes it all the more surprising that no data is given on the power or reliability of the tests carried out.

 

4. There is confusion in the data notation, with decimals separated by a full stop once and a comma once. In many places 0 is missing (required in line with APA). The authors should also unify the number of decimal places given.

5. In the discussion section, the Authors have shown citations. This is good but it was not at all mentioned in the earlier sections of introduction or methods. It is good to ensure all citations are introduced in your earlier sections of the paper prior to the discussion so as to allow the reader(s) connect the literature otherwise there is a literature disconnection.

Author Response

Comment 1: 1. The introduction section needs improving. It is predominantly focused on general gamification issues rather than the proper focus of the article indicated in both the title and the research questions. Gender issues should be significantly deepened in the introduction.    
Response 1: The introductory section has been improved, more literature has been added, and paragraphs have been rewritten to ensure coherence with the research questions, findings, and the article's title.

Comment 2: Breathe of literature could show citations for 2024 in the introduction section since this is missing. Gamification issues are developing rapidly. Much of the literature cited is now out of date, while new items are missing.    
Response 2: The literature has been updated by integrating articles from 2023 and 2024.

Comment 3: The research sample is very small, which makes it all the more surprising that no data is given on the power or reliability of the tests carried out.    
Response 3: The indicated points have been addressed in a new section called "Limitations."

Comment 4: There is confusion in the data notation, with decimals separated by a full stop once and a comma once. In many places 0 is missing (required in line with APA). The authors should also unify the number of decimal places given.    
Response 4: This issue has been corrected.

Comment 5: In the discussion section, the Authors have shown citations. This is good but it was not at all mentioned in the earlier sections of introduction or methods. It is good to ensure all citations are introduced in your earlier sections of the paper prior to the discussion so as to allow the reader(s) connect the literature otherwise there is a literature disconnection.    
Response 5: The references used in the discussion have been added to the upper sections. To do this, the contribution of the article was reviewed and integrated into the relevant sections.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

The paper presents an interesting topic, digging deeper into a gamification domain. A lot of presumptions are made in the field of gamification. Therefore, any experiment in the field adds to the existing body of knowledge. 

The authors provide a lot of related work and theoretical background. However, I propose that the sections that cover related work and background be joined into separate sections, not under Section 1. 

RQs should be presented earlier in the paper, at the end of the introduction. Also, the structure of the paper should be added at the end of the introduction section. 

The aim of the paper aligns with RQs and with observed values. Experimental design is well described, but, I propose a graphical representation of the methodology is added. 

The experiment consisted of two groups - experimental and control. However, in line 359, the authors claim that both groups use the EducaPlay tool. So, both groups were gamified? This is not clear and should be cleared further. 

The inserted TAM model (figure) should be replaced with the table. Providing a screenshot does not add any value. 

The discussion should be linked to RQs - so the answers are visible at first sight. 

 

Author Response

Comment 1: However, I propose that the sections that cover related work and background be joined into separate sections, not under Section 1.    
Response 1: A Chapter 1: Introduction has been created, which contains a preview, the research questions, and the structure of the document; and a Chapter 2: Literature Review, which includes the state of the art and related works.

Comment 2: RQs should be presented earlier in the paper, at the end of the introduction.    
Response 2: These have been placed at the end of the introduction, before the document structure.

Comment 3: Also, the structure of the paper should be added at the end of the introduction section.     Response 3: These have been placed at the end of the introduction.

Comment 4: El objetivo del artículo se alinea con las preguntas de investigación y con los valores observados.    
Response 4: A figure has been integrated (Figure 1).

Comment 5: The experiment consisted of two groups - experimental and control. However, in line 359, the authors claim that both groups use the EducaPlay tool. So, both groups were gamified? This is not clear and should be cleared further.    
Response 5: For the control group, access was granted without user registration, so their access and scores were not recorded. This was clarified in the rewritten explanation of gamification.

Comment 6: The inserted TAM model (figure) should be replaced with the table. Providing a screenshot does not add any value.    
Response 6: The questions have been included as part of the text.

Comment 7: The discussion should be linked to RQs - so the answers are visible at first sight.     Response 7: The responses to the research questions have been explicitly included: RQ1 is addressed in "5.1. Learning," RQ2 in "5.2. Motivational Perceptions," and RQ3 in "5.3. Perceptions of Ease of Use."

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

Authors made a good job to improve the manuscript, answering all my doubts. Thank you for your effort.

I only have minor comments:

- A part from Oliveira et al., I was unable to found any of the references suggested. However authors found some more updated references. I think some of those references are important to be in line with the current literature (I understand that the last reference, Toccafondi et al., was difficult to find, but all the others references are well identified in my last report).
I'm suspecting that authors insert the references in the text but not in the "Reference" section. If this is the case, fix it.

- In line 470, authors use "Time 5" or "Time 6" to describe the "moments" of the experiment. I believe using "Moments" as done before is better and is cleared to the readers.

Thanks for your work, I believe more studies in this direction are needed (with more participants would be better!)

Author Response

Comments 1: - A part from Oliveira et al., I was unable to found any of the references suggested. However authors found some more updated references. I think some of those references are important to be in line with the current literature (I understand that the last reference, Toccafondi et al., was difficult to find, but all the others references are well identified in my last report). I'm suspecting that authors insert the references in the text but not in the "Reference" section. If this is the case, fix it.B6
Response 1: 

From the suggested references, we have successfully integrated: Kalogiannakis, M., Papadakis, S., & Zourmpakis, A.-I. (2021). Gamification in Science Education. A Systematic Review of the Literature. Education Sciences, 11(1), Article 1. https://doi.org/10.3390/educsci11010022; Oliveira, W., Hamari, J., Shi, L., Toda, A. M., Rodrigues, L., Palomino, P. T., & Isotani, S. (2023). Tailored gamification in education: A literature review and future agenda. Education and Information Technologies, 28(1), 373–406. https://doi.org/10.1007/s10639-022-11122-4; Chiarello, F., Fioretti, D., Antognozzi, T., Bertani, F. R., Bilancini, E., Bisanti, M., Boncinelli, L., Businaro, L., Paolo, R. D., Gerardino, A., & Iurescia, S. (2023). The Language of Life: A Game-Based Workshop for Introducing Protein Biosynthesis. European Conference on Games Based Learning, 17(1), Article 1. https://doi.org/10.34190/ecgbl.17.1.1587; Wulantari, N. P., Rachman, A., Sari, M. N., Uktolseja, L. J., & Rofi’i, A. (2023). The Role Of Gamification In English Language Teaching: A Literature Review. Journal on Education, 6(1), Article 1. https://doi.org/10.31004/joe.v6i1.3328.
The changes have been highlighted in yellow.

Comments 2: - In line 470, authors use "Time 5" or "Time 6" to describe the "moments" of the experiment. I believe using "Moments" as done before is better and is cleared to the readers.
Response 2: The changes have been corrected and marked in yellow.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

Even though the manuscript has been revised extensively, the previous comments have not been taken into account. Unfortunately, I don't notice a significant improvement on the last draft. It is up to the Editors to agree to publish as is, but the article really needs a qualitative improvement.

1. The introduction section needs improving. It is predominantly focused on general gamification issues rather than the focus of the article indicated in both the title and the research questions. The introduction should be much more in-depth. It should focus on gender issues, side issues should be removed.

2. Gamification issues are developing rapidly. Much of the cited literature on gender issues is outdated and new items are missing.

3. The research sample is very small, which makes it all the more surprising that no data is given on the power or reliability of the tests carried out. Although it is described as a limitation of the study, the results section should begin with the exact power and reliability values of the tests. If statistical tools are used, it must be demonstrated that they are not manipulative.

4. There is confusion in the data notation, with decimals separated by a full stop once and a comma once. In many places 0 is missing (required in line with APA). The authors should also unify the number of decimal places given, which now is not in line with APA style.

Author Response

Comments 1:  1. The introduction section needs improving. It is predominantly focused on general gamification issues rather than the focus of the article indicated in both the title and the research questions. The introduction should be much more in- depth. It should focus on gender issues, side issues should be removed.

Response 1: The introduction has been rewritten, providing a focused and precise perspective, addressing the requested gender issues, and eliminating secondary matters.

Comments 2: 2. Gamification issues are developing rapidly. Much of the cited literature on gender issues is outdated and new items are missing.

Response 2: The suggested literature has been added, along with additional updated literature.

Comments 3: 3. The research sample is very small, which makes it all the more surprising that no data is given on the power or reliability of the tests carried out. Although it is described as a limitation of the study, the results section should begin with the exact power and reliability values of the tests. If statistical tools are used, it must be demonstrated that they are not manipulative.

Response 3: The requested statistical elements have been added to the results section.

Comments 4: There is confusion in the data notation, with decimals separated by a full stop once and a comma once. In many places 0 is missing (required in line with APA). The authors should also unify the number of decimal places given, which now is not in line with APA style.

Response 4: Corrected according to APA 7th edition standards

Author Response File: Author Response.pdf