Training and Certification of Competences through Serious Games
Round 1
Reviewer 1 Report
Comments and Suggestions for Authors1.The abstract should explain how the results obtained through the Triadic Certification Method (TCM) are sufficient to represent the participants' competencies.
2.In the "1. Introduction" section, appropriate references should be added. For example, "Many studies have explored the use of serious games for skill certification, and game design needs to consider learning objectives and performance indicators." Which studies are these? How do these studies use serious games for skill certification?
3.In the "1. Introduction" section, it is recommended to list the four guiding key questions in a bullet-point format for clarity.
4.In the "2.3.1. Assessment of digital learning" section, the RETAIN framework (Relevance, Embedding, Transfer, Adaptation, Immersion, and Naturalization) is mentioned but not further explained. It should be elaborated on or related to subsequent content.
5.In the "2.3.1. Assessment of digital learning" section, Gagné's Nine Events of Instruction model is mentioned. What is this model?
6.The "3. Training, Competences, and Certification" section does not seem very necessary. Can you explain why this section is included?
7.In the "4.1. Relationship between game taxonomy and competencies development survey" section, decision and problem-solving quality (SS3), technical learning (SS8), organizational competence (OpS5), timely decision making (OpS9), results-oriented capabilities, action orientation (R1), and results mobility (R2) are mentioned. Where do these abbreviations or codes come from? This should be clarified.
8.There is a numbering error for the figures; Figure 2 is missing.
9.In the "4.2. Design of Triadic Certification Method" section, it is mentioned that the last step includes summarizing players' performance data in the game and comparing it with predefined certification standards. Who designed these predefined certification standards, and do they have credibility?
10.The certification method of this study seems more suitable for skill cultivation, but many gamified learning processes are subject-based. Does this study have limitations in this regard? What necessary factors are required to achieve the certification method proposed in this study?
11.The study could include more statistical analyses, such as significance tests, to support the validity and reliability of the results.
12.It is recommended to check the timeliness of the references cited and supplement with the latest relevant research to ensure the foundational relevance and cutting-edge nature of the study.
13.The conclusion should highlight the innovative aspects of the research and distinguish it from existing literature, emphasizing the study's uniqueness and academic value.
14.This study appears to have traces of being written by AI.
Author Response
Firstly, the authors would like to express their deep gratitude to the reviewers and editor for their time in suggesting improvements to the article. Corrections made were highlighted in red in the resubmitted manuscript
Comments 1: The abstract should explain how the results obtained through the Triadic Certification Method (TCM) are sufficient to represent the participants' competencies.
Response 1: We include the application case issue in the abstract, which has enabled us to confirm the effectiveness of the driving training and certification component.
Comments 2: In the "1. Introduction" section, appropriate references should be added. For example, "Many studies have explored the use of serious games for skill certification, and game design needs to consider learning objectives and performance indicators." Which studies are these? How do these studies use serious games for skill certification?
Response 2: We've added studies that already systematically address this issue in the use of serious games, both in design and development as well in evaluation, mostly resulting in the creation of frameworks.
Comments 3: In the "1. Introduction" section, it is recommended to list the four guiding key questions in a bullet-point format for clarity.
Response 3: The four key questions were structured as suggested.
Comments 4: In the "2.3.1. Assessment of digital learning" section, the RETAIN framework (Relevance, Embedding, Transfer, Adaptation, Immersion, and Naturalization) is mentioned but not further explained. It should be elaborated on or related to subsequent content.
Response 4: In line with the recommendation, the necessary explanations have been added to frame/explain the frameworks referenced and thus contextualize the state of the art of previous approaches related to the assessment of learning through games.
Comments 5: In the "2.3.1. Assessment of digital learning" section, Gagné's Nine Events of Instruction model is mentioned. What is this model?
Response 5: Following the recommendation, the explanations considered necessary to frame/explain the model and its context regarding the assessment of learning carried out through games have been added.
Comments 6: The "3. Training, Competences, and Certification" section does not seem very necessary. Can you explain why this section is included?
Response 6: Given the question posed, it's important to keep this chapter because it's the topic where we provide the theoretical framework for the words in the article's title. It represents the real context in which the purposes of the research are carried out with applied serious games. I think it's an important section to keep because it defines a set of necessary concepts such as skills, aptitudes and knowledge that are then framed with the competencies to be trained and show the results, measuring this learning and results through reference matrices recognised by scientific literature and in the context of training and education.
Comments 7: In the "4.1. Relationship between game taxonomy and competencies development survey" section, decision and problem-solving quality (SS3), technical learning (SS8), organizational competence (OpS5), timely decision making (OpS9), results-oriented capabilities, action orientation (R1), and results mobility (R2) are mentioned. Where do these abbreviations or codes come from? This should be clarified.
Response 7: It has already been mentioned that this terminology corresponds to identifying the competencies of each category in Table 3. Educational Competency Wheel was also in the explanatory text preceding Table 3.
Comments 8: There is a numbering error for the figures; Figure 2 is missing.
Response 8: All the text's image numbering, captions and call-outs have been corrected.
Comments 9: In the "4.2. Design of Triadic Certification Method" section, it is mentioned that the last step includes summarizing players' performance data in the game and comparing it with predefined certification standards. Who designed these predefined certification standards, and do they have credibility?
Response 9: The last step of the method allows you to build a visual model that fits the training elements (competencies) with the mechanics to be used based on an actual training plan. This training or learning plan is validated by specialists in the area to be certified in the second step of the method, who analyse the training scenarios and design the entire sequential training structure based on the certification standards to be applied. The basis for creating these training plans is always guaranteed recognition and credibility by the expert who collaborates with their knowledge as a recognised professional in the area in question.
In this way, we can guarantee that the actions, tasks, and learning activities are necessary and sufficient to achieve the certification objectives based on the results obtained.
Comments 10: The certification method of this study seems more suitable for skill cultivation, but many gamified learning processes are subject-based. Does this study have limitations in this regard? What necessary factors are required to achieve the certification method proposed in this study?
Response 10: While the method proposed in this study is indeed generic, it therefore must be customized to each individual subject and implementation to effectively address the specific requirements and learning objectives of different educational contexts. This involves considering the alignment with the learning objectives and competences to be developed, a design of subject-specific tasks and mechanics and integrating adaptive feedback mechanisms.
Comments 11: The study could include more statistical analyses, such as significance tests, to support the validity and reliability of the results.
Response 11: The study presented a more methodological objective with the Triadic Certification methodology and its theoretical underpinning in state of the art, where a gap in knowledge was identified to obtain learning results with in-game evaluation in serious games. For this reason, the article presented was written around its rationale and structure as an application process, with the main objective being an in-depth discussion of the results, as seen in the doctoral dissertation.
Comments 12: It is recommended to check the timeliness of the references cited and supplement with the latest relevant research to ensure the foundational relevance and cutting-edge nature of the study.
Response 12: Some new references to more recent studies have been added to frame the analyses and results obtained to validate and justify some of the decisions. They were included in the introduction (point 1) and points 2.3.1 and 2.3.2, most of which are frameworks both in the design and development and in the evaluation of serious games and are also referred to and used in research after this work.
Comments 13: The conclusion should highlight the innovative aspects of the research and distinguish it from existing literature, emphasizing the study's uniqueness and academic value.
Response 13: We highlight not only the results obtained in answering the questions raised but also how research can be distinctive as a presentation of a method and innovation as a replication of good practices carried out as a basis for support.
Comments 14: This study appears to have traces of being written by AI.
Response 14: The corresponding author is not a native English speaker, so to avoid some writing fluency issues, Grammarly was used to improve the text and overcome this situation.
Reviewer 2 Report
Comments and Suggestions for AuthorsThe idea of making serious games more relevant to educational goals through Certification of Competences looks interesting and promising. The implementation no longer looks so rosy, despite numerous (and clearly unnecessary) statements from the authors that this is "a significant advancement".
"Design of Triadic Certification Method" is an absolutely ordinary selection of competencies and tasks that correspond to them. Yes, it's needed to know which tasks develop which competencies - but this is a trivial idea. There are a lot of articles about the design of serious games, where this topic is covered very fully. Moreover, the idea that already AFTER “determining the most suitable actions within the game to achieve the learning objectives" you can choose the genre of the game shows, it looks more than strange. It is very interesting how the author would develop in the given example, for example, role-pelying.
Figure 4 is difficult to read, especially the top row (even if it is implied by the example, there cannot be unreadable words), and is not very clear. The most incomprehensible part is on the right - “heritage area", etc. is not explained in any way in the text.
It is not clear why, when analyzing the practical application of certification, what the article is devoted to is not considered in detail. Tasks, situations, growth in the level of competencies, etc., but simply some incomplete statistics are given on the results of passing the test. In this case, it is important to show the process, not the result. Since the purpose of the article is to show that the result is truly based on competencies.
Author Response
Firstly, the authors would like to express their deep gratitude to the reviewers and editor for their time in suggesting improvements to the article. Corrections made were highlighted in red in the resubmitted manuscript
Comments 1: The idea of making serious games more relevant to educational goals through Certification of Competences looks interesting and promising. The implementation no longer looks so rosy, despite numerous (and clearly unnecessary) statements from the authors that this is "a significant advancement".
"Design of Triadic Certification Method" is an absolutely ordinary selection of competencies and tasks that correspond to them. Yes, it's needed to know which tasks develop which competencies - but this is a trivial idea. There are a lot of articles about the design of serious games, where this topic is covered very fully. Moreover, the idea that already AFTER “determining the most suitable actions within the game to achieve the learning objectives" you can choose the genre of the game shows, it looks more than strange. It is very interesting how the author would develop in the given example, for example, role-pelying.
Response 1: Thank you for your comment. However, the assertion that there are numerous articles about the design of serious games addressing this topic fully is somewhat misleading. While many articles discuss the connection between game actions and learning objectives, there are no established procedures that drive the selection of game mechanics following a desired set of competences. The Triadic Certification Method offers a more structured and validated approach to measuring competencies, incorporating both qualitative and quantitative metrics. By focusing first on the most suitable actions to achieve learning objectives, the method allows for a more tailored and effective learning experience. This approach ensures that the learning objectives drive the game design, rather than constraining it within a predetermined genre. And this is valid for RPG or any other game genre.
We do agree that there was an excessive number of occurrences of the term “significant” and removed most of them.
Comments 2: Figure 4 is difficult to read, especially the top row (even if it is implied by the example, there cannot be unreadable words), and is not very clear. The most incomprehensible part is on the right - “heritage area", etc. is not explained in any way in the text.
Response 2: The images have now been linked to the text, and some complementary and explanatory information about the simulation room's context has been added.
Comments 3: It is not clear why, when analyzing the practical application of certification, what the article is devoted to is not considered in detail. Tasks, situations, growth in the level of competencies, etc., but simply some incomplete statistics are given on the results of passing the test. In this case, it is important to show the process, not the result. Since the purpose of the article is to show that the result is truly based on competencies.
Response 3: The article does indeed delve into certification, but it places a significant emphasis on training as well and it explores the tasks, situations, and competency growth involved in the training process, providing a comprehensive overview of how these elements contribute to the development of competencies.
The presented statistics are supported by a discussion of the underlying processes and it demonstrates how the training leads to competency development, making it clear that the certification results are truly based on these competencies. By doing so, it establishes a linkage between the training process and the certification outcomes, effectively proving that the results are grounded in the competencies developed during training.
Round 2
Reviewer 1 Report
Comments and Suggestions for AuthorsThank you for your thorough and thoughtful responses to the reviewers' comments. The revisions and additions made to address the reviewers' comments have significantly improved the clarity and depth of the manuscript. For the abstract, it would be beneficial to explicitly state the key findings that support the effectiveness of the Triadic Certification Method (TCM) in demonstrating participants' competencies. In the "1. Introduction" section, the added references are helpful. Consider providing a brief summary of how each referenced study uses serious games for skill certification to give readers a better understanding. Listing the four guiding key questions in a bullet-point format as suggested is clear and effective.
In the "2.3.1. Assessment of digital learning" section, the additional explanations for the RETAIN framework help contextualize it. Ensure these explanations are directly linked to your study's objectives and findings. The explanation of Gagné's Nine Events of Instruction model is clear. It would be helpful to briefly discuss how this model applies specifically to your study's assessment methodology.
For the "3. Training, Competences, and Certification" section, the justification for keeping this section is well-argued. In the "4.1. Relationship between game taxonomy and competencies development survey" section, mentioning the terminology's origin in Table 3 is good. Ensure that each abbreviation and code is defined the first time it appears in the text for better readability.
Thank you for correcting the figure numbering errors. Double-check the entire manuscript to ensure all figures and references are correctly numbered.
In the "4.2. Design of Triadic Certification Method" section, the explanation regarding the validation by specialists adds credibility. Make sure this process is described in detail in the methods section to enhance transparency. Discussing the customization of the certification method for different subjects is useful. It may help to provide a specific example of how the method could be adapted to a different subject area.
While the methodological focus is clear, adding a brief discussion on potential future statistical analyses could strengthen the manuscript's implications for broader application. The addition of recent studies to check the timeliness and relevance of cited references is commendable. Ensure that the most critical references for your arguments are up-to-date to maintain the study's cutting-edge relevance.
The revisions made to the conclusion effectively emphasize the study's uniqueness and academic value. Addressing the concern regarding traces of AI writing by acknowledging the use of Grammarly for language improvement is understandable. It might be beneficial to include a brief acknowledgment of this in the manuscript to address potential concerns upfront.
Thank you for your hard work in revising the manuscript. These aim to further enhance the clarity, credibility, and impact of your study.
Author Response
Firstly, the authors would like to express their deep gratitude to the reviewers and editor for their time in suggesting improvements to the article. Corrections made were highlighted in red in the resubmitted manuscript
Comment 1: Thank you for your thorough and thoughtful responses to the reviewers' comments. The revisions and additions made to address the reviewers' comments have significantly improved the clarity and depth of the manuscript. For the abstract, it would be beneficial to explicitly state the key findings that support the effectiveness of the Triadic Certification Method (TCM) in demonstrating participants' competencies. In the "1. Introduction" section, the added references are helpful. Consider providing a brief summary of how each referenced study uses serious games for skill certification to give readers a better understanding. Listing the four guiding key questions in a bullet-point format as suggested is clear and effective.
Response 1: Thank you for your comment. We have presented some conclusions regarding the tools that have been implemented in response to the questions posed and a reference idea of the studies presented in this section.
Comment 2: In the "2.3.1. Assessment of digital learning" section, the additional explanations for the RETAIN framework help contextualize it. Ensure these explanations are directly linked to your study's objectives and findings. The explanation of Gagné's Nine Events of Instruction model is clear. It would be helpful to briefly discuss how this model applies specifically to your study's assessment methodology.
Response 2: In this section, two contexts are presented separately: firstly, how learning assessment is carried out, particularly in game-based learning, by identifying some assessment structures/frameworks; secondly, the identification of references of valid assessment design approaches in serious games to be able to verify other good practices/systematisation of the assessment process in games. The aim is to present a validated part of the ecosystem of the use of serious games for the assessment of game-based learning. Still, its analysis focuses on the structuring process rather than its aspects, such as the motivational component, which still needs to be addressed in this study.
Comment 3: For the "3. Training, Competences, and Certification" section, the justification for keeping this section is well-argued. In the "4.1. Relationship between game taxonomy and competencies development survey" section, mentioning the terminology's origin in Table 3 is good. Ensure that each abbreviation and code is defined the first time it appears in the text for better readability.
Response 3: Thank you for your comment.
Comment 4: Thank you for correcting the figure numbering errors. Double-check the entire manuscript to ensure all figures and references are correctly numbered.
Response 4: Thank you for your comment and recommendation. I've double-checked the figures and references in the manuscript.
Comment 5: In the "4.2. Design of Triadic Certification Method" section, the explanation regarding the validation by specialists adds credibility. Make sure this process is described in detail in the methods section to enhance transparency. Discussing the customization of the certification method for different subjects is useful. It may help to provide a specific example of how the method could be adapted to a different subject area.
Response 5: Following the recommendation, the above section has briefly described the algorithm applied to a context other than the case study.
Coment 6: While the methodological focus is clear, adding a brief discussion on potential future statistical analyses could strengthen the manuscript's implications for broader application. The addition of recent studies to check the timeliness and relevance of cited references is commendable. Ensure that the most critical references for your arguments are up-to-date to maintain the study's cutting-edge relevance.
Response 6: Thank you for your comment. I've added a brief description of validating the results with various statistical tools that enabled the following results to be presented as suggested.
Comment 7: The revisions made to the conclusion effectively emphasize the study's uniqueness and academic value. Addressing the concern regarding traces of AI writing by acknowledging the use of Grammarly for language improvement is understandable. It might be beneficial to include a brief acknowledgment of this in the manuscript to address potential concerns upfront.
Response 7: Thank you for your comment.
Comment 8: Thank you for your hard work in revising the manuscript. These aim to further enhance the clarity, credibility, and impact of your study.
Response 8: Thank you for your feedback.
