A Cloud Game-Based Educative Platform Architecture: The CyberScratch Project^†

Round 1

Reviewer 1 Report

The submitted manuscript presents the developed game-based educative platform as a teaching tool that can suit any subjects. Even though the title is referred to cybersecurity, a reader could obviously be mislead since this subject is missing. 

I find the article well-written and almost ready for publication except for three minor issues:

1. Authors shall delete word Cybersecurity from the title. 
2. A sentence in line 26-27 and 152 should be self-explanatory, i.e., provide some examples/evidence and not to put readers in a situation to browse for the additional literature/references.
3. The language used in Figures should also be English.

Once this is completed, I consider the manuscript ready for publication.

Author Response

Comment:

The submitted manuscript presents the developed game-based educative platform as a teaching tool that can suit any subjects. Even though the title is referred to cybersecurity, a reader could obviously be mislead since this subject is missing.

I find the article well-written and almost ready for publication except for three minor issues:

1. Authors shall delete word Cybersecurity from the title.

Answer:

We agree with this suggestion. For this reason, the word Cybersecurity has been deleted from the title of the manuscript. The new title is “A Cloud Game-based Educative PlatformArchitecture: the CyberScratch Project”.

Comment:

2. A sentence in line 26-27 and 152 should be self-explanatory, i.e., provide some examples/evidence and not to put readers in a situation to browse for the additional literature/references.

Answer:

We agree with this suggestion. The new version of the manuscript includes relevant information about the mentioned references, in order to be self-explanatory, as suggested. Some of them have been moved to the new Background and Discussion section.

Comment:

3. The language used in Figures should also be English.

Once this is completed, I consider the manuscript ready for publication.

Answer:

We agree with this suggestion. Two figures have been removed according the suggestion of another reviewer. Language texts of figures are now in English, even, the new ones added, as suggested by other reviewers.

Reviewer 2 Report

• Is the CyberScratch project the main subject (or contribution) in this paper. or the overall discussion on Game-based Platforms Architectures (in the context of Cloud Computing) is a bolder topic?
• The Research Method is descriptive for the "LA Process" and is not quite a "clear" RM based on hypothesis and experimental or analytical validation.
• The CyberScratch Platform has a very interesting architecture (based on cloud services), but to be described as a research (engineering) artifact a more formal-modeling approach could be used in RM.
• A (more) user/client-oriented validation is necessary to prove the viability of the proposed platform. In the proposed architectural model there is an analytical component (namely "Cloud Game Monitor" with LA responsibilities) that could produce the analytical data to be used in the validation study.
• Overall, the text is readable, but still, there are some corrections to be done (e.g, page 9, line 285, Figure 4 reference is misleading - replace the reference with Figure 6).

Author Response

Comment:

Is the CyberScratch project the main subject (or contribution) in this paper. or the overall discussion on Game-based Platforms Architectures (in the context of Cloud Computing) is a bolder topic?

Answer:

A modular architecture (cloud-based) has been provided to allow the development and implementation of serious games for distance education. 

For this reason, the Abstract description has been improved, and additional explanations have been added into the Introduction section, to enforce this concept and the main objective of this work; the platform development and a use case (CyberScratch). It is essential to consider that issues related to data privacy management have been considered in the architecture design itself. This way, the architecture inherently provides this feature. Therefore, the game designer must only focus on designing the game instead of managing data privacy. For this reason, we emphasize so much this feature, not present in other platforms. More description of components and layers are included in Section 4.3 and how to use the architecture for other cases/games.

The Conclusions section has been reformulated in order to clarify the work done in the paper, and the main insights of this contribution.

Comment:

The Research Method is descriptive for the "LA Process" and is not quite a "clear" RM based on hypothesis and experimental or analytical validation.

Answer:

We are currently validating our cloud game-based educative platform architecture from the point of view of users (faculty and students). For this reason, we incorporate a new Section 4.4 about user satisfaction and an Appendix with an opinion survey based on the TAM methodology. The factors to be validated and their corresponding statements are included in it. The managed data during the LA process is the specified in Section 5 about data privacy with GDPR and user interactions with the ISO/IEC TR 20748-1:2016 standard. This section and the rest of the manuscript have been revised in order to clarify this point.

Comment:

The CyberScratch Platform has a very interesting architecture (based on cloud services), but to be described as a research (engineering) artifact a more formal-modeling approach could be used in RM.

Answer:

We agree with this suggestion. For this reason, the description about the CyberScratch Platform has been extended in Sections 4.1 and 4.3. First, Section 4.1 formalizes the pedagogical model and dynamics, even, including a game pattern diagram using Machinations for the game dynamics (Figure 2), as suggested by another reviewer. The game dynamics have also extended with a schematic representation of the game flow (Figure 5). On the other hand, Section 4.3 has also been extended with additional explanations about the cloud architecture.

Comment:

A (more) user/client-oriented validation is necessary to prove the viability of the proposed platform. In the proposed architectural model there is an analytical component (namely "Cloud Game Monitor" with LA responsibilities) that could produce the analytical data to be used in the validation study.

Answer:

The new version of the manuscript has been revised and extended to clarify how our cloud game-based platform architecture can help game designer (one of the roles of the client side), by avoiding managing data privacy and supporting a LA process with standards. These management issues will be transparent for users. The new Section 2.2 discusses motivation and privacy challenges. Section 4.1 also clarifies the pedagogical model and game dynamics.

The collected data is basically the recommended by the ISO/IEC TR 20748-1:2016 standard. Our main objective here is to establish a correlation about the LA phases and the GDPR rights. Figure 9 shows an example about “The right to data portability” characteristic.

Comment:

Overall, the text is readable, but still, there are some corrections to be done (e.g, page 9, line 285, Figure 4 reference is misleading - replace the reference with Figure 6).

Answer:

The manuscript has exhaustively been revised, and new additions have been included in red color. Additionally, some minor typos and English mistakes have been corrected.

Reviewer 3 Report

The paper describes the development of a game-based educational platform in the cloud for teaching cybersecurity related topics.

Comments:

1. The novelty and contribution of this paper should be explicitly stated in the introduction section.
2. The article lacks detailed analysis and comparison with the recent works such as “An interactive serious mobile game for supporting the learning of programming in JavaScript in the context of eco-friendly city management” and “Serious game iDO: Towards better education in dementia care”. How is your game different from other games known in the scientific literature?
3. The educational/pedagogical backgrounds of gamification and serious games must be presented.
4. The paper should address the development of serious games in more detail as the implementation has a great effect on the acceptance of the game and the results achieved. Discuss the methodological backgrounds of serious game development such as gamification patterns described in “Gamification Patterns for Gamification Applications”.
5. In the materials and methods section, there is not much information about the methodology related to the research. The authors should give more details of the methodology of game development. While the game is the focus, the article does not explain the game scenario and gamification techniques employed to support player engagement and motivation to play. Describe the game mechanics more formally using Machinations diagrams (https://machinations.io/)
6. Figures 4 and 5 are very informal and do not present enough details about architecture. Use a more formal diagram such as UML to present the architecture.
7. Figures 6 and 9: screenshot does not have much scientific value and is suggested to be removed from the paper.
8. The validation of the game is missing. The game should be evaluated by a user survey. You can use a well-established questionnaire such as the technology acceptance model (TAM) or a more game-oriented questionnaire such as the Technology-Enhanced Training Effectiveness Model (TETEM). Is the game more effective than traditional education methods?
9. The discussion section in this paper is missing Where is the discussion based on the bibliographic references? Present a critical discussion section and discuss the limitations of the serious game for education as well as any threats to the validity of the results.
10. Conclusions should be improved as it lacks insights and recommendations for further research in this domain. The claims are not supported. For example, “Gamification will help students to increase their motivation and engagement in practical activities.” I did not find the evaluation of gamification in this paper.

Author Response

Comment:

The paper describes the development of a game-based educational platform in the cloud for teaching cybersecurity related topics.

Comments:

1. The novelty and contribution of this paper should be explicitly stated in the introduction section.

Answer:

According to this suggestion, the new version of the manuscript has been clarified to include the novelty and contribution explicitly in the Abstract, Introduction and Conclusions sections. Additionally, the new version of the manuscript has been revised and extended according to the reviewers’ suggestions. All relevant changes are marked in red color.

Comment:

2. The article lacks detailed analysis and comparison with the recent works such as “An interactive serious mobile game for supporting the learning of programming in JavaScript in the context of eco-friendly city management” and “Serious game iDO: Towards better education in dementia care”. How is your game different from other games known in the scientific literature?

Answer:

We agree with this comment. The Background section has been revised and extended in order to highlight lack of previous works and the open challegues and contrubutions. They are related to our proposed modular and flexible architecture and transparent data privacy management during the LA process. GDPR regulations and ISO/IEC TR 20748-1:2016 standard are employed. These references have been included and revised to improve Section 4 in the new version of the manuscript.

Comment:

3. The educational/pedagogical backgrounds of gamification and serious games must be presented.

Answer:

We agree with this comment. Section 2 has been improved in this sense, and a new Sections 4.1.1 (Pedagogical model) and 4.1.2 (Game mechanics) have been included in the new version of the manuscript.

Comment:

4. The paper should address the development of serious games in more detail as the implementation has a great effect on the acceptance of the game and the results achieved. Discuss the methodological backgrounds of serious game development such as gamification patterns described in “Gamification Patterns for Gamification Applications”.

Answer:

We agree with this comment. Following with the previous comment and these guidelines, we have included the game pattern diagram of our cloud platform architecture (Figure 2) and the schematic representation of the game flow (Figure 5) in the new Sections 4.1.1 (Pedagogical model) and 4.1.2 (Game mechanics).

The proposed reference has been included and related with our current work in the new version of the manuscript.

Comment:

5. In the materials and methods section, there is not much information about the methodology related to the research. The authors should give more details of the methodology of game development. While the game is the focus, the article does not explain the game scenario and gamification techniques employed to support player engagement and motivation to play. Describe the game mechanics more formally using Machinations diagrams (https://machinations.io/)

Answer:

We agree with this comment. The whole Section 4 has been extended and improved to tackle these suggestions. Apart from this, a Game pattern diagram using Machinations (Figure 2).

Comment:

6. Figures 4 and 5 are very informal and do not present enough details about architecture. Use a more formal diagram such as UML to present the architecture.

Answer:

We agree with this comment. For this reason, more details about the architecture are presented in the Section 4 of the new version of the manuscript. More specifically, to make more formal this Section, Machinations is used, and a formal schematic representation of the game flow is presented.

Comment:

7. Figures 6 and 9: screenshot does not have much scientific value and is suggested to be removed from the paper.

Answer:

These figures have been removed in the new version of the manuscript.

Comment:

8. The validation of the game is missing. The game should be evaluated by a user survey. You can use a well-established questionnaire such as the technology acceptance model (TAM) or a more game-oriented questionnaire such as the Technology-Enhanced Training Effectiveness Model (TETEM). Is the game more effective than traditional education methods?

Answer:

The main objective of this paper is to develop a modular and flexible cloud game-based educative architecture for distance education for CyberScratch, by including the mentioned advantages of other proposals and minimizing their theoretical limitations found in the literature. As an additional benefit of our proposed cloud platform is that both GDPR regulations and ISO/IEC TR 20748-1:2016 standard about the LA process are considered. These objectives have been emphasized along the new version of the manuscript. The data management is performed in a transparent way from the point of view of the game editor, faculty, and so on.

In addition to this, we are currently immersing in the process of evaluation with a TAM approach, from both the points of view of faculty and students. The TAM survey is included as an Annex in the new version of the manuscript, considering the perceived ease of use, usefulness, attitude towards the platform, and intention to use. We have experience in proposing and validating TAM and UTAUT models for new technologies for distance education, so analyzing their goodness in distance education.

Comment:

9. The discussion section in this paper is missing Where is the discussion based on the bibliographic references? Present a critical discussion section and discuss the limitations of the serious game for education as well as any threats to the validity of the results.

Answer:

We agree with this comment, although some discussion is already detailed in the background section. We have revised this Background section to clarify the limitations of serious games for education, as well as justify the necessity of proposing a cloud game-based platform for the CyberScratch project. According to this, the Abstract and Introduction section has also been revised. In addition to this, as previously explained, the platform is being evaluating with TAM. A Section 4.4 has been included in the new version of the manuscript and the TAM survey has been incorporated to the manuscript as an Annex.

Comment:

10. Conclusions should be improved as it lacks insights and recommendations for further research in this domain. The claims are not supported. For example, “Gamification will help students to increase their motivation and engagement in practical activities.” I did not find the evaluation of gamification in this paper.

Answer:

We agree with this comment. We have refactored the paper's Conclusions section to clarify the main contributions and further research related to the paper. At the end of the section, future work has been included. Additionally, among the new bibliographic references included in this version, one of them justifies the increment of motivation and engagement claim: 

Jayalath, J.; Esichaikul, V. Gamification to Enhance Motivation and Engagement in Blended eLearning for Technical and Vocational Education and Training. Technology, Knowledge and Learning. 2020. Doi:10.1007/s10758-020-09466-2.

This reference has also been incorporated in the Introduction section, to justify the claim about engagement and motivation. So, in the final version of paper maybe we can remove this reference in the Conclusions section.

Apart from this, another contribution of our work concerning data privacy management is establishing a correlation about the LA phases and the GDPR rights, by following the ISO/IEC TR 20748-1:2016 standard. Our cloud game-based platform can autonomously manage data, without the intervention of the game editor, faculty and so on.

Round 2

Reviewer 2 Report

Maybe the research design and method could be slightly improved, but, overall, the article quality endorses publication.

 

Reviewer 3 Report

The article was revised according to all my comments and the quality of description and presentation has improved.

I have no further comments and recommend the article to be accepted.