Modern Aspects of Cyber-Security Training and Continuous Adaptation of Programmes to Trainees

Round 1

Reviewer 1 Report

The paper describes a setting for cyber-security trainings. It claims to combine Bloom's taxonomy with STRIDE and cyber-ranges. All three topics are not new and their combination is straight forward and especially the observance of Bloom's taxonomy should be a matter of course for any good training. And the integration of STRIDE is not explained enough. How STRIDE influences the training setting remains unclear.

The author claim to talk about a "formalism called Cyber threat and training preparation (CTTP) modelling". But there is no formalism whatsoever in the paper - at least in a scientific sense. In my opinion this is the main problem with the paper: It is not a scientific paper. It seems that some people had some straight forward ideas about setting up a good cyber-security training, which is not a bad thing at all, but it contains nothing new. Many of the features described in chapter 2 are offered by various vocational training platforms.

The paper does not cover the state of the art in a satisfactory way. There are no references to similar models od training.

Another problem is the description of the Smart Shipping Use Case: it is very superficial and one does not understand how the systems work, what user groups there are and what their specific training needs are. And hence, one cannot understand, why and how the CPPT model will work in this environment. For example the list of risks to be covered by the training at the end of chapter 3 is very general and generic and could be used in any edp-environment. By the way: a good training should not only cover the risks, but countermeasures, too. Fig. 12 at the end of chapter 3 is not good for anything, because you can read nothing except the main heading.

Chapter 4 does not deserve to be called "Discussion". There are no advantages and disadvantages discussed; no comparisons with other methods are made; and it is really short.

Chapter 5, Conclusions, gives a very short summary of the work and repeats an idea for future extensions from chapter 4.

Sorry, but this paper does not fulfill the requirements of a scientific paper.

Author Response

The detailed response for the overall review process can be found in the attacked PDF file.

Reviewer #1

• The paper describes a setting for cyber-security trainings. It claims to combine Bloom's taxonomy with STRIDE and cyber-ranges. All three topics are not new and their combination is straight forward and especially the observance of Bloom's taxonomy should be a matter of course for any good training. And the integration of STRIDE is not explained enough. How STRIDE influences the training setting remains unclear.

In the new version, we have better described all the pedagogical aspects of our proposal and how they are integrated in the overall THREAT-ARREST approach. The role of STRIDE is also better explained. It is used in order to define the security objectives that we want to cover in a training programme (e.g. knowledge on specific threats and related countermeasures). The completion of the underlying tasks for all the layers of the Bloom’s taxonomy reveal the level of understanding that the trainee has accomplished for these security features.

Although the methods are mainstream and not novel, the research effort lays in the integration in the platform and the materialization of a dynamic training adaptation which takes into consideration the pedagogical perspective and drives the skill development features in an automated fashion.

• The author claim to talk about a "formalism called Cyber threat and training preparation (CTTP) modelling". But there is no formalism whatsoever in the paper – at least in a scientific sense. In my opinion this is the main problem with the paper: It is not a scientific paper. It seems that some people had some straight forward ideas about setting up a good cyber-security training, which is not a bad thing at all, but it contains nothing new. Many of the features described in chapter 2 are offered by various vocational training platforms.

We have better describe the overall CTTP aspects, the modelling phases, and their exploitation for the implementation of the dynamic adaptation functionality (e.g. subsection 2.3). A novel feature of THREAT-ARREST is the post-training evaluation. Thus, for the full application of the platform in a piloting environment, we continue auditing the trainees after the training and try to figure out if the really apply what they have learnt in the actual system. Feedback from this operation is utilized for the evaluation of the training programme and deployed educational modules.

• The paper does not cover the state of the art in a satisfactory way. There are no references to similar models of training.

The literature review has been significantly improved. The subsection 2.2 is now surveying the landscape for cyber-security training. Similar methodologies for security programmes are also overviewed.

• Another problem is the description of the Smart Shipping Use Case: it is very superficial and one does not understand how the systems work, what user groups there are and what their specific training needs are. And hence, one cannot understand, why and how the CPPT model will work in this environment. For example, the list of risks to be covered by the training at the end of chapter 3 is very general and generic and could be used in any edp-environment. By the way: a good training should not only cover the risks, but countermeasures, too. Fig. 12 at the end of chapter 3 is not good for anything, because you can read nothing except the main heading.

We devote the Section 4 in order to discuss solely the establishment of the training programme for the smart shipping pilot. Specifically, we followed the structure that was proposed by the Reviewer 2 and we are now describing the procedures for: i) description of the training programme, ii) learning outcome of the training module, iii) teaching and learning strategies, iv) student participation, v) overview of assessments and training levels, vi) study plan (learning schedule), vii) resources required to complete the training, and viii) bench marking of the module.

Figure 12 (now Figure 9) is better visualized and Table 2 was added explaining in more details the main scenarios that have been implemented so far.

• Chapter 4 does not deserve to be called "Discussion". There are no advantages and disadvantages discussed; no comparisons with other methods are made; and it is really short. Chapter 5, Conclusions, gives a very short summary of the work and repeats an idea for future extensions from chapter 4.

The discussion section (now Section 5) has been updated discussing the features of modern cyber-ranges platforms as well as the potential extension of the dynamic training adaptation techniques with machine learning.

• Sorry, but this paper does not fulfill the requirements of a scientific paper.

We have further detailed our proposed framework for dynamic training adaptation. We have better highlighted the need for embodying pedagogical methods in cyber-security courses and their exploitation for the materialization of a dynamic adaptation mechanism that guide the development of cyber-security skills. All these features are incorporated in the THREAT-ARREST cyber-ranges platform, benefiting from the model-driven design approach which is actually enabling the realization of the overall concept in an efficient manner. With this approach we can produce dynamically a high volume of diverse scenarios and cover the educational adaptation needs for various trainee types.

Author Response File: Author Response.pdf

Reviewer 2 Report

The paper works on building the process of dynamic adaptation of cyber-security training programmes. 

The problem formulation was well defined and I can see the benefit of introducing cyber training and indeed it an essential process for the lifelong personnel education in organizations.

I do have two major recommendations for this paper which will add more benefit to the reader.

1: Literature based on teaching pedagogy in the field of cyber awareness need to be improved. It will be nice if we can have bit more understanding of how the existing training programs delivers their learning materials and how they achieved their learning outcome set. 

A detailed framework is missing in the paper. While proposing the framework I request the authors to include the following

• Description of the training program
• Learning outcome of the training module. 
• Teaching and learning strategies
• Student participation
• Overview of assessments
• Study plan - (learning schedule)
• Resources required to complete the training
• Bench marking of the module
• The framework should also capture the training level starting from beginner to advanced.

Author Response

The detailed response for the overall review process can be found in the attacked PDF file.

Reviewer #2

The paper works on building the process of dynamic adaptation of cyber-security training programmes. The problem formulation was well defined and I can see the benefit of introducing cyber training and indeed it an essential process for the lifelong personnel education in organizations. I do have two major recommendations for this paper which will add more benefit to the reader.

• • Literature based on teaching pedagogy in the field of cyber awareness need to be improved. It will be nice if we can have bit more understanding of how the existing training programs delivers their learning materials and how they achieved their learning outcome set.

The related works concerning pedagogical and teaching aspects have been improved. Section 2 is now discussing these issues, presents state-of-the-art approaches, and compares the results with our method.

• A detailed framework is missing in the paper. While proposing the framework I request the authors to include the following:

-Description of the training program

-Learning outcome of the training module.

-Teaching and learning strategies

-Student participation

-Overview of assessments

-Study plan - (learning schedule)

-Resources required to complete the training

-Bench marking of the module

-The framework should also capture the training level starting from beginner to advanced.

We better present our framework, explaining in more detail the referred modules. Specifically, we are now describing the procedures for: i) description of the training programme, ii) learning outcome of the training module, iii) teaching and learning strategies, iv) student participation, v) overview of assessments and training levels, vi) study plan (learning schedule), vii) resources required to complete the training, and viii) bench marking of the module.

Author Response File: Author Response.pdf

Reviewer 3 Report

This paper describes an interesting platform for cybersecurity e-learning.

I have only a comment: There are so much similarities with a previous publication, cited as [20]. Some paragraphs (including complete sections) could be reduced (or eliminated) using that article as main reference.This fact could increase space to explain better some of the practical examples.

Author Response

The detailed response for the overall review process can be found in the attacked PDF file.

Reviewer #3
This paper describes an interesting platform for cybersecurity e-learning.

• • I have only a comment: There are so much similarities with a previous publication, cited as [20]. Some paragraphs (including complete sections) could be reduced (or eliminated) using that article as main reference. This fact could increase space to explain better some of the practical examples.

As the reviewer states, we are not referring to the previous paper. We reduce the relevant content, leaving only the main discussion points. We have added several new subsections presenting practical issues concerning our method, pedagogical and teaching perspectives, etc. Refer to our previously-mentioned answers for more details.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The new version of the paper is a coniderable improvement.

I would suggest the following (minor) improvements:

a) Do not use the term "formalism"; what you suggest is only slightly formal (in the strict sense of the word as generally used in computer science). May be the term methodology fits better.

b) In the shipping use case - which is better described now - it would be good, if you could exemplify the use of the STRIDE model (section 4.3).  Giving one or two examples of the STRIDE results would be valuable.

c) It is not clear whether the system has been tested with real users and what are the outcomes. Maybe just state that a real world test is still pending.

Author Response

The new version of the paper is a considerable improvement.

I would suggest the following (minor) improvements:

• Do not use the term "formalism"; what you suggest is only slightly formal (in the strict sense of the word as generally used in computer science). May be the term methodology fits better.

As the reviewer stated, we changed the wording concerning the CTTP modelling from “formalism” into “methodology” to better describe the overall concept.

• In the shipping use case - which is better described now - it would be good, if you could exemplify the use of the STRIDE model (section 4.3). Giving one or two examples of the STRIDE results would be valuable.

We provided an example of the STRIDE model in our framework and the Bloom taxonomy in our framework, in the subsection 4.5.1. There, we detailed an example of a social engineering scenario, that involves the ‘tampering’ and spoofing’ threats. We also define the training process for learning the adequate defenses and achieve the related ‘integrity’ and ‘authentication’ security properties, respectively.

• It is not clear whether the system has been tested with real users and what are the outcomes. Maybe just state that a real world test is still pending.

We better clarified in the text (at the end of the discussion section) that the platform is currently under evaluation and actual training sessions with real employees from the shipping company are conducted this summer.

Author Response File: Author Response.pdf

Reviewer 2 Report

I congratulate the authors for bringing such a major change in the paper.

I am happy to see my concerns addressed nicely.

Author Response

Thank you for your time and effort and the feedback that you provided to us.

Author Response File: Author Response.pdf