A Study of the Ordinal Scale Classification Algorithm for Cyber Threat Intelligence Based on Deception Technology
Round 1
Reviewer 1 Report
This paper addresses an important problem of how to "sort the wheat from chaff" in cybersecurity data collected in a honeypot. The english language in the paper is a little stilted, but quite understandable, so generally OK. The main issue is a tendency to introduce technical terms/concepts without explanation, which inhibits understanding by readers not intimately familiar with the subject area. I note a few of these below.
My biggest beef is with the mathematical presentation, which is particularly confused. I accept that naive bayes is somehow equivalent with the max log likelihood method, but I think the presentation would be clearer if expressed in terms of max log likelihood.
My notes from reading the paper:
Line 27: spell out CTI on first use (Computer Threat Intelligence).
Deception technology, and deceptive environment not really
defined. I'm assuming the authors mean the use of something like honey
pots to monitor cyber security threats.
What is "ordinal scale"? I think what is mean is a priority value
assigned to each attack source, with the idea to filter out low
priority attacks from the data.
Table 1 has a Korean column header. Replace with "Target"
Lemmas 1-2 These are more models than lemmas. The purported proofs are
more handwavy justifications for the models. I think it more honest to
present these as "conjectures" or "models" of the situation.
Note in Lemma 1, you haven't described what x is, or how the mean μ
and standard deviation σ are calculated.
In lemma 2, the right hand side of (20 does not depend on fáµ¢, except
in as much as the destination set Dáµ¢ depends on fáµ¢. I could imagine a
situation where dâ‚€ ∈ Dáµ¢ has a lot of flows, from a lot of sources, but
is relatively weakly important to the attacker sáµ¢.
Lemma 3 - don't you mean ... is of a low-risk ordinal scale...?
Eq (6) - do you mean P_{low}?
I get that we expect the flows to different destinations to be
independent, so the product rule will apply, but I don't get the
overall factor of 1/|Dáµ¢|? Your explanation doesn't make sense.
Similarly in the second equation (5) - please number your equations
uniquely and consecutively - your explanation of the normalisation
coefficient 1/((e-1)|D_i|!) doesn't make sense.
In terms of algorithm 1 - surely the correct approach should be to
calculate the log likelihood of your source data for each model distribution?
The distribution with higher log likelihood should indicate whether
the threat is low or high.
The english language in the paper is a little stilted, but quite understandable, so generally OK. The main issue is a tendency to introduce technical terms/concepts without explanation, which inhibits understanding by readers not intimately familiar with the subject area.
Author Response
Dear Reviewer,
We are grateful for your insightful and constructive feedback on our paper, "A Study of the Ordinal Scale Classification Algorithm for Cyber Threat Intelligence Based on Deception Technology." We appreciate the time and effort that you have invested in reviewing our manuscript and providing your valuable insights. I have attached a separate response to the comments you provided.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Sun Mo Yoo and and co-authors proposed a Naive Bayesian (NB) discriminant analysis-based ordinary scale classification model to quickly classify low-risk/high-risk attacks, while consuming less computation/storage resources.
The main assumption in authors' NB model is that the risk level can be modeled by "the distribution of attack sources', namely the concentration level of deceptive terminals targeted by the source IP and the level of flow generated.
Overall, the authors presented enough information of their experiment setup and results. However, I cannot overlook the fact that the significance of this study is reduced because the authors' assumption is too strong by relying on a single feature --- "the distribution of attack sources". This can easily leads to a model working specifically on a particular dataset, which is not robust enough with large performance variance in more general cases.
The drawback of making prediction based on a single feature is actually reflected from authors' own experiment. On line 244, the author stated that "false positive rate is 9.19\%, which cannot be considered a low value.". Then, on line 255/256, the authors' optimal configuration yields "false negative rate is 7.56\%." Based on authors' own standard, their optimally tuned model does not achieve impressive low false negative rate as well, which is a direct consequence of a simple model with a single feature.
This manuscript in its current form is not up to the standard of a scientific research paper for this journal. To improve the significance of their study, I would suggest the authors to consider the following:
1. Making a slight more sophisticated multi-stage model without significantly increase of resources.
For example, the authors' NB model can be used as a first-stage model to identify low/high risk attack candidates with relatively high accuracy. Then, since the number of the selected candidates is much fewer than the entire dataset, we can afford to build a more complicated second-stage model to achieve both high accuracy and high recall on the classification job. We can even use the probability score generated by the NB model as soft labels for the second-stage model to achieve knowledge transfer in the learning process.
2. NB models are famous for its cold start issue. Its success requires the statistically significant distribution in the dataset being already established in the first place. Otherwise, NB's performance is known to be poor. The authors need to discuss this issue and provide a proposal to mitigate it.
Typo/errors that affecting readability of the manuscript:
1. line 179: ...... is of a high-risk ordinal scale .....
I believe the authors intend to refer the low-risk ordinal scale here in Lemma 3.
2. Similarly, in equation (6), the probability defined is for low risk, so the notation should be P_low instead of P_high
3. In the Proof of Lemma 4, the index for the two equations overlap with those in the Lemma 3. Here, they should be annotated as equation (8) and (9).
Quality of English is ok for a general audience to read and understand the manuscript.
Author Response
Dear Reviewer,
We are grateful for your insightful and constructive feedback on our paper, "A Study of the Ordinal Scale Classification Algorithm for Cyber Threat Intelligence Based on Deception Technology." We appreciate the time and effort that you have invested in reviewing our manuscript and providing your valuable insights. I have attached a separate response to the comments you provided.
Author Response File: Author Response.pdf
Reviewer 3 Report
The paper describes a model that aims to quickly classify low-risk attacks, including collecting and scanning information that is extensively and repeatedly carried out, as well as high-risk attacks, instead of classifying specific types of attacks. Most of the existing tech-based CTI generation deceptive research is of limited use in real-world environments because the data labeling, learning, and detection processes using AI algorithms take time for computational resources. In the proposed approach, a simple-scale classification model based on naive Bayesian discriminant analysis showed higher accuracy in classifying low-risk attacks while using significantly less resources than other studies. It is a very interesting and developmental approach, which should be further explored by the authors.
Comments:
Correct the column headings in Table 1.
Move table 3 to a new page.
Illegible formula number 1, you can't see the minus well and it merges with the fraction.
Figure 4 - auxiliary lines only blur the picture, besides the data in the table below are clearer and the figure does not add anything to the article.
Author Response
Dear Reviewer,
We are grateful for your insightful and constructive feedback on our paper, "A Study of the Ordinal Scale Classification Algorithm for Cyber Threat Intelligence Based on Deception Technology." We appreciate the time and effort that you have invested in reviewing our manuscript and providing your valuable insights. I have attached a separate response to the comments you provided.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
Authors have addressed all the discussion points. I would recommend for its publication.
Quality of English Language has been improved.
