Machine Learning-Based Adaptive Synthetic Sampling Technique for Intrusion Detection

Round 1

Reviewer 1 Report

In this manuscript the authors investigate the challenging topic of managing IoT network risks, by exploiting and combining various DL approaches to find anomalies in IDS. Overall the manuscript is well written, lacking however some crucial details across the applied methods and their captured results, which currently are not quite clear documented. Furthermore, the manuscript lacks the relevant scientific research evidence upon a plethora of statements, which is a shortcoming. Rather than those drawbacks, the authors should address the following issues for improving both the content and the quality of their work:

1. Introduction

* Currently the Introduction part includes a lot of information that better fits in Section 2 of the manuscript. Hence, all the state of the art concepts that are outlined in the Introduction Section (i.e., details of intrusion detection systems and techniques) should be removed in the literature review Section (Section 2). In sequel, the Introduction part should be enhanced with additional details regarding the investigated problem. Among the additional details, I suggest to the authors to also include relevant statistics to verify the problem’s statement.

* The authors should provide some relevant references for the provided information in Section 1, since currently this Section is not evidenced-based. The authors should make sure that they also include up-to-date-references.

* In the final paragraph of the Introduction Section, the authors should keep a homogeneous numbering formatting for the underlying Sections. Currently they are speaking for example for Section 2 and Section V.

* The authors should make clear in the Introduction part what exactly they are proposing in this manuscript, making clear their research contributions.

2. Literature Review

* The authors state “As a result, the fundamental obstacle to supervised learning is a need for more labeled data.”. What does this mean for IDS? The authors should better elaborate on this part. The same comment goes for the relevant statement for the unsupervised learning: “On the other hand, unsupervised learning draws out useful feature information from unlabeled data, making it considerably simpler to gather training data. However, unsupervised learning approaches typically do less well at detection than supervised learning methods”.

* The authors should better organize the information in Table 1. Currently the provided information is kind of messy (e.g., in the accuracy column the filled in information is provided into different ways). Also, the authors should add an additional column, depicting the type of learning that was applied in each research (i.e., supervised/unsupervised).

3. Dataset

* In the context of explaining Figure 3, the authors should also describe what does each attack mean.

* The authors should outline the specific data pre-processing actions that they performed upon their experimentation.

* The sentence “Due to the potential for underperformance and sensitivity to the size of the input characteristics, several machine-learning algorithms.” doesn’t make sense. The authors should correct this sentence.

* In One Hot-Encoding part, the authors state “Following one-hot encoding, the final dataset has a total of 128 features”. What was the initial number of the features? What was their specific type? The authors should provide such information. Moreover, it should be beneficial to also include a table shortly depicting/describing those features (e.g., their name/short description, their type, possible values’ restrictions).

* In Target Balancing part, the authors must provide a reference for SMOTE.

4. Methodology

* I suggest merging Section 3 with Section 4, adding all the information of Section 3 as a new sub-Section of Section 4. Then, the authors should provide a figure of the overall architecture of the followed approach to help in better understanding the whole process. The authors should make sure that all the input data/procedures analyzed in the Section are illustrated in the figure.

* Why XGBoost was used for feature extraction? The authors should provide such explanation.

* In the Model Architecture part, the authors state “MLP architecture is used in the first one. The second architecture combines CNN-1D and LSTM into a hybrid model.”. Why did they conclude on using such methods? They should better elaborate on this part.

5. Results

* The authors should mention the working environment and the implementation choices of their applied experiments.

* The authors should provide some relevant references for the criteria of recall, accuracy, F1 score, precision, and confusion matrix outlining the reason behind using those criteria. Two up-to-date references that include the explanation of all those metrics and can be stated are:

- Zafeiropoulos, Nikolaos, et al. "Interpretable Stroke Risk Prediction Using Machine Learning Algorithms." Intelligent Sustainable Systems: Selected Papers of WorldS4 2022, Volume 2. Singapore: Springer Nature Singapore, 2023. 647-656.

- Mavrogiorgou, Argyro, et al. "A Catalogue of Machine Learning Algorithms for Healthcare Risk Predictions." Sensors 22.22 (2022): 8615.

* How did the captured results of Table 2 derive? Did they derive by applying under the same circumstances both models? What was the specific timeframe of the experiments? How many times did the experiments take place? The authors should definitely provide such information. The same comment goes for the results of Table 3 and Table 4.

* For the comparison results of Table 5 and Table 6, the authors outline that their experiments/methods provided better results, not however outlining the reason behind such conclusion (rather than just stating the numbers). The authors should better describe this part, making clear their evidence/proposed method’s advancements behind such final results.

6. Discussion

* This Section should also outline possible identified limitations based on the captured results.

7. Conclusion

* The authors should include their future steps – perhaps some steps in order to address their limitations.

-

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

The authors propose a supervised ML-based method for Intrusion Detection, including an adaptive synthetic algorithm to generate synthetic data. 

The manuscript presents important errors that prevent its publication. 

1. The most important aspect is that the data in the results section are not consistent. For example the values of the metrics (e.g. Table 2) do not match the data presented in the confusion matrices (e.g. Figs. 10 and 11). This calls into question, mainly the content of sections 5 and 6, but in general the whole manuscript. 

2. Information on the dataset and the application with Adasyn is missing. For example, class balancing is performed only for training?, how the distribution of the dataset was performed?, the number of features is not specified (the article refers to X features)

3. Figures 1 and 2 do not directly contribute to the content of the article, since they are general about IDS methods and ML methods. 

4. The work was done by one person or was actually co-authored: "Line 132: ... the individual contributions I made to this work"

5. What is really the contribution of the article with respect to the state of the art (Table 1). 

6. What type of diagram is shown in Figure 5? Is it a flowchart?, and if so, it presents errors in its structure

7. What is the reason for making a two, four and five-class incremental model?

8. Data with categorical representation are used for both tSNE and ADASYN? What are the implications of this?

9. The context of use of XGBoost is missing. Is it to calculate feature importance?

Review some minor aspects. Eg.  

Lines 152-154: "The analysis is performed using a detector. The model is constructed using many sensors and detectors. It gathers data in real-time from all the sources that are accessible, which is then analyzed by a detector. The detector recognizes..."

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

I have reviewed your work titled "Machine Learning-Based Adaptive Synthetic Sampling Technique for Intrusion Detection" in detail. Items that I see missing are presented below.

Researchers used CNN and LSTM networks in their study. First of all, spelling errors in the study should be reviewed. Words like we, our should be used as little as possible. I think one of the biggest innovations in the study is the data balancing step. However, no details are given about the two methods used here. Smote and Adasyn techniques need to be elaborated.

I understand some of the topics presented in the study. For example, a title called 3.1 Data Pre-processing was opened and general information was given. No explanation is given about how this title applies to the dataset. A second example can be given in headings 3.2 and 3.3. It is possible to expand this.

4.1. Feature selection is mentioned in the Feature Selection title. Is it a part of the proposed model in this step, if it is, you should design the proposed model and related figures accordingly. In addition, CNNs essentially extract features. For example, the size can be reduced by max-pooling. I don't understand the logic of using the XGBoost classifier here. If we look at it with this logic, wouldn't it be more accurate to use NCA, LDA, mRMR, Relief, or metaheuristic algorithms? Confusion matrices look too bad to read. It is important to present the accuracy and loss curves to see if there is memorization in the models. When I read the article starting from the introduction to the conclusion, I can say that there are too many repetitive statements. For this reason, the relevant chapters are very long and it is very difficult to catch fluency while reading. I do not refuse to work because the experimental results seem successful. But it is important that you eliminate the major deficiencies that I have mentioned.

Spelling errors in the study should be reviewed.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The authors have successfully addressed the majority of the raised comments. However, some changes are still unclear. More particular, it is unclear where in the manuscript the authors have provided the figure of the overall architecture of the followed approach. It seems that such figure is currently missing from the manuscript in order to make clear the entire followed process. The authors are kindly requested to create a new figure for such purpose.

Author Response

"Please see the attachment."

Author Response File: Author Response.pdf

Reviewer 2 Report

The authors present a review of the manuscript, giving responses to the points. However the manuscript still presents important errors that prevent its publication. 

1. The most important aspect is that the data in the results section are not consistent. The data in the text does not match the values in the tables, the calculated metrics do not match the values in the confusion matrix, nor do the calculations within the tables match the data in the same table (e.g. F1-Score).

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

I reviewed your work titled "Machine Learning-Based Adaptive Synthetic Sampling Technique for Intrusion Detection" after the revision. Thank you for addressing the deficiencies mentioned in the revision in detail and fixing these problems. But we still have a serious problem in title 3.4 Feature Selection. I agree with what you said about XGBoost. XGBoost is a classifier. I don't understand why you put the title as Feature selection. Feature selection means the selection of important features from a specific feature map. Please examine Figure 3 and Figure 6 in the related study "https://doi.org/10.3390/diagnostics13071299". Also, the resolution of confusion matrices in the article is still bad. I think these confusion matrices originate from the application environment you have received. But it is very difficult to read as it is. You can draw these confusion matrices by hand yourself. My aim is not to force you, it is to eliminate the problems that will cause confusion in the study. Thank you again for the quality revision you did in the previous round.

.

Author Response

"Please see the attachment."

Author Response File: Author Response.pdf