A Review of Machine Learning and Deep Learning Techniques for Anomaly Detection in IoT Data

Round 1

Reviewer 1 Report

In this review article, the authors discuss all the aspects of IoT data processing by providing a complete image of various state-of-the-art techniques, how they solve the major problems, and how they address the core challenges.
The authors focused on machine learning techniques for anomaly detection in the data stream, specifically on evolving data stream.

This review article is well-organized, with proper structure, flow and length.
The article includes lots of references to other relevant studies and, the review of the state-of-the-art is sufficient.
The purpose and the findings of the research are clear and relatively concise.
The research is well-designed, a clear objective is set, and the motivation of this study is defined.
The authors identify the research questions and mention the main challenges.
The main contributions of this review article are the following:
1)Examination of state-of-the-art studies centred on machine learning techniques for anomaly detection in the data stream.
2)Proposal of taxonomy to define current literature based on nature of the data, anomaly types, detection learning modes, window model, data source and evaluation criteria.
3)Analyzing the existing techniques based on the proposed taxonomy.
4)Highlighting research challenges that form the future directions.

I think this article has good potential but, a few suggestions will improve its quality. Specifically, I have to point out the following:
1)I have noticed several syntactic and grammatical errors in the article.
In addition, the article has many extensive sentences. Punctuation is essential.
2)The Conclusions section is weak and needs a better connection with the research findings. Summarize them in a more targeted way. Moreover,  I would like to know the limitations and the potential issues of this study.
3)Reference [19] is not accessible.
4)Many references chosen by the authors do not have the equivalent doi.
This makes them difficult to find and access.
5)The authors did not select journals from MDPI in their research.

Author Response

Dear Editor,

The authors would also like to sincerely thank the reviewer for his/her valuable remarks and careful feedback which helped them to significantly enhance this manuscript and its presentation. Regardless of the final outcome, the authors sincerely thank the editor and reviewers for supporting their work and improving its quality. The productive and valuable remarks enabled us to update many parts of the manuscript as shown by the responses to each comment below. Besides, all the updated parts in the manuscript were highlighted in yellow color in order to be easily tracked by the editor and reviewers.

Reviewer #1:

“Comments to the Authors”

1. I have noticed several syntactic and grammatical errors in the article. In addition, the article has many extensive sentences. Punctuation is essential.

Response

The authors are really very grateful to the reviewer for suggesting these significant issues. We updated the manuscript by sending it to a professional editing service to settle. In the revised version of the manuscript, all syntactic and grammatical errors have been corrected. In addition, proofreading has been made for the whole of the manuscript. We believe the manuscript language is quite good according to the English Language editor.

 

“Comments to the Authors”

 

2. The Conclusions section is weak and needs a better connection with the research findings. Summarize them in a more targeted way. Moreover, I would like to know the limitations and the potential issues of this study.

 

Response

Thank you for this suggestion. The conclusion section has been revised to include a summary of the findings as well as highs the limitations and potentional issues of the study. Please refer to page 19 lines (707-726)

“Comments to the Authors”

 

3. Reference [19] is not accessible.

 

Response

Thank you for pointing that. Reference [19] has been updated and its accessible now. Please refer to page 1 lines 38

“Comments to the Authors”

 

4. Many references chosen by the authors do not have the equivalent doi. This makes them difficult to find and access.

 

Response

Thank you for highlighting this, it’s very important comment.  Authors have gone through the references one by one to make sure that all have the equivalent doi to make it easier for readers to find and access the cited papers. Please refer to pages (20-23)

“Comments to the Authors”

 

5. The authors did not select journals from MDPI in their research.

Response

Thank you again for pointing at this concern, MDPI is a well-reputed journal.  Authors have included some related journals from MDPI to the research paper as followed:

• Kozitsin, I. Katser, and D. Lakontsev, “Online Forecasting and Anomaly Detection Based on the ARIMA Model,” Appl. Sci., vol. 11, no. 7, p. 3194, 2021, doi: 10.3390/app11073194.
• B. Baydargil, J. S. Park, and D. Y. Kang, “Anomaly analysis of alzheimer’s disease in pet images using an unsupervised adversarial deep learning model,” Appl. Sci., vol. 11, no. 5, pp. 1–18, 2021, doi: 10.3390/app11052187.

Please refer to page 2 Line 59 and page 3 Line 130

Author Response File: Author Response.pdf

Reviewer 2 Report

This paper conducts a survey and reviews on the machine learning techniques for anomaly detection in IoT evolving data stream. The main contributions include proposal of a taxonomy to define current literature based on nature of the data, anomaly types, detection learning modes, window model, data source and evaluation criteria, analyzing the existing techniques based on the proposed taxonomy, and presenting research challenges. There is one concern that Section 4 is very short, while section 5 is very long. Both sessions mention about C-LOF, AutoCloud, TEDA Clustering, Evolving spiking neural network, Combination of (BDLMs) & (RBPF), KPI-TSAD, HTM, ensembles neural networks, Multiple kernel learning, xStream, CEDAS, MuDi-Stream, and Long Short-Term Memory. It may be good to reorganize these two sessions so that they are closely relevant and more balance in the terms of section length.

There are some other aspects for authors to consider when improving their manuscript.

• Line 44, “or noise [5], [6]. Where outliers are the data points that considers out of the ordinary” should be “or noise [5], [6], where outliers are the data points that are considered out of the ordinary”.
• Line 46, “anomalies are special kind of outliers that has actionable piece of information” should be “anomalies are a special kind of outliers that has actionable piece of information”.
• Line 54, “Anomaly detection plays a vital role in the analysis of anomalies in  multiple” should be “Anomaly detection plays a vital role in the analysis of anomalies in multiple”.
• Line 80, “To form a good quality clusters” should be ““To form a good quality cluster” or “To form good quality clusters”.
• Line 100, “clusters' quality won't be affected” should be “clusters' quality is affected” or “clusters' quality will not be affected”.
• Line 111, please check the “cantered” in the sentence of “Examination of state-of-the-art studies cantered on machine learning techniques”.
• Line 147 and Line 148, “such as smart cities, such as water leakage” should be “such as smart cities, water leakage”.
• Line 157, “anomalous point or events that differ” should be “anomalous points or events that differ”.
• Line 166, “flow of regular patterns and classify any suspected data that varies” should be “flow of regular patterns and classify any suspected data that vary”.
• Line 170, “Shallow learning methods utilized the selected features” should be “Shallow learning methods utilize the selected features”.
• Line 176, this paragraph has only one sentence. It should join the previous paragraph.
• Line 199, “result in this type of anomaly detection. “Collective” should be “result in this type of anomaly detection - “Collective”.
• Line 230, “Additionally, [39], used the” should be “Additionally, [39] used the”.
• Line 233, “The K-Nearest Neighbor (kNN) algorithm” should be “The KNN algorithm”.
• Line 238, please add a full stop for the sentence.
• Line 246, “[30], The Conditional Gradient Boosting Decision Tree (GBDT) was used for the early detection of anomalies in the problem of breaking wind turbine bolts” should be “The Conditional Gradient Boosting Decision Tree (GBDT) was used for the early detection of anomalies in the problem of breaking wind turbine bolts [30]”.
• Line 286, could “The k-means clustering (GMM)” be changed into “The GMM” since this paragraph discusses about the GMM, instead of the k-means clustering?
• In Figure 6, the font size is too small; “Cumulative Local Outlier Factor (C-LOF)” should be “C-LOF”; “long-short-term memory (LSTM)” should be “LSTM”; “Hierarchical Temporary Memory (HTM)” should be “HTM”; “Extreme Learning Machine” should be “ELM”; “(OFAT) deep neural network” should be “OFAT”; “(OeSNN-UAD)” should be “OeSNN-UAD”; “(e-SREBOM)” should be “e-SREBOM”;
• Figure 6 lists OeSNN-UAD, Gryphon and Extra-adaptive robust online subspace tracker for anomaly detection. However, there are no descriptions about them in the content.
• Section 5 discusses about the BDLMs and RBPF and SVM. However, they are not listed in Figure 6.
• Line 454, “A full online method has been introduced” should be “A full online method CEDAS has been introduced”.
• Line 473, “ELM” should be “Extreme Learning Machine (ELM)”.
• Table 1 is showed without any descriptions. Please add some descriptions about Table 1 in the content.
• Line 547, “capability requirements. [13]” should be “capability requirements [13]”.
• Line 657, “R.A.;R.K investigation, R.A.;A.F resources, M.M.;A.A;M.A” should be “R.A. and R.K.; investigation, R.A. and A.F.; resources, M.M., A.A. and M.A.”
• Line 658, “draft preparation, R.A.;A.F writing—review and editing, R.A.;R.K.;M.A” should be “draft preparation, R.A. and A.F.; writing—review and editing, R.A., R.K. and M.A.”.
• Line 659, “funding acquisition, M.M.; A.A” should be “funding acquisition, M.M. and A.A.”.

Author Response

Dear Editor,

The authors would also like to sincerely thank the reviewer for his/her valuable remarks and careful feedback which helped them to significantly enhance this manuscript and its presentation. Regardless of the final outcome, the authors sincerely thank the editor and reviewers for supporting their work and improving its quality. The productive and valuable remarks enabled us to update many parts of the manuscript as shown by the responses to each comment below. Besides, all the updated parts in the manuscript were highlighted in yellow color in order to be easily tracked by the editor and reviewers.

Reviewer #2:

 

“Comments to the Authors”

1. Line 44, “or noise [5], [6]. Where outliers are the data points that considers out of the ordinary” should be “or noise [5], [6], where outliers are the data points that are considered out of the ordinary”.

Response

Thank you so much for your feedback, it is highly. We have updated the manuscript by sending it to a professional editing service then updated the manuscript accordingly and now its language is quite good according to the English Language editor. Please refer to the revised version of the manuscript.  

 

“Comments to the Authors”

2. Line 46, “anomalies are special kind of outliers that has actionable piece of information” should be “anomalies are a special kind of outliers that has actionable piece of information”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

3. Line 54, “Anomaly detection plays a vital role in the analysis of anomalies in multiple” should be “Anomaly detection plays a vital role in the analysis of anomalies in multiple”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

4. Line 80, “To form a good quality clusters” should be ““To form a good quality cluster” or “To form good quality clusters”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

5. Line 100, “clusters' quality won't be affected” should be “clusters' quality is affected” or “clusters' quality will not be affected”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

6. Line 111, please check the “cantered” in the sentence of “Examination of state-of-the-art studies cantered on machine learning techniques”.

Response

Thank you so much for your feedback. The typo has been corrected, please refer to page 3 Line 102

“Comments to the Authors”

 

7. Line 147 and Line 148, “such as smart cities, such as water leakage” should be “such as smart cities, water leakage”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

8. Line 157, “anomalous point or events that differ” should be “anomalous points or events that differ”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

9. Line 166, “flow of regular patterns and classify any suspected data that varies” should be “flow of regular patterns and classify any suspected data that vary”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

10. Line 170, “Shallow learning methods utilized the selected features” should be “Shallow learning methods utilize the selected features”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

11. Line 176, this paragraph has only one sentence. It should join the previous paragraph.

Response

Thank you so much for your feedback, as recommended by the reviewers the paper structure has been updated and this sentence has been removed from the revised version of the manuscript.

“Comments to the Authors”

 

12. Line 199, “result in this type of anomaly detection. “Collective” should be “result in this type of anomaly detection - “Collective”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

13. Line 230, “Additionally, [39], used the” should be “Additionally, [39] used the”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

14. Line 233, “The K-Nearest Neighbor (kNN) algorithm” should be “The KNN algorithm”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

15. Line 238, please add a full stop for the sentence.

Response

Thank you so much for your feedback. A full stop has been added. Please refer to page 12 Line 441

“Comments to the Authors”

 

16. Line 246, “[30], The Conditional Gradient Boosting Decision Tree (GBDT) was used for the early detection of anomalies in the problem of breaking wind turbine bolts” should be “The Conditional Gradient Boosting Decision Tree (GBDT) was used for the early detection of anomalies in the problem of breaking wind turbine bolts [30]”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

 

17. Line 286, could “The k-means clustering (GMM)” be changed into “The GMM” since this paragraph discusses about the GMM, instead of the k-means clustering?

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

18. In Figure 6, the font size is too small; “Cumulative Local Outlier Factor (C-LOF)” should be “C-LOF”; “long-short-term memory (LSTM)” should be “LSTM”; “Hierarchical Temporary Memory (HTM)” should be “HTM”; “Extreme Learning Machine” should be “ELM”; “(OFAT) deep neural network” should be “OFAT”; “(OeSNN-UAD)” should be “OeSNN-UAD”; “(e-SREBOM)” should be “e-SREBOM”;

Response

Thank you so much for your feedback. The figure has been resigned with a bigger font size. All recommendation in designing the figure have been reflected in the revised version. Please refer to page 4 Line155

“Comments to the Authors”

 

19. Figure 6 lists OeSNN-UAD, Gryphon and Extra-adaptive robust online subspace tracker for anomaly detection. However, there are no descriptions about them in the content.

Response

Thank you so much for highlighting this. The Figure has been updated and all the information is the figure is updated according to the sections provided in the manuscript. Please refer to page 4 Line155

“Comments to the Authors”

 

20. Section 5 discusses about the BDLMs and RBPF and SVM. However, they are not listed in Figure 6

Response

Thank you so much for highlighting this. The Figure has been updated and all the information is the figure is updated according to the sections provided in the manuscript. Please refer to page 4 Line155

“Comments to the Authors”

 

21. Line 454, “A full online method has been introduced” should be “A full online method CEDAS has been introduced”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

22. Line 473, “ELM” should be “Extreme Learning Machine (ELM)”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

23. Table 1 is showed without any descriptions. Please add some descriptions about Table 1 in the content

Response

Thank you so much for highlighting this, description of the table has been added as recommended. Please refer to page 7 Line 297

“Comments to the Authors”

 

24. Line 547, “capability requirements. [13]” should be “capability requirements [13]”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

25. Line 657, “R.A.;R.K investigation, R.A.;A.F resources, M.M.;A.A;M.A” should be “R.A. and R.K.; investigation, R.A. and A.F.; resources, M.M., A.A. and M.A.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

26. Line 658, “draft preparation, R.A.;A.F writing—review and editing, R.A.;R.K.;M.A” should be “draft preparation, R.A. and A.F.; writing—review and editing, R.A., R.K. and M.A.”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

“Comments to the Authors”

 

27. Line 659, “funding acquisition, M.M.; A.A” should be “funding acquisition, M.M. and A.A.”.

Response

Thank you so much for rising up this, your valuable recommendation has been reflected in the revised version. Please refer to the revised version of the manuscript. 

 

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments are in the review report below.

Comments for author File: Comments.pdf

Author Response

Dear Editor,

The authors would also like to sincerely thank the reviewer for his/her valuable remarks and careful feedback which helped them to significantly enhance this manuscript and its presentation. Regardless of the final outcome, the authors sincerely thank the editor and reviewers for supporting their work and improving its quality. The productive and valuable remarks enabled us to update many parts of the manuscript as shown by the responses to each comment below. Besides, all the updated parts in the manuscript were highlighted in yellow color in order to be easily tracked by the editor and reviewers.

 

Reviewer #3:

 

Major Concern

“Comments to the Authors”

1. I’m confused with the logical division for section 3-5. I know classifying ML/DL methods, techniques, applications, etc.. is very hard. But I believe it would make more logical sense to have one section that starts with Figure 6, the taxonomy, then discuss each aspects as a subsection.

Response

Thank you for this suggestion. The paper structure has been revised based on your valuable  recommendation. The revised version of the paper structure is as followed: 1) Introduction; 2) Background; 3) Taxonomy; 4) Research challenges and future direction; 5) Conclusion. The Taxonomy section contains 7 sub-sections that elaborate every component of the taxonomy highlighted in Figure 2. Please refer to page 4 – page 15.

 

 

“Comments to the Authors”

2. I’m confused with what should be the key takeaway for this review paper, it feels like an exhaustive collection of facts optimizing for topic coverage but each point is lacking in depth. While such initiative itself is no doubt commemorable, the paper has great potential to have more scientific impact. For example,
   6. Can the paper be expanded to guide selection of algorithm based on constraints and problem? A possible “results” section could combine methods discussed in section 5 and how they are ideal for the challenges discussed in 6. Or maybe some challenges in 6 are fundamentally difficult and can’t be solved with any of these methods?
   7. Can the paper be expanded to include a section that is “problem/use case driven” rather than “methods driven”? For data scientists, solving the problem is more important than what is the hottest ML/DL method. Possible areas include
3. Computer vision
4. Natural language processing

iii. Self-driving

1. Recommendation systems

Response

Thank you for this suggestion. Based on your recommendation we have added a new section “Section 4” to discuss the results of the review paper and we illustrated the section with summary table to highlight the research gab found within the reviewed techniques. The table also highlights the techniques capabilities and guide the researchers to the research challenges for future work.  Please refer to Section 4 pages 15-17 Lines 552-591

Further Discussions:

“Comments to the Authors”

1. 2021 is considered to be the year for edge-driven, federated-learning where data is processed on the IoT and the model is deployed on the IoT. Can you discuss the prospects and limitations of anomaly detection on these, “less-capable (compared to online clusters)” machines?

Response

Thank you for raising this point. The paper new structure is organized in a way that highlight the importance of IoT in modern technology and we believe that the revised version of the manuscript now offers an insight toward the prospects and limitation of IoT in Anomaly detection field.

“Comments to the Authors”

2. Please address the proper data handling prior to model training. The model is only as good as the data. Without proper data preprocessing and validation steps, feature engineering and feature selection, the method of discussion may simply not work, ie. Garbage in garbage out. Do the use cases that you cite properly handle these steps?

Response

Thanks for bring that, it is indeed an important process prior to model training. However, pre-processing is out of the scope of this review paper, and it’s within the limitation of the paper. Please refer to Conclusion section, Page 19 Lines 723-728.

“Comments to the Authors”

3. Please expand on how these methods scale with respect to ever increasing volume of data.

Response

Scalability is another major requirement for anomaly detection algorithm, and based on the techniques discussed it was found that scalability is one of the major issues faced by many anomaly detection techniques. Thus it was highlighted in the results section page 16 Line 578-583, and in the research challenges and future work section. Please refer to page 19 Lines 689-691.

 

“Comments to the Authors”

4. Please discuss issues with model deployment and fault tolerance.

Response

Thank you for the suggestion. However, as the limitation of this review paper, Model deployment and fault tolerance are out of the scope. Yet, we intend to included it within our future work.

 

“Comments to the Authors”

5. Please discuss anomaly detection wrt. adversarial attacks such as deepfake

Response

Thank you for highlighting this. However, as mentioned earlier, the limitation of this review paper, addressing different types of attacks such as deepfake are out of the scope. Yet, we intend to included it within our future work.

Small Issues:

“Comments to the Authors”

1. Need more descriptive figure captions

Response

Thank you for raising this point. As per the revised version of the manuscript more descriptive to the figure are added. Please check the revised version of the manuscript.  

“Comments to the Authors”

2. Avoid words like “some”, “several”, “very”, they are too vague. Support them with numerical or interval values.

Response

Thanks for highlighting that, it has been applied in the revised version of the paper where applicable

 

 

Abstract:

“Comments to the Authors”

1. well written with clear challenges and goals and the area of discussion is sufficiently complete.

Response

Thank you.

 

Introduction:

“Comments to the Authors”

1. Missing citation for “IoT has become one of the biggest data sources in the last few years.”

Response

Thank you for raising this point. Citation has been added. Please refer to page 1 Line 43-44

“Comments to the Authors”

2. Explain why non-ML anomaly detection techniques may work/fail?

Response

Thanks for highlighting that, the scope of the paper is limited to machine learning and deep learning techniques. Other techniques such as statistical methods are out of the scope of the paper. Please refer to  Conclusion section, Page 19 Lines 704-728.

 

“Comments to the Authors”

3. Line 70 -105 discuss good clustering algorithm criteria, but it’s quite a lot to take it. Are the points organized based on order of importance? Some of the points are overlapping (fast data, memory, data streams discuss similar things) Can you organize it to approx. 3-4 main points (if applicable) with 2-3 sub bullet points?

Response

Thank you for raising this point. As recommended overlapped points have been removed and the number of points have been reduced from 6 to 4 point. Please refer to page 2 Lines 80-96.

“Comments to the Authors”

4. Why is the latter part of the introduction so heavily describing clustering while ignoring other aspects of anomaly detection algorithms? Are you only reviewing clustering algorithms? But many of these points are not only pertain to clustering. I’m confused.

Response

Thanks for pointing that, Introduction section has been revised to include other aspects of anomaly detection to be comprehensive and reflecting the scope of the paper. Please refer to pages 1-3 Lines 35-113

 

 

Motivation:

“Comments to the Authors”

1. Some overlap with previous section, is it possible to merge intro and motivation.

Response

Thank you for raising this point. Motivation section has been merged with Introduction section as recommended and overlapped sentences and points were eliminated. Please refer to pages 1-3 Lines 35-113.

 

 

Anomaly Detection:

“Comments to the Authors”

1. Discussion of non-ML anomaly detection needed for a complete story.

Response

Thank you for pointing this. Deep learning techniques have been also reviewed in this paper to make it more comprehensive and to cover a non-ML techniques. Please refer to pages 8-10 Lines 303-359.

“Comments to the Authors”

“Comments to the Authors”

2. For Line 216 “a series of clustering rules” and line 218 “clustering and regression: do you mean classification? Clustering generally used for unsupervised learning

Response

Thank you for pointing this. Description of supervised anomaly detection have been revised. Please refer to page 12 Lines 413-446.

“Comments to the Authors”

3. Two 3.2.2 section, I assume the latter is 3.2.3.

Response

Thank you for highlighting  this. Section latter has been updated. Please refer to page 14 Line 457.

 

“Comments to the Authors”

4. Major concern: Between 3.2.1 – 3.2.3, you gave a very brief description of around a dozen machine learning algorithms and their use cases. However, you need to go in depth with why these methods work for the use case. It is a common problem (dangerzone) in data science where one use a method out of the blackbox without understanding the underlying assumptions and math for these methods. For example, with Naïve Bayes assumes the parameters are independent of each other, is that the case?

Response

Thank you for highlighting  this. The entire sub-section has been revised in a way that will give the refer an insight about the techniques and the major contribution it achieves as well as their main limitations. Please refer to pages 5- 10 Lines 159-359

 

Results:

“Comments to the Authors”

1. In the introduction, you mentioned clustering algorithm “should” have # characteristics. The evaluation of these characteristics should be present in your results on whether or not an algorithm contains those characteristics.

Response

Thank you for pointing this. In the old version we did wrongly mentioned clustering algorithms. However, in the revised version of the manuscript “Introduction section” we have updated the section to include also other methods which are presented in the taxonomy. Please refer to the updated version of the Introduction section pages 1-3 Lines 35-113

 

Taxonomy of Anomaly Detection:

“Comments to the Authors”

1. how does deep learning fit into the taxonomy? I see it dotted in the techniques part. The techniques parts need one or two more layer of organization

Response

Thank you for pointing this. As mentioned earlier, deep learning techniques have been also reviewed in this paper to make it more comprehensive and to cover a non-ML techniques. Please refer to pages 8-10 Lines 303-359.

“Comments to the Authors”

2. Again, this section is a huge dump of information without much organization and unpleasant to read through.
3. Think about what the reader want to get out of this section?
4. Can you sort the algorithms or provide some logical structure (subsections, subsubsections?)
5. Excessive amount of sentence start with “recently”… how recent? Can you provide a year? In the ML space, 4-5 years can be considered “ancient”.

Response

Thank you for highlighting this.

a/b. This section has been restructured as part of the taxonomy and it has been resigned in a structured manner that reviews state-of-the-art anomaly detecting techniques for IoT data by grouping them into two categories namely machine learning and deep learning techniques. In the taxonomy section, we discussed the machine learning techniques, nature of data, anomaly types, detection learning modes, window models, the datasets used and the evaluation criteria. Please refer to pages 4-15 Lines 147- 550.

1. Thanks for pointing this, years have been provided to replace the word “Recently” in the revised version of the manuscript.

“Comments to the Authors”

3. I’m not sure what I’m supposed to take away from this section except ok, I learned a bunch of random facts. Can you help the reader distill some key takeaways? For example:
4. What are some key metrics? (like speed, compatibility, scalability, etc. all those in section 6 but select 3-4 key metrics) How do all of these algorithm rank for each of these metrics?
5. What methods should I use?
6. How does x compare to y?

Response

Thank you for highlighting this. We believe with the new section added to the manuscript in section 4 “Results”, readers can have a summary of the research findings and also they will be able to compare the techniques reviewed and clearly linked it with the research challenges. Please refer to pages 15-17 Lines 551- 590

Research Challenges:

“Comments to the Authors”

1. Well, how does all the methods you described in section 5 fair/rank wrt to each other with regards to the 13 points discussed in section 6?

Response

Thank you for pointing this. The research challenges section is originally extracted from the limitations of the techniques discuses in the taxonomy. With the new structure of the revised version of the manuscript, readers will be able to link all the methods described with the research challenges

“Comments to the Authors”

2. The take away for the paper should be very clear: If I care about X,Y,Z, then I should use method 1, if I care about A,B,C, then I should use method 2… etc.

Response

Thank you for highlighting this. The revised version of the manuscript has made it more clear to the readers to understand the fundamental of the techniques reviewed and connecting that will the finding extracted in the research challenges. This will enable the future researchers to have a guidelines on what technique to be used.

 

Questions:

“Comments to the Authors”

1. Discussion about adversarial attacks?

Response

Thank you for asking this. Attacks are out of the scope of the paper. The paper is focusing on techniques used for anomaly detection using ML and DL.

 

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

The authors have edited their manuscript according to the previous comments. There is one aspect for authors to consider: 

In Line 398 and Line 399, please rewrite ““Collective anomaly” over consecutive time intervals relating to complete data patterns” since it is not a sentence.

 

 

Author Response

Dear Editor,

The authors would also like to sincerely thank the reviewers for their valuable remarks and careful feedback which helped us to significantly enhance this manuscript. Regardless of the final outcome, the authors sincerely thank the editor and reviewers for supporting their work and improving its quality. In the second round of the review, we have addressed the reviewers comments. Besides, all the updated parts in the manuscript were highlighted using the track change function in order to be easily tracked by the editor and reviewers.

Reviewer #2:

 

“Comments to the Authors”

The authors have edited their manuscript according to the previous comments. There is one aspect for authors to consider:

 

1. In Line 398 and Line 399, please rewrite ““Collective anomaly” over consecutive time intervals relating to complete data patterns” since it is not a sentence.

Response

Thank you so much for your feedback, it is highly appreciated. We have updated the definition of collective anomaly as well as the example used to elaborate the concept of a collective anomaly as per your request.

 

 

Reviewer 3 Report

Excellent! I'm happy with most of the changes made by the author, all major concerns have been addressed. The new tables are especially helpful and highlight key takeaways for the paper (Table 4 column header could be 45degree instead of 90degree). I commend the tremendous effort the authors put into this review and I'm excited about the official publication. This is a review I will bookmark for future references. Good Luck!

Author Response

Dear Editor,

The authors would also like to sincerely thank the reviewers for their valuable remarks and careful feedback which helped us to significantly enhance this manuscript. Regardless of the final outcome, the authors sincerely thank the editor and reviewers for supporting their work and improving its quality. In the second round of the review, we have addressed the reviewers comments. Besides, all the updated parts in the manuscript were highlighted using the track change function in order to be easily tracked by the editor and reviewers.

Reviewer #3:

 

“Comments to the Authors”

1. In Excellent! I'm happy with most of the changes made by the author, all major concerns have been addressed. The new tables are especially helpful and highlight key takeaways for the paper (Table 4 column header could be 45degree instead of 90degree). I commend the tremendous effort the authors put into this review and I'm excited about the official publication. This is a review I will bookmark for future references. Good Luck!.

Response

Thank you so much for your feedback, it is highly appreciated. We have updated Table 4 column header.