Handling Imbalanced Datasets for Robust Deep Neural Network-Based Fault Detection in Manufacturing Systems
Round 1
Reviewer 1 Report
The paper presents a fault detection approach in manufacturing systems based on a Deep Learning model. The main contribution is the use of the proposed approach in the case of unbalanced data. The proposed approach relies on the Deep Learning model backed by coordinated reinforcements from human experts.
The paper is well written and the method is interesting. Please find here below my major comments.
- My first comment regards the absence of a competitor. The authors mentioned that a wide literature has been devoted to data-driven fault detection methods. I believe that identifying a benchmark and including a comparison analysis would allow the reader to better understand the actual value of the proposed approach.
- Although the paper is well written, some parts are, in my opinion, not that easy to follow. I would have a few suggestions: 1) add a summary table in Section 2 to provide the reader with a synthetic recap about the most relevant literature, the pros/cons of different streams in the research, and their application limits; 2) include in Section 3 a list of assumptions the proposed approach relies on (it would be very helpful to understand when the applicability framework of the proposed approach). Please, provide some examples of actual faults, more details about the anomaly scenarios. In my opinion, this would help the reader to find connections between the proposed case study in manufacturing and other real-world problems where the proposed approach can find some application.
- I would suggest including other techniques in the analysis, such as the Transfer Learning approach for an unbalanced data set. This approach of using transfer learning for unbalanced data has been recently proposed in the literature and it should be included in the discussion of the current manuscript.
Author Response
Absence of a competitor: We identified a benchmark; GPR-based GAN framework, implemented and evaluated on the APS Failure at Scania Trucks dataset. We introduce the method in the literature review section (pg 3, line 119), summary wide-table (pg 4), and subsequently, a comparative analysis was carried out in table 1 (pg 12) and discussed further in section 4 of results and discussion (pg 15, lines 542 -- 557).
Add a summary table in Section 2: We have added a summary wide-table 1 (pg 4) discussing the different streams of research considered and their application limits.
Include in Section 3 a list of assumptions the proposed approach relies on:
The proposed framework applicability discussed in Section 3.7, including considerations relied upon by the proposed approach - (i) Subsection 3.7.1 (pg 11, lines 378 -- 389) discuss the APS Failure at Scania Trucks dataset as a safety-related dataset and the justification for prioritizing recall in safety-critical systems. (ii) (pg 12, lines 395 --413) discuss the shortcomings in the dataset and the justification for the selection of a deep learning-based fault detection framework.
Examples of actual faults: In Section 3.7, we discuss the application and implementation of the framework on two real-world imbalanced datasets. (i) APS Failure at Scania Trucks dataset: where the actual fault is the detection of failures in a Scania Truck Air Pressure System (APS) (ii) Steel Plates Faults dataset where the actual fault is the detection of faulty steel plates determined from measured attributes representing the plates geometric shape of the fault and its contour. We discuss the datasets, actual faults, challenges, and system objectives, followed by the implementation details of a safety-related fault detection system.
Transfer Learning: We introduce Transfer learning in the literature review, citing a recent study (pg 3 line 135-138). We also include it in the summary wide-table 1 (pg 4).
Author Response File: 
Author Response.pdf
Reviewer 2 Report
This paper used deep learning method for fault detection using steel plates faults dataset.
- This paper showed experimental results from steel plates faults dataset. Please try more dataset for proposed method for reliability.
- Please add figure showing overall model architecture.
- Please add Precision and F1-score in Table 1 and Table 2 for better understanding.
Author Response
Thank you for your excellent review: We believe it will improve the paper in clarity and structure. We have addressed the concerns as follows:
Additional dataset for the proposed method for reliability: We have included experimental results from APS Failure at Scania Trucks dataset. See section 3.7.1 (pg 11) and in table 1 (pg 12), including a further discussion in section 4 of results and discussion (pg 15, line 542 -- 557).
Figure showing overall model architecture: We have included a labeled diagram of the system architecture outlining the framework. See Figure 1 (pg 6).
Add Precision and F1-score in Table 1 and Table 2: Following major revisions to the paper, we have included a comparison table 1 (pg 12), which features the Precision and F1-score. Addition results can be viewed in the appendix section Tables 1 and 2.
Author Response File: Author Response.pdf
Reviewer 3 Report
The comments -
1) The proposed methodology seems difficult to understand. An overall diagram, algorithm, or pseudocode might be helpful for the readers.
2) As the authors proposed deep learning-based model - the below paper might be helpful to highlight the contributions in terms of deep learning modeling.
Deep Learning: A Comprehensive Overview on Techniques, Taxonomy, Applications and Research Directions, SN Computer Science, Springer.
3) Experimental results can be shown/compared with Chart/figure for easy visualization.
Author Response
Thank you for your excellent review. We believe it will improve the paper in clarity and structure. We have addressed the concerns as follows:
An overall diagram: We have included a labeled diagram of the system architecture outlining the framework. See Figure 1 (pg 6).
Ref paper for deep learning-based model: We have included a reference in connection to our paper highlighting the contribution to deep learning (pg 12 lines 405 -- 413).
Experimental results visualization: We have included three figures visualizing the experimental results: figure 3 (pg 9) visualizing the effect of switching the logit weights during training, figure 5 (pg 16) visualizing statistics for type I and type II errors, and figure 6 (pg 17) visualizing misclassification costs dispersion for the methods applied.
Author Response File: Author Response.pdf
Round 2
Reviewer 3 Report
This reviewer has a concern regarding the title "Deep Learning for Industry 4.0". Actually, Industry 4.0 represents many application areas of The Fourth Industrial Revolution. However, in this paper, the authors focus only on fault detection in manufacturing systems. Thus the title should be specific.
The authors can think and change the title accordingly.
Author Response
Thank you for your excellent review. We have addressed the concerns as follows:
The title has been edited into the following:
Handling Imbalanced Datasets for Robust Deep Neural Network-based Fault Detection in Manufacturing Systems
Author Response File: Author Response.pdf
