A Holistic Quality Assurance Approach for Machine Learning Applications in Cyber-Physical Production Systems

Round 1

Reviewer 1 Report

Engineers dedicate the study to the study and problem solving of improving data quality to ensure the efficiency and transparency of the use of artificial intelligence methods and tools in practice. The topic and content of the study at the present stage of development of production systems, in the conditions of Industry 4.0 and the need for the use for analysis and making effective decisions in conditions of uncertainty of production parameters is very relevant for science and practice. Section 2 examines the current state of data quality assurance problems, describes the structure of the mechatronic system and cyber-physical production systems (CPPS), studies data processing and data mining algorithms. Models of data quality are analyzed, the main criteria for assessing data quality are highlighted and described. The essence of the methodology and quality assurance methods used is outlined in Section 3. Section 4 outlines the concept of the generated V-model to ensure the quality of data used in machine learning applications for CPPS. All stages of the proposed V model for ensuring data quality are described and criteria for assessing data quality for these stages of the V-model are assigned. Thus, the main results of the article can be noted:

- based on the system analysis methodology, an integrated approach to ensuring the quality of not only basic data, but also transaction data (signal data) with variable values is proposed;

- investigated and resolved the main issues of ensuring quality and transparency in data processing, timely avoiding incorrect results and decisions, and increasing the confidence of engineers in artificial intelligence methods;

-  on the basis of the proposed integrated approach to data quality assurance, a V-model has been built and described, designed to ensure data quality, which is of theoretical and practical importance;

- research results can contribute to the transparency and reliability of machine learning methods, and can also ensure safety in the processes of using CPPS.

On the work, you can note some minor remarks, recommendations and wishes, which, in our opinion, can improve the content of the work:

1. The content and forms of table 1 - Data quality criteria and examples of corresponding methods for quality inspection as well as for quality restoration using the example of signal data, given in section 2, could be improved, for example, the first column (column) of the table, which is empty, it is not clear why, in the remaining columns it was also possible to provide more meaningful information.
2. In Section 3, it is desirable to present the problem of data quality assurance in an even more formalized form.
3. In the Introduction or in Section 4, it was possible to consider situations of uncertainty that negatively affect the quality of data and initial information, including the problems of fuzziness of initial information, which are also recently used in solving optimization problems, decision-making and management of production facilities and when creating intelligent systems. If necessary, you can see, for example, work from the journal Applied Sciences: https://www.scopus.com/record/display.uri?eid=2-s2.0-85114691906&origin=resultslist

To that end, I would recommend to the authors to make the above-suggested corrections prior to be accepted in the Journal.

Comments for author File: Comments.pdf

Author Response

Thank you for your valuable comments. We have incorporated these and also the similar comments from the other reviewer as follows: 

Note 1: You were absolutely right. Now I also like the new table 2 much better. I think it provides quite good clues for understanding in the subject matter.

Note 2: We have tried to show their question in Figure 6 as an example application.

Note 3: We have considered this point in Chapter 1. Thank you for the useful reference. Some examples have been cited. This opens a very wide field, so it should be enough to use the references you recommended.

Best regards from Dresden
Hajo Wiemer

Translated with www.DeepL.com/Translator (free version)

Author Response File: Author Response.pdf

Reviewer 2 Report

This paper proposes A Holistic Quality Assurance Approach for Machine Learning Applications in Cyber-physical Production Systems. In general, this paper is well presented. The following issues can be further considered.

1. The motivation of proposing such algorithm on the specific application scenario should be more highlighted. Why introducing this method and what is the major difference from the existing ones?

2. The tables should be modified to be in better viewing shape.

3. Some related works on this topic should be reviewed, such as "Open Set Domain Adaptation In Machinery Fault Diagnostics Using Instance-Level Weighted Adversarial Learning", "Deep Learning-Based Prognostic Approach for Lithium-ion Batteries with Adaptive Time-Series Prediction and On-Line Validation" etc.

4. The experimental part should be enhanced. More ablation studies should be added to examine the effects of the proposed method in different situations.

5. A flow chart of the proposed method should be added to be clear about the algorithm.

Author Response

Thank you for your valuable comments. We have incorporated these and also the similar comments from the other reviewer as follows: 

Note 1: We have considered this point in Chapter 1. Thank you for the useful comment. Some examples have been cited. This opens a very wide field, so it should be sufficient to use the references you recommended.

Note 2: You were absolutely right. Now I also like the new Table 2 much better. I think it provides quite good references for understanding in the subject matter.

Note 3: I tried to answer these hints together with note 1 in chapter 1, so that the paper does not become even longer.

Note 4: This is difficult because we are talking about an abstraction. The abstraction is based on many different projects that have been worked on in the institutes. Some are enumerated.

Note: 5 There are currently no detailed flowcharts. Nevertheless, we have tried to show their question in Figure 6 as an exemplary application. If you like, the proposed V-model is a flowchart, only on the meta-level. It is a process model that is used to derive specific workflows. Detailed use-case specific flowcharts are currently being developed and will be published in follow-up papers.

Best regards from Dresden
Hajo Wiemer

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

All my comments are well addressed. It can be accepted.