A Dataset and a Comparison of Classification Methods for Valve Plate Fault Prediction of Piston Pump

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This article evaluates the application of machine learning methods in predicting hydraulic pump valve plate faults through five methods. The effectiveness ranking of various methods is presented, but this article still needs major revisions. 

1、The title of the article is too broad. Machine learning is a very macro category, and this article only uses a few methods, which cannot represent the entire machine learning. The title of the article should focus on the work done, which should involve a comparison of theoretical and experimental methods.

2. The article seems to have applied several methods to the dataset of hydraulic pumps and drawn conclusions through analysis of the results. The innovation of the article is not prominent and may not be suitable for publication in this journal.

 

 

Comments on the Quality of English Language

This article also requires extensive English editing. For example, the first two sentences of the paper are not in a transitional relationship, but they use the conjunction “However”. In addition, the wording is also more colloquial, such as "in public opinion" in the first sentence

Author Response

Dear Reviewer

First we’d like to thank you for your insight and comments which allows us to improve the manuscript.

During the revision process, we’ve made several updates, and sections 4 and 5 were significantly changed by restructuring and adding new subsections concerning sensitivity analysis. In addition, we have found a mistake in the evaluation script (available on github) which caused some changes in the results obtained for the Failure 2, and Failure 3 test sets. That error appeared due to the global properties of Python variables which influenced values in one of the functions. Respectively we updated the results in section 5.1, where the results for Failure 2 got worse but for Failure 3 got improved.

Below we provide a detailed answers to the reviewer’s comments :

• The title of the article is too broad. Machine learning is a very macro category, and this article only uses a few methods, which cannot represent the entire machine learning. The title of the article should focus on the work done, which should involve a comparison of theoretical and experimental methods.

Answer to the comment:

Thank you for the comment. We have updated the title of the manuscript. The current version is: “A Dataset and a Comparison of Classification Methods for Valve Plate Fault Prediction of Piston Pump.”

Indeed the family of machine learning methods is very large and it includes classification methods as well as anomaly detection methods and other approaches. Therefore the current title indicates more precisely the goal of the manuscript which is introduction of a new dataset and the comparison of classification methods in application to failure state detection.

• The article seems to have applied several methods to the dataset of hydraulic pumps and drawn conclusions through analysis of the results. The innovation of the article is not prominent and may not be suitable for publication in this journal.

Answer to the comment:

Thank you for the comment. We searched in the literature and we couldn’t find any real dataset of this type of failure, therefor there were two goals and novelties of this article: first to introduce a dataset which comes from real experiments and the second goal was to give the readers on overview of what can be achieved with this dataset, that is how accurate the classical classification methods can be. We also wanted to evaluate the model if it meets the expert’s knowledge. All these findings we added to the introduction section.

We also want to note, that there are dedicated journals aimed at delivering datasets but then we wouldn’t have the opportunity to evaluate how well various classification methods perform and whether the properties learned by the models meets the experts knowledge. The article shows also a recommended method for evaluation of the extracted knowledge.
These factors influenced our decision to publish this article in the Applied Sciences journal, which is very broad and allows us to publish interdisciplinary articles covering not only novel findings within one domain but more broad aspects covering multiple disciplines.

Reviewer 2 Report

Comments and Suggestions for Authors

I suggest the authors expand chapter 3 with a cross-sectional drawing of the pump, especially marking the valve plate and providing explanations:

1. Was the piston pump with an inclined plate tested or was it a pump with an inclined cylinder block?

2. What are the working and basic design parameters of the pump?

3. What is the number of pistons and what is their influence on low and high pressure pulsations?

4. What materials is the valve plate made of?

5. Which operating and construction parameters of the pump significantly influence the damage to the valve plate?

6. What is the gap between the cylinder block and the valve plate?

7. Is it possible to determine the voltage state of the valve plate and analyze the damage using the finite element method?

Author Response

Dear Reviewer

First we’d like to thank you for your insight and comments which allows us to improve the manuscript.

During the revision process, we’ve made several updates, and sections 4 and 5 were significantly changed by restructuring and adding new subsections concerning sensitivity analysis. In addition, we have found a mistake in the evaluation script (available on github) which caused some changes in the results obtained for the Failure 2, and Failure 3 test sets. That error appeared due to the global properties of Python variables which influenced values in one of the functions. Respectively we updated the results in section 5.1, where the results for Failure 2 got worse but for Failure 3 got improved.

Below we provide a detailed answers to the reviewer’s comments :

1. I suggest the authors expand chapter 3 with a cross-sectional drawing of the pump, especially marking the valve plate and providing explanations:

Thank You for this suggestion. We agree and the drawing as well as description was added to the manuscript, please see Figure 1.

2. Was the piston pump with an inclined plate tested or was it a pump with an inclined cylinder block?

The pump used during research is a pump with inclined plate. Appropriate update with this information was added to the manuscript.

3. What are the working and basic design parameters of the pump?

Nominal parameters of the pump are:

• displacement 45cm³/rev.
• continuous pressure 280bar; peak pressure 350bar

Manuscript was updated accordingly.

4. What is the number of pistons and what is their influence on low and high pressure pulsations?

Thank You for this point. There is nine pistons pump used – information was added to the manuscript. We are aware of the occurrence of pulsation but during this part of research average values of process parameters, including pressure, were used and pressure pulsations were not analyzed. In future we plan to include vibration data in research and also pulsations will be taken into account.

5. What materials is the valve plate made of?

Unfortunately, manufacturer does not provide this information.

6. Which operating and construction parameters of the pump significantly influence the damage to the valve plate?
   
   

We did not conduct any research into the causes of damage, but based on our experience, it can be assumed that a big impact on damage to pump components is caused by contamination of the working fluid, as well as high operating parameters such as pressure or too high oil temperature (significantly lowering oil viscosity) or air dissolved in oil. Low viscosity and/or air in oil can cause cavitation damage which was the cause of damage in one of the valve plates used in the research.

7. What is the gap between the cylinder block and the valve plate?
   
   

There is no technological gap between cylinder block and valve plate. The cylinder block slides on the valve plate surface on oil film. Manuscript was updated with this description.

8. Is it possible to determine the voltage state of the valve plate and analyze the damage using the finite element method?

We are not sure if we understand right Your question, but if You mean mechanical strain/stress in material then probably it could be possible. This way of analysis needs a detailed 3D model of the pump and information about materials for each pump component. We do not have these data and this research did not assume such scope of work.

Reviewer 3 Report

Comments and Suggestions for Authors

Review process of the manuscript

Encoded as: applsci-3103491.

Entitled as: Machine Learning for Valve Plate Fault Prediction of Piston Pump

 

The manuscript is interesting technically because treats a topic related with integrity of mechanical components. Nevertheless, there are many aspects related with writing and scientifical detail must be corrected, I am attaching a *.pdf file with some of them; please check it. In addition, I have some doubts and comments for authors.

 

1. 1     Why do you recorded more data for the test with no failure sample, than for the other 3 where a failure.

2.  2   Authors mention 6.6 million data were collected, what kind of data are integer or float, moreover they do not mention if these are from temperature, fluid, pressure or other variable they worked. This must be explained

3.  3    The document do not have any equation about the model they used or an explanation about how authors treat these data.

4.   4   The most important relationship for calculate lifetime, endurance or resistance of a mechanical workpiece is strain-stress, the authors do not mention if they consider this fact.

5.  5    O.k. I understand the authors are using artificial intelligence to forecast the failure of a mechanical component, it is interesting, of course the most used method is to simulate computationally the component under any effort. The authors presented an alternative way for it, it is nice, maybe they can compare their results with that in future works   

 

 

 

Comments for author File: Comments.pdf

Comments on the Quality of English Language

Author Response

Dear Reviewer

First we’d like to thank you for your insight and comments which allows us to improve the manuscript.

During the revision process, we’ve made several updates, and sections 4 and 5 were significantly changed by restructuring and adding new subsections concerning sensitivity analysis. In addition, we have found a mistake in the evaluation script (available on github) which caused some changes in the results obtained for the Failure 2, and Failure 3 test sets. That error appeared due to the global properties of Python variables which influenced values in one of the functions. Respectively we updated the results in section 5.1, where the results for Failure 2 got worse but for Failure 3 got improved.

Below we provide a detailed answers to the reviewer’s comments :

1. Why do you recorded more data for the test with no failure sample, than for the other 3 where a failure.

The size of the recorded data varies due to our learning process, and natural breaks resulted from the availability of the Ponar research center facility - a place where the experiments were conducted. First, it was impossible to keep identical experiment conditions. For example, we had to warm up the oil which has different initial temperatures between trials conducted in different periods of the year. We first experimented with a proper (not broken) valve plate, and then we waited until the temperature drops down to start another experiment. This process takes hours. In between we had some breaks/end of working hours or we had to postpone the experiments for a month because other experiments were conducted in the Ponar Research Center, so we had to disassemble our system and reassemble it again after the break.

Secondly, we've been learning about how to conduct the experiments. Our first experiments assumed the change of the load following a step-like function, but it turned out that such a load is inappropriate because inaccuracies in the values of load settings can cause a simple decision tree-based model to classify samples with almost perfect matches. Therefore we changed the characteristics of the load of the pomp to sinusoidal to avoid such an information leak. Finally when we knew how to operate the pump and how to conduct the experiments we collected the remaining data of failure 2 and failure 3. After an internal discussion, we decided to use all of the collected data in our experiments. This allows us to even better validate the model and check if the models are not corrupted. We did some small updates to the text to emphasize the differences in the data sizes.

2. Authors mention 6.6 million data were collected, what kind of data are integer or float, moreover they do not mention if these are from temperature, fluid, pressure or other variable they worked. This must be explained

Thank you for the comment, we updated the manuscript accordingly, we added the information on what type of data was collected.  The 6.6 million data samples resulted from high sampling rate, but later it turned out that this frequency was too high and the following samples represent the same value. Therefore the initial data was resampled by averaging values within 1s.

3. The document do not have any equation about the model they used or an explanation about how authors treat these data.

Thank you for the comment, but if we correctly understand your question in most machine learning applications, the equations describing the model are not provided because the models can be derived from various theories. Moreover, we study here a comparison of various models so we would need to provide a very detailed description of all of the models.

4. The most important relationship for calculate lifetime, endurance or resistance of a mechanical workpiece is strain-stress, the authors do not mention if they consider this fact.

Mechanical strain-stress analyze has, indeed, important role but we do not consider this fact. This is because conducted research aim is to find a way to implement predictive system in real industrial application using common process sensors. Strain-stress analyze seems to be mostly related to mechanical engineering and materials science. This paper is focused on analyze of external process parameters of the pump, which is rather considered as a black-box.

5. O.k. I understand the authors are using artificial intelligence to forecast the failure of a mechanical component, it is interesting, of course the most used method is to simulate computationally the component under any effort. The authors presented an alternative way for it, it is nice, maybe they can compare their results with that in future works.

This is interesting idea. In future we consider to create a digital twin of the pump and conduct  further research.

Comments to the issues raised in the peer-review-38330669.v1.pdf:

We’d like to thank the reviewer for his significant work in correcting the manuscript. The PDF with the correction helps us significantly improve the quality of the manuscript.

Below are responses to the  selected comments from the PDF document.

• The diagram presented in Figure 1 is now explained in the manuscript:

“The test HPU built for the research uses a pump (10) driven by an electric motor (3). As a load for the pump, hydraulic motor (14) was used. Further, motor (14) was driving another electric motor (16), which was acting as an adjustable brake for which a specific torque value can be set. Torque value was measured by torque-meter (15). Process parameters for pump were measured by temperature (1, 4, 8), pressure (2, 5, 9) and flow (6, 11) sensors. Check valve (7) is used as a safety relief valve for pump leak port in case of flow-meter(6) clogging. Temperature and pressure sensors(12,13,18,19) for hydraulic motor(14) are used for diagnostic and safety monitoring purposes. As a standard element in hydraulic systems, return oil filter(17) is also used.”

• Figure 3 (after update Figure 4) is now explained:.

“Both types of load were applied randomly to better fit behavior to a real hydraulic system. Graphs showing the applied load torque are shown in Figure \ref{fig:torque_time}. The torque value was set in the range of 20 - 220Nm, which corresponds to the obtained output pressure value in the range around 25 - 235bar. For the stepped type value of the applied torque was changed manually in periods ca. 4-5 minutes. For the sine wave, the period of sine was set to 300s.”

 

• We have moved figure 5 to the proper location. Although it is often further edited by the MDPI correctors who update the manuscript according to the precise editorial rules.
• Regarding figure 6 we updated the caption to reflect the text in the manuscript indicating what is class 0 and 1
• We added the description to figure 7:

In the figures, the $X$ axis represents the drop in accuracy and the $y$ axis represents the name of the features.  The whiskers represent statistics of the data collected for 10 random permutations of the values for each attribute.

 

Reviewer 4 Report

Comments and Suggestions for Authors

Comments to the Authors

applsci-3103491-peer-review-v1: "Machine Learning for Valve Plate Fault Prediction of Piston Pump. ", Marcin Rojek and Marcin Blachnik.  

The author has conducted research based on a laboratory setup was developed that allowed the creation of a training dataset containing both the normal operating state of the pump and the operating state with three different valve plate damages.

 

1. Experimental methodology

In Table 1 the accuracies should be added for the following analysis. Additionally Error Analysis should also be included for discussion.

 

2. Results and discussion

Figure 7 shows the results for 6 parameters, but Table 6 shows 8 parameters. Why there are no results for the other 2 parameters.  Besides, why T_diff is minus and positive?

 

3. Conclusion

2-4 numbered items should be listed for fast understand of the readers.

Author Response

Dear Reviewer

First we’d like to thank you for your insight and comments which allows us to improve the manuscript.

During the revision process, we’ve made several updates, and sections 4 and 5 were significantly changed by restructuring and adding new subsections concerning sensitivity analysis. In addition, we have found a mistake in the evaluation script (available on github) which caused some changes in the results obtained for the Failure 2, and Failure 3 test sets. That error appeared due to the global properties of Python variables which influenced values in one of the functions. Respectively we updated the results in section 5.1, where the results for Failure 2 got worse but for Failure 3 got improved.

Below we provide a detailed answers to the reviewer’s comments :

1. Experimental methodology

In Table 1 the accuracies should be added for the following analysis. Additionally Error Analysis should also be included for discussion.

Answer:

We consider the topic raised by the reviewer to be important; therefore, the article has been expanded with subsection 5.3, which describes the sensitivity analysis of the model with respect to noise that simulates sensor measurement errors.

2. Results and discussion

Figure 7 shows the results for 6 parameters, but Table 6 shows 8 parameters. Why there are no results for the other 2 parameters.  Besides, why Tdiff is minus and positive?

Answer:

Thank you for raising this issue. We have tried to explain it in lines 336 and 343, but we agree that this description was insufficient therefore the paragraph was rewritten to better emphasize why there are fewer attributes.

“(…)As described above, a typical approach is based on the PCA analysis, so that the three attributes with a correlation coefficient above 0.9 were separated from the remaining attributes, and the PCA analysis on those tree attributes was performed. From the PCA the first principal component was used to replace the three temperature attributes. Therefore, the results presented in Figure \ref{fig:feat_importance} represent only 6 attributes (with a single temperature attribute) instead of 8.(…)”

Thank you for pointing out that we didn’t explained the negative value of T_diff in the feature importance plot. In general, the negative value of the $T_{diff}$ obtained for the Failure 2 dataset indicates that permuting this attribute improves the performance. That means that the attribute is useless as it contains pure noise and that the attribute should be rejected. But as pointed out at the beginning we found a mistake in one of the functions which due to the global properties of Python variables influenced the obtained results. We updated the script and now all importances are >= 0

3. Conclusion

2-4 numbered items should be listed for fast understand of the readers.

Answer:

Thank you for the suggestion. We added a bullet list at the end of conclusions.

The results obtained can be summarized as follows.

• three classification datasets were developed which represent 3 levels of valve plate damages,
• the best performance out of the evaluated models was obtained by a neural network consisting of two hidden layers containing 100 and 10 neurons respectively,
• the system accuracy depends on the level of damage,
• the most important attributes are the flow in the leak line, the system output pressure, the pressure on the leak line and the temperature,
• the system performance starts to be sensitive to the input parameters when the level of noise in the input data is higher than 5%.

Reviewer 5 Report

Comments and Suggestions for Authors

This manuscript evaluates machine learning methods to predict valve plate failures in hydraulic pumps. It describes a laboratory setup to generate training data representing both normal and faulty operating states, evaluates five predictive models, and identifies key parameters for accurate prediction. Overall, the manuscript presents a relevant study with clear objectives using machine learning. However, I have some questions for the authors:

1. The authors states that data were collected from both normal and faulty operating states. How many samples were collected for each state, and what was the duration of each data collection session? Please provide a summary of the dataset, including the number of samples and the distribution of each class.

2. The authors discusses the model training process but didn’t provide details on the training and validation split. What proportion of the data was used for training versus validation? Were any cross-validation techniques used to ensure the robustness of the model evaluations?

3. In predictive maintenance tasks, data can often be imbalanced, with more samples of normal states than faulty states. Did you encounter any class imbalance in your dataset? If so, what techniques did you employ to address this issue, such as oversampling, undersampling, or using weighted loss functions?

4. Actually, the real-world data can be noisy, did you test the robustness of your models against noise in the input data? If so, how did you simulate noise, and what was the impact on the model performance? If not, how do you plan to address this potential issue in future work?

 

Author Response

Dear Reviewer

First we’d like to thank you for your insight and comments which allows us to improve the manuscript.

During the revision process, we’ve made several updates, and sections 4 and 5 were significantly changed by restructuring and adding new subsections concerning sensitivity analysis. In addition, we have found a mistake in the evaluation script (available on github) which caused some changes in the results obtained for the Failure 2, and Failure 3 test sets. That error appeared due to the global properties of Python variables which influenced values in one of the functions. Respectively we updated the results in section 5.1, where the results for Failure 2 got worse but for Failure 3 got improved.

Below we provide a detailed answers to the reviewer’s comments :

1. The authors states that data were collected from both normal and faulty operating states. How many samples were collected for each state, and what was the duration of each data collection session? Please provide a summary of the dataset, including the number of samples and the distribution of each class.

Additional tables (Table 3 and 4) were added that describe the number of samples in each dataset and in each class. Data are made public where the exact time of sample collection is given in column ‘Czas’. Link for data: https://www.kaggle.com/datasets/mbjunior/valve-plate-failure-prediction-in-hydraulic-pumps

Approx. time taken for data collection was as follows (+ additional time needed to set-up the data collection system):

No-failure class: ca.  20 hours

Failure 1: ca. 14,5 hours

Failure 2: ca. 4 hours

Failure 3: ca. 4,5 hours

2. The authors discusses the model training process but didn’t provide details on the training and validation split. What proportion of the data was used for training versus validation? Were any cross-validation techniques used to ensure the robustness of the model evaluations?

We performed a two-level evaluation. First, we performed model selection using the cross-validation procedure (stratified k fold test using 10 splits)on the Failure 1 dataset. Here we didn’t use randomized samples to avoid information leak that happens when processing signals – where nearby samples fall into training and testing subsets. After selecting the best classifier with the best hyperparameters we evaluated the obtained models on Failure 2 and Failure 3 which have never been used for training. This allowed us for proper validation of the model.

In the manuscript, we updated parts of the "Model evaluation procedure" and more significantly extended one of the paragraphs:

An important factor worth highlighting was the model evaluation method during cross-validation. The traditional approach to using cross-validation involves randomly splitting the data into individual folds. This procedure was abandoned in this study due to the nature of the data. The recorded data should be treated to some extent as a signal, which means that with randomized data and the step-wise nature of the pump load operation, we would obtain almost identical training and test data. This would consequently lead to an overestimation of the model's quality. Therefore, a linear partitioning approach was adopted in this work. In this schema a dataset is divided into k subsets (we used $k=10$), but not randomly, rather divided into k following subsets where each subset contains $n\/k$  samples ordered by time. In each of $k$ iterations, one of these subsets was used for testing and the remaining were used for training.

3. In predictive maintenance tasks, data can often be imbalanced, with more samples of normal states than faulty states. Did you encounter any class imbalance in your dataset? If so, what techniques did you employ to address this issue, such as oversampling, undersampling, or using weighted loss functions?

Thank you for that question which is often an issue. As presented in Table 4 which we added to the manuscript, we didn’t encounter that problem. This problem often occurs when processing data collected from live objects, but here, since we performed experiments in a laboratory condition we were able to collect a comparable number of samples for failure and none-failure classes. We also pointed it out in the manuscript:

While collecting data from an operational system does not generally seem to be a difficult task, collecting data during system failures are practically impossible because companies strive to prevent failures from occurring. There are two solutions to this problem: one is the creation of digital twins \cite{blachnik2023development, rivera2018towards}, which through computer simulations allow the creation of any failure states, and the other is conducting experiments on actual equipment but in the laboratory conditions.

4. Actually, the real-world data can be noisy, did you test the robustness of your models against noise in the input data? If so, how did you simulate noise, and what was the impact on the model performance? If not, how do you plan to address this potential issue in future work?

Thank you for the question. Initially, we didn’t simulate the impact of the noise/robustness of the model but we found it as an important issue. It was also pointed out (not directly) by another reviewer, therefore we added an additional section 5.3 concerning this issue.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have revised the title, abstract, and overview to make them more relevant to the work of the article. Moreover, in the introduction, the contribution of this article is emphasized, and the highlights of this article are more clear.

In the conclusion of the article, it is suggested that the author looks forward to future research work from the perspective of digital twins, and can refer to the following literature.

1、Digital twin model of gas turbine and its application in warning of performance fault

2、Digital twin driven prognostics and health management for complex equipment

3、Make more digital twins.

Author Response

Dear Reviewer,

We are very grateful for taking Your time to review this manuscript and insightful comments. Your comments on this paper improved it significantly and made it more sound and readable.

Best regards,

Marcin Rojek and Marcin Blachnik

 

According to Your comments these articles have been added to the references list:

29. Minghui, H.; Ya, H.; Xinzhi, L.; Ziyuan, L.; Jiang, Z.; Bo, M. Digital twin model of gas turbine and its application in warning of performance fault. Chinese Journal of Aeronautics 2023, 36, 449–470.
30. Tao, F.; Qi, Q. Make more digital twins. Nature 2019, 573, 490–491

Reviewer 3 Report

Comments and Suggestions for Authors

this paper has been notoriously improved.

Author Response

Dear Reviewer,

We are very grateful for taking Your time to review this manuscript and insightful comments. Your comments on this paper improved it significantly and made it more sound and readable.

Best regards,

Marcin Rojek and Marcin Blachnik

Reviewer 4 Report

Comments and Suggestions for Authors

Romove the period at the end of the title please. 

Author Response

Dear Reviewer,

We are very grateful for taking Your time to review this manuscript and insightful comments. Your comments on this paper improved it significantly and made it more sound and readable.

Best regards,

Marcin Rojek and Marcin Blachnik

 

According to Your comment period at the end of the title has been removed.

Reviewer 5 Report

Comments and Suggestions for Authors

This manuscript can be accepted now.

Author Response

Dear Reviewer,

We are very grateful for taking Your time to review this manuscript and insightful comments. Your comments on this paper improved it significantly and made it more sound and readable.

Best regards,

Marcin Rojek and Marcin Blachnik