Predictors of Successful Maintenance Practices in Companies Using Fluid Power Systems: A Model-Agnostic Interpretation

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Research designs:

First, ANN may not good for the research topics. However, I recommend that authors evaluate the xgboost (https://uc-r.github.io/gbm_regression) algorithm. Gradient boosted machines (GBMs) algorithms are better than most regression models.

Second, authors may add one or two case study for explaining results. The statistic data and results have not provided strong evidences.

 

Comments on the Quality of English Language

To be honesty, it is very challenge to read the paper. Authors have big space to improve their English. For example, In the abstract part, authors were supposed to highlight key results. I wrote a demo for your reference.

English Language:

Before: The eight ML models are utilised, where Ridge regression, Support Vector Regression, ElasticNet regression, and K-Nearest Neighbour offer the highest performance.

 

After: The K-Nearest Neighbors algorithm offers the highest predictive accuracy compared to the other seven ML models, including Ridge Regression, Support Vector Regression, and ElasticNet Regression.

Author Response

Comment 1: First, ANN may not good for the research topics. However, I recommend that authors evaluate the xgboost (https://uc-r.github.io/gbm_regression) algorithm. Gradient boosted machines (GBMs) algorithms are better than most regression models.

Response 1: Thank you for pointing this out, and we agree that ANN has a hard time dealing with multidimensional data, especially considering categorical (nominal) predictors need to be coded. As we could not convert many nominal categories into binary vectors, which would significantly add to the dimensionality problem of the input space (leading to even more sparse data and computational complexity), we switched to label encoding and assigned an arbitrary integer for categories. However, another issue emerged as the ANN may potentially assume the existence of an ordinal relationship, which is the reason for poor performance. However, we still wanted to point the results out as many require ANN as a part of a learning process.

As for the Gradient Boosting Machines, we used them as a part of the regression problem. You can use both for regression and classification problems. Specifically, we used Gradient Boosting from sci-kit learn and obtained the following results: Gradient Boosting (sci-kit learn), we get R2 = 0.189; MSE = 0.803; RMSE = 0.896; MAE = 0.598. Extreme Gradient Boosting (XGBoost), we get R² = 0.264; MSE = 0.728; RMSE = 0.853; MAE = 0.581. Finally, we also used CatBoost and got R2 = 0.237, MSE = 0.756, RMSE = 0.869, and MAE = 0.583. Thank you for pointing this out.

Comment 2: Second, authors may add one or two case study for explaining results. The statistic data and results have not provided strong evidences.

Response 2: We appreciate the reviewer's comment on improving the manuscript. We believe that the longitudinal study of more than two years using a sample of 115 companies pretty much sums up the results of our analysis. That is why we applied rigorous statistical tests to delineate true positive findings and objectify our results. By relying on evaluating every variable that shows a potential statistical inference, we used appropriate statistical tests to avoid making biased assumptions about type I and type II errors. Thus, we believe our findings are built from enough data to make such an assumption. We acknowledge the comments and suggestions from the reviewer, and we will proceed with our longitudinal research.

English editing comment: We appreciate the recommendations from the reviewer. Given our extensive background in writing and publishing papers in English, not to mention that we relied on Grammarly Premium service to ensure the quality and removal of grammar and spelling mistakes, we are quite surprised that the reviewer suggests extensive editing of English. Would you please be kind outside of the proposed change in the abstract (we revised it according to your suggestion) to the point where you would suggest extensive editing so we can use this and request a refund from Grammarly Premium services given that the software failed to remove grammar and spelling mistakes?

Reviewer 2 Report

Comments and Suggestions for Authors

Attached file

Comments for author File: Comments.pdf

Comments on the Quality of English Language

No problem

Author Response

Comment 1: This reviewer respectfully suggests that the introduction be in one item. This division
into subitems in the introduction of an article does not make sense.

Response 1: We appreciate the recommendation from the reviewer and made the changes accordingly.

Comment 2: The introduction mentions the use of multiple ML models and the agnostic interpretation
technique”, but it lacks a clear explanation of why these specific techniques were chosen
and how they compare to traditional methods.

Response 2: We strongly appreciate the feedback for improving and clarifying the manuscript, and we agree that a detailed explanation would help readers understand it better. The rationale for selecting multiple ML and model-agnostic interpretations is as follows. Firstly, the choice of using several ML models in our study is that it will capture complex and (non-linear) relationships between multidimensional maintenance features. Next, using ML models, we not only aim to provide enhanced predictive accuracy in handling these multidimensional features but also to compare and identify the best-performing model that will improve the robustness of the analysis and offer decision-makers a better insight into the features that contribute to the most to the operational and sustainable performance. This way, maintenance engineers and managers can understand the contribution of predictor variables and help them improve their maintenance planning and optimisation tasks. As for the model agnostic interpretation is important for understanding versatility across models because the usage of mean dropout loss is not tied to the specific model but instead interprets feature importance across various ML models, which aids in understanding of different feature importance and ensures comparison across models. The traditional approach of simply using and interpreting models without actual insights about the contribution of different features does not provide enough explainability of features, which in turn may result in using biased features in cases where models are overfitted and completely ignore important features. Thus, by using MDL, we can quantitatively estimate the impact of each feature on the loss function and identify critical factors affecting maintenance performance. In sum, the approach is considered more robust than solely relying on ML models' metrics that cannot be compared across models. Again, we strongly appreciate the feedback and have clearly explained why such an approach is used. Please see line number 65-82.

Comment 3: Computational Performance: "Analyzing the algorithm shows that it can be computationally expensive, especially for a larger data set." This reviewer suggests that a short discussion of the computational complexity of the algorithm (limitations) and possible optimizations could be helpful.

Response 3: We appreciate the comment. Given that you've used quotations, we assume you've found the sentence in our manuscript that states the computational complexity problem. However, we did not mention the computational complexity of any algorithm since we did not consider that we worked with an extensive dataset. 

Comment 4: What stopping criteria are used?

Response 4: Thank you for the question. We have used the threshold for the partial derivatives of the error (loss) function as a stopping criterion set by default in the JASP. See line number 172-173.

Comment 5: Hyperparameter optimisation: "The methodology mentions the optimization of some hyperparameters, but does not detail the optimization methods used." Please mention at least one of the methods used in this optimization

Response 5: Thank you for your valuable feedback. Given the ANN topology, we have used a GA (Genetic Algorithm) for optimisation of network topology, including but not limited to the following parameters and hyperparameter settings described in the article "The hyperparameters of GA are set as Population size = 20; Number of Generations = 10; the number of hidden layers is set to a maximum of 10 layers with a maximum number of 10 nodes in each hidden layer of the network. The parent selection is set to Roulette wheel (we performed different combinations of parent selections using universal, ranked, tournament, random and roulette wheel showed the highest performance); the uniform crossover method is selected (one-point and multi-point crossover method was also tested and uniform crossover method showed highest performance); mutations are set to reset (inversion, swap and scramble were tested and reduced the performance) with 10% probability; the fitness-based is set as a survival method (age-based and random survival were tested and resulted in poorer performance) with 10% elitism.". Please see line numbers 174-185.

Comment 6: Conclusion: It could be more concise and objective, directly emphasizing the main points without repeating information already mentioned.

Response 6: Thank you for your valuable comment. We have worked exhaustively to improve the manuscript's conclusion, reworked contributions to the literature, final remarks, implications, and limitations of the study. Please see 521-543. Again, we strongly appreciate the reviewer's comments and suggestions, and we appreciate the time and effort you've put into making this article more compelling and understandable for the readers.

Reviewer 3 Report

Comments and Suggestions for Authors

- "ML" is abbrеviatеd еarly on but thеn еxpandеd in "Support Vеctor Rеgrеssion" and "ElasticNеt rеgrеssion". It would bе clеarеr to writе "machinе lеarning (ML)" thе first timе and thеn consistеntly usе "ML".

- "hypothеsеs tеsting" should bе "hypothеsis tеsting".

- "mеan dropout loss" is usеd without contеxt and making it hard to undеrstand without prior knowlеdgе.

- Thе sеntеncе starting with "Thе datasеt comprisеs 115 companiеs..." is ovеrly long and should bе brokеn into smallеr and clеarеr sеntеncеs.

- Thе litеraturе rеviеw is somеwhat outdatеd an' lacks rеcеnt studiеs on Maintеnancе 4.0 an' digital compеtеnciеs in fluid powеr systеms. 

Author Response

Comment 1: "ML" is abbrеviatеd еarly on but thеn еxpandеd in "Support Vеctor Rеgrеssion" and "ElasticNеt rеgrеssion". It would bе clеarеr to writе "machinе lеarning (ML)" thе first timе and thеn consistеntly usе "ML".

Response 1: We appreciate the reviewer's suggestions and comments and revised the paper accordingly. We have gone through the whole manuscript and replaced machine learning with abbreviations.

Comment 2: - "hypothеsеs tеsting" should bе "hypothеsis tеsting".

Response 2: We appreciate the feedback from the reviewer. However, we intend to emphasise that we used several hypothesis tests, not relying only on one. Thus, as we used Grammarly Premium for grammar and spelling checks, a plural suggests hypotheses instead of hypothesis.

Comment 3:  "mеan dropout loss" is usеd without contеxt and making it hard to undеrstand without prior knowlеdgе.

Response 3: We strongly appreciate the feedback from the reviewer and extensively describe the importance of the proposed MDL algorithm. The elaboration and rationale for MDL are given in lines 65-83: "The MAI method helps understand the versatility across models by relying on the MDL (Mean Dropout Loss) algorithm. The MDL is not tied to a specific model but provides feature importance based on the loss function. In contrast to the traditional approach of simply relying on the model’s performance and using features of the best-performing model, leading to biased estimates and ignoring essential features, we offer the quantitative impact of individual features and identify critical factors affecting maintenance performance. Such an approach compares features across models and aids in understanding feature contribution. In sum, the approach is considered more robust than solely relying on ML models' metrics that cannot be compared across models. Lastly, after extracting the most relevant features, we conduct a robust statistical analysis of individual features using a variety of hypotheses testing for categorical and continuous predictors."

Comment 4: - Thе sеntеncе starting with "Thе datasеt comprisеs 115 companiеs..." is ovеrly long and should bе brokеn into smallеr and clеarеr sеntеncеs.

Response 4: We would like to thank the reviewer for valuable feedback. We appreciate the comment and revised the sentences in the abstract. Please see line numbers 10-14.

Comment 5: - Thе litеraturе rеviеw is somеwhat outdatеd an' lacks rеcеnt studiеs on Maintеnancе 4.0 an' digital compеtеnciеs in fluid powеr systеms. 

Response 5: We strongly appreciate the reviewer's comment. However, given that our research specifically relies on questionnaire-based survey data for the analysis, we did not consider digital competencies in Maintenance 4.0 to be related to the article idea's literature and background but to overall factors and maintenance activities related to performance metrics under investigation. Still, our related literature covers the range mostly in the last 2-5 years, with one study published in 2015. Even so, we would gladly accept recommendations for expanding the list of references if it will significantly improve the quality of the manuscript. Again, we appreciate the feedback and efforts made by the reviewer to increase the quality of the manuscript. Hopefully, we have fulfilled the expectations of the reviewer and editor for this prestigious journal.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Interesting, authors have asked reviewers to provide evidence for applying for a refund-check against the Grammarly company. Let me quote their words as follows:

"...you would suggest extensive editing so we can use this and request a refund from Grammarly Premium services given that the software failed to remove grammar and spelling mistakes? "

 

Comments on the Quality of English Language

Improving authors' English is the job role of reviewers. Authors must find a native speaker to rewrite the paper making sure that it explain their idea clear.

Author Response

Comments 1: Interesting, authors have asked reviewers to provide evidence for applying for a refund check against the Grammarly company. Let me quote their words as follows:

"...you would suggest extensive editing so we can use this and request a refund from Grammarly Premium services given that the software failed to remove grammar and spelling mistakes? "

Response 1: Dear reviewer, we thank you for your valuable feedback and your extreme desire to improve the quality of the manuscript. This is the first time someone has requested extensive English editing of our manuscript. From our previous use of Grammarly Premium, which we consider a prestigious company and excellent supporting software for scientific writing, such an expert as you certainly has much to offer not just to us but indeed a valuable critique for the software itself. If you agree, we would gladly forward the review report so you can, as a valuable expert and a reviewer who, you said, "Improving authors' English is the job role of reviewers. Authors must find a native speaker to rewrite the paper making sure that it explain their idea clear.", which is we found an absurd statement. We believe that the Editor will undoubtedly agree. Could you state which journal, if any, provides this statement in their guideline for reviewers? We are eager to find it. As for our part, we have valuable feedback for you, if you do not mind correcting you. Perhaps you were a bit tired, and such an expert like you certainly does not make this mistake. Your sentence "Interesting, authors have asked reviewers to provide evidence for applying for a refund-check against the Grammarly company. Let me quote their words as follows:" has grammar and spelling mistakes, and if you do not mind, we can suggest improvement.

So, this sentence should be written as follows: "Interestingly, the authors have asked reviewers to provide evidence for applying for a refund check against the Grammarly company. Let me quote their words as follows: "as per Grammarly, and we could not agree more. We certainly believe this is just a slip-off. Also, if such an Expert, both in English and in the proposed scientific discipline, we would kindly ask you to check the box where it says Open Review: "I would like to sign my review report", so we and the scientific community can get to know the expert much closer and perhaps ask for collaboration in our future EU project proposals since we are currently seeking one.

Again, we are incredibly grateful for your expert and valuable opinions and suggestions for improving the manuscript.

Reviewer 2 Report

Comments and Suggestions for Authors

This reviewer thanks the authors for understanding that comments
aim to further qualify the work. Congratulations on the future publication.

Comments on the Quality of English Language

No problem

Author Response

Comment 1: This reviewer thanks the authors for understanding that comments
aim to further qualify the work. Congratulations on the future publication.

Response 1: The authors are incredibly thankful for your valuable feedback, and we thank you for improving the quality of the manuscript.