Comparing Approaches for Evaluating Digital Interventions on the Shop Floor^†

Round 1

Reviewer 1 Report

Very interesting topic with the high potential to develop and further research. I have only minor suggestions. 

I think it will be better to explain shortcuts HMI and HDI same like for AR/VR/HMD. 

Figures 5, 6, 7 and Table 1 are blurry (should be in the higher resolution). 

Improve graphics in figures 8 and 12 to unify them like figures 9-11. Add the unit on the left side of the charts. 

Do not capitalize words after a semicolon (line 466-468). 

I think there is no necessity to present heuristic items in chapter 4.2 as a figure (figure 13) - just list them in the text. 

Basing on the text in the last chapter it should be named as "summary" rather than "conclusions". 

Author Response

Dear reviewer,

Thanks for your suggestions.  I think they are included on attached document.

1)       I think it will be better to explain shortcuts HMI and HDI same like for AR/VR/HMD.

Response:

We did it and review the paper for others possible shortcuts requiring explanation.

2)       Figures 5, 6, 7 and Table 1 are blurry (should be in the higher resolution).

Response:

We did it an review all the picture trying to avoid the same problem.

3)       Improve graphics in figures 8 and 12 to unify them like figures 9-11. Add the unit on the left side of the charts.

Response:

We did the improvement of the graphics.  About the units I Introduced an explanation before the first charts and link it to the quantification process explained in chapter 2.1.1 of the paper.  Basically as our framework consider multiple sources que transform raw data to a value not considering units normalized to [0,1] in order to be able to operate with different data.

4)        Do not capitalize words after a semicolon (line 466-468).

Response:

We did it an review the rest of the paper looking for the same issue.

5)       I think there is no necessity to present heuristic items in chapter 4.2 as a figure (figure 13) - just list them in the text.

Response:

Thanks for the suggestion, done.

6)       Basing on the text in the last chapter it should be named as "summary" rather than “conclussions”.

Response:

Thanks done.

Please give us your feedback  and BR.

Author Response File: Author Response.pdf

Reviewer 2 Report

- The abstract mainly focuses on the research question and the method, but not on the results. It should be rewritten accordingly.

- Sentences are far too long. One sentence, for instance, runs from line 42 to line 46. The sentences must be divided and shortened throughout the manuscript.

- Authors mention the concept of Industry 4.0, but do not relate the requirements to integrate humans into the concept. This link must be drawn clearly. For an overview, please consult, e.g.

Ansaria, F., Erola, S., & Sihna, W. (2018). Rethinking Human-Machine Learning in Industry 4.0: How Does the Paradigm Shift Treat the Role of Human Learning?. Education & Training, 2351, 9789.

Kiel, D., Müller, J. M., Arnold, C., & Voigt, K. I. (2017). Sustainable Industrial Value Creation: Benefits and Challenges of Industry 4.0. International Journal of Innovation Management, 21(8). or

Müller, J. M., Kiel, D., & Voigt, K. I. (2018). What Drives the Implementation of Industry 4.0? The Role of Opportunities and Challenges in the Context of Sustainability. Sustainability, 10(1), 247.

Wittenberg, C. (2016). Human-CPS Interaction-requirements and human-machine interaction methods for the Industry 4.0. IFAC-PapersOnLine, 49(19), 420-425.

- The introduction turns into a literature review before even explaining the broader picture. Here, the general context, also including Industry 4.0 as a concept as mentioned above, must be highlighted.

- Figures do not have the same resolution and size, foremost figures 8 to 11. Figures 5 to 7 should be larger in order to make them readable. Table seems to be squeezed (line 2, left column).

- Subheadings appear twice, e.g., section 2.2, 2.3 comes after 2.1.

- It should be deduced from academic literature what heuristics are and why they are used.

- Authors talk a lot about AR and VR and then present a tablet solution. Why was this solution chosen?

- The authors promise "several contexts-of-use of the industrial partners" and then present a single case. The entire chapter 3 must be better introduced.

- Y-axis labels are missing in several figures. Labels "Innovation and Problem S." (what is S.?), what is the last data row in figure 12?

- A clear limitation is the small sample size, data only from a single case, and we do not know what tasks the empirical results relate to. In what tasks were the workers supported by the tablet, what tasks does the empirical data relate to?

- Who was chosen to be interviewed and why?

- The results in section 4.2. would be much more understandable in a table or alike, not presenting plain text.

- In total, the text seems like a lot to read for (at least sometimes) little content. This is also a result of too long sentences. The authors should try to streamline their text accordingly and highlight their central findings more clearly.

- The conclusion should better highlight limitations of the study and suggestions for future research.

Author Response

Dear Reviewer,

Thanks for your comments.  We think they contribute to improve our paper.

1.        The abstract mainly focuses on the research question and the method, but not on the results. It should be rewritten accordingly.

Response:

We modified the abstract trying to follow your suggestions.

2.        Sentences are far too long. One sentence, for instance, runs from line 42 to line 46. The sentences must be divided and shortened throughout the manuscript.

Authors mention the concept of Industry 4.0, but do not relate the requirements to integrate humans into the concept. This link must be drawn clearly.

The introduction turns into a literature review before even explaining the broader picture. Here, the general context, also including Industry 4.0 as a concept as mentioned above, must be highlighted.

Response:

We agree with your suggestion.  We included the relation of the Industry 4.0 and the kwnoledge workers in the first paragraphs of the introduction.

3.        Figures do not have the same resolution and size, foremost figures 8 to 11. Figures 5 to 7 should be larger in order to make them readable. Table seems to be squeezed (line 2, left column).

Response:

Corrected

4.        Subheadings appear twice, e.g., section 2.2, 2.3 comes after 2.1.

Response:

Corrected

5.        It should be deduced from academic literature what heuristics are and why they are used.

Response:

We reference the paper in which the process for selecting the eight heuristics is presented.

6.        Authors talk a lot about AR and VR and then present a tablet solution. Why was this solution chosen?

Response:

We explained the problems for introducing AR/VR on the shop-floor in the introduction.

7.        The authors promise "several contexts-of-use of the industrial partners" and then present a single case. The entire chapter 3 must be better introduced.

Response:

The presented Industrial Challenge is one of the four and they are linked to one main UC and several “adopting” use cases.  We try to explain it and we reference project reports and papers introducing them, their link and results.

8.        - Y-axis labels are missing in several figures. Labels "Innovation and Problem S." (what is S.?), what is the last data row in figure 12?

Response:

In the paragraph introducing the first figure we explain why we do not use units in the Y axis.  It is due to the fact FACTS4WORKERS framework use multisource data in consequence, as explained in chapter 2.1.1, we need to normalize data in order to be able to operate with them.  The normalization process “suprime” the units and express each assessment as a real value in [0,1] without unit.

9.        A clear limitation is the small sample size, data only from a single case, and we do not know what tasks the empirical results relate to.

Response:

We include an explanation in the Summary chapter.

10.     In what tasks were the workers supported by the tablet, what tasks does the empirical data relate to?

Response:

We try to explain better in the introduction of chapter 3.4

11.     -Who was chosen to be interviewed and why?

Response:

It was determined by the pilot workplace in which the solution is deployed.  As explained in chapter 4.1

12.     - The results in section 4.2. would be much more understandable in a table or alike, not presenting plain text.

Response:

We included the findings link to the solutions in a table.  However, we keet the sentence because we think they present them in a logic way.

13.     In total, the text seems like a lot to read for (at least sometimes) little content. This is also a result of too long sentences. The authors should try to streamline their text accordingly and highlight their central findings more clearly.

Response:

We reviewed and try to improve some paragraphs

14.     The conclusion should better highlight limitations of the study and suggestions for future research.

Response:

Try to include future research.  Other reviewer suggested to modify the title to summary.

BR.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Dear authors, thank you for your revisions. From my point of view, two major points remain:

1) The revision seems to have been done a bit too quickly, please see three examples:

- Line 61: "current technology readiness level of AR systems is still below 8" What is 8, on what scale?

- Line 431: "After logging" -> "After logging in?" (There is also a lot of double spacing in this new paragraph.)

- Line 488: "they participated workers" -> Who is "they"? Or is it "the participating workers"?

The authors must carefully revise and re-check the new parts, as some of the new parts have serious grammatical errors and are hard to understand.

2) The introduced new references in the introduction are often references from the same project. Apart from that, the majority of new references are not from established scientific journals. For instance, Kagermann et al. (2011) is a German trade magazine, I would at least use the 2013 version of Kagermann et al. in English.

Further, the authors did not inlude the references suggested in the last comment, or find more suitable and scientifcally established ones. Again, I suggest to include, for instance:

Ansaria, F., Erola, S., & Sihna, W. (2018). Rethinking Human-Machine Learning in Industry 4.0: How Does the Paradigm Shift Treat the Role of Human Learning?. Education & Training, 2351, 9789.

Kagermann, H.; Wahlster, W.; Helbig, J. Recommendations for Implementing the Strategic Initiative Industrie 4.0—Final Report of the Industrie 4.0 Working Group; Acatech—National Academy of Science and Engineering: Frankfurt am Main, Germany, 2013.

Kiel, D., Müller, J. M., Arnold, C., & Voigt, K. I. (2017). Sustainable Industrial Value Creation: Benefits and Challenges of Industry 4.0. International Journal of Innovation Management, 21(8).

Müller, J. M., Kiel, D., & Voigt, K. I. (2018). What Drives the Implementation of Industry 4.0? The Role of Opportunities and Challenges in the Context of Sustainability. Sustainability, 10(1), 247.

Wittenberg, C. (2016). Human-CPS Interaction-requirements and human-machine interaction methods for the Industry 4.0. IFAC-PapersOnLine, 49(19), 420-425.

Also, the references inserted are not in the correct order according to the referencing guidelines.

Author Response

Dear Reviewer,

We attended your suggestions

1) the suggested corrections and others were made

2) we review references and we also rebuild introduction considering the papers you suggested.

Thanks for your comments.