Methodology for Rationalization of Pre-Production Processes Using Virtual Reality Based Manufacturing Instructions

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

After reviewing this article, some comments are list as follows:

1. In the abstract, I cannot find the clearly description for the main contribution significantly.

2. Some keywords cannot be found in the Abstract.

3. In the Section2, from line 283 to 310, the Z score should be clearly defined by some references. As shown in Table 5, can it be transferred for forecasting performance?

4. In line 539, how can we define the P7 (i.e. P71 and P72 are distinguished as 80%). Next, the  values of P8 should also be clarified.

5. Can you try to state that the major index to conclude that your designed researches are well-performed?

Author Response

See attached Word file. Thank you very much.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

- 30ff: Pre-production phase is stated as the most influencing phase of the production life-cycle. Rather than the development phase? I don't agree if whole product life cycle is the basis. cp. Ehrlenspiel et. al. Cost-Efficient Design.

 - 57: Interactivity is not a unique selling point of AR. With H5P also video tutorials can be interactive.

- 72: What is the definition of VR in your opinion? Following the 3i approach can 2D VR be considered as VR? cp. Nanjappan et al. EFFECTS OF IMMERSIVE VIRTUAL REALITY IN ENHANCING CREATIVITY
==> The different tools and the exact distinction between them is not clearly explained.

- 73: There are several evaluation methods presented, e.g. in the field of engineering education cp. Balzerkiewitz et. al Evaluation of the Learning Effect of VR on Engineering Education – Case Study in Machine Elements.

- 92: There are some papers presenting methods to choose the right tool at least in the field of engineering education cp. Lyrath et al. Application of Augmented Reality (AR) in the Laboratory for Experimental Physics

-124: There is a difference whether it is a good tool (usability) or whether the tool is suited to the use case (failure rate, effectiveness, efficiency,...). This must be analysed separately.

-726: How was the validation created? Someone used the tool and said: "Yes, that was fun."? What is the validation data? Objective vs. subjective questioning? Usability of the tool vs. learning effect. Also, the fact that 3D VR was suitable does not mean that paper would give worse results. It is not a validation of the "selection method".

-753: This is not a proof.

-996: You cannot draw this conclusion! The authors have shown that the developed use cases work. This is a good result. But we do not know if the selection method works. Perhaps other tools could even be used with greater efficiency. This has not been tested. Since it has been emphasised that the main aim of this publication is to prove that the selection method works, I regret to say that this aim has not been achieved.

- Revise the layout wherever figures or tables are cited. Check the page breaks.

Author Response

See attached Word file. Thank you very much.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The proposed paper explores the use of modern visualization tools for rationalization of the pre-production process.

The abstract is well-written and clearly summarizes the paper's content.

The introduction clearly described the research domain, the open problems, and the state of the art, as well as the goals of the proposed study and the research hypotheses.

Section 2 describes in detail the design and development of the proposed methodology. Since there are many subsections, I recommend starting this section with the section organization. Then, the first step should be to clarify how the design process of the proposed methodology takes into account the sub-objectives described at the end of the previous section. For example, defining the different types of available instructions involves sub-objectives 4 and 5, and the authors take them into account in the evaluation step; thus, it is not convenient to describe them at the beginning of section 2. Moreover, the Implementation phase subsection is quite short and should be expanded to explain better the steps required in this phase. Otherwise, the readers should infer this information from the application of the proposed methodology described in section 3.

The state of the art/references should be improved: it seems to me that the papers pertaining to the usage of AR in the industry are quite recent (2018-2023), whereas instead, there are many research papers investigating the usage of AR for assembly, maintenance and repair procedures published between 2013 and 2018. Some of these papers also address the problem of AR procedure creation and workflow problem and the comparison with paper, video, and remote call instructions. The paper is good in terms of proposed methodology and verification, but I am not quite sure pertaining to the novelty of the research questions: based on my experience and knowledge of the state of the art, most of the time video is better than paper, AR is better than video or VR, AR with multiple options (animations, video, remote call) is the best option; even so, video through AR is usually preferred by the users, and technicians need remote calls only if the real case scenario is somehow different from the one proposed in the video.

Section 3 describes the usage of the proposed methodology in real use cases.

I would suggest improving this section's organization; there are a lot of subsections, and the readability could be improved. First of all, I would recommend providing a section organization at the very beginning, even if it follows the pipeline defined in Section 2.  Moreover, the Validation of the methodology should be a separate section, with a different subsection for each use case, describing the creation and verification for each one of them.  

I would suggest slightly resizing Figure 4; Tables 3, 5, 6, and 9 are quite huge and could be resized greatly; Table 8 should be positioned on a single page. Figure 6 is low-res and should be improved.

Finally, Verification results and summary should be expanded (maybe a separate section, just ‘Results’) since it cannot be limited to the verification/validation part of each use case but should summarize the results pertaining to the usage of the whole methodology to the different use cases (so, the results obtained from each step of the proposed pipeline).

Section 4 briefly discusses the obtained results. This section should be organized into subsections to highlight the answers to the proposed research hypothesis, how the proposed systems address the different sub-objectives and the limitations of the proposed methodology.

Section 5 summarizes the proposed paper. However, I would recommend describing possible Future Works.

Comments on the Quality of English Language

English language and grammar are okay, but minor errors and typos should be fixed, e.g., thesis instead of paper in the last section.

Author Response

See attached Word file. Thank you very much.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

After the revision, I recommend this article to be published in the Journal of Machines.

Author Response

R1: After the revision, I recommend this article to be published in the Journal of Machines.

Response:

We very much appreciate the time you have invested in working together to make this article better. All suggestions were very helpful. Thank you.

Reviewer 2 Report

Comments and Suggestions for Authors

Many thanks for the responses to my review. It is a good piece of work and advances research in this field. I think that the proposed method can work in practice. In particular, it can give inexperienced and unsure people a guideline and confidence in development. Unfortunately, we always have the problem that we cannot prove positive effects due to the diverse influences and lack of reproducibility. We can only describe tendencies. But you've illustrated this well. I wish you continued success in your research work.

Author Response

R2: Many thanks for the responses to my review. It is a good piece of work and advances research in this field. I think that the proposed method can work in practice. In particular, it can give inexperienced and unsure people a guideline and confidence in development. Unfortunately, we always have the problem that we cannot prove positive effects due to the diverse influences and lack of reproducibility. We can only describe tendencies. But you've illustrated this well. I wish you continued success in your research work.

Response:

Thank you also very much for the many valuable comments and suggestions. I'm very glad that you were able to look at the article with fresh eyes and thus we have improved it together. We are of course continuing to work on the research, I am especially glad that we already have a great team. Thank you very much.

Reviewer 3 Report

Comments and Suggestions for Authors

The authors addressed most of my suggestions.

In the introduction, I would argue that “Although the technological immaturity of hardware has so far prevented the deployment of augmented reality for maintenance and service in industrial practice, there are a number of scientific studies that confirm that this direction promises great innovation” is quite a strong sentence. I would change it a bit because technological immaturity and/or cost have limited the widespread of these technologies, but even so, these technologies have been used by big manufacturing companies (e.g., Stellantis, BMW, etc.).  

The overall paper organization has been improved. However, as the authors themselves wrote, section 3 describes the implementation process in detail; thus, I would title it implementation instead of result.

Figures and tables have been adjusted according to my suggestions.

The results’ discussion has been improved, and Section 5 has been improved as well with possible future works.

Comments on the Quality of English Language

I would suggest revising the English language since some sentences are quite long and difficult to read, as well as minor mistakes could be found. For example, the first sentence of the abstract is way too long:

“This paper deals with the rationalization of manufacturing processes within the product life cycle, with an emphasis on the pre-production phase of manufacturing and presents a novel methodology for assessing the applicability of modern visualization tools in production processes, featuring a modified Z-score to categorize manufacturing processes and validated through the successful implementation of real-world projects, ultimately providing a practical decision-making aid for manufacturing companies in deploying such Computer Aided Instruction tools.”

The paper deals…  and presents… featuring a modified Z-score (what is it?)… and validated (what has been validated? The methodology? The modified Z-score?) … providing (who is providing?) …

Author Response

R3C1:

The authors addressed most of my suggestions.

Response:

Thank you very much and we really appreciate the time you spent on our article ergo helping us to make it better.

R3C2:

In the introduction, I would argue that “Although the technological immaturity of hardware has so far prevented the deployment of augmented reality for maintenance and service in industrial practice, there are a number of scientific studies that confirm that this direction promises great innovation” is quite a strong sentence. I would change it a bit because technological immaturity and/or cost have limited the widespread of these technologies, but even so, these technologies have been used by big manufacturing companies (e.g., Stellantis, BMW, etc.).  

Response:

Yes, you are right the statement is too strong. The hardware immaturity of see-through displays in particular was meant here. This has been rephrased in the article: Within the service and maintenance, it is possible to observe many global applications that are implemented using an indirect view, i.e., for example, using a tablet or a mobile phone. For direct view applications, for example using semi-transparent glasses (e.g. MS Hololens 2), we still face many technological limitations. It seems that the use of glasses that mediate the final image by means of cameras (e.g. Meta Quest 3) may be the right way forward, but here we can face legislative restrictions (from an occupational safety point of view, it is usually not possible to work with opaque glasses).

R3C3:

The overall paper organization has been improved. However, as the authors themselves wrote, section 3 describes the implementation process in detail; thus, I would title it implementation instead of result.

Figures and tables have been adjusted according to my suggestions.

The results’ discussion has been improved, and Section 5 has been improved as well with possible future works.

Response:

Based on your comments, we have improved the structure and the formal aspect. Thank you for them. It definitely helped mainly the readability of the article. The Results section has been renamed to Implementation and Results.

R3C4:

I would suggest revising the English language since some sentences are quite long and difficult to read, as well as minor mistakes could be found. For example, the first sentence of the abstract is way too long:

“This paper deals with the rationalization of manufacturing processes within the product life cycle, with an emphasis on the pre-production phase of manufacturing and presents a novel methodology for assessing the applicability of modern visualization tools in production processes, featuring a modified Z-score to categorize manufacturing processes and validated through the successful implementation of real-world projects, ultimately providing a practical decision-making aid for manufacturing companies in deploying such Computer Aided Instruction tools.”

The paper deals…  and presents… featuring a modified Z-score (what is it?)… and validated (what has been validated? The methodology? The modified Z-score?) … providing (who is providing?) …

Response:

You're quoting the original version of the abstract. However, we have divided the sentences into individual sentences to increase clarity. Other subtle corrections have already been made.

Thank you again for everything