Three-Dimensional Coordinate Calibration Models for Augmented Reality Applications in Indoor Industrial Environments

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The manuscript deals with a proposition of a new approach to handle and improve coordinate calibration of HoloLens-based AR application in industrial indoor-scenarios. The manuscript is clearly written and has a good structure. There are however a couple of points the authors should reflect when preparing a revised version of the manuscript:

1)      The background section lacks an introduction of studies that were dedicated to the improvement of HoloLens functionalities. Examples are (just to mention a few): https://ieeexplore.ieee.org/abstract/document/8951907 or https://doi.org/10.1007/s42489-019-00025-z

 

2)      The introduction section ends very abruptly. In ll. 68-71, you mention “the lack of studies related to the calibration of AR systems”. Your paragraphs above do not go in the details what previously published studies offer as connecting points to your research. Where exactly are the gaps in terms of calibration? Could these papers help? https://doi.org/10.1115/MSEC2018-6660 or https://doi.org/10.1117/12.2255831

 

3)      The manuscript lacks a discussion section. Without this, the readers do not get a compact overview of the study outcome and its relations to other research. Please write a discussion in a revised version.

Comments on the Quality of English Language

Moderate editing of English language required

Author Response

Dear Reviewer,

We are pleased to resubmit the manuscript of our paper entitled Three-Dimensional Coordinate Calibration Models for Aug-mented Reality Applications in Indoor Industrial Environments, revised according to your recommendations, which considerably contributed to the improvement of our work.

Please find a detailed description of the changes made below:

Point 1 - The background section lacks an introduction of studies that were dedicated to the improvement of HoloLens functionalities. Examples are (just to mention a few): https://ieeexplore.ieee.org/abstract/document/8951907 or https://doi.org/10.1007/s42489-019-00025-z

Response 1 – Thank you for bringing this to our attention. In the revised version, we added a paragraph (lines 48-55) and the two following studies that approached improvements of HoloLens features.

11. S. Knopp, P. Klimant and C. Allmacher, "Industrial Use Case - AR Guidance using Hololens for Assembly and Disassembly of a Modular Mold, with Live Streaming for Collaborative Support," 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Beijing, China, 2019, pp. 134-135, doi: 10.1109/ISMAR-Adjunct.2019.00-63.
12. Keil, J., Edler, D. & Dickmann, F. Preparing the HoloLens for user Studies: an Augmented Reality Interface for the Spatial Adjustment of Holographic Objects in 3D Indoor Environments. KN J. Cartogr. Geogr. Inf. 69, 205–215 (2019). https://doi.org/10.1007/s42489-019-00025-z

 

Point 2 - The introduction section ends very abruptly. In ll. 68-71, you mention “the lack of studies related to the calibration of AR systems”. Your paragraphs above do not go in the details what previously published studies offer as connecting points to your research. Where exactly are the gaps in terms of calibration? Could these papers help? https://doi.org/10.1115/MSEC2018-6660 or https://doi.org/10.1117/12.2255831.

Response 2 – We appreciate your recommendation. In the revised version we included a new paragraph in the Introduction section (lines 64-71) and the two following references to summarize the contributions of previous studies to our research.

15. Reid Vassallo, Adam Rankin, Elvis C. S. Chen, and Terry M. Peters "Hologram stability evaluation for Microsoft HoloLens", Proc. SPIE 10136, Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment, 1013614 (10 March 2017); https://doi.org/10.1117/12.2255831
16. Radkowski, R, & Kanunganti, S. "Augmented Reality System Calibration for Assembly Support With the Microsoft HoloLens." Proceedings of the ASME 2018 13th International Manufacturing Science and Engineering Conference. Volume 3: Manufacturing Equipment and Systems. College Station, Texas, USA. June 18–22, 2018. V003T02A021. ASME. https://doi.org/10.1115/MSEC2018-6660

 

Point 3 - The manuscript lacks a discussion section. Without this, the readers do not get a compact overview of the study outcome and its relations to other research. Please write a discussion in a revised version.

Response 3 – Thank you for your comment. In the revised version we created the new 3.4 Discussion section (lines 346-374) to discuss the study’ outcomes and its relations to current research. We also rewrote the Abstract and the Conclusions section to make the contributions of our research clearer.

Reviewer 2 Report

Comments and Suggestions for Authors

This manuscript evaluates the importance of accuracy and precision of a calibration model in which used to address the distortions caused by incorrect recognition of the marker. Overall, this research is very interesting and was conducted in a well-organized manner. The manuscript is well-written a good read. However, there are major weak spots that either need a more elaborate clarification or a better justification of choices.

- The abstract seems to be a little misguiding from the content in the main manuscript. It can be rewritten to make the readers understand better. 
- The introduction is missing a clear research question/ purpose of the manuscript.
- The introduction seems too long and includes unnecessary statements which are not required. It is better if point to point statement is made related to the objective of the study.
- Authors should elaborate more on the accuracy and precision and make it clear to the reader because current manuscript does not provide information what is the difference between both. 
- I suggest authors to add discussion section and discuss their outcomes instead of just reporting the results without interpreting the real meaning of the results. Because current section of results seems to be too long with the inability to summarize the authors’ point of view.
- The conclusion is too arbitrary. Suggest rewriting the conclusion. Also, the section enumeration is wrong: 3--> 4. 
- What was the experimental design and statistical analysis performed? 
- The whole manuscript lacks vision, and it is suggested for the authors to clearly summarize what message the readers will draw post reading this manuscript.

 

Author Response

Dear Reviewer,

We are pleased to resubmit the manuscript of our paper entitled Three-Dimensional Coordinate Calibration Models for Aug-mented Reality Applications in Indoor Industrial Environments, revised according to your recommendations, which considerably contributed to the improvement of our work.

Please find a detailed description of the changes made below:

     Point 1 - The abstract seems to be a little misguiding from the content in the main manuscript. It can be rewritten to make the readers understand better.

 

Response 1 – Thank you for bringing this to our attention. We rewrote the Abstract to make the research question, the methods, the outcomes, and the contributions of our research clearer.

 

Point 2 - The introduction is missing a clear research question/ purpose of the manuscript.

 

Response 2 – We appreciate the feedback. In the revised version we improved the introduction (starts at line 74) to clarify the question and purpose of the study.

 

Point 3 - The introduction seems too long and includes unnecessary statements which are not required. It is better if point to point statement is made related to the objective of the study.

 

Response 3 – Thank you for your comments. In the revised version we excluded the paragraphs 2, 3, and 4 (previously lines 38-50) from the introduction. We understand that the studies cited there support the statement: "The technological advances observed over the last decade have stimulated the implementations of several AR applications in industry" (lines 33-35). Consequently, we include the studies 2-8 as reference to this statement.

 

Point 4 - Authors should elaborate more on the accuracy and precision and make it clear to the reader because current manuscript does not provide information what is the difference between both.

 

Response 4 – Thank you for bringing this to our attention. In the revised version we included the concept of accuracy and precision from ISO 5725-1:2023 (lines 142-145).

 

Point 5 - I suggest authors to add discussion section and discuss their outcomes instead of just reporting the results without interpreting the real meaning of the results. Because current section of results seems to be too long with the inability to summarize the authors’ point of view.

 

Response 5 – Thank you for your comments. In the revised version we created the new 3.4 Discussion section (lines 346-374) to discuss the study’ outcomes and its relations to current research. We also rewrote the Abstract and the Conclusions section to make the contributions of our research clearer. 

 

Point 6 - The conclusion is too arbitrary. Suggest rewriting the conclusion. Also, the section enumeration is wrong: 3--> 4.

 

Response 6 – We appreciate the feedback. In the revised version we improved the conclusion, as well we adjusted the enumeration.

 

Point 7 - What was the experimental design and statistical analysis performed?

 

Response 7 – Thank you for your comments. We improved the section "2. Materials and Methods" to clarify that our study adopted a Design Science Research approach, we added the following reference related to DSR approach and described in detail each one of the six steps we followed (lines 85-134).

 

16a. GREGOR, S.; HEVNER, A. R. Positioning and presenting design science research for maximum impact. MIS quarterly, JSTOR, p. 337–355, 2013.

 

Point 8 - The whole manuscript lacks vision, and it is suggested for the authors to clearly summarize what message the readers will draw post reading this manuscript.

 

Response 8 – Thank you for bringing this to our attention. In the Conclusion, we added a summary of our findings and the contributions of our work (lines 376-388) and we also rewrote the Abstract to make the research question, the methods, the outcomes, and the contributions of our research clearer.

Reviewer 3 Report

Comments and Suggestions for Authors

The text discusses the complexity of calibrating three-dimensional (3D) coordinates in augmented reality systems, particularly in large industrial environments. Authors explore the development of calibration models for Microsoft HoloLens mixed reality devices and presents results indicating enhanced precision and accuracy in determining 3D point coordinates within the studied environment, primarily affecting the observed point positions.

The article is a typically technical item regarding a specific model of the HoloLens 1 device (from 2016). It should be noted that AR technology and the devices themselves are constantly being improved, and currently there is already a HoloLens 3 model. Therefore, it is worth checking and comparing other, newer models. Nevertheless, the proposed method is interesting from the point of view of users of augmented reality devices. It is worth emphasizing the convincing comparisons of the simple positioning method with the method proposed by the authors of the publication using violin charts. Unfortunately, the reader may not be satisfied with the discussion of the obtained results.

Author Response

Dear Reviewer,

We are pleased to resubmit the manuscript of our paper entitled Three-Dimensional Coordinate Calibration Models for Aug-mented Reality Applications in Indoor Industrial Environments, revised according to your recommendations, which considerably contributed to the improvement of our work.

Please find a detailed description of the changes made below:

 

Point 1 - AR technology and the devices themselves are constantly being improved, and currently there is already a HoloLens 3 model. Therefore, it is worth checking and comparing other, newer models. The proposed method is interesting from the point of view of users of augmented reality devices. It is worth emphasizing the convincing comparisons of the simple positioning method with the method proposed by the authors of the publication using violin charts.

Response 1 – Thank you for bringing this to our attention. In the revised version we added a new paragraph (lines 332-337) and we rewrote the Methods Sections (lines 86-186) to clarify that two-step calibration is the proposed method and the HoloLens 1 was just a tool used to prove that concept. In the future the same concept can be applied with any other devices that share the same augmented reality concepts (including non-Microsoft devices).

 

Point 2 - Unfortunately, the reader may not be satisfied with the discussion of the obtained results.

Response 2 – Thank you for your comments. In the revised version we created the new 3.4 Discussion section (lines 346-374) to discuss the study’ outcomes and its relations to current research. We also rewrote the Abstract and the Conclusions section to make the contributions of our research clearer.  We also added a new paragraph in the Conclusion to clarify this point.

Reviewer 4 Report

Comments and Suggestions for Authors

The paper addresses the complexity of calibrating 3D coordinates in augmented reality (AR) system. For this, it is necessary to know and understand how positions and orientations are computed by the AR device, according to its technology and to environmental features. In industrial context, performance capabilities are impacted with long range measures. Here, the Microsoft HoloLens is used as sensors set for 3D acquisition. The proposed approach is to use a double calibration to enhance precision and accuracy within the case study. Finally, even if experiment description could be more detailed and results presentation can be improved, results show clearly improvements in determining axes defining the 3D space, in relation with the observed points position.

 

Remark: paragraph #3 is used twice (results and conclusions)

Author Response

Dear Reviewer,

We are pleased to resubmit the manuscript of our paper entitled Three-Dimensional Coordinate Calibration Models for Aug-mented Reality Applications in Indoor Industrial Environments, revised according to your recommendations, which considerably contributed to the improvement of our work.

Please find a detailed description of the changes made below:

Point 1 – […] experiment description could be more detailed and results presentation can be improved.

Response 1 – We appreciate the feedback. In the revised version, we rewrote the Methods Sections (lines 86-186) to make the design of our experiment clearer; we also created the new 3.4 Discussion section (lines 346-374) to discuss the study’ outcomes and its relations to current research, and rewrote the Abstract and the Conclusions section to make the contributions of our research clearer. 

 

Point 2 - Paragraph #3 is used twice (results and conclusions).

 Response 2 – Thank you for bringing this to our attention. In the revised version we fixed this issue.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors submitted a revised manuscrpt version. The new manuscript and response letter consider the points mentioned in the first review round. The argumentation in the response letter is sound. Against this background, I would like to recommend this manuscript version for publication.

Comments on the Quality of English Language

Moderate editing of English language required