Development and Usability Evaluation of Augmented Reality Content for Light Maintenance Training of Air Spring for Electric Multiple Unit
Round 1
Reviewer 1 Report
Comments and Suggestions for Authors
The paper introduces a system for light maintainance training of air spring for electric multiple unit. The paper claims the method being efficient. However, it uses a questionnaire human scoring metric with a weird grading system for evaluation. Nothing can be concluded from such evaluations as it lacks objective measures and compared methods.
Comments on the Quality of English Language
It seems that the English needs minor improvements. For example, line 545 is grammarly incorrect.
Author Response
Thank you for considering my research for publication in applied sciences. I am grateful to you and the reviewers for the valuable suggestions provided.
Here are responses to reviewer comments:
Reviewer #1:
Comment 1: The paper introduces a system for light maintenance training of air spring for electric multiple unit. The paper claims the method being efficient. However, it uses a questionnaire human scoring metric with a weird grading system for evaluation. Nothing can be concluded from such evaluations as it lacks objective measures and compared methods.
Response 1: Although somewhat unfamiliar, the evaluation of AR content developments similar to software development is one of the methods in which the user evaluation method is generally accepted among various techniques. Please understand that in this study, for the usability evaluation technique that enables the evaluation based on the user's experience or knowledge after the user uses and experiences the development product himself, we chose to prepare and evaluate applicable surveys based on the existing techniques. The academic basis related to this method was reinforced in Chapter 1 to help readers understand.
As the reviewer's advice, we updated the references [21-24] that contain examples of using surveys to objectively evaluate previously developed AR content on the line number 104-115.
Reviewer 2 Report
Comments and Suggestions for Authors
Overall, the paper offers a useful analysis of the application of augmented reality (AR) in railway maintenance training, complete with creative additions and encouraging empirical results. Its impact and accessibility could be further enhanced by addressing the identified limitations.
· The introduction should provide comprehensive explanations of essential technical terminology and acronyms to guarantee comprehension for readers who are not familiar with the subject matter.
· Although the related work section is extensive, it would be advantageous to include a broader analysis of the issues faced in railway maintenance and the special advantages of using augmented reality (AR) in this particular context.
· To strengthen the validity of the study, it is recommended to provide additional thorough information about the survey participants, including the selection criteria and demographic data.
· Although the new algorithms (FDG and OAR) are acknowledged as important advancements, their intricate mechanisms are not adequately elucidated. Providing additional details regarding the operational mechanisms and practical use of these algorithms would enhance the paper.
· The survey comprised a group of 40 railway maintenance personnel, but, the paper does not delve into the diversity of this particular sample. Details regarding their origins, levels of expertise, and specific jobs could offer an improved understanding of the results.
· There are minor grammatical errors and awkward phrasings that could be improved with careful editing to enhance readability.
Comments on the Quality of English Language
There are minor grammatical errors and awkward phrasings that could be improved with careful editing to enhance readability.
Author Response
Thank you for considering my research for publication in applied sciences. I am grateful to you and the reviewers for the valuable suggestions provided.
Here are responses to reviewer comments:
Reviewer #2:
Comment 1: The introduction should provide comprehensive explanations of essential technical terminology and acronyms to guarantee comprehension for readers who are not familiar with the subject matter.
Response 1: As the reviewer's point out, we updated the comprehensive explanations of essential technical terminology and acronyms to guarantee comprehension in Chapter 1, Introduction on the line number 39-53.
Comment 2: Although the related work section is extensive, it would be advantageous to include a broader analysis of the issues faced in railway maintenance and the special advantages of using augmented reality (AR) in this particular context.
Response 2: As the reviewers' advice, we updated the manuscript in Discussion regarding major contributions and innovations on the line number 656-666.
Comment 3: To strengthen the validity of the study, it is recommended to provide additional thorough information about the survey participants, including the selection criteria and demographic data.
Response 3: As the reviewer's advice, we updated Chapter 6 Evaluation condition to provide additional information about survey participants on the line number 565-572.
Comment 4: Although the new algorithms (FDG and OAR) are acknowledged as important advancements, their intricate mechanisms are not adequately elucidated. Providing additional details regarding the operational mechanisms and practical use of these algorithms would enhance the paper.
Response 4: As the reviewers' advice, we updated the manuscript by adding Figures 9 and 11 regarding the mechanisms of the new algorithms (FDG and OAR) on the line number 350-359 and 380-378.
Comment 5: The survey comprised a group of 40 railway maintenance personnel, but, the paper does not delve into the diversity of this particular sample. Details regarding their origins, levels of expertise, and specific jobs could offer an improved understanding of the results.
Response 5: As the reviewer's advice, we updated the evaluation criteria in Chapter 6 to provide more information about the origins, level of expertise, and specific occupations of survey participants on the line number 565-572.
Reviewer 3 Report
Comments and Suggestions for Authors
Review Report:----applsci-3131328
Title: Development and Usability Evaluation of Augmented Reality Content for Light Maintenance Training of Air Spring for Electric Multiple Unit
The authors present a special effect algorithm that displays the dispersion and diffusion of fluid, and an algorithm that allows objects to be rotated at various angles, were proposed to increase the visual learning effect of fluid flow for maintenance. The FDG algorithm can increase the training effect by visualizing the leakage of air at a specific location when the air spring is damaged.
However, there are some problems, which must be solved before it is considered for publication. I suggest a major revision. Some problems are as follows
1. In Abstract, “ New visual technologies have been proposed and evaluated to support the management and training of air spring maintenance tasks and to create effective augmented reality content. ” should be revised.
2.Page 4, “Industrial AR is a technology that enhances productivity and improves work effi- ciency by combining real-world and virtual information for on-site workers [27-28].”should be revised.
3. Major contributions and innovations should be added to the Introduction.
4.Studies on usability evaluation, such as user experience assessments. Moreover, the authors should refer the recent work reference in detail. e.g. Finite difference method: An efficient alternating direction implicit finite difference scheme for the three-dimensional time-fractional telegraph equation. Spectral methods: Unconditional convergence of linearized orthogonal spline collocation algorithm for semilinear subdiffusion equation with nonsmooth solution; Superconvergence Analysis of a Robust Orthogonal Gauss Collocation Method for 2D Fourth-Order Subdiffusion Equations. Finite element method: Second-order BDF ADI Galerkin finite element method for the evolutionary equation with a nonlocal term in three-dimensional space.. Finite volume method: Analysis of a new NFV scheme preserving DMP for two-dimensional sub-diffusion equation on distorted meshes.
5.Page 16, “Where, Qi is each question” should be “where, Qi is each question”.
6.Page 16, Eq (2) should be explained in detail..
7. Some words have extra spaces between them.
8. It is suggested to comma in equation.
9. There may be problems with the format of the reference, and it is recommended to check it.
10. Further work, authors can consider preserving some important physical properties and physical structures. The authors can refer some work in section conclusion. Such as, Simple positivity-preserving nonlinear finite volume scheme for subdiffusion equations on general non-conforming distorted meshes; The uniform l1 long-time behavior of time discretization for time-fractional partial differential equations with nonsmooth data;The finite volume scheme preserving maximum principle for two-dimensional time-fractional Fokker–Planck equations on distorted meshes.
Comments on the Quality of English Language
it is ok
Author Response
Thank you for considering my research for publication in applied sciences. I am grateful to you and the reviewers for the valuable suggestions provided.
Here are responses to reviewer comments:
Reviewer #3:
Comment 1: In Abstract, “New visual technologies have been proposed and evaluated to support the management and training of air spring maintenance tasks and to create effective augmented reality content.” should be revised.
Response 1: As the reviewer's advice, we updated the manuscript in the summary that needed correction have been updated on the line number 13-14.
Before modification: New visual technologies have been proposed and evaluated to support the management and training of air spring maintenance tasks and to create effective augmented reality content.
After modification: This study developed augmented reality content that proposes a novel visual technology to effectively support the training of air spring maintenance tasks.
Comment 2: Page 4, “Industrial AR is a technology that enhances productivity and improves work efficiency by combining real-world and virtual information for on-site workers [27-28].” should be revised.
Response 2: As the reviewer's advice, we updated the manuscript in the summary that needed correction have been updated on the line number 168-170.
Before modification: Industrial AR is a technology that enhances productivity and improves work efficiency by combining real-world and virtual information for on-site workers [27-28].
After modification: Industrial AR enhances productivity and efficiency by combining real-world and virtual information, as shown by the System Usability Scale (SUS) and systematic literature reviews covering AR applications, industry interest, development, benefits, and challenges [24-25].
Comment 3: Major contributions and innovations should be added to the Introduction.
Response 3: As the reviewers' advice, we updated the manuscript in Section 2 of the Introduction regarding major contributions and innovations on the line number 158-165.
Comment 4: Studies on usability evaluation, such as user experience assessments. Moreover, the authors should refer the recent work reference in detail. e.g. Finite difference method: An efficient alternating direction implicit finite difference scheme for the three-dimensional time-fractional telegraph equation. Spectral methods: Unconditional convergence of linearized orthogonal spline collocation algorithm for semilinear subdiffusion equation with nonsmooth solution; Superconvergence Analysis of a Robust Orthogonal Gauss Collocation Method for 2D Fourth-Order Subdiffusion Equations. Finite element method: Second-order BDF ADI Galerkin finite element method for the evolutionary equation with a nonlocal term in three-dimensional space. Finite volume method: Analysis of a new NFV scheme preserving DMP for two-dimensional sub-diffusion equation on distorted meshes.
Response 4: As the reviewer's advice, we updated the references [21-24] to include recent studies evaluating user experiences with surveys used in developed content on the line number 104-115.
Comment 5: Page 16, “Where, Qi is each question” should be “where, Qi is each question”.
Response 5: As the reviewer's advice, we modified the manuscript on the line number 555.
Before modification: Where, Qi is each question
After modification: where, Qi is each question
Comment 6: Page 16, Eq (2) should be explained in detail.
Response 6: As the reviewer’s point, we updated the description of the calculation method to better explain equation (2) on the line number 543-549.
Comment 7: Some words have extra spaces between them.
Response 7: As the reviewer's point, we confirmed that there is extra space between some words and have updated the revision accordingly.
Comment 8: It is suggested to comma in equation.
Response 8: As the reviewer's point, we updated the equation by adding a comma on the line number 524 and 553.
Comment 9: There may be problems with the format of the reference, and it is recommended to check it.
Response 9: As the reviewer's advice, we modified the reference format according to MDPI format on the line number 728-825.
Comment 10: Further work, authors can consider preserving some important physical properties and physical structures. The authors can refer some work in section conclusion. Such as, Simple positivity-preserving nonlinear finite volume scheme for subdiffusion equations on general non-conforming distorted meshes; The uniform l1 long-time behavior of time discretization for time-fractional partial differential equations with nonsmooth data;The finite volume scheme preserving maximum principle for two-dimensional time-fractional Fokker–Planck equations on distorted meshes.
Response 10: As the reviewer's advice, we updated the manuscript to include additional information in the conclusion regarding future studies that would consider preserving the physical properties and physical structure of the air spring on the line number 710-712.
Round 2
Reviewer 1 Report
Comments and Suggestions for Authors
The authors don't address the stated problems and do not include baselines for comparison and insist on their weird grading system (84/100=A+, for example), hence I suggest to reject the paper.
Author Response
Thank you for your valuable reviewer's comment.
We have tried to make corrections to address the issues the reviewer mentioned, and have also added statements to explain the criteria for comparison and the weird grading system.
We hope these changes meet your expectations and demonstrate the validity of our findings.
Comment 1: The authors don't address the stated problems and do not include baselines for comparison and insist on their weird grading system (84/100=A+, for example), hence I suggest to reject the paper.
Stated problems 1: Object measure
Response 1: We have updated the manuscript on lines 151-157 to demonstrate, using previous studies, that the UMUX [26-27] and CSUQ [28-29] surveys offer a comprehensive framework for evaluating the usability of AR-based educational products. These surveys have been shown to serve as reliable and objective measures for usability evaluation.
- Brooke, J. SUS: A retrospective. Usability Stud. 2013, 8, 29–40.
- James, R. L. IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction. 1995, 7, 57-78. https://doi.org/10.1080/10447319509526110
- Kraig, F. The usability metric for user experience. Interacting with computers. 2010, 22, 323-327. https://doi.org/10.1016 j.intcom.2010.04.004
- Raymund, D.; Gali, M. A.; Llavore, D. Usability Test of Moodle LMS Using Empirical Data and Questionnaire for User Interface Satisfaction. Proceedings of the 2022 11th International Conference on Software and Computer Applications. Melaka, Malaysia, February 24-26, 2022.
Stated problem 2: Baselines for comparison
Response 1: We have updated our original study (Reference Numbers: 48-61) in lines 657-667, which involved a usability evaluation based on UMUX and CSUQ surveys for AR content developed across various domains. Using this data, we constructed Table 8, which compares the usability evaluation scores from previous studies with the scores of the air spring AR content developed in our research. This comparison allowed us to analyze and compare the effectiveness of the AR content in supporting maintenance and training.
- Stefanidi, Z.; Margetis, G.; Ntoa, S.; Papagiannakis, G. Real-time adaptation of context-aware intelligent user interfaces, for enhanced situational awareness. IEEE Access. 2022, 10, 23367-23393.
- Varela-Aldás, J.; Buele, J.; López, I.; Palacios-Navarro, G. Influence of hand tracking in immersive virtual reality for memory assessment. International Journal of Environmental Research and Public Health. 2023, 20, 4609.
- Eversberg, L.; Lambrecht, J. Evaluating digital work instructions with augmented reality versus paper-based documents for manual, object-specific repair tasks in a case study with experienced workers. The International Journal of Advanced Manufacturing Technology. 2023, 127, 1859-1871.
- Pittman, C. Evaluating Augmented Reality Tools for Physics Education. Doctor of Philosophy, University of Central Florida, Florida USA, 2021.
- Stefanidi, E. Transferring Traditional Crafts from the Physical to the Virtual World: An Authoring and Visualization Method and Platform. ACM Journal on Computing and Cultural Heritage. 2022, 15, 35:1-35:24
- Tärning, J. Giving the Audience a Perspective: Investigating the opportunities and challenges of introducing independent audience controlled cameras to a collaborative virtual environment. degree of Master, KTH ROYAL INSTITUTE OF TECHNOLOGY, Stockholm Sweden, 2021.
- Criollo-C, S.; Abad-Vasquez, D.; Martic-Nieto, M.; Velasquez-G, F. A.; Perez-Medina, J. L.; Lujan-Mora, S. Towards a new learning experience through a mobile application with augmented reality in engineering education. Applied Sciences. 2021, 11, 4921.
- Denisart, L.; Zapata-Dominguez, D.; David, X.; Leclere, A.; Lelong, R.; Liu, C.; Franco, A. A. Combining virtual reality with mixed reality for efficient training in battery manufacturing. Batteries & Supercaps. 2024, 7, e202300268.
- Caria, M.; Todde, G.; Sara, G.; Piras, M.; Pazzona, A. Performance and usability of smartglasses for augmented reality in precision livestock farming operations. Applied Sciences. 2020, 10, 2318.
- Criollo-C, S.; Govea, J.; Jativa, W.; Pierrottet, J.; Guerrero-Arias, A.; Jaramillo-Alcazar, A.; Lujan-Mora, S. Towards the integration of emerging technologies as support for the teaching and learning model in higher education. Sustainability. 2023, 15, 6055.
- Olmos-Pineda, I. An Augmented Reality-Based Application Relying on the Use of Tangible User Interfaces for English Teaching to Children Between. Springer. 2019, 1114, 97-112.
- Oh, J. Y.; Park, J. H.; Park, J. M. Virtual object manipulation by combining touch and head interactions for mobile augmented reality. Applied sciences. 2019, 9, 2933.
- Santiago, C. C.; Andrea G. A.; Buewnano F. D.; Lujan M. S. Usability and workload evaluation of a cybersecurity educational game application: a case study. IEEE Access. 2024, 12, 12771-12784.
- Choi, K. S.; Schmutz, B. Usability evaluation of 3D user interface for virtual planning of bone fixation plate placement. Informatics in Medicine Unlocked. 2020, 19, 100348.
Stated problem 3: weird grading system
Response 1: We have updated the manuscript (lines 629-656) to provide a detailed explanation of the grading systems mentioned by the reviewer. This includes the Curved grading system, the Absolute grading system, and a comparative grading scale. Additionally, we have elaborated on the CGS rating system, which was developed by Sauro et al. as a curve rating scale based on percentiles and average scores. This system is designed to ensure fair rating allocation by analyzing data from 446 surveys and usability studies. The description has been revised to clarify these points.
Reviewer 2 Report
Comments and Suggestions for Authors
Paper may be accepted in its present form.
Author Response
I really appreciate your decision.
I will carefully review the paper based on your valuable comments so that I can write a more complete paper.
Reviewer 3 Report
Comments and Suggestions for Authors
Review Report:----applsci-3131328
Title: Development and Usability Evaluation of Augmented Reality Content for Light Maintenance Training of Air Spring for Electric Multiple Unit
The article has undergone a major revisions. I suggest the author revise the article carefully according to the comments. If the quality of the article does not improve again, I recommend to reject.
1. In Abstract, “ To verify the usefulness of the developed training content, a survey was conducted on 40 railway maintenance personnel ” should be revised.
2. The authors can refer some work, e.g. An efficient alternating direction implicit finite difference scheme for the three-dimensional time-fractional telegraph equation; Unconditional convergence of linearized orthogonal spline collocation algorithm for semilinear subdiffusion equation with nonsmooth solution; Superconvergence Analysis of a Robust Orthogonal Gauss Collocation Method for 2D Fourth-Order Subdiffusion Equations; Second-order BDF ADI Galerkin finite element method for the evolutionary equation with a nonlocal term in three-dimensional space; Construction of a new time-space two-grid method and its solution for the generalized Burgers' equation.
3.Page 13, ”The hierarchical structure section is designed to facilitate easy recognition and learning by organizing the 7 components of the air spring as intermediate items.” should be revised.
4. Further work, authors can consider preserving some important physical properties and physical structures. The authors can refer some work in section conclusion. Such as, Simple positivity-preserving nonlinear finite volume scheme for subdiffusion equations on general non-conforming distorted meshes; The uniform l1 long-time behavior of time discretization for time-fractional partial differential equations with nonsmooth data;The finite volume scheme preserving maximum principle for two-dimensional time-fractional Fokker–Planck equations on distorted meshes.
Comments on the Quality of English Language
it can be improved
Author Response
Thank you for your valuable reviewer's comment.
I have updated the manuscript and carefully revised the article based on the reviewer's comments.
Comment 1: In Abstract, “To verify the usefulness of the developed training content, a survey was conducted on 40 railway maintenance personnel.” should be revised.
Response 1: As the reviewer's advice, we updated the manuscript in the summary that needed correction have been updated on the line number 21-22.
Before modification: To verify the usefulness of the developed training content, a survey was conducted on 40 railway maintenance personnel
After modification: The UMUX and CSUQ surveys were conducted with 40 railway maintenance workers to evaluate the effectiveness of the developed educational content.
Comment 2: The authors can refer some work, e.g. An efficient alternating direction implicit finite difference scheme for the three-dimensional time-fractional telegraph equation; Unconditional convergence of linearized orthogonal spline collocation algorithm for semilinear subdiffusion equation with nonsmooth solution; Superconvergence Analysis of a Robust Orthogonal Gauss Collocation Method for 2D Fourth-Order Subdiffusion Equations; Second-order BDF ADI Galerkin finite element method for the evolutionary equation with a nonlocal term in three-dimensional space; Construction of a new time-space two-grid method and its solution for the generalized Burgers' equation.
Response 2: As per the reviewer's advice, we updated references [36-37] to include recent studies relevant to the developed content, at lines 229-236.
Comment 3: Page 13, “The hierarchical structure section is designed to facilitate easy recognition and learning by organizing the 7 components of the air spring as intermediate items.” should be revised.
Response 3: As the reviewer's advice, we updated the manuscript in the summary that needed correction have been updated on the line number 430-434.
Before modification: The hierarchical structure section is designed to facilitate easy recognition and learning by organizing the 7 components of the air spring as intermediate items.
After modification: The hierarchical structure section includes seven intermediate items, corresponding to the seven sub-components of the air spring: bellows, upper plate, lower plate, upper seat, lower seat, auxiliary spring, and slide board. This structure provides an effective learning framework for understanding the overall combined structure of these components and sub-components.
Comment 4: Further work, authors can consider preserving some important physical properties and physical structures. The authors can refer some work in section conclusion. Such as, Simple positivity-preserving nonlinear finite volume scheme for subdiffusion equations on general non-conforming distorted meshes; The uniform l1 long-time behavior of time discretization for time-fractional partial differential equations with nonsmooth data;The finite volume scheme preserving maximum principle for two-dimensional time-fractional Fokker–Planck equations on distorted meshes.
Response 4: As the reviewer's advice, we updated the manuscript and added references [62-63] to include additional information in the conclusions regarding future studies that would consider preserving the physical properties and physical structure of the air springs in lines 756-762.
Round 3
Reviewer 1 Report
Comments and Suggestions for Authors
What I mean is to do a head-to-head comparison between alternative methods and your method on the same topic.
As you claim it is a "developed educational content", it shall at least perform better than the previous educational content on the same topic, which you fail to prove by the experiments. This information is necessary. No need to do cross-domain comparisons like you do in table 8.
Author Response
Thank you for your valuable reviewer's comment.
We appreciate your suggestion to conduct a head-to-head comparison between our developed educational content and alternative methods on the same topic.
Comment 1: What I mean is to do a head-to-head comparison between alternative methods and your method on the same topic.
As you claim it is a "developed educational content", it shall at least perform better than the previous educational content on the same topic, which you fail to prove by the experiments. This information is necessary. No need to do cross-domain comparisons like you do in table 8.
Response 1:
We have updated the manuscript by presenting results that compare our method (AR content) with an alternative method (e-file format document manual) using the same topic, as shown in lines 574–636.
This comparison was chosen because the document manual represents the widely-used educational method in this domain. As shown by the UMUX and CSUQ scores, our results indicate that AR content improves learning efficiency and usability compared to traditional methods.
We also deleted the cross-domain comparisons you mentioned and the content in Table 8, but we will include this direct comparison in our future work. We plan to further verify the benefits of the developed educational content.
The survey scores presented in this study were evaluated based on the AR content developed. Please understand that the assessment of these scores is limited, as creating high-quality AR content involves continuous upgrades through feedback from users.
We hope these changes meet your expectations and demonstrate the validity of our findings.
Reviewer 3 Report
Comments and Suggestions for Authors
The paper have improved, I suggest to accept.
Comments on the Quality of English Language
it is ok
Author Response
I really appreciate your decision.
I will carefully review the paper based on your valuable comments so that I can write a more complete paper.
