Objective Evaluation of Motion Cueing Algorithms for Vehicle Driving Simulator Based on Criteria Importance through Intercriteria Correlation (CRITIC) Weight Method Combined with Gray Correlation Analysis

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

• The research discusses how to evaluate motion cueing algorithms (MCAs) and proposes an objective evaluation scheme for their perceptual fidelity in motion-based simulators. It also discusses the development of an evaluation method that replaces subjective evaluations, making the analysis more reasonable and reliable.
• Using the CRITIC weight method with gray correlation analysis to evaluate MCAs is a technique applied in this work that can provide a specific solution and a systematic process for quantifying the performance of motion signaling algorithms, but it depends on evaluation methods.
• This article adds a measurable approach to evaluating MCAs using models (CRITIC and gray correlation analysis) to evaluate MCA performance.
• The conclusion only summarizes what is described in the text without bringing any new elements for reflection.
• As I wrote in my review, there are typos in figure 1 and table 3 and 4 should be checked.
• I could not understand the phrase in section 3.2. "In total 10 participants, 3 females, ...". How about the rest of the 7 participants?

 

Comments on the Quality of English Language

There are typos in some of the papers, such as Figure 1, "Motion cueing algorithm". Tables 3 and 4 have problems with a strange inclination line

Author Response

Dear Reviewer,

Thank you for taking the time to provide feedback on our paper. We appreciate your positive evaluation and the thoughtful suggestions you have provided. We believe that these insights will greatly contribute to improving the quality and impact of our research. This paper proposes an evaluation method for driving simulators based on CRITIC and grey correlation analysis, aiming to quantify the performance of motion cueing algorithms from multiple perception fidelity indicators. In this response, we address each of the points raised in your review and provide explanations or revisions where necessary. Reply as follows:

1.The conclusion only summarizes what is described in the text without bringing any new elements for reflection.

In lines 406-425 of the article, based on your suggestion, we have made changes to the conclusion section, adding some new elements for reflection to enrich the content as follow:

The paper proposes an objective evaluation method for the perception fidelity of MCAs based on a visual-vestibular interaction system and head tilt angle perception system model. Compared with the existing PFE method, it not only considers acceleration and angular velocity perception but also integrates linear velocity perception and tilt angle perception, effectively improving the accuracy and reliability of the evaluation results. Experimental results demonstrate that human drivers' responses to MCA are generally consistent with the NPC index in the longitudinal, lateral, vertical, pitch, roll, and yaw channels. In the pitch, roll, and yaw directions, the ED index trend is also generally consistent with subjective evaluation. Furthermore, the overall performance of MCA perception fidelity, as reflected by the combination of CRITIC and grey correlation analysis, aligns with the trend of human drivers' responses to MCA.

Certainly, the proposed evaluation indicator system and overall performance indica-tors in this article are primarily designed for MCAs in automotive simulators. The objective weight of each indicator is determined by the information contained in the indicator data (standard deviation and correlation coefficient). The indicators listed in the article are commonly used indicators in current driving simulators. Further analysis is needed for future new evaluation indicators to obtain better overall performance indicators.

Future work will further research and validate objective evaluation methods for the motion fidelity of different types of vehicle simulators as well as different types (flight, maritime) of driving simulators.

2. As I wrote in my review, there are typos in figure 1 and table 3 and 4 should be checked.

The typos in figure 1 have been rectified, and tables 3 and 4 have been thoroughly checked for accuracy.

3. I could not understand the phrase in section 3.2. "In total 10 participants, 3 females, ...". How about the rest of the 7 participants?

There are a total of 10 participants, including 3 females and 7 males in the experiment.

We believe that these changes have strengthened our manuscript and thank you for guiding us in making these improvements. We hope that the revised version meets with your approval.

Thank you once again for your valuable feedback and time. We look forward to hearing from you soon.

Sincerely,

Jiang Xue

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The original achievement of the authors of the paper is the synthesis of a method for assessing the perceptual fidelity of flight object motion cueing algorithms.

The manuscript in its current form is more of a research report than a scientific article.

Comments:

1. In the title of the paper and in the abstract, please include that it concerns an flight simulator.

2. Move the content of Abstract to Introduction, and take the content of Conclusions as Abstract.

3. In the Introduction, formulate the thesis and the resulting objectives of the work, which will become its chapters.

4. In Figure 1, the "Input" designation should be supplemented with "where" or this output should be omitted.

5. On line 81, it is marked Figure 2, but it should be Figure 3.

6. In Conclusions, specific quantitative and qualitative conclusions from the research should be presented.

7. In Conclusions, present a plan for further research on the topic of the paper.

Author Response

Dear Reviewer,

Thank you for taking the time to provide feedback on our paper” Objective Evaluation of Motion Cueing Algorithms For Vehicle Driving Simulator Based on CRITIC Weight Method Combined with Grey Correlation Analysis”. We appreciate your positive evaluation and the thoughtful suggestions you have provided. We believe that these insights will greatly contribute to improving the quality and impact of our research. In this response, we address each of the points raised in your review and provide explanations or revisions where necessary. Reply as follows:

1. In the title of the paper and in the abstract, please include that it concerns an flight simulator.

We primarily focused on simulating vehicle driving in this study, while the simulation of aircraft piloting will be addressed in future research. I have updated the title of the paper and included mention of "flight simulator" in the conclusions as future work.

2. Move the content of Abstract to Introduction, and take the content of Conclusions as Abstract.

We have implemented the changes as requested. The content of the abstract has been moved to the introduction, and the content of the conclusions section is now included in the abstract. I believe this restructuring enhances the flow and clarity of the paper.

3. In the Introduction, formulate the thesis and the resulting objectives of the work, which will become its chapters.

We have revised the Introduction section to clearly formulate the thesis and outline the resulting objectives of the work.

4. In Figure 1, the "Input" designation should be supplemented with "where" or this output should be omitted.

The output has been omitted.

 

5. On line 81, it is marked Figure 2, but it should be Figure 3.

We have made the necessary adjustment, changing the label in line 81 from "Figure 2" to "Figure 3".

6. In Conclusions, specific quantitative and qualitative conclusions from the research should be presented.

In lines 406-416 of the article, we have revised the conclusions section to include specific quantitative and qualitative conclusions derived from the research.

7. In Conclusions, present a plan for further research on the topic of the paper.

In lines 417-425 of the article, we have improved the conclusion as requested.

We believe that these changes have strengthened our manuscript and thank you for guiding us in making these improvements. We hope that the revised version meets with your approval.

Thank you once again for your valuable feedback and time. We look forward to hearing from you soon.

Sincerely,

Jiang Xue

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The paper contributes valuable insights into the evaluation of motion cueing algorithms, addressing a significant gap in current research. However, enhancing the methodological rigor and broadening the scope of validation can significantly enhance the impact and applicability of the findings. The proposed revisions should help in refining the paper to meet the high standards of your journal.

 

1.       Abstract: The background section is excessively lengthy, necessitating careful pruning for conciseness. Additionally, the method and experimental procedure descriptions require elaboration.

2.       Grammar: The manuscript must undergo rigorous grammatical review to correct any syntactical errors.

3.       While the paper proposes a novel evaluation method, the explanation and justification of the chosen methods (CRITIC and grey correlation analysis) could be expanded. The manuscript would benefit from a more detailed discussion on why these particular methods were chosen over others and how they complement each other in this specific application.

4.       The validation section primarily focuses on subjective evaluations to support the objective methods introduced. Including additional objective validation metrics or comparing the proposed methods against established benchmarks could strengthen the credibility of the results.

5.       The study focuses on motion cueing algorithms in vehicle simulators. Discussing the potential adaptability or limitations of the proposed evaluation method in other types of simulators (e.g., flight, virtual reality) would provide a clearer understanding of its general applicability.

6.       Elaborate on the selection and integration of the CRITIC and grey correlation analysis methods. A comparative analysis with other potential methods could clarify the benefits and limitations of the current approach.

7.       Integrate more robust objective validation techniques to corroborate the subjective assessments. This might include quantitative performance metrics or comparisons with known benchmarks in simulator technology.

8.       Include a section discussing how the evaluation method can be adapted or needs modification for different types of simulators beyond the vehicle motion simulators discussed.

9.       Equations should be mentioned in the main text.

10.    Some references seem outdated or are not properly cited according to the journal's guidelines.

11.    Future Work: The discussion should extend to the method's potential for generalization across different data sets and types of plant silicates, to affirm its practicality and robustness.

12.    Conclusion: The concluding remarks need to openly recognize the study's limitations, promoting academic honesty and integrity.

Comments on the Quality of English Language

Moderate editing of English language required

Author Response

Dear Reviewer,

Thank you for taking the time to provide feedback on our paper” Objective Evaluation of Motion Cueing Algorithms For Vehicle Driving Simulator Based on CRITIC Weight Method Combined with Grey Correlation Analysis”. We appreciate your positive evaluation and the thoughtful suggestions you have provided. We believe that these insights will greatly contribute to improving the quality and impact of our research. In this response, we address each of the points raised in your review and provide explanations or revisions where necessary. Reply as follows:

1. Abstract: The background section is excessively lengthy, necessitating careful pruning for conciseness. Additionally, the method and experimental procedure descriptions require elaboration.

We have improved the abstract as requested, as follows:

Abstract: Perception-based fidelity evaluation metrics are crucial in driving simulators, as they play a key role in the automatic tuning, assessment, and comparison of motion cueing algorithms. Nevertheless, there is presently no unified and effective evaluation framework for these algorithms. To tackle this challenge, our study initially establishes a model rooted in visual-vestibular interaction and head tilt angle perception systems. We then employ metrics like the Normalized Average Absolute Difference (NAAD), Normalized Pearson Correlation (NPC), and Estimated Delay (ED) to devise an evaluation index system. Furthermore, we use a combined approach incorporating CRITIC and grey relational analysis to ascertain the weights of these indicators. This allows us to consolidate them into a comprehensive evaluation metric that reflects the overall fidelity of motion cueing algorithms. Subjective evaluation experiments validate the reasonableness and efficacy of our proposed Perception Fidelity Evaluation (PFE) method.

2. Grammar: The manuscript must undergo rigorous grammatical review to correct any syntactical errors.

Thank you for your feedback. The manuscript has been subjected to rigorous grammatical review to correct any syntactical errors, as requested.

3. While the paper proposes a novel evaluation method, the explanation and justification of the chosen methods (CRITIC and grey correlation analysis) could be expanded. The manuscript would benefit from a more detailed discussion on why these particular methods were chosen over others and how they complement each other in this specific application.

In lines 296-302 of the article, we have supplemented the respective advantages of CRITIC and grey correlation analysis.

4. The validation section primarily focuses on subjective evaluations to support the objective methods introduced. Including additional objective validation metrics or comparing the proposed methods against established benchmarks could strengthen the credibility of the results.

In lines 406-416 of the article, we compared and analyzed the proposed method with common objective evaluation metrics.

5. The study focuses on motion cueing algorithms in vehicle simulators. Discussing the potential adaptability or limitations of the proposed evaluation method in other types of simulators (e.g., flight, virtual reality) would provide a clearer understanding of its general applicability.

In lines 417-422, we provide additional explanations on the potential adaptability or limitations of the proposed evaluation method in other types of simulators such as flight and virtual reality.

6. Elaborate on the selection and integration of the CRITIC and grey correlation analysis methods. A comparative analysis with other potential methods could clarify the benefits and limitations of the current approach.

In lines 280-301 of the article, we supplemented the advantages of the combination of CRITIC and grey correlation analysis. We also introduced the limitations of other methods and clarified the benefits of the proposed method.

7. Integrate more robust objective validation techniques to corroborate the subjective assessments. This might include quantitative performance metrics or comparisons with known benchmarks in simulator technology.

In lines 406-416, we compared and analyzed the subjective value with the objective standard of known motion cues using the combination of CRITIC and grey correlation analysis.

8. Include a section discussing how the evaluation method can be adapted or needs modification for different types of simulators beyond the vehicle motion simulators discussed.

In lines 303-307 of the article, we have supplemented the objective weight calculation method for each indicator.

9. Equations should be mentioned in the main text.

We have addressed this concern by incorporating references to equations within the main text where appropriate.

10. Some references seem outdated or are not properly cited according to the journal's guidelines.

We have updated some of the references and checked their formatting to comply with the journal's guidelines.

11. Future Work: The discussion should extend to the method's potential for generalization across different data sets and types of plant silicates, to affirm its practicality and robustness.

In lines 423-425 of the article, we have improved the Future Work in conclusion as requested.

12. Conclusion: The concluding remarks need to openly recognize the study's limitations, promoting academic honesty and integrity.

In lines 417-422 of the article, we have improved the conclusion as requested.

We believe that these changes have strengthened our manuscript and thank you for guiding us in making these improvements. We hope that the revised version meets with your approval.

Thank you once again for your valuable feedback and time. We look forward to hearing from you soon.

Sincerely,

Jiang Xue

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

Since the authors of the manuscript took into account all my comments, I propose its publication in Machines Journal.

Reviewer 3 Report

Comments and Suggestions for Authors

My comments on the initial version of the manuscript have been sufficiently addressed by the authors in this revised version. I have no further comments on the technical aspects. The manuscript may be considered for publication after a proofreading.

Comments on the Quality of English Language

Minor editing of English language required.