Predicting Cybersickness Using Machine Learning and Demographic Data in Virtual Reality

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The authors attempt to explore the potential of demographic variables as standalone predictors for predicting cybersickness through mechine learning. However, it appears that insufficient explaination of research methods and data analysis. 

 

Concerns are listed as follows:

 

1.It does not provide a thorough introduction to its background, especially in sections 1 and 2, where the discussion mainly revolves around cyberscikness in virtual reality environment. the author aims to present some approach different from the traditional methods which often requires invasive data collection techniques, it is suggested that the narration about these traditional methods and the corresponding data collection techniques is needed.

2.As described in Section 3.1, the online surveys requires more detailed explanation of its validation, design and impletation. The design of the survey should be articulated, such as the types of data it collects and the selection of the data, which may also be explained by the prior work. Besides, pictures of the senario of the survey or the presentation of the questionnaireshould be added to enhance the rigor of the survey.

3.A machine learning model’s applicability is heavily dependent upon the data fed into it. Therefore, further investigation into the experimental design, method of data selection and data labelling in studies is needed. Besides, Section 3.3 suggests a refined data set of 148 participants selected after rigorous criteria. However, the lack of detailed explanation of these selection criteria may hamper the transparency of the research data processing phase.

4.The demographics of study participants suggest potential biases that may hinder the generality of the findings. It is incumbent upon the authors to discuss the implications of this skewed population distribution and its potential implications for the generality of the study. 

5.Although the manuscript commendably employs a variety of machine learning model classifiers and utilizes leave-one cross-validation (LOOCV) to eveluate the accuracy of the models, it does not justify the selection of these specific classifiers and LOOCV method for eluvation. Besides, the selection of the specific metrics for evaluation should also be explained, which may be proved by citing prior work. While F1 scores and balanced accuracy are useful metrics for evaluating binary classification problems, they should be used in conjunction with other metrics for a more comprehensive evaluation of model performance.  

6.It is suggested to provide illustration of the process of data analysis, from data processing to evaluating the efficacy of the models.

7.The manuscript correctly acknowledges the ethical dimensions of the research, including approval and informed consent. Nonetheless, a discourse on the ethical consequences of using demographic data in predictive models - particularly with regard to privacy concerns and bias is required. 

Author Response

Dear reviewer, thank you for your detailed feedback on the manuscript your comments are invaluable for improving the paper.

Here are my thoughts in response to your comments: 

1.It does not provide a thorough introduction to its background, especially in sections 1 and 2, where the discussion mainly revolves around cyberscikness in virtual reality environment. the author aims to present some approach different from the traditional methods which often requires invasive data collection techniques, it is suggested that the narration about these traditional methods and the corresponding data collection techniques is needed.

Based on your suggestion we have added a more comprehensive introduction with traditional methods and in the background more detailed review about cybersickness and ML.

 

2.As described in Section 3.1, the online surveys requires more detailed explanation of its validation, design and impletation. The design of the survey should be articulated, such as the types of data it collects and the selection of the data, which may also be explained by the prior work. Besides, pictures of the senario of the survey or the presentation of the questionnaireshould be added to enhance the rigor of the survey.

For the comments about the types of data collected, the rationale behind the selection of data, and how they align with previous work will certainly add depth to the study. Including images of the survey scenarios. We have added more details as suggested, relevant to the study.

3.A machine learning model’s applicability is heavily dependent upon the data fed into it. Therefore, further investigation into the experimental design, method of data selection and data labelling in studies is needed. Besides, Section 3.3 suggests a refined data set of 148 participants selected after rigorous criteria. However, the lack of detailed explanation of these selection criteria may hamper the transparency of the research data processing phase.

We have included a figure detailing the whole process and also updated the sections3.3 and 3.4 with more details.

4.The demographics of study participants suggest potential biases that may hinder the generality of the findings. It is incumbent upon the authors to discuss the implications of this skewed population distribution and its potential implications for the generality of the study. 

A discussion about the implications of a skewed population has been listed in the limitations the limitations sections for an improved application of findings.

5.Although the manuscript commendably employs a variety of machine learning model classifiers and utilizes leave-one cross-validation (LOOCV) to eveluate the accuracy of the models, it does not justify the selection of these specific classifiers and LOOCV method for eluvation. Besides, the selection of the specific metrics for evaluation should also be explained, which may be proved by citing prior work. While F1 scores and balanced accuracy are useful metrics for evaluating binary classification problems, they should be used in conjunction with other metrics for a more comprehensive evaluation of model performance.  

We have included the justifications, supported by references to prior work where appropriate, to ensure a more robust evaluation framework.

6.It is suggested to provide illustration of the process of data analysis, from data processing to evaluating the efficacy of the models.

We have provided a figure which details the research methodology and the ML part of the research.

7.The manuscript correctly acknowledges the ethical dimensions of the research, including approval and informed consent. Nonetheless, a discourse on the ethical consequences of using demographic data in predictive models - particularly with regard to privacy concerns and bias is required. 

A discussion on privacy concerns, potential biases, and the ethical implications of the research methods has been added to the limitations section.

 

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The present work reflects a new opportunity to test VR solutions, although the description of the test applied needs to be improved.

Author Response

Dear Reviewer,

Thank you for your comments on our manuscript. We appreciate your feedback and recognize the importance of refining the test description.

In the revised manuscript, we will change the description of our testing methodology and provide more detailed explanations of the procedures involved, including any parameters or variables measured during the testing process.

Regards,
Ananth

Reviewer 3 Report

Comments and Suggestions for Authors

The literature review presents (2. Background and Related Work) 7 sources, which is not enough. You wrote that: "Section 2 provides a comprehensive overview of VR technology, the emergence of cyber sickness as a concern, and a thorough review of existing research." The section 2 has only short description of the related research. I advise you to change the cites sentence with accordance with content of the section 2.

I recommend you mention not only the problematic of cyber disease in literature review, but also make a smaller review of machine learning models so that the reader can appreciate the completeness of your research.

In section 3 you write about the gender, age and other parameters of the respondents, but there is no information about their work or affiliation.

Readers will be interested to know who these people are and their affiliations. From a practical point of view, it is important to understand the industry or type of activity of the respondent in order to assess the practical application of the proposed research. Also, the key role plays the time that respondents spend daily with VR devices.

In subsection 3.4. Machine Learning Mod, you should pay more attention to the model. Mention mathematical expressions, variable dependencies, etc.

In References in source 15 you forgot to include the authors of the article and information about the journal. Check the formatting and completeness of each reference.

16 literary sources for this kind of article are not enough.

Author Response

Response to Reviewer 2 Comments

 

Dear reviewer, thank you for your detailed feedback on the manuscript your comments are invaluable for improving the paper.

The literature review presents (2. Background and Related Work) 7 sources, which is not enough. You wrote that: "Section 2 provides a comprehensive overview of VR technology, the emergence of cyber sickness as a concern, and a thorough review of existing research." The section 2 has only short description of the related research. I advise you to change the cites sentence with accordance with content of the section 2.

The intention was to provide a broad overview, but I recognize now that the section falls short of delivering a comprehensive review of VR technology and cybersickness. Considering the suggestion, we have improved by adding more to the literature review.

I recommend you mention not only the problematic of cyber disease in literature review, but also make a smaller review of machine learning models so that the reader can appreciate the completeness of your research.

We have also incorporated a subsection dedicated to discussing various machine learning models relevant to understanding and mitigating cybersickness in VR environments.

In section 3 you write about the gender, age and other parameters of the respondents, but there is no information about their work or affiliation. Readers will be interested to know who these people are and their affiliations. From a practical point of view, it is important to understand the industry or type of activity of the respondent in order to assess the practical application of the proposed research. Also, the key role plays the time that respondents spend daily with VR devices.

For the user privacy, we have not reported these details, while submitting the IRB we have only mentioned that the age gender and other parameters needed for VR sickness prediction will only be reported.

In subsection 3.4. Machine Learning Mod, you should pay more attention to the model. Mention mathematical expressions, variable dependencies, etc.

We think that providing detail expressions would divert the reader form the point we are trying to make, but we have added the variable dependencies in the section.

In References in source 15 you forgot to include the authors of the article and information about the journal. Check the formatting and completeness of each reference. 16 literary sources for this kind of article are not enough.

We have updated the 15 previously cited reference, and added many more based on the changes made to the article.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The contribution is trifle, yet concerns have been addressed. The work is way too simple.

Author Response

Dear Reviewer,

We appreciate your thoughtful review and helpful suggestions for our work. We value the chance to respond to your concerns and make our manuscript better.

We agree that our method is simple. Our purpose was to create a framework that was clear and easy to understand so it could be a trusted starting point for other research in our field.

We believe our contribution will be helpful to others and help them build on our results in the future.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors In lines 205-209 you refer to the Figure 1. It is difficult to determine where  is the “online survey methodology”. In my opinion, an additional comment should be written on the Figure 1 and something about each stage depicted in it.   The Figure 2 is also not informative. Complete it with accompanying text.  

 

In the Figures 3 and 4, the text is too small and the axes are not labeled.

Author Response

Dear Reviewer,

We appreciate your thoughtful review and helpful suggestions for our work. We value the chance to respond to your concerns and make our manuscript better.

As suggested, we have made all the

• Changes to the figures and
• Updated the methodology to reflect the online survey methodology and also
• Added descriptions for all the figures in the sections and updated Fig. 3 and 4 with axes and sizes.

Author Response File: Author Response.pdf