Next Article in Journal
A Comparison Study on the Learning Effectiveness of Construction Training Scenarios in a Virtual Reality Environment
Previous Article in Journal
Characterization of Functional Connectivity in Chronic Stroke Subjects after Augmented Reality Training
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Virtual Worlds
Volume 2
Issue 1
10.3390/virtualworlds2010002
virtualworlds-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Cybersickness in Virtual Reality Questionnaire (CSQ-VR): A Validation and Comparison against SSQ and VRSQ

by
Panagiotis Kourtesis
1,2,*,
Josie Linnell
2,
Rayaan Amir
2,
Ferran Argelaguet
3,4 and
Sarah E. MacPherson
2
1
Department of Psychology, National and Kapodistrian University of Athens, 15772 Athens, Greece
2
Department of Psychology, University of Edinburgh, Edinburgh EH8 9YL, UK
3
Institut National de Recherche en Sciences et Technologies du Numérique (Inria), de l’Université de Rennes, 35042 Rennes, France
4
Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), de Centre National de la Recherche Scientifique (CNRS), 35042 Rennes, France
*
Author to whom correspondence should be addressed.
Virtual Worlds 2023, 2(1), 16-35; https://doi.org/10.3390/virtualworlds2010002
Submission received: 19 December 2022 / Revised: 9 January 2023 / Accepted: 17 January 2023 / Published: 29 January 2023
(This article belongs to the Topic Simulations and Applications of Augmented and Virtual Reality)
Browse Figures
Versions Notes

Abstract

:
Cybersickness is a drawback of virtual reality (VR), which also affects the cognitive and motor skills of users. The Simulator Sickness Questionnaire (SSQ) and its variant, the Virtual Reality Sickness Questionnaire (VRSQ), are two tools that measure cybersickness. However, both tools suffer from important limitations which raise concerns about their suitability. Two versions of the Cybersickness in VR Questionnaire (CSQ-VR), a paper-and-pencil and a 3D–VR version, were developed. The validation of the CSQ-VR and a comparison against the SSQ and the VRSQ were performed. Thirty-nine participants were exposed to three rides with linear and angular accelerations in VR. Assessments of cognitive and psychomotor skills were performed at baseline and after each ride. The validity of both versions of the CSQ-VR was confirmed. Notably, CSQ-VR demonstrated substantially better internal consistency than both SSQ and VRSQ. Additionally, CSQ-VR scores had significantly better psychometric properties in detecting a temporary decline in performance due to cybersickness. Pupil size was a significant predictor of cybersickness intensity. In conclusion, the CSQ-VR is a valid assessment of cybersickness with superior psychometric properties to SSQ and VRSQ. The CSQ-VR enables the assessment of cybersickness during VR exposure, and it benefits from examining pupil size, a biomarker of cybersickness.

1. Introduction

Virtual reality (VR) is a promising form of technology that facilitates applications in many areas, such as education [1], professional training [2], cognitive assessment [3], mental health therapy [4], and entertainment [5]. Nevertheless, beyond the advantages that VR brings to these fields, a limitation of VR is the presence of cybersickness that affects a percentage of users [6]. Cybersickness symptomatology includes nausea, disorientation, and oculomotor symptoms. Although there are similarities between cybersickness and simulator sickness, cybersickness differs from simulator sickness in terms of the frequency and severity of the types of symptoms [7]. Specifically, users experiencing cybersickness report increased general discomfort due to nausea and disorientation-related symptoms [7]. Cybersickness also differs from motion sickness as cybersickness is triggered by visual stimulation, while motion sickness is triggered by actual movement [8].
Although there is not a comprehensive theoretical framework for cybersickness, the most frequent and predominant one is the sensory conflict theory [6,8,9]. This theoretical framework suggests that cybersickness symptomatology stems from a sensorial conflict between the vestibular (inner ear) and the visual system [6,9]. In simple terms, the perception of postural balance relies on a combination of visual, vestibular, and proprioceptive input. Conflicting motion perception cues of the visual, proprioception, and vestibular systems postulate to cause cybersickness. The technological reason for this conflict is vection, an illusory sense of motion that occurs in VR. Vection is one of the main reasons for experiencing cybersickness in VR [10,11]. Specifically, motions such as linear and angular accelerations appear to induce cybersickness in the user.

1.1. Cybersickness, Cognition, and Motor Skills

Beyond the obvious decrease in user experience in VR, cybersickness may also negatively affect the cognitive and/or motor performance of the user. Given that VR is used for applications that require intact cognitive and motor abilities (e.g., educational, research, clinical, and training applications), the presence of cybersickness has serious consequences for the implementation of VR in these applications. Recent systematic reviews of the literature suggest that cybersickness may substantially, yet temporarily, decrease the cognitive and/or motor performance of the user in immersive VR studies [12,13,14]. Dahlman et al. [15] postulated that motion sickness significantly decreases users’ verbal working memory. Comparably, in immersive VR, Varmaghani et al. [16] conducted a study (N = 47) in which the participants formed two groups: a VR group (N = 25) and a control group (N = 22; playing a board game). The results indicated that the VR group did not show an increase in visuospatial processing ability, while the control group did. Thus, this outcome postulated that cybersickness affects visuospatial processing and/or learning ability.
In another study, Mittelstaedt et al. [17] examined cybersickness and cognition (reaction time, spatial processing, visuospatial working memory, and visual attention processing) in pre- and post-sessions in VR. The findings showed that cybersickness modulated a slower reaction speed and prevented an expected improvement in visual processing speed [17]. These results suggest that cybersickness has a negative effect on attentional processing and reaction times, while spatial abilities and visuospatial memory remain intact. In the same vein, the studies of Nalivaiko et al. [18] (N = 26) and Nesbitt et al. [10] (N = 24) examined the effect of cybersickness on reaction times. In both studies, reaction speed substantially slowed. Interestingly, slower reaction times were significantly correlated with an increase in the intensity of cybersickness [10,18], indicating that cybersickness intensity may be associated with temporary cognitive and/or motor decline. However, no study has examined whether cybersickness intensity predicts cognitive or motor decline. Finally, while the above studies support the notion that cybersickness may decrease cognitive and/or motor skills, they all evaluated cybersickness after VR exposure. No study has assessed cybersickness during exposure.

1.2. Cybersickness Questionnaires

The Simulator Sickness Questionnaire (SSQ) is a 4-point Likert scale that was designed to assess simulator sickness in aviators [19]. The SSQ is the tool that has been used most frequently to measure cybersickness due to exposure to VR [13]. However, simulator sickness differs from cybersickness symptomatology. In the latter, disorientation and nausea symptoms are more frequent and intense [7]. Thus, despite its use in VR studies, the SSQ is not specific to cybersickness symptoms that a user may experience in VR. Indeed, a recent study showed that the SSQ does not have adequate psychometric properties to evaluate cybersickness in VR [20]. However, there is a variant of the SSQ, namely the VR sickness questionnaire (VRSQ), that was recently developed [21] using items directly derived from the SSQ. In the development and validation study of the VRSQ, researchers attempted to isolate the items of the SSQ that are pertinent to cybersickness [21]. Nevertheless, this development and validation study suffered from serious limitations. Firstly, the sample size was small (i.e., 24 participants), and the stimuli diversity was limited. Notably, the factor analyses accepted only items pertinent to oculomotor and disorientation symptoms, while they rejected all items pertinent to nausea (i.e., 7 items) [21]. The latter is very problematic because it is well-established that nausea is the second (after disorientation) most frequent type of symptom of cybersickness [7,22,23,24]. Furthermore, both the SSQ and the VRSQ examine symptoms after VR exposure (not during) and produce scores that cannot be easily interpreted. Finally, when developing the SSQ and the VRSQ, the available guidelines for designing and developing a Likert scale tool were not considered.
There is scientific consensus regarding the design of Likert scale questionnaires. The literature suggests that a 7-point Likert scale is substantially better than a 5-point (or less) one [25,26,27,28]. The 7-point design offers a greater variety of responses, which better captures the diversity of the individuals’ views or experiences. Furthermore, combining numbers (e.g., 6) with corresponding text (e.g., strongly disagree) facilitates a better understanding of the differentiation between the available responses [25,26,27,28]. These suggestions have been considered and adopted in the development of the VR Neuroscience Questionnaire (VRNQ) and the Cybersickness in VR Questionnaire (CSQ-VR) [29]. The CSQ-VR is derived from the VR Induced Symptoms and Effects (VRISE) section of the VR Neuroscience Questionnaire (VRNQ), which has been found to have very good structural and construct validity [29]. Additionally, the VRISE section of the VRNQ has been validated against the SSQ and the Fast Motion Sickness Scale [30]. The advantages of the VRISE over the SSQ pertain to its short administration (only 5 items/questions) and its production of easily comprehensible outcomes [30]. However, the scoring of the VRISE is inverse (i.e., higher scores indicate milder symptom intensity). In addition, oculomotor symptoms were assessed by only one question in the VRISE.
The CSQ-VR was designed in line with the aforementioned guidelines (i.e., using a 7-point scale and combining text with numbers), while also addressing the previous shortcomings (i.e., inverse scoring and one oculomotor question) of the VRISE section of the VRNQ. The CSQ-VR assesses the whole range of cybersickness symptoms, including nausea, disorientation, and oculomotor symptoms. There are two questions for each type of symptom. Each question is presented on a 7-item Likert Scale and the responses are offered in the form of combined text and numbers, ranging from “1-absent feeling” to “7-extreme feeling”. The CSQ-VR produces a total score and three sub-scores: nausea, disorientation, and oculomotor. Each sub-score corresponds to a type of symptom and is calculated by adding the two corresponding responses. The total score is the sum of the three sub-scores. The design of the CSQ-VR yielded the maintenance of the advantages associated with the VRISE of the VRNQ (i.e., very short administration, easy and interpretable scoring, comprehensible questions and responses, and an examination of all types of cybersickness symptoms) and the improvement of the weaker aspects (i.e., the addition of one more oculomotor question and positive scoring, where larger numbers indicate stronger symptoms). Finally, the CSQ-VR was not only developed in a paper-and-pencil form, but also in a 3D form that can be used in any virtual environment to examine cybersickness while the user is in VR. This VR version of the CSQ-VR also benefits from eye tracking to measure gaze fixations and pupil size (i.e., pupillometry). Because pupil size is associated with negative emotions [31], pupillometry may offer a physiological metric of cybersickness intensity.

1.3. Research Aims

This study aims to examine the validity of the paper-and-pencil version and the VR version of the CSQ-VR in detecting and evaluating cybersickness symptoms. The validity is examined against that of the SSQ and the VRSQ, which are considered valid tools to measure cybersickness. Furthermore, as there is an association between cybersickness and cognitive and motor performance, this study offers a comparison between the CSQ-VR (both versions), the SSQ, and the VRSQ in detecting temporary cognitive and/or motor decline due to cybersickness. The VR version of the CSQ-VR is expected to facilitate an ongoing examination of cybersickness while the user is immersed. Finally, the utility of pupillometry in predicting cybersickness intensity is also be explored.

2. Materials and Methods

2.1. Virtual Environment Development

The virtual environment was developed using the Unity3D game engine. The interactions with the environment were developed using SteamVR SDK. Because gaming experience may modulate task performance [3], the virtual hands/gloves of SteamVR SDK were used to ensure an ergonomic and effortless interaction. Notably, none of the interactions required button presses. Instead, interactions were facilitated by simply touching the object (initial selection) and continuously touching the object (to confirm the selection). In addition, SteamVR virtual hands/gloves do not represent any gender or race, so their utilization prevents confounding effects from these variables [32].
To ensure an understanding and the seamless completion of the tasks, users received instructions in video, audio, and written form. For each task’s instructions, audio clips with neutral naturalistic voices were produced using Amazon Polly. The audio feedback was spatialized using the SteamAudio plugin. SRapinal SDK was used for eye tracking and facilitating pupillometry. Finally, randomization of the experimental blocks within and between participants and the extraction of the data into a CSV file, as well as the facilitation of the experimental design and control, were achieved using bmlTUX SDK [33].

Linear and Angular Accelerations in VR

Based on the relevant literature, linear and angular accelerations are efficient in inducing significant cybersickness symptoms in users in a relatively short time (e.g., 5–10 min) [10,12,13,18,24,29,34]. Correspondingly, a ride of 5 min was developed. Because the ride had to be repeated three times (i.e., a total 15 min ride) for each participant, a 5 min duration was preferred. The ride was designed as an animation of the platform that the user was standing on (see Figure 1). The direction of motion was always forward (except in the last stage; see reversed z-axis). The movements of the platform were similar to those of a roller coaster. The ride included the following accelerations in this specific order: (1) linear (z-axis); (2) angular (z- and y-axes); (3) angular (z-, x-, and y-axes); (4) angular (roll axis); (5) extreme linear (z-axis); (6) angular (yaw axis); and (7) extreme linear (y-axis followed by reversed z-axis). The environment had simple black-and-white surroundings (see Figure 1). This background was used to ensure that the symptoms were strictly induced by vection and not due to other reasons, such as intense colors. Additionally, having the squared/tiled design offered cues for the participants to perceive vection and altitude changes.

2.2. Cognitive and Psychomotor Skills’ Assessment

The aims of this study required the examination of cybersickness, cognition, and motor skills to be repeated while the user was immersed in VR. For these reasons, immersive VR versions of well-established tests were developed. For the development of these VR cognitive and psychomotor tasks, the specific design and development guidelines and recommendations for cognitive assessments in immersive VR were followed [35].

2.2.1. Verbal Working Memory

A VR version of the Backward Digit Span Task (BDST; [36]) was developed and used. The VR BDST requires participants to listen to a series of digits and remember and recall them in the reverse order of their presentation. For example, when the digits were 2, 4, and 3, then participants had to respond in the reverse order (i.e., 3, 4, and 2). Therefore, the first step involved listening to the digits. After this step, a keypad appeared in front of the participants. Using the keypad, users had to provide the digits in the reverse order. To indicate a number, participants had to touch the white box button displaying the equivalent number (see Figure 2). Continuous touch of the button for one second confirmed the response. After confirmation, if the response was correct, the button turned green and made a positive sound. In contrast, if the response was incorrect, then the button turned orange and made a negative sound. When a mistake was made or all the digits were provided correctly, the trial ended. In every second successful trial, the length of the digit sequence increased. When the participant made two subsequent mistakes within the same digit sequence length (e.g., 3 digits), or when they finished the last trial (i.e., second trial with a sequence length of 7 digits), then the task ended. The total score of the VR BDST was determined by adding together the total number of correct trials and the highest digit sequence length that was performed in at least one trial. A video displaying the task and its procedures can be found here: https://www.youtube.com/watch?v=1H8cqci-lFs (accessed on 16 January 2023).

2.2.2. Visuospatial Working Memory

Visuospatial working memory was assessed using the Backward Corsi Block Test (BCBT) [37]. A VR version of the BCBT was developed. This task consists of 27 white boxes where each one is placed in a different position based on the x-, y-, and z-axes. Nevertheless, only 9 boxes out of the 27 possible boxes were shown to the participants at one time (see Figure 2). The 9 boxes were presented at the beginning of each trial. Then, a number of these boxes (depending on the current sequence length) were randomly presented (turning blue and making a bell sound) in sequential order, with each box presented for one second. After the presentation of the sequence, participants had to select the boxes in reverse order. Participants had to touch a cube (the cube turned blue on touch) and keep touching it for one second to select the cube. When a cube was selected, it either turned green and made a positive sound (i.e., correct response), or it turned orange and made a negative sound (i.e., an error). The trial ended when the participants either made a mistake or they correctly selected all the targets in their reverse order. The sequence lengths were initially two boxes, with two trials for each length. The number of boxes in the sequence was increased by one box when at least one of the two trials of the same length/span was correct. When the participant incorrectly recalled two sequences of the same length, the task ended. Equally, when the second trial of the last length/span (i.e., 7 cubes) was performed, the task ended. The sequence lengths increased by up to seven cubes. The total score is the sum of the span (the longest correct sequence length) and the total number of correct sequences. A video displaying the task and its procedures can be found here: https://www.youtube.com/watch?v=MLilvkyMt-g (accessed on 16 January 2023).

2.2.3. Psychomotor Skills

To assess reaction times, a VR version of the Deary–Liewald Reaction Time (DLRT) task [38] was developed and used. The DLRT encompasses two tasks. One task assesses simple reaction time (SRT), and the other task examines choice reaction time (CRT). For the SRT task, participants had to observe a white box and touch it as soon as the box changes color to blue (see Figure 2). There are 20 trials/repetitions in the SRT task. In the CRT task, there are four boxes, which are aligned horizontally (see Figure 2). Randomly, one of the four boxes changes its color to blue. When the box turns blue, participants are required to touch the box as fast as possible (see Figure 2). The CRT task includes 40 trials/repetitions. For both the SRT and CRT, the participants were instructed to touch the boxes as fast as possible using the most convenient hand. There was a practice session at the start of both the SRT and the CRT to ensure that the instructions were understood by the participants. A video displaying the task and its procedures can be found here: https://www.youtube.com/watch?v=wXdrt0PjNsk (accessed on 16 January 2023).
As in the original version, the SRT produces a score that is the average reaction time across the 20 trials. Similarly, the CRT produces a score that is the average reaction time across the 40 trials, for the correct responses only. However, in addition, given that the VR version of the CRT is enhanced by eye tracking, the time required to attend to the target was also measured (attentional time, i.e., the time from the appearance of the target until the gaze of the user falls on it). Additionally, eye tracking facilitated the calculation of the time required to touch the target once it had been attended to (motor time). Finally, similarly to the original version, the overall time between the target’s presentation and its selection (reaction time) was also calculated. Thus, the VR version of the CRT produces three scores:
(1)
the reaction time (RT) to indicate overall psychomotor speed,
(2)
the attentional time (AT) to indicate attentional processing speed,
(3)
the motor time (MT) to indicate movement speed.

2.3. Cybersickness Questionnaires

The Motion Sickness Susceptibility Questionnaire (MSSQ) [39] was completed prior to enrolment to reduce the likelihood of a participant experiencing severe symptoms of cybersickness. The MSSQ is a 3-point Likert scale with 18 items/questions examining the experience of motion sickness using diverse means of transport. Nine items refer to the experience of motion sickness as a child, and the other nine items refer to such experiences as an adult. The nine questions are hence repeated in both sections. The MSSQ produces three scores: a child score; an adult score; and a total score, which is the addition of the previous two scores.
The SSQ and the VRSQ were administered pre- and post-exposure to VR to assess the intensity of cybersickness symptoms. Both the SSQ and VRSQ are 4-point Likert scales. The SSQ was developed for individuals that are trained using simulators (e.g., aviators) [19]. The SSQ has 16 questions, which are grouped under three categories: nausea; disorientation; and oculomotor. Four scores are produced, including one for each category and a total score. The calculation of the scores is made by a formula offered by the developers of the SSQ [19]. The maximum score is 100 for each category, and 300 for the total score. On the other hand, the VRSQ is derived from the SSQ, and it contains 9 items (i.e., approximately half of the SSQ items), which are grouped under two categories: disorientation and oculomotor (i.e., the nausea items are excluded) [21]. The VRSQ produces three scores, including one for each category and a total score, which is the sum of the two sub-scores divided by two. The maximum score for each sub-score is 100. As discussed above (see Section 1.2), while both the SSQ and VRSQ appear to be valid tools, they suffer from certain limitations:
  • The SSQ is not specific to cybersickness, and the frequency and intensity of symptoms substantially differ between simulator sickness and cybersickness.
  • The VRSQ does not consider nausea symptoms, and nausea symptoms are the second most frequent type of symptoms in cybersickness.
  • VRSQ validation was performed in a study with a small sample size and a limited diversity of stimuli.
  • Both the SSQ and VRSQ, being 4-point Likert scales, were not designed in line with the design guidelines for Likert scale questionnaires.

Cybersickness in VR Questionnaire

The CSQ-VR is an improved version of the VRISE section of the VRNQ. The VRISE section has been found to have very good structural validity in a study where participants were exposed to three diverse kinds of VR software and environments [29]. Additionally, the VRISE section of the VRNQ was previously validated and compared against the SSQ [30]. The VRISE of the VRNQ appeared superior to the SSQ due to its shorter administration time (i.e., 5 items instead of 16 items) and the enhanced interpretability of the scores (i.e., scores calculated by a simple addition, instead of a complex formula). However, the VRISE section of the VRNQ had only one item for oculomotor symptoms and the score was inversed (i.e., a higher score indicated a weaker intensity of that symptom). To address these limitations, the CSQ-VR was developed based on the VRISE section of the VRNQ. Comparably to the VRNQ, the CSQ-VR was designed and developed by following the design guidelines for Likert scales, i.e., a 7-point Likert scale, and combining text with numbers in the responses (see [25,26,27,28]). The CSQ-VR is a 7-point Likert scale that includes six questions for the assessment of the three types of symptoms of cybersickness, which form the following respective sub-scores: nausea; vestibular; and oculomotor. Each category includes two questions. The total score is the sum of the three scores, which a maximum score of 42 (14 for each sub-score). The paper-and-pencil version of the CSQ-VR can be found in the Supplementary Materials.
Moreover, a 3D version of the CSQ-VR has also been developed to assess cybersickness while the user is immersed in VR. A user interface (UI) for the VR version of the CSQ-VR was designed and developed. In the UI, the question appears in the upper area and the response (in red letters) appears in the middle area. The users change their response by touching the corresponding number or sliding along the slider (see Figure 3). Furthermore, based on the established link between pupil size and affective/emotional state [31], eye tracking was integrated to facilitate ophthalmometry and pupillometry. To measure fixation duration, invisible eye-tracking targets were placed in front of the text, while their height and width were always matched to the displayed text per line (see Figure 3). Moreover, the measurement of pupil size was continuous while the user responded to the CSQ-VR questions. Pupillometry yields measurements of average pupil size (right and left), which can be used as a physiological metric of negative emotion. Finally, a video showing the procedures of a questionnaire in VR can be found here: Link to the Video (accessed on 16 January 2023).

2.4. Participants and Procedures

Thirty-nine participants (22 females, 17 males) were recruited with a mean age of 25.28 years [SD = 3.25, Range = 22–36] and a mean education of 17.23 years [SD = 1.60, Range = 13–20]. The recruitment was performed via opportunity sampling using the University of Edinburgh’s internal mailing lists, alongside advertisements on social media. The study was approved by the School of Philosophy, Psychology, and Language Sciences (PPLS) Ethics Committee of the University of Edinburgh. Informed and written consent was obtained from all participants prior to their participation. Participants were compensated with 20 GBP each for their time and effort.
The MSSQ was completed before enrolment to reduce the likelihood of severe symptoms following VR exposure. In line with the MSSQ author’s suggestions [39], the 75th percentile was used as a parsimonious cut-off score for inclusion in the study. This allowed us to exclude individuals who are susceptible to experiencing strong cybersickness symptomatology (i.e., the upper 25th percentile of the population). The included participants were then invited to attend the experiment. Upon arrival, participants were informed of the study’s aims and procedures, and the adverse effects that they may experience. The participants then provided informed consent in written form.
Firstly, an induction on how to wear the headset and use and hold the controllers was offered to every participant. An HTC Vive Pro Eye was used, which embeds an eye-tracker with a 120 Hz refresh rate and a tracking accuracy of 0.5–1.1°. Secondly, the participants provided the following demographic data by responding to a questionnaire: age; sex; gender; education; dominant eye; VR experience; computing experience; and gaming experience. The dominant eye was determined using the Miles test [40]. Note that VR/computing/gaming experiences were calculated by adding the scores from two questions (6-item Likert scale) for each one. The first question was pertinent to the participant’s ability (e.g., 5: highly skilled) to operate a VR/computer/game, and the second one was pertinent to the frequency of operating them (e.g., 4: once a week).
Before VR exposure, participants responded to the CSQ-VR (paper version), SSQ, and VRSQ. Participants were then immersed in VR. Note that for the assessments and rides, participants were always in a standing position in the middle of the VR area (see the X mark in Figure 1). The first part included the tutorials, during which a video tutorial for each task was offered, alongside the corresponding verbal and written instructions. After each tutorial, the participant performed the corresponding task. This part formed the baseline assessment of each participant. The baseline assessment included the following: the VR version of CSQ-VR (Cybersickness); the verbal working memory task (BDST); the visuospatial working memory task (BCBT); and the reaction time task (DLRT; see Figure 2). After the baseline assessment, the first ride started. After each ride, the participants performed an assessment identical to the baseline (i.e., CSQ-VR, BDST, BCBT, DLRT). On top of the baseline assessment, the participants were exposed to three rides and three respective assessments. The whole procedure in VR lasted approximately 100 min for each participant. After the VR session, participants responded to the CSQ-VR (paper version), SSQ, and VRSQ. Then, refreshments rich in electrolytes were offered to the participants. Moreover, the participants rested for 10–15 min before leaving the premises. The participants were instructed to avoid driving and using heavy machinery for the rest of the day.

2.5. Statistical Analyses

Descriptive statistical analyses were performed to provide an overview of the sample. Reliability analyses were conducted to examine the internal consistency of the CSQ-VR. The recommended thresholds for Cronbach’s α were used to interpret the internal consistency (i.e., adequate = 0.6–0.7, good = 0.7–0.8, and very good = 0.8–0.95) [41]. Pearson’s correlational analyses were performed to examine the validity of the CSQ-VR versions against the SSQ and the VRS post-exposure (i.e., after the VR session). Because the SSQ is considered the gold standard and it has a structure (i.e., three sub-scores: nausea; oculomotor; and disorientation) similar to the CSQ-VR, the convergent validity (i.e., correlations) of the CSQ-VR was assessed against the SSQ. Receiver operating characteristic (ROC) and area under the curve (AUC) analyses were performed to appraise the psychometric properties of the CSQ-VR, SSQ, and VRSQ in detecting temporary cognitive and motor decline due to cybersickness. The thresholds of AUC > 0.7 and metric score > 1.5 were used in line with the respective recommendations for determining the suitability of the tool [42,43]. The temporary decline was based on the performance on the assessment after each ride. In agreement with the consensus of the American Academy of Clinical Neuropsychology for determining a substantial decrease in performance, two standard deviations from the mean were used [44]. Thus, when the performance (i.e., score) on the assessment after the respective ride was 2 standard deviations from the mean of the baseline assessment, the performance was defined as abnormal (i.e., temporary decline). Note that the two standard deviations had to indicate a worse performance and thus be greater for reaction and motor times (i.e., slower reaction or motor speed) and smaller for the verbal and visuospatial working memory (i.e., poorer performance). Finally, the predictive ability of pupil size was examined by performing a mixed model regression analysis. The analysis was performed using Jamovi statistical software (descriptive statistics, reliability, ROC, and AUC analyses) [45], as well as R (transforming the data, plots design, and correlation and regressions analyses) [46]. As the variables violated the normality assumption, we used the bestNormalize R package [47] to transform and centralize the data. The distribution of the data was then normal. The transformed data were used for parametric analyses (i.e., correlations and mixed regression analysis). Furthermore, the psych (correlational analyses) [48], the ggplot2 (plots) [49], and the lme4 (regression analyses) [50] R packages were used to perform the respective analyses.

3. Results

The descriptive statistics of the sample are displayed in Table 1. Concerning the intensity of cybersickness symptoms, it can be observed that the participants predominantly experienced moderate symptoms. There were no dropouts during the experiment. The descriptive statistics for the VR version of the CSQ-VR, per experimental stage, are presented in Table 2.

3.1. Reliability and Validity

Interpretation of the outcomes was based on the recommendations offered by Ursachi et al. [41]. Based on them, Cronbach’s α of 0.6–0.7 is an acceptable score, 0.7–0.8 is a good score, and 0.8–0.95 is a very good score. The overall internal consistency of the questionnaire (i.e., the total score’s reliability) was evaluated by considering each questionnaire’s sub-scores. The internal consistency of the sub-categories (i.e., the reliability of the sub-score) was examined by considering the respective items/questions. The reliability analyses revealed that all sub-scores of the CSQ-VR had good internal consistency (see Table 3). Specifically, the total score and the vestibular sub-score showed very good internal consistency, while the nausea and oculomotor sub-scores revealed good internal consistency. However, Cronbach’s α of the oculomotor sub-score was at the margins between good and adequate internal consistency.
Both the SSQ and VRSQ total scores showed good internal consistency; however, both were substantially lower than the internal consistency of the CSQ-VR total score (see Table 3). The nausea score of the SSQ showed an acceptable internal consistency, which was significantly lower than the almost very good internal consistency of the nausea score of the CSQ-VR. The disorientation score of the SSQ revealed marginally very good internal consistency, while the disorientation score of the VRSQ indicated marginally good internal consistency. Both disorientation scores (SSQ and VRSQ) were significantly lower than the almost excellent internal consistency of the CSQ-VR. Finally, the oculomotor score of the SSQ showed good internal consistency that was higher than the marginally good internal consistency of the oculomotor score of the CSQ-VR. On the other hand, the oculomotor score of the VRSQ revealed adequate internal consistency. The oculomotor score of the VRSQ and the nausea score of the SSQ were the two scores that were below the parsimonious threshold of 0.7. Overall, the CSQ-VR appeared to have superior internal consistency compared to the SSQ and the VRSQ.
The scores of the CSQ-VR (both versions) were significantly correlated with the corresponding scores of the SSQ. Overall, the analyses revealed moderate to strong correlations between the scores. The paper-and-pencil version of the CSQ-VR was strongly associated with the SSQ (see Figure 4). Their total scores especially, as well as their oculomotor scores, revealed a very strong correlation between them. Although the correlations for the nausea and vestibular scores were weaker than those observed above, they were still strong correlations (see Figure 4). Similarly, the VR version of the CSQ-VR was strongly associated with the SSQ (see Figure 5). In particular, their total scores indicated a strong correlation between them. While the correlations for their sub-scores were weaker than those between the total score, they were still moderate to strong correlations (see Figure 5). These results postulate the convergent validity of both versions of the CSQ-VR. Additionally, given that all sub-scores were substantially associated, the construct validity of the CSQ-VR is strongly supported.
Furthermore, the scores for both versions of the CSQ-VR were strongly associated with the VRSQ scores (see Table 4). The total scores of the CSQ-VR versions showed the strongest correlations with the total score of the VRSQ. The oculomotor scores of the CSQ-VR and VRSQ equally revealed robust associations between them. Although the vestibular scores indicated weaker correlations compared to the other scores, the correlations were moderate (see Table 4). These outcomes further support the convergent and construct validity of both versions of the CSQ-VR.

3.2. Detection of Temporary Decline due to Cybersickness

As mentioned above, the temporary decline was defined by two standard deviations from the mean of the baseline assessment. This definition is in line with the guidelines of the American Academy of Clinical Neuropsychology for determining whether performance is abnormal [44]. Eleven observations were detected which met the criterion for temporary cognitive/motor decline. Six of these were pertinent to reaction speed (i.e., longer reaction times) and five of them were applicable to motor speed (i.e., slower). Thus, all temporary declines were found for the DLRT task. A trend was also observed where, when motor speed was substantially slower (i.e., a decline), reaction speed also substantially declined. Finally, these declines were all found in three participants. Thus, only three participants experienced a temporary decline in their psychomotor skills. These results indicate that susceptibility to experiencing a temporary decline due to cybersickness may be attributed to individual differences.
The ROC–AUC analyses provide cut-off scores for each questionnaire where the optimal sensitivity (i.e., the detection of true positives) and specificity (i.e., the exclusion of true negatives) are achieved. Following the recommendations for ROC–AUC analyses and psychometrics [42,43] to determine the suitability of a questionnaire to detect temporary decline, two criteria were set as follows: (1) AUC > 70% and (2) metric score > 1.5, both of which had to be met. The ROC–AUC analyses for declines in reaction time (i.e., slower reaction times) showed that only the total scores for both versions of the CSQ-VR met the criteria (see Table 5). Similarly, the ROC–AUC analyses for motor speed decline indicated that only the total scores for both versions of the CSQ-VR met the criteria. Furthermore, the two versions of the CSQ-VR showed the best sensitivity and specificity in detecting a temporary decline in reaction time and motor speed, while the VRSQ and SSQ showed significantly smaller psychometric properties (see Table 5 and Table 6 and Figure 6). These results postulate that the total scores for both versions of the CSQ-VR have superior psychometric properties to the total scores of the SSQ and VRSQ. Additionally, only the CSQ-VR total scores are suitable for detecting a temporary decline in reaction speed and/or motor speed.
The psychometric properties of the sub-scores of each questionnaire were also examined. In detecting a temporary decline in reaction speed or motor speed, only the vestibular/disorientation scores of the questionnaires met the criteria of suitable psychometric properties (see Table 7 and Table 8). However, the nausea score of the VR version of the CSQ-VR also met the criteria for detecting both. The best sensitivity and specificity in detecting a temporary decline in either reaction or motor speed was observed for the vestibular score of the paper-and-pencil version of the CSQ-VR, closely followed by the same score for the VR version of the CSQ-VR (see Table 7 and Table 8 and Figure 7). The sensitivity and specificity of the disorientation scores of the SSQ and VRSQ were substantially lower compared to the CSQ-VR. However, the sensitivity and specificity of the disorientation score of the SSQ were significantly higher than the ones for the disorientation score of the VRSQ.

3.3. Mixed Model Regression Analysis

A mixed model regression analysis was conducted to determine whether pupil size can be a biomarker/predictor of cybersickness. The analysis indicated that the model with pupil size as a predictor of the total score on the VR version of the CSQ-VR was significant. Pupil size also revealed a relatively high beta (negative) coefficient, postulating that cybersickness intensity substantially increases as pupil size decreases (see Figure 8). Furthermore, the fixed effects of pupil size, alongside the random effects of the participants, appear to explain 50% of the variance in the intensity of cybersickness. These outcomes postulate that pupil size is a significant predictor of the intensity of cybersickness, and it can therefore be considered as a biomarker of cybersickness.

4. Discussion

The CSQ-VR is an adapted and enhanced version of the VRISE section and sub-score of the VNRQ. Based on the recommendations by Ursachi et al. [41] for Cronbach’s α, the CSQ-VR displayed good to very good internal consistency. This finding is aligned with the high structural validity and internal consistency of the VRNQ and its VRISE sub-score, which have previously been observed [29]. Additionally, the total scores and sub-scores of both versions of the CSQ-VR showed robust correlations with their respective sub-scores and total scores of the SSQ and VRSQ. This finding supports the findings of Somrak et al. [30] in which the VRISE sub-score of the VRNQ was significantly correlated with the SSQ total score. Nevertheless, the current study meticulously examined the reliability and validity of the total scores and the sub-scores of both versions of the CSQ-VR. Beyond their convergent validity, the associations between the sub-scores of the CSQ-VR (both versions) (i.e., nausea, vestibular, and oculomotor) and the equivalent sub-scores of the SSQ support the construct validity of both versions of the CSQ-VR in examining the whole range of cybersickness symptomatology. Therefore, both the paper-and-pencil and the VR versions of the CSQ-VR are highly reliable and are valid tools for measuring the presence and intensity of cybersickness symptoms in VR.

4.1. Comparison of CSQ-VR, SSQ, and VRSQ

Several studies have reported that the SSQ does not have adequate psychometric properties for measuring cybersickness in VR [20,51,52]. The findings of this current study are aligned with the previous literature. The inadequacy and inappropriateness of the SSQ for measuring cybersickness in VR have also been confirmed. Specifically, the overall and sub-scores for the SSQ and the VRSQ displayed internal consistency which was substantially inferior to the respective internal consistency of the CSQ-VR total score and sub-scores. Moreover, the nausea item of the SSQ revealed internal consistency that was below the parsimonious threshold of 0.7 for Cronbach’s α, which is required for a tool to be used in research and professional settings [53]. Likewise, the oculomotor sub-score of the VRSQ was well below this threshold. Given that the VRSQ has only two sub-scores (i.e., disorientation and oculomotor), half of the test was found to be unreliable. This finding agrees with the serious limitations reported in VRSQ development and validation, which was conducted using smartphone VR (i.e., Samsung Gear VR) and not PC or standalone VR, a very simplistic task (i.e., target selection) and stimuli (i.e., small and large buttons), which were not efficient in inducing adequate levels of cybersickness in a relatively small sample [21]. As a result, all the items pertinent to nausea, which is the second most frequent symptom of cybersickness in VR [7,22,23,24,51], were dropped. Thus, it comes as no surprise that both the SSQ and VRSQ displayed problematic consistencies in certain sub-scores and overall inferior reliability for the total and sub-scores of the CSQ-VR.
Furthermore, the SSQ has received criticism for its highly complex structure and scoring [30,51]. The CSQ-VR has previously been strongly preferred over the SSQ because of its easily calculated and interpretable scores [30]. The VRSQ, derived from the SSQ, has predominantly maintained the SSQ structure and scoring system, although the VRSQ scoring system requires somewhat simpler calculations. Nevertheless, as was also seen in this study, both the SSQ and VRSQ suffer in terms of structure. Additionally, given that the design of the questions and available responses use a Likert scale that is essential for collecting reliable and informative data [25,26,27,28], the CSQ-VR has an advantage over the SSQ and VRSQ. Both the SSQ and VRSQ use a 4-point Likert scale, while the relevant literature suggests that a 7-point Likert scale, especially when combining a number with textual information (e.g., “6–Very Intense Feeling”) like in the CSQ-VR, are substantially more efficient in providing useful and representative self-reports [25,26,27,28]. The design of the general instructions (i.e., “Please, from 1 to 7, circle the response that better corresponds to the presence and intensity of the symptom.”) and questions (e.g., “Nausea A: Do you experience nausea (e.g., stomach pain, acid reflux, or tension to vomit)?”) are also more explicit in the CSQ-VR than the equivalent design in the SSQ and VRSQ (i.e., general instruction: “Circle how much each symptom below is affecting you now.”; question: “Nausea”). Finally, the SSQ has 16 questions measuring the whole range of symptoms. The VRSQ has nine questions, but it measures only the vestibular and oculomotor-related symptoms, while the CSQ-VR is shorter, measuring the whole range of cybersickness with only six questions. Therefore, the CSQ-VR is a shorter questionnaire with an overall superior design to the SSQ and VRSQ, which was also reflected in the psychometric properties examined in this study.
The previous literature has shown that cybersickness, particularly when symptoms are strong, may affect the cognitive and/or motor skills of the user [12,13,14], especially their reaction speed [10,17,18]. It is thus assumed that a questionnaire designed to measure cybersickness would also be effective in detecting relevant declines in performance. The total score for both versions of the CSQ-VR showed high sensitivity and specificity in detecting these temporary declines in performance due to cybersickness, while the psychometric properties of the total scores of the SSQ and VRSQ were substantially lower and inadequate. Furthermore, two sub-scores (nausea and vestibular) of the CSQ-VR were also highly sensitive and specific in the detection of temporary declines, while the equivalent scores of the SSQ and VRSQ (which does not include a nausea score) were either significantly inferior or inadequate. Thus, the CSQ-VR is the only questionnaire that is effective in detecting these temporary declines in performance modulated by cybersickness. Given that VR is implemented in education [1], professional training [2], neuropsychological assessments [54], and therapy [4], where cognitive and motor skills should be reliable, it is essential that a tool should be able to provide information that these skills may have been compromised by cybersickness symptomatology. Finally, beyond these applications, VR is gradually becoming established as a research tool in scientific fields, such as human–computer interactions [55] and psychological sciences [3], where cybersickness may compromise the reliability of the scientific findings [12]. Thus, the detection of a participant whose performance has been compromised by cybersickness enables the exclusion of this participant or observation from the analyses and assures the data’s reliability.
Nevertheless, as was also observed in this study, a participant’s performance may not be affected throughout the experiment. In previous studies, changes in the intensity of cybersickness during exposure can occur in terms of an increase due to aggravation [22] or a decrease due to cultivated tolerance [56]. Therefore, the continuous or repetitive assessment of cybersickness is required while the participant/user is immersed. Instead of excluding all of a participant’s observations, the CSQ-VR allows a researcher to drop only those particular observations where a participant’s performance was affected by cybersickness. Considering that the VR version of the CSQ-VR has shown comparable (and sometimes superior) psychometric properties to its paper-and-pencil version, it can detect specific compromised observations/performance and suggest its exclusion from analyses. Nevertheless, beyond self-reports, there are other neuro and biomarkers that have been used to detect and measure cybersickness [8]. Specifically, researchers have efficiently implemented electroencephalography [57,58] and eye tracking [59,60] to detect and appraise the occurrence and intensity of cybersickness. The VR version of the CSQ-VR also benefits from eye-tracking metrics. In this study, pupil size was found to be a significant predictor of cybersickness. A decrease in pupil size indicated higher intensity cybersickness and vice versa, a pattern that has been previously observed between pupil size and negative emotions [31]. Previously, pupil size has been included in a deep fusion model for predicting cybersickness [60]; however, its relationship, predictive ability and contribution to this model were not evaluated, preventing a conclusion of whether pupil size is a biomarker of cybersickness. This study provides evidence postulating that pupil size is indeed a biomarker of cybersickness, as well as its intensity. The VR version of the CSQ-VR thus has an additional advantage of incorporating pupillometry.

4.2. Limitations and Future Studies

The current study also has limitations that should be considered. The sample consisted of young adults, which prevented the examination of cybersickness in a more age-diverse population. Future studies should attempt to examine cybersickness in a sample with a greater age spectrum to enable the study of age differences in tolerance and/or susceptibility towards cybersickness. Additionally, this study implemented a parsimonious inclusion criterion based on the MSSQ scores (i.e., excluding individuals who scored higher than the 75th percentile and could experience substantially more frequent and stronger cybersickness symptomatology). Given that the intensity and prevalence of cybersickness substantially differ across individuals, future studies should explore the effects of cybersickness on cognitive and motor skills in a sample that may experience stronger symptoms. Finally, the assessment included only working memory and psychomotor tests. Future studies should strive to examine more complex cognitive functions (e.g., episodic memory or decision making) and motor skills (e.g., tasks that require fine motor skills and accuracy).

5. Conclusions

The CSQ-VR is a short, valid and reliable tool of cybersickness, which has superior psychometric properties to the SSQ and VRSQ. Additionally, the paper-and-pencil and the VR versions of the CSQ-VR were highly sensitive and specific in detecting temporary performance declines that were modulated by cybersickness. The VR version of the CSQ-VR provides further advantages by facilitating an assessment of cybersickness in the virtual environment while the participant/user is immersed. Finally, the VR version of the CSQ-VR benefits from pupillometry (i.e., measurement of pupil diameter), which was found to predict the presence and intensity of cybersickness. Pupillometry may thus be applied in VR as a biomarker of positive (e.g., amusement) and negative (e.g., frustration) emotions.

Supplementary Materials

The Cybersickness in Virtual Reality Questionnaire (CSQ-VR) can be downloaded at: http://dx.doi.org/10.13140/RG.2.2.36571.03362.

Author Contributions

Conceptualization, P.K., F.A. and S.E.M.; methodology, P.K., F.A. and S.E.M.; software, P.K.; validation, P.K., F.A. and S.E.M.; formal analysis, P.K.; investigation, J.L. and R.A.; resources, P.K.; data curation, J.L. and R.A.; writing—original draft preparation, P.K., J.L. and R.A.; writing—review and editing, P.K., F.A. and S.E.M.; visualization, P.K.; supervision, P.K. and S.M; project administration, P.K. and S.E.M.; funding acquisition, J.L. and R.A. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki, and approved by the Philosophy, Psychology and Language Sciences Research Ethics Committee of the University of Edinburgh (269-2122/4; 14 June 2022).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to the ethical approval requirements.

Acknowledgments

This work was funded by the School of Philosophy, Psychology and Language Sciences of the University of Edinburgh. The authors would like to thank the uCreate Studio of the University of Edinburgh for providing them with the VR equipment and tech support.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Radianti, J.; Majchrzak, T.A.; Fromm, J.; Wohlgenannt, I. A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Comput. Educ. 2019, 147, 103778. [Google Scholar] [CrossRef]
  2. Xie, B.; Liu, H.; Alghofaili, R.; Zhang, Y.; Jiang, Y.; Lobo, F.D.; Li, C.; Li, W.; Huang, H.; Akdere, M.; et al. A Review on Virtual Reality Skill Training Applications. Front. Virtual Real. 2021, 2, 645153. [Google Scholar] [CrossRef]
  3. Kourtesis, P.; MacPherson, S.E. How immersive virtual reality methods may meet the criteria of the National Academy of Neuropsychology and American Academy of Clinical Neuropsychology: A software review of the Virtual Reality Everyday Assessment Lab (VR-EAL). Comput. Hum. Behav. Rep. 2021, 4, 100151. [Google Scholar] [CrossRef]
  4. Emmelkamp, P.M.; Meyerbröker, K. Virtual Reality Therapy in Mental Health. Annu. Rev. Clin. Psychol. 2021, 17, 495–519. [Google Scholar] [CrossRef] [PubMed]
  5. Cruz-Neira, C.; Fernández, M.; Portalés, C. Virtual Reality and Games. Multimodal Technol. Interact. 2018, 2, 8. [Google Scholar] [CrossRef] [Green Version]
  6. Rebenitsch, L.; Owen, C. Estimating cybersickness from virtual reality applications. Virtual Real. 2020, 25, 165–174. [Google Scholar] [CrossRef]
  7. Stanney, K.M.; Kennedy, R.S.; Drexler, J.M. Cybersickness is Not Simulator Sickness. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting; SAGE Publications: New York, NY, USA, 1997; Volume 41, pp. 1138–1142. [Google Scholar]
  8. Davis, S.; Nesbitt, K.; Nalivaiko, E. A Systematic Review of Cybersickness. In Proceedings of the 2014 Conference on Interactive Entertainment, Newcastle, Australia, 2 December 2014; Blackmore, K., Nesbitt, K., Smith, S.P., Eds.; ACM: New York, NY, USA, 2014. [Google Scholar] [CrossRef]
  9. LaViola, J.J. A discussion of cybersickness in virtual environments. ACM SIGCHI Bull. 2000, 32, 47–56. [Google Scholar] [CrossRef]
  10. Nesbitt, K.; Davis, S.; Blackmore, K.; Nalivaiko, E. Correlating reaction time and nausea measures with traditional measures of cybersickness. Displays 2017, 48, 1–8. [Google Scholar] [CrossRef]
  11. Kim, J.; Palmisano, S.; Luu, W.; Iwasaki, S. Effects of Linear Visual-Vestibular Conflict on Presence, Perceived Scene Stability and Cybersickness in the Oculus Go and Oculus Quest. Front. Virtual Real. 2021, 2, 582156. [Google Scholar] [CrossRef]
  12. Kourtesis, P.; Collina, S.; Doumas, L.A.A.; MacPherson, S.E. Technological Competence Is a Pre-condition for Effective Implementation of Virtual Reality Head Mounted Displays in Human Neuroscience: A Technological Review and Meta-Analysis. Front. Hum. Neurosci. 2019, 13, 342. [Google Scholar] [CrossRef] [Green Version]
  13. Saredakis, D.; Szpak, A.; Birckhead, B.; Keage, H.A.D.; Rizzo, A.; Loetscher, T. Factors Associated with Virtual Reality Sickness in Head-Mounted Displays: A Systematic Review and Meta-Analysis. Front. Hum. Neurosci. 2020, 14, 96. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  14. Conner, N.O.; Freeman, H.R.; Jones, J.A.; Luczak, T.; Carruth, D.; Knight, A.C.; Chander, H. Virtual Reality Induced Symptoms and Effects: Concerns, Causes, Assessment & Mitigation. Virtual Worlds 2022, 1, 130–146. [Google Scholar] [CrossRef]
  15. Dahlman, J.; Sjörs, A.; Lindström, J.; Ledin, T.; Falkmer, T. Performance and Autonomic Responses during Motion Sickness. Hum. Factors: J. Hum. Factors Ergon. Soc. 2009, 51, 56–66. [Google Scholar] [CrossRef]
  16. Varmaghani, S.; Abbasi, Z.; Weech, S.; Rasti, J. Spatial and attentional after effects of virtual reality and relations to cybersickness. Virtual Real. 2021, 26, 659–668. [Google Scholar] [CrossRef]
  17. Mittelstaedt, J.M.; Wacker, J.; Stelling, D. VR aftereffect and the relation of cybersickness and cognitive performance. Virtual Real. 2018, 23, 143–154. [Google Scholar] [CrossRef]
  18. Nalivaiko, E.; Davis, S.L.; Blackmore, K.; Vakulin, A.; Nesbitt, K. Cybersickness provoked by head-mounted display affects cutaneous vascular tone, heart rate and reaction time. Physiol. Behav. 2015, 151, 583–590. [Google Scholar] [CrossRef]
  19. Kennedy, R.S.; Lane, N.E.; Berbaum, K.; Lilienthal, M.G. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. Int. J. Aviat. Psychol. 1993, 3, 203–220. [Google Scholar] [CrossRef]
  20. Sevinc, V.; Berkman, M.I. Psychometric evaluation of Simulator Sickness Questionnaire and its variants as a measure of cybersickness in consumer virtual environments. Appl. Ergon. 2019, 82, 102958. [Google Scholar] [CrossRef] [PubMed]
  21. Kim, H.K.; Park, J.; Choi, Y.; Choe, M. Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment. Appl. Ergon. 2018, 69, 66–73. [Google Scholar] [CrossRef]
  22. Sharples, S.; Cobb, S.; Moody, A.; Wilson, J.R. Virtual reality induced symptoms and effects (VRISE): Comparison of head mounted display (HMD), desktop and projection display systems. Displays 2008, 29, 58–69. [Google Scholar] [CrossRef]
  23. Bohil, C.; Alicea, B.; Biocca, F.A. Virtual reality in neuroscience research and therapy. Nat. Rev. Neurosci. 2011, 12, 752–762. [Google Scholar] [CrossRef]
  24. Palmisano, S.; Mursic, R.; Kim, J. Vection and cybersickness generated by head-and-display motion in the Oculus Rift. Displays 2017, 46, 1–8. [Google Scholar] [CrossRef] [Green Version]
  25. Dawes, J. Do Data Characteristics Change According to the Number of Scale Points Used? An Experiment Using 5-Point, 7-Point and 10-Point Scales. Int. J. Mark. Res. 2008, 50, 61–104. [Google Scholar] [CrossRef]
  26. Joshi, A.; Kale, S.; Chandel, S.; Pal, D.K. Likert Scale: Explored and Explained. Br. J. Appl. Sci. Technol. 2015, 7, 396–403. [Google Scholar] [CrossRef]
  27. Taherdoost, H. What is the best response scale for survey and questionnaire design; review of different lengths of rating scale/attitude scale/Likert scale. Int. J. Acad. Res. Manag. 2019, 8, 1–10. [Google Scholar]
  28. Wakita, T.; Ueshima, N.; Noguchi, H. Psychological Distance between Categories in the Likert Scale. Educ. Psychol. Meas. 2012, 72, 533–546. [Google Scholar] [CrossRef]
  29. Kourtesis, P.; Collina, S.; Doumas, L.A.A.; MacPherson, S.E. Validation of the Virtual Reality Neuroscience Questionnaire: Maximum Duration of Immersive Virtual Reality Sessions without the Presence of Pertinent Adverse Symptomatology. Front. Hum. Neurosci. 2019, 13, 417. [Google Scholar] [CrossRef] [Green Version]
  30. Somrak, A.; Pogačnik, M.; Guna, J. Suitability and Comparison of Questionnaires Assessing Virtual Reality-Induced Symptoms and Effects and User Experience in Virtual Environments. Sensors 2021, 21, 1185. [Google Scholar] [CrossRef]
  31. Partala, T.; Surakka, V. Pupil size variation as an indication of affective processing. Int. J. Hum.-Comput. Stud. 2003, 59, 185–198. [Google Scholar] [CrossRef]
  32. Schwind, V.; Knierim, P.; Tasci, C.; Franczak, P.; Haas, N.; Henze, N. “These Are Not My Hands!”: Effect of Gender on the Perception of Avatar Hands in Virtual Reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 6–11 May 2017; pp. 1577–1582. [Google Scholar] [CrossRef]
  33. Bebko, A.O.; Troje, N.F. bmlTUX: Design and Control of Experiments in Virtual Reality and Beyond. I-Perception 2020, 11. [Google Scholar] [CrossRef]
  34. Bonato, F.; Bubka, A.; Palmisano, S. Combined Pitch and Roll and Cybersickness in a Virtual Environment. Aviat. Space Environ. Med. 2009, 80, 941–945. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  35. Kourtesis, P.; Korre, D.; Collina, S.; Doumas, L.A.A.; MacPherson, S.E. Guidelines for the Development of Immersive Virtual Reality Software for Cognitive Neuroscience and Neuropsychology: The Development of Virtual Reality Everyday Assessment Lab (VR-EAL), a Neuropsychological Test Battery in Immersive Virtual Reality. Front. Comput. Sci. 2020, 1. [Google Scholar] [CrossRef] [Green Version]
  36. Wechsler, D. Wechsler Bellevue Adult Intelligence Scale; Williams & Wilkins: Baltimore, MD, USA, 1939. [Google Scholar]
  37. Corsi, P.M. Human memory and the medial temporal region of the brain. Diss. Abstr. Int. 1972, 34, 891. [Google Scholar]
  38. Deary, I.J.; Liewald, D.; Nissan, J. A free, easy-to-use, computer-based simple and four-choice reaction time programme: The Deary-Liewald reaction time task. Behav. Res. Methods 2010, 43, 258–268. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  39. Golding, J.F. Predicting individual differences in motion sickness susceptibility by questionnaire. Pers. Individ. Differ. 2006, 41, 237–248. [Google Scholar] [CrossRef]
  40. Roth, H.L.; Lora, A.N.; Heilman, K.M. Effects of monocular viewing and eye dominance on spatial attention. Brain 2002, 125, 2023–2035. [Google Scholar] [CrossRef] [Green Version]
  41. Ursachi, G.; Horodnic, I.A.; Zait, A. How Reliable Are Measurement Scales? External Factors with Indirect Influence on Reliability Estimators. Procedia Econ. Financ. 2015, 20, 679–686. [Google Scholar] [CrossRef] [Green Version]
  42. Power, M.; Fell, G.; Wright, M. Principles for high-quality, high-value testing. Evid.-Base. Med. 2013, 18, 5–10. [Google Scholar] [CrossRef] [Green Version]
  43. Streiner, D.L.; Cairney, J. What’s under the ROC? An Introduction to Receiver Operating Characteristics Curves. Can. J. Psychiatry 2007, 52, 121–128. [Google Scholar] [CrossRef] [Green Version]
  44. Guilmette, T.J.; Sweet, J.J.; Hebben, N.; Koltai, D.; Mahone, E.M.; Spiegler, B.J.; Stucky, K.; Westerveld, M.; Participants, C. American Academy of Clinical Neuropsychology consensus conference statement on uniform labeling of performance test scores. Clin. Neuropsychol. 2020, 34, 437–453. [Google Scholar] [CrossRef]
  45. The Jamovi Project [Computer Software]. Available online: http://www.jamovi.org (accessed on 1 January 2020).
  46. R Core Team. A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2022; Available online: https://www.R-project.org/ (accessed on 12 December 2022).
  47. Peterson, R.A.; Cavanaugh, J.E. Ordered quantile normalization: A semiparametric transformation built for the cross-validation era. J. Appl. Stat. 2019, 47, 2312–2327. [Google Scholar] [CrossRef] [PubMed]
  48. Revelle, W. psych: Procedures for Psychological, Psychometric, and Personality Research; Northwestern University: Evanston, IL, USA, 2018; Available online: https://CRAN.R-project.org/package=psych (accessed on 12 December 2022).
  49. Wickham, H. ggplot2: Elegant Graphics for Data Analysis; Springer-Verlag: New York, NY, USA, 2016; Available online: https://ggplot2.tidyverse.org (accessed on 12 December 2022)ISBN 978-3-319-24277-4.
  50. Bates, D.; Mächler, M.; Bolker, B.; Walker, S. Fitting Linear Mixed-Effects Models Using lme4. J. Stat. Softw. 2015, 67, 48. [Google Scholar] [CrossRef]
  51. Bouchard, S.; Berthiaume, M.; Robillard, G.; Forget, H.; Daudelin-Peltier, C.; Renaud, P.; Blais, C.; Fiset, D. Arguing in Favor of Revising the Simulator Sickness Questionnaire Factor Structure When Assessing Side Effects Induced by Immersions in Virtual Reality. Front. Psychiatry 2021, 12, 739742. [Google Scholar] [CrossRef] [PubMed]
  52. Iii, W.B.S. Psychometric Evaluation of the Simulator Sickness Questionnaire as a Measure of Cybersickness. Ph.D. Thesis, Iowa State University, Ames, IA, USA, 2017. [Google Scholar] [CrossRef] [Green Version]
  53. Nunnally, J.C. Psychometric Theory—25 Years Ago and Now. Educ. Res. 1975, 4, 7–21. [Google Scholar] [CrossRef]
  54. Kourtesis, P.; Collina, S.; Doumas, L.A.; MacPherson, S.E. Validation of the Virtual Reality Everyday Assessment Lab (VR-EAL): An Immersive Virtual Reality Neuropsychological Battery with Enhanced Ecological Validity. J. Int. Neuropsychol. Soc. 2020, 27, 181–196. [Google Scholar] [CrossRef]
  55. Karray, F.; Alemzadeh, M.; Abou Saleh, J.; Arab, M.N. Human-Computer Interaction: Overview on State of the Art. Int. J. Smart Sens. Intell. Syst. 2008, 1, 137–159. [Google Scholar] [CrossRef] [Green Version]
  56. Stanney, K.; Lawson, B.D.; Rokers, B.; Dennison, M.; Fidopiastis, C.; Stoffregen, T.; Weech, S.; Fulvio, J.M. Identifying Causes of and Solutions for Cybersickness in Immersive Technology: Reformulation of a Research and Development Agenda. Int. J. Hum.–Comput. Interact. 2020, 36, 1783–1803. [Google Scholar] [CrossRef]
  57. Jeong, D.; Yoo, S.; Yun, J. Cybersickness Analysis with EEG Using Deep Learning Algorithms. In Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 23–27 March 2019; pp. 827–835. [Google Scholar] [CrossRef]
  58. Krokos, E.; Varshney, A. Quantifying VR cybersickness using EEG. Virtual Real. 2021, 26, 77–89. [Google Scholar] [CrossRef]
  59. Lopes, P.; Tian, N.; Boulic, R. Eye Thought You Were Sick! Exploring Eye Behaviors for Cybersickness Detection in VR. In Proceedings of the 13th ACM SIGGRAPH Conference on Motion, Interaction and Games, New York, NY, USA, 16–18 October 2020. [Google Scholar] [CrossRef]
  60. Islam, R.; Desai, K.; Quarles, J. Cybersickness Prediction from Integrated HMD’s Sensors: A Multimodal Deep Fusion Approach using Eye-tracking and Head-tracking Data. In Proceedings of the 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Bari, Italy, 4–8 October 2021; pp. 31–40. [Google Scholar] [CrossRef]
Virtualworlds 02 00002 g001 550
Figure 1. Examples of Linear (Left) and Angular (Centre and Right) Accelerations during the Ride.
Figure 1. Examples of Linear (Left) and Angular (Centre and Right) Accelerations during the Ride.
Virtualworlds 02 00002 g001
Virtualworlds 02 00002 g002 550
Figure 2. Digit Span Task (Upper Left), Corsi Block Task (Upper Right), and Deary–Liewald Reaction Time Tasks (Bottom).
Figure 2. Digit Span Task (Upper Left), Corsi Block Task (Upper Right), and Deary–Liewald Reaction Time Tasks (Bottom).
Virtualworlds 02 00002 g002
Virtualworlds 02 00002 g003 550
Figure 3. User Interface and Eye-Tracking (ET) Targets of the VR version of CSQ-VR. Note: Eye-tracking targets were not visible to the user.
Figure 3. User Interface and Eye-Tracking (ET) Targets of the VR version of CSQ-VR. Note: Eye-tracking targets were not visible to the user.
Virtualworlds 02 00002 g003
Virtualworlds 02 00002 g004 550
Figure 4. Correlations between the scores of the CSQ-VR (paper-and-pencil version) and the SSQ.
Figure 4. Correlations between the scores of the CSQ-VR (paper-and-pencil version) and the SSQ.
Virtualworlds 02 00002 g004
Virtualworlds 02 00002 g005 550
Figure 5. Correlations between the scores of the CSQ-VR (VR version) and the SSQ.
Figure 5. Correlations between the scores of the CSQ-VR (VR version) and the SSQ.
Virtualworlds 02 00002 g005
Virtualworlds 02 00002 g006 550
Figure 6. Sensitivity and Specificity of the CSQ-VR, SSQ, and VRSQ Total Scores in detecting Reaction Time (Left) and Motor Speed (Right) Decline. Note: (VR) = VR version.
Figure 6. Sensitivity and Specificity of the CSQ-VR, SSQ, and VRSQ Total Scores in detecting Reaction Time (Left) and Motor Speed (Right) Decline. Note: (VR) = VR version.
Virtualworlds 02 00002 g006
Virtualworlds 02 00002 g007 550
Figure 7. Sensitivity and Specificity of the CSQ-VR, SSQ, and VRSQ Vestibular/Disorientation Scores in detecting Reaction Time (Left) and Motor Speed (Right) Decline. Note: (VR) = VR version.
Figure 7. Sensitivity and Specificity of the CSQ-VR, SSQ, and VRSQ Vestibular/Disorientation Scores in detecting Reaction Time (Left) and Motor Speed (Right) Decline. Note: (VR) = VR version.
Virtualworlds 02 00002 g007
Virtualworlds 02 00002 g008 550
Figure 8. Mixed Regression Model of Pupil Size Predicting Cybersickness Intensity.
Figure 8. Mixed Regression Model of Pupil Size Predicting Cybersickness Intensity.
Virtualworlds 02 00002 g008
Table
Table 1. Descriptive Statistics.
Table 1. Descriptive Statistics.
Mean (SD)RangeMax. Score
Sex (22F/17M)---
Age25.28 (3.22)22–36 -
Years of Education15.14 (5.18)13–20-
VR Experience2.67 (0.92)2–614
Computing Experience10.36 (0.80)9–1214
Gaming Experience5.54 (2.97)2–1214
MSSQ Child Score4.69 (3.34)0–13.50 27
MSSQ Adult Score3.91 (3.20)0–11.2527
MSSQ Total Score8.60 (5.23)0–20.1354
Pupil Size (mm)5.37 (0.90)3.70–8.32-
CSQ-VR (VR) Total Score *10.63 (4.97)6–2842
CSQ-VR (VR) Nausea Score *3.18 (1.56)2–914
CSQ-VR (VR) Vestibular Score *3.66 (2.43)2–1314
CSQ-VR (VR) Oculomotor Score *3.79 (1.70)2–914
CSQ-VR Total Score12.23 (4.96)6–2742
CSQ-VR Nausea Score3.51 (1.68)2–914
CSQ-VR Vestibular Score3.97 (2.41)2–1014
CSQ- VR Oculomotor Score4.74 (1.81)2–1014
SSQ-Total Score67.24 (48.09)0–223.66300
SSQ-Nausea Score24.22 (22.09)0–95.40100
SSQ-Disorientation Score9.40 (9.98)0–44.88100
SSQ-Oculomotor Score33.62 (21.84)0–83.38100
VRSQ-Total Score19.17 (13.27)0–59.17100
VRSQ-Disorientation Score11.62 (13.27)0–60.00100
VRSQ-Oculomotor Score26.71 (15.63)0–58.33100
* (VR) = VR version; Pupil Size measured while responding to the VR version of CSQ-VR.
Table
Table 2. Descriptive Statistics of the VR version of CSQ-VR per Experimental Stage.
Table 2. Descriptive Statistics of the VR version of CSQ-VR per Experimental Stage.
Experimental
Stage		CSQ-VR Scores *Mean (SD)RangeMax. Score
BaselineTotal Score7.59 (2.09)6–1642
Nausea Score2.23 (0.54)2–414
Vestibular Score2.38 (0.85)2–614
Oculomotor Score2.79 (1.11)2–614
Ride 1Total Score10.79 (4.35)6–24 42
Nausea Score3.41 (1.37)2–8 14
Vestibular Score3.97 (2.47)2–12 14
Oculomotor Score3.41 (1.41)2–8 14
Ride 2Total Score11.87 (5.03)6–23 42
Nausea Score3.54 (1.57)2–8 14
Vestibular Score4.13 (2.56)2–12 14
Oculomotor Score4.21 (1.73)2–9 14
Ride 3Total Score12.26 (6.19)6–28 42
Nausea Score3.54 (2.02)2–9 14
Vestibular Score4.15 (2.91)2–13 14
Oculomotor Score4.56 (2.00)2–914
* Scores of the VR version of CSQ-VR during the exposure to VR.
Table
Table 3. Reliability (Internal Consistency) of CSQ-VR, SSQ, and VRSQ.
Table 3. Reliability (Internal Consistency) of CSQ-VR, SSQ, and VRSQ.
QuestionnaireScoresCronbach’s α
CSQ-VRTotal Score0.865
Nausea0.792
Vestibular0.934
Oculomotor0.704
SSQTotal Score0.810
Nausea0.676
Disorientation0.809
Oculomotor0.744
VRSQTotal Score0.806
Disorientation0.718
Oculomotor0.654
The internal consistency was based on the sub-scores of the total score, and the sub-scores were based on their respective items. Based on [41], Cronbach’s α of 0.6–0.7 is acceptable, 0.7–0.8 is good, and 0.8–0.95 is very good.
Table
Table 4. Correlations between the scores of CSQ-VR (both versions) and the VRSQ.
Table 4. Correlations between the scores of CSQ-VR (both versions) and the VRSQ.
Correlation PairPearson’s rp-Value
CSQ-VR–Total ScoreVRSQ–Total Score0.77<0.001
CSQ-VR–OculomotorVRSQ–Oculomotor0.75<0.001
CSQ-VR–VestibularVRSQ–Disorientation0.55<0.001
CSQ-VR (VR)–Total ScoreVRSQ–Total Score0.65<0.001
CSQ-VR (VR)–OculomotorVRSQ–Oculomotor0.62<0.001
CSQ-VR (VR)–VestibularVRSQ–Disorientation0.52<0.001
(VR) = VR version.
Table
Table 5. Psychometric Properties of the CSQ-VR, SSQ, and VRSQ in detecting Reaction Speed Decline.
Table 5. Psychometric Properties of the CSQ-VR, SSQ, and VRSQ in detecting Reaction Speed Decline.
Cybersickness ScoreCut-OffSensitivity (%)Specificity (%)PPV (%)NPV (%)AUC (%)Metric Score
CSQ-VR–Total Score10100%75%15.15%100%87%1.75
CSQ-VR (VR)–Total Score9100%75%15.15%100%86.5%1.75
SSQ–Total Score83.3680%68.75%10.26%98.72%66.1%1.49
VRSQ–Total Score20100%53.57%8.77%100%66.6%1.54
(VR) = VR version. Based on [42] and [43], the following thresholds were set and had to be met: AUC > 70% and metric score > 1.5.; PPV = positive predictive value (i.e., the ratio of true positives); NPV = negative predictive value (i.e., the ratio of true negatives).
Table
Table 6. Psychometric Properties of the CSQ-VR, SSQ, and VRSQ in detecting Motor Speed Decline.
Table 6. Psychometric Properties of the CSQ-VR, SSQ, and VRSQ in detecting Motor Speed Decline.
Cybersickness ScoreCut-offSensitivity (%)Specificity (%)PPV (%)NPV (%)AUC (%)Metric Score
CSQ-VR–Total Score10100%75.68%18.18%100%86.9%1.76
CSQ-VR (VR)–Total Score9100%75.68%18.18%100%88%1.76
SSQ–Total Score83.3683.33%69.37%12.82%98.72%68%1.53
VRSQ–Total Score20100%54.05%10.53%100%67.53%1.54
(VR) = VR version. Based on [42] and [43], the following thresholds were set and had to be met: AUC > 70% and metric score > 1.5; PPV = positive predictive value (i.e., the ratio of true positives); NPV = negative predictive value (i.e., the ratio of true negatives).
Table
Table 7. Psychometric Properties of the CSQ-VR, SSQ, and VRSQ Vestibular/Disorientation Scores in detecting Reaction Speed Decline.
Table 7. Psychometric Properties of the CSQ-VR, SSQ, and VRSQ Vestibular/Disorientation Scores in detecting Reaction Speed Decline.
Cybersickness ScoreCut-OffSensitivity (%)Specificity (%)PPV (%)NPV (%)AUC (%)Metric Score
CSQ-VR–Nausea360%67.86%7.69%97.44%65.3%1.28
CSQ-VR–Vestibular5100%77.68%16.67%100%92.6%1.78
CSQ-VR–Oculomotor740%93.75%22.22%97.22%65.8%1.34
CSQ-VR(VR)–Nausea3100%66.96%11.09%100%83.6%1.67
CSQ-VR(VR)–Vestibular4100%70.54%13.16%100%86.7%1.71
CSQ-VR (VR)–Oculomotor640%90.18%15.38%97.12%61.2%1.30
SSQ–Nausea47.740%88.39%13.33%97.06%60.04%1.28
SSQ–Disorientation11.22100%64.29%11.11%100%70.1%1.64
SSQ–Oculomotor45.4880%58.04%7.84%98.48%67.9%1.38
VRSQ–Disorientation2080%74.01%12.12%98.81%73.06%1.54
VRSQ–Oculomotor33.33100%53.57%8.77%100%63.4%1.54
(VR) = VR version. Based on [42] and [43], the following thresholds were set and had to be met: AUC > 70% and metric score > 1.5; PPV = positive predictive value (i.e., the ratio of true positives); NPV = negative predictive value (i.e., the ratio of true negatives).
Table
Table 8. Psychometric Properties of the CSQ-VR, SSQ, and VRSQ Vestibular/Disorientation Scores in detecting Motor Speed Decline.
Table 8. Psychometric Properties of the CSQ-VR, SSQ, and VRSQ Vestibular/Disorientation Scores in detecting Motor Speed Decline.
Cybersickness ScoreCut-OffSensitivity (%)Specificity (%)PPV (%)NPV (%)AUC (%)Metric Score
CSQ-VR–Nausea2100%32.43%7.41%100%62.6%1.32
CSQ-VR–Vestibular5100%78.38%20%100%94.4%1.78
CSQ-VR–Oculomotor733.33%93.69%22.22%96.3%61%1.27
CSQ-VR(VR)–Nausea3100%67.57%14.29%100%85.1%1.68
CSQ-VR(VR)–Vestibular4100%71.17%15.79%100%89.3%1.71
CSQ-VR (VR)–Oculomotor633.33%90.09%15.38%96.15%56.5%1.23
SSQ–Nausea47.750%89.19%20%97.06%65.08%1.39
SSQ–Disorientation11.22100%64.86%13.33%100%70.3%1.65
SSQ–Oculomotor45.4883.33%58.56%9.8%98.48%67.8%1.42
VRSQ–Disorientation2082.3%74.77%15.15%98.81%75%1.58
VRSQ–Oculomotor33.33100%54.05%10.53%100%63.5%1.54
(VR) = VR version. Based on [42] and [43], the following thresholds were set and had to be met: AUC > 70% and metric score > 1.5; PPV = positive predictive value (i.e., the ratio of true positives); NPV = negative predictive value (i.e., the ratio of true negatives).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Kourtesis, P.; Linnell, J.; Amir, R.; Argelaguet, F.; MacPherson, S.E. Cybersickness in Virtual Reality Questionnaire (CSQ-VR): A Validation and Comparison against SSQ and VRSQ. Virtual Worlds 2023, 2, 16-35. https://doi.org/10.3390/virtualworlds2010002

AMA Style

Kourtesis P, Linnell J, Amir R, Argelaguet F, MacPherson SE. Cybersickness in Virtual Reality Questionnaire (CSQ-VR): A Validation and Comparison against SSQ and VRSQ. Virtual Worlds. 2023; 2(1):16-35. https://doi.org/10.3390/virtualworlds2010002

Chicago/Turabian Style

Kourtesis, Panagiotis, Josie Linnell, Rayaan Amir, Ferran Argelaguet, and Sarah E. MacPherson. 2023. "Cybersickness in Virtual Reality Questionnaire (CSQ-VR): A Validation and Comparison against SSQ and VRSQ" Virtual Worlds 2, no. 1: 16-35. https://doi.org/10.3390/virtualworlds2010002

APA Style

Kourtesis, P., Linnell, J., Amir, R., Argelaguet, F., & MacPherson, S. E. (2023). Cybersickness in Virtual Reality Questionnaire (CSQ-VR): A Validation and Comparison against SSQ and VRSQ. Virtual Worlds, 2(1), 16-35. https://doi.org/10.3390/virtualworlds2010002

Article Metrics

Back to TopTop