Reliability of Scores Computed by a Commercial Virtual Reality System and Association with Indices of Cognitive Performance in Male Elite Rugby Players
Round 1
Reviewer 1 Report
The paper is well written, the experiment was taken in a good way and the conclusions are supported by the researching.
However, the research could be improved using data mining techniques in order to classify several types of players or their behaviors.
Author Response
The paper is well written, the experiment was taken in a good way and the conclusions are supported by the researching.
However, the research could be improved using data mining techniques in order to classify several types of players or their behaviors.
R : Dear reviewer, I understand your point which is very pertinent by the way. The submitted paper is the beginning of a larger project about the 1) cognitive fatigue evaluation after high intensity exercise and 2) the development of cognitive performance throughout VR training with rugby union players. For sure, we will improve our data mining in the future work, however it seems difficult in the current project in view of the delay of review required and the nature of the paper (reliability study).
Reviewer 2 Report
The purpose of the trial is to examine the reliability of scores calculated from virtual reality (VR) games and their association with inhibitory control and congitive flexibility in young elite rugby players.
Methods and results are quality described.
Considering metrological properties and association level with inhibition and flexibility, the sensibility scores of the Beat Master-Never Stop and Tracker Master games should be prefferd for monitoring trained load, provided at least two familiarization sessions precede them.
Author Response
The purpose of the trial is to examine the reliability of scores calculated from virtual reality (VR) games and their association with inhibitory control and congitive flexibility in young elite rugby players.
Methods and results are quality described.
Considering metrological properties and association level with inhibition and flexibility, the sensibility scores of the Beat Master-Never Stop and Tracker Master games should be prefferd for monitoring trained load, provided at least two familiarization sessions precede them.
R : The final part of the conclusion has been modified according to your suggestion.
Reviewer 3 Report
This is an interesting work using a novel technology (VR) in rugby players and comparing its outcomes with the one obtained from cognitive tests.
The following comments/suggestions must be taken into consideration:
- The introduction should be extended and integrated with a deeper background and larger state of art, about both cognitive tests (Takahashi S, Grove PM. Front Psychol. 2020; Ryu K, Kim J, et al. Percept Mot Skills. 2015) and virtual reality (Caporaso T, et al. Sensors 2022; Kittel A, et al. J Sci Med Sport. 2019), and a focus on their role in the assessed population.
-section 2.2: please clarify that subjects repeated VR games two times for each exercise
-statistical analysis: "A 2-way factorial analysis of variance (group x time) with repeated measures 120 on the time factor was performed to test the null hypothesis that measures were similar 121 between groups and at each time point". Were participants divided into two groups? Moreover, references should be added on the used interpretation of Hedges’ g (g), intraclass correlation coefficient (ICC) and standard error of measurement (SEM). How was association with Stroop test derived?
- Discussion: please adequate the following sentence to results "Interestingly, the scores were strong to very strong associated with executive performance". Please refer to similar works in the literature, even in different populations, for comparison and interpretation of results.
Author Response
This is an interesting work using a novel technology (VR) in rugby players and comparing its outcomes with the one obtained from cognitive tests.
The following comments/suggestions must be taken into consideration:
- The introduction should be extended and integrated with a deeper background and larger state of art, about both cognitive tests (Takahashi S, Grove PM. Front Psychol. 2020; Ryu K, Kim J, et al. Percept Mot Skills. 2015) and virtual reality (Caporaso T, et al. Sensors 2022; Kittel A, et al. J Sci Med Sport. 2019), and a focus on their role in the assessed population.
R : According to your suggestion, one paragraph has been added to extend the state of art with the suggestion made by yourself. Please see below :
With the development of digital technologies, new devices such as virtual reality (VR) offer a friendly strategy to assess or develop cognitive performance. In view of their accessibility and the strong application which represent the immersive virtual environment, an increasing number of studies focus on the ergonomic aspect and assess the metrological properties of such systems in the context of work place [5] or elite team sport [6]. To our knowledge, Kittel et al. [6] are among the first to assess the reliability and validity of a VR system in the context of professional sport. In their study, authors reported better performance of elite versus sub elite population and a better perception of environment compared with standard video system concomitant with a strong relative reliability of the VR system. These results bring out the VR systems as a promising tool in the context of decision making assessment, but their use for cognitive performance monitoring still needs to be established.
-section 2.2: please clarify that subjects repeated VR games two times for each exercise
R : according to your suggestion the following sentence was added : “Therefore, each participant performed one familiarization and two testing sessions for each VR exercise”.
-statistical analysis: "A 2-way factorial analysis of variance (group x time) with repeated measures 120 on the time factor was performed to test the null hypothesis that measures were similar 121 between groups and at each time point". Were participants divided into two groups? Moreover, references should be added on the used interpretation of Hedges’ g (g), intraclass correlation coefficient (ICC) and standard error of measurement (SEM). How was association with Stroop test derived?
R : I totally agree with your comment and the entire 2.5 paragraph has been modified according to your suggestion : “Standard statistical methods were used for the calculation of means and standard deviations. A 2-way factorial analysis of variance (group x time) with repeated measures on the time factor was performed to test the null hypothesis that measures were similar between groups and at each time point. Multiple comparisons were made with Tukey's post hoc test. The magnitude of the difference was assessed by the Hedges’ g (g), considered as small (0.2 £ ½g½< 0.5), moderate (0.5 £ ½g½ < 0.8) or large (½g½ ³ 0.8) [6]. Relative and absolute reliability were assessed with the intraclass correlation coefficient (ICC; model 2.1) and the standard error of measurement (SEM). Both the ICC and SEM were computed from the breakdown two-way ANOVA (trials x subjects) with repeated measures. The ICC was considered moderate (0.50 < ICC < 0.69), large (0.70 < ICC < 0.89) or very large (ICC > 0.90) [7]. Standard error measurement was also used to determine the minimum difference to be considered real (MD) [8]. The Pearson linear correlation (r) was used to determine the association between the VR system scores and Stroop test variables. The strength of a relationship was considered “good” with 0.69 ³ ½r½³ 0.50, “strong” with 0.89 ³ ½r½³ 0.70 or “very strong” with ½r½³ 0.90 [7]. Statistical significance was set at p < 0.05 for all analyses. All calculations were made with Statistica 6.0 (StatSoft, Tulsa, OK, USA).”
- Discussion: please adequate the following sentence to results "Interestingly, the scores were strong to very strong associated with executive performance". Please refer to similar works in the literature, even in different populations, for comparison and interpretation of results.
R : According to your suggestion, the sentence has been modified to : “Interestingly, the scores were strongly to very strongly associated with executive performance.”
Furthermore, one last paragraph has been added to contextualize the metrological results, as you can see below :
When considering their reliability characteristics and their level of association with executive performance, sensibility scores measured during the Beat Master – Never Stop game (MD = 8.3% and r = -0.82) or during the Tracker Master game (MD = 11.1% and r = -0.64) represent the best compromise. The absence of a learning effect during the test-retest (p>0.05 and g < 0.58) suggests they can be used without familiarization. Furthermore, the moderate relative reliability reported for both indices (ICC = 0.54 and 0.60 for Beat Master- Never Stop and Tracker Master, respectively) are lower than results reported with a comparable VR system assessed with Elite Australian Football Players [6]. These differences can be explained by the lack of experience in VR environment by the players included in our study. It suggests that VR familiarization, or acclimation, sessions are required in order to accustom players with this special environment.
Round 2
Reviewer 3 Report
Authors worked to improve the manuscript according to the provide suggestion. I am satisfied with the updates in the current version.
