Next Article in Journal
Gamification of Culture: A Strategy for Cultural Preservation and Local Sustainable Development
Next Article in Special Issue
Systematic Review for Knowledge Transfer at International Sport Mega-Events
Previous Article in Journal
Scientometrics-Based Research Status and Hot Topics Analysis of Chinese Private Colleges under Policy Guidance
Previous Article in Special Issue
Accessibility of Inclusive Sports Facilities for Training and Competition in Indonesia and Malaysia
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sustainability
Volume 15
Issue 1
10.3390/su15010648
sustainability-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Ruler Drop Method in Virtual Reality as an Accurate and Reliable Tool for Evaluation of Reaction Time of Mixed Martial Artists

by
Alan Langer
1,
Jacek Polechoński
2,*,
Piotr Polechoński
1 and
Jarosław Cholewa
2
1
Student Scientific Circle of Physical Activity and Tourism in Virtual Reality “ACTIVE VR”, The Jerzy Kukuczka Academy of Physical Education in Katowice, 40-065 Katowice, Poland
2
Institute of Sport Sciences, The Jerzy Kukuczka Academy of Physical Education in Katowice, 40-065 Katowice, Poland
*
Author to whom correspondence should be addressed.
Sustainability 2023, 15(1), 648; https://doi.org/10.3390/su15010648
Submission received: 27 October 2022 / Revised: 16 December 2022 / Accepted: 27 December 2022 / Published: 30 December 2022
(This article belongs to the Special Issue Physical Activity and Sustainable Healthy Lifestyles)
Browse Figures
Versions Notes

Abstract

:
Reaction time (RT) is one of the key factors in combat sports. Its high level is a predictor of sporting success. Therefore, RT tests are an important diagnostic tool in combat sports. The implementation of some conventional psychomotor tests in virtual settings can facilitate research and improve the objectivity and standardization of the measurement procedure. The main aim of the present study was to evaluate the reliability and validity of RT measurements, using the ruler drop method (RDM) implemented within immersive virtual reality (VR). Twenty-eight professional mixed martial arts (MMA) fighters were examined. The validity of the new VR measurement method was estimated by comparing the results obtained using the computer test and the conventional catch-the-ruler test. The reliability of the measurements was evaluated using an intraclass correlation procedure. Analysis of variance was used to examine the differences in RT in MMA fighters obtained from different tests. Significant correlations were found between the results of measurements in VR and the results of other tests, with the highest values observed between the tests performed in a virtual environment and computer-based tests. The values of the ICC intraclass correlation coefficients for all the reaction time tests conducted in the group of MMA fighters were at an adequate or high level. The analysis of variance showed that the reaction time of MMA fighters differed significantly between the tests. In VR and computer-based tests, reaction times were significantly longer than during conventional RDM measurements. RT did not depend on the hand used during the test. In conclusion, the VR environment allows for designing and conducting valid reaction time tests reliably and objectively, using standard testing procedures, while reducing the effect of human factors on the measurement results.

1. Introduction

Mixed martial arts (MMA) is a sport that has been growing rapidly in popularity in recent years [1]. Due to the combination of multiple fighting styles, competition takes place in different planes and positions. Competitors utilize stand-up, clinch, and ground fighting. MMA uses techniques derived from kickboxing, Muay Thai, wrestling, judo, and Brazilian jiu-jitsu, among others [2]. The fight consists of three rounds, with each lasting, depending on the sports skill level, three or five minutes. The exception is championship fights, which usually consist of five rounds [3]. Various factors affect the results of a bout: the level of aerobic endurance [4,5], maximum strength, anaerobic capacity [6], or training experience [7]. Good perception, efficient processing, and choosing the right response to the opponent’s actions are also of key importance [8,9]. Therefore, a major part of strength and conditioning programs for people who practice combat sports is focused on the development of reaction speed [10], which consists of reaction time (RT) and movement time (MT). RT is the time it takes to initiate an appropriate response to a stimulus [11]. A high level of RT allows the athlete to react quickly to the movements of the opponent, a significant factor affecting the speed of initiation of movements, which impacts sporting success [12]. In MMA, fighters respond to visual stimuli that depend on their opponent’s activity and behavior. The fighter must adapt movements as quickly as possible to the situations occurring during the fight, such as an attack, defense, or counterattack [13]. MMA fighters perform a variety of techniques during a fight. Their success is determined by the effective use of dominant techniques (accurate hand and leg strikes, takedowns, and control of the opponent on the ground) while neutralizing the dangerous actions of the opponent [14,15]. RT affects the effectiveness of offensive and defensive actions in both ground fighting [16] and stand-up fighting [17]. Athletes characterized by higher levels of RT are more successful in sports [18]. RT is also a parameter that distinguishes professional athletes from amateurs or non-athletes [19]. Therefore, accurate and reliable evaluation of RT is important in controlling training effects in athletes and predicting their future performance.
There are various methods for evaluating RT. Popular online applications include Human Benchmark and the Hit-the-Dots Reaction Test [20]. However, these are not the most accurate testing methods due to the latency of data transmission over the Internet. RT is also often assessed using Red and Green Light tests [21], which involve reacting or refraining from reacting, depending on the color of the light presented. In the context of evaluating the reactions of MMA fighters, it seems that the ruler drop method (RDM) is more relevant to the specifics of the sport. This test involves catching a ruler (30 cm long, dropped by the investigator) with the thumb and index finger [22,23]. This is because athletes involved in combat sports must respond to movement-related stimuli rather than changes in light color.
However, when comparing RDM to computer-based RT tests, some shortcomings of this method can be addressed. Unlike a randomized electronic system operating as in the case of computer-based tests, it is the investigator who drops the ruler and is responsible for initiating the stimulus. The participant, after several repetitions, can predict the time when the investigator drops the ruler from his or her hand. Furthermore, the ruler used in the test is considered a low-precision testing tool (with a measurement accuracy of ±1 cm). Therefore, the task performed can be considered unreliable as it does not meet the basic criterion of measurement tests, i.e., objectivity. Objectivity is determined by the degree of independence of the test results from the investigator and the test procedure. In other words, the test should be designed in such a way that the investigator does not affect the results.
The aforementioned problems with RDM can be solved if it is supplemented with a randomized stimulus generation system and a more precise electronic evaluation system. This is now possible with immersive virtual reality (VR) technology, an artificially created environment using information technology, in which a person is deprived of visual and auditory stimuli of the real world, and instead receives the images, sound, and even tactile sensations of a simulated world. In recent years, this technology has not only become very popular, but is also developing extremely rapidly. VR finds its application in various areas of human life, such as education, military technologies, and entertainment [24]. Its potential is also being recognized in the context of the possibility of promoting and practicing health-enhancing physical activity [25,26,27,28], use in physical education [29], training of cognitive functions [30,31,32], and various applications in sports [33]. Thanks to VR technology, it is possible to create realistic conditions to participate in various sports. It is possible to implement exercise accessories and devices into VR and create realistic sports facilities. A great advantage is also the possibility of training at a time convenient for the user in a small space, even at home, and with the use of appropriate software, generating an accurate assessment of training effects. At the same time, it should be noted that the potential of VR has not yet been fully exploited. For example, gloves and haptic costumes that enable the reception of tactile stimuli, which is important in sports, are still rarely used. Studies conducted over the past few years have shown that exercises in VR can improve fitness and physical capacity [34,35]. This is possible through special training programs or active video games (AVGs). Some applications can be used with trainers and ergometers, such as multidirectional treadmills, cycle ergometers, rowing ergometers, etc. This promotes the intensity of exercise, deepens immersion, and is positively evaluated by users [26,36,37,38,39]. There have also been studies on locomotion analysis in VR [40,41,42]. Initial attempts are being made to use a virtual experience for RT evaluation. In a study by Vahle et al. [43], participants performed a computer-based test in the laboratory room and in an identical digital copy of the room in a VR environment. The authors yielded similar results for both environments. This demonstrates the feasibility of implementing standard tests that assess human cognitive functions in the VR environment. Due to the fact that in the virtual world the user can perform motor tasks, there is also the possibility of implementing motor tests in VR, which can be widely used in the diagnosis of athletes. Therefore, there is a need to create such measurement tools and evaluate their usefulness.
Taking into account all the aforementioned issues, we created a simulation of a virtual RT examination based on the well-known test used by athletes. The main aim of our study was to evaluate the reliability and validity of RT measurements using RDM implemented into VR in MMA fighters, who were considered a representative group of athletes for various combat sports. It was hypothetically assumed that the reliability of the newly developed test in VR would be adequate, and its results would correlate with other tests used to assess RT. Furthermore, an attempt was made to answer the question of whether RT in MMA fighters differs between the tests (test in VR, computer-based test, catch-the-ruler test). Furthermore, the reaction times of the athletes were compared between the right and left upper limbs.

2. Materials and Methods

2.1. Participants

The study included twenty-eight men who are MMA fighters (age: 26.7 ± 5.3 years; body mass: 76.8 ± 10.2 kg; body height: 178.9 ± 6.1 cm; training experience: 7.2 ± 3.8 years). All fighters had participated in at least one professional MMA bout and were right-handed. The tests were performed at the MMA & Performance training studio in Świętochłowice, Poland. Participants had to meet the following inclusion criteria: good general health status (the athletes had valid medical examinations entitling them to participate in sports competitions), no contraindications to participation in the study (in particular, no motion sickness, epileptic episodes, sensitivity to flashing lights), no physical limitations (e.g., injuries), no intense physical exertion for 12 h preceding the examination, and no use of medications that could affect reaction time. All athletes were notified of the purpose and procedure of the tests. The study procedures were reviewed and approved by the Research Ethics Committee of the Jerzy Kukuczka Academy of Physical Education in Katowice (protocol #9/2018). All participants took part in the study voluntarily and could discontinue their participation at any time. The athletes were familiar with VR before the examination, but none of them declared regular use of the technology. Participants had not previously used the software employed in the study to assess reaction time.

2.2. Procedure

The Oculus Quest 2 wireless VR headset (Facebook Technologies, LLC. 1 Hacker Way, Menlo Park, CA 94025, USA), consisting of VR glasses and controllers, was used for virtual reality projection.
The fighters were tested using the RDM method in a VR environment by following a procedure temporarily named ‘catch-the-ruler test’. The test was performed in a standing position. This position was used because, during an MMA fight, the fighters spend the most time on stand-up fighting [44]. The use of the standing position during RT tests using RDM has also been recommended in previous studies [45]. Before the main part of the test, the athletes performed one trial set consisting of 6 repetitions. During the main part, they performed three series of 12 repetitions. The ruler was released at randomized intervals (2–6 s). For each test, the mean reaction time was calculated, with the two extreme values rejected. The calculated means were used to evaluate test reliability. RT was measured to the nearest 1 ms, from the moment the ruler began to fall until it was caught by the athlete. The falling virtual ruler moved downward at an acceleration equal to gravitational acceleration (9.8 m/s²).
Each study participant received the following instructions: ”Stand up with your arm extended forward [right or left]. Press the trigger with your index finger, which will bring the ruler hanging in space closer to your hand. As soon as the ruler begins to fall, press the trigger with your index finger, which will cause the ruler to stop.” Figure 1 presents a visualization of the catch-the-ruler test in VR.
To determine the validity of the reaction time test in VR, similar measurements were made using a conventional computer-based test [46] and a conventional variation of RDM. The computer-based tests consisted of pressing the appropriate buttons on the computer keyboard as soon as light stimuli (white squares) appeared on the screen. A simple reaction test was performed separately for the right and left upper limbs. The participants pressed the labeled keyboard key with their index finger when a white square appeared at randomly varying intervals in the center of the screen.
The conventional catch-the-ruler test was performed in a standing position, according to the commonly accepted methodology [45]. The athletes were instructed to catch a ruler (50 cm in length), which was dropped by the investigator. During the test, the participant’s upper limb was flexed at the elbow, and the thumb and index finger were about 5 cm apart on both sides of the ruler. Catching occurred by joining the thumb and forefingers. The investigator held the ruler vertically and dropped it, trying not to signal his or her intention. The ruler was released at randomized intervals (2–6 s) (Figure 2). The result was recorded in centimeters and then converted to time in milliseconds. The following formula was used for this purpose: t = 2 d / g ; derived from the formula: d = 1 2   g t 2 ; where: d = distance in meters, g = acceleration due to gravity = 9.81 m/s2, t = time in seconds [22,45,47]. The testing procedure for the computer-based tests and the conventional catch-the-ruler test was the same as that used for RDM in VR.

2.3. Statistical Analysis

The validity of the test task created in the virtual environment was evaluated using Pearson correlation analysis. The results obtained using RDM in VR were compared with the results of the computer-based test and the conventional RDM test. The intraclass correlation procedure was used to assess the reliability of the measurements [48,49]. Also used was a 2-factor model with mixed effects, in which the object effects are random, the position effects are constant, and the intraclass correlation type uses the definition of absolute agreement. Reliability was assessed for a single measurement, according to the formula:
ICC   ( 3 , 1 ) = M S R − M S E M S R + k − 1 M S E + k n M S C − M S E  
where MSR—mean sum of squares between objects, MSE—error mean sum of squares, MSC—mean sum of squares between measurements, n—number of objects, k—number of repeated measurements of each object. The F test was used to assess the significance of the intraclass correlation coefficient. Reliability coefficients equal to or greater than 0.90 are considered very high, 0.80–0.89 are considered high, 0.70–0.79 are considered adequate, 0.60–0.69 are marginal, and 0.59 or lower are considered low [50,51]. The results of reaction times obtained from the athletes in all tests performed with the right and left upper limbs were also analyzed. For this purpose, a two-factor analysis of variance was used. The effect size was calculated with eta square η2.The significance of the differences in results between the different tests was estimated using Tukey’s post-hoc tests. The level of significance was set at p < 0.05. The normality of distribution was evaluated using the Shapiro–Wilk test. Statistical calculations were performed using IBM SPSS Statistics and Statistica software (StatSoft, Inc., Tulsa, OK, USA).

3. Results

Analysis of the results obtained from MMA fighters in tests performed in VR using the RDM method and other tests indicated the presence of significant correlations (Table 1). The highest correlations were observed between tests performed in the virtual environment and the computer-based tests. For the right hand, the value of the correlation coefficient was r = 0.767, whereas for the left hand, it was slightly lower (r = 0.683). Correlations between the other tests were weaker. The values of the correlation coefficients were in the range of 0.402 ≤ r ≤ 0.474. Significant correlations were also observed between the results obtained in tests performed with the right and left hand. The largest correlations were found for computer-based tests (r = 0.871) and those conducted in VR (r = 0.740). The lowest correlation between the right and left hands (r = 0.634) was recorded for the conventional catch-the-ruler test (Table 1).
The values of the ICC intraclass correlation coefficients for all the reaction time tests conducted in the group of MMA fighters were at an adequate or high level. Higher reliability was found (ICC = 0.783) for catching a ruler in VR with the right hand compared to the test performed with the left hand (ICC = 0.710). An inverse relationship was observed in computer tests. The intraclass correlation coefficient reached a higher value for the right hand (ICC = 0.845), while for the left hand, the ICC was ca. 0.800. In the conventional catch-the-ruler test, higher measurement reliability was found for the RT of the left upper limb. The intraclass correlation coefficient for the left hand was ICC = 0.852. For the right hand, it was slightly lower (ICC = 0.734) (Table 2).
The analysis of variance of the results obtained from MMA fighters in the tests showed that reaction times [F(2, 26) = 1900.866, p < 0.001, η2 = 0.972] differed significantly depending on the test. However, a detailed analysis of the results using post-hoc tests demonstrated that the results of the conventional catch-the-ruler test differed significantly (p < 0.001) from those obtained in computer-based tests and in VR (Figure 3). It is worth noting that in the case of conventional RDM, the reactions of MMA fighters were nearly twice as fast. In contrast, the reaction time of MMA fighters did not depend on the hand they used to perform the tests [F(1, 27) = 1.361, p = 0.249, η2 = 0.25].

4. Discussion

Many factors determine success in combat sports, among which are physical fitness, muscle strength, training experience, tournament previous performance, age, a record of fights, and mental attitude [52,53]. Darby et al. [54] showed that boxers who placed higher in the competitions had statistically significantly shorter reaction times than those who dropped out of the competition earlier. Therefore, RT can be considered a predictor of future success in boxing competitions, and reaction tests are an important diagnostic tool in sports involving striking.
For several decades, computer technology has been used to evaluate RT in response to various stimuli [55]. Computerized tests are standard tools used in research to measure the time of reaction to visual stimuli. Therefore, the significant correlations found in our study between the results of tests implemented within VR and computer-based tests demonstrate the validity of the new tool in assessing simple RT. Based on the classification proposed by Hinkle et al. [56], the correlation between the results of computer-based and VR tests should be considered high for the right hand and moderate for the left. It is difficult to interpret unequivocally the higher values of the correlation coefficient for the right hand. Perhaps this was due to handedness, as all the athletes were right-handed. Very similar correlations were observed in a study by Polechoński and Langer [31], in which the authors compared the results of the tests of simple and complex RT in response to light stimuli in VR with those obtained in computer-based tests. In the group of combat sports athletes examined by these researchers, the vast majority declared to be right-handed.
However, the correlations between the results obtained in the virtual catch-the-ruler test and its conventional variant, despite their statistical significance, were relatively low. This may be surprising, given the similarity of the assumptions of both tests. Presumably, the human factor present in the conventional RDM may have played a role, affecting the objectivity of the measurements.
The results of the present study indicated the adequate measurement reliability of both the RDM implemented within VR and its conventional version. In all the tests performed, the ICC was higher than 0.7. Studies by other authors on RDM conducted in real-world settings also showed similar results [22,23,45]. Therefore, it turns out that RDM can be used successfully in both real-world and virtual environments. It is worth noting that in our previous research on the reliability of a similar RT test implemented within VR, we also yielded promising results [31].
As it is possible to create reliable reaction tests using VR technology, it is reasonable to assume that it will also be possible to design tools to reliably diagnose other cognitive and motor abilities. Furthermore, new programs and simulators developed in VR offer new forms of training. They open up unprecedented opportunities in athlete training by controlling the perceptual inputs, measuring the kinematic-based outputs, measuring how the athlete interacts with the created environment, and providing concurrent audio-visual-haptic feedback [57]. There are also more and more applications for learning movement. These applications use virtual reality technology to create simulations of various sports or physical exercises, allowing the user to exercise in near-real-life conditions and improve their movement technique. Some applications for learning movement also offer additional functions, such as a virtual trainer or analysis of data collected during exercises, which allows for ongoing assessment of progress and identification of possible weak points. In general, VR applications for learning movement are an interesting and effective tool for people who want to improve their physical fitness and technique in various sports. There is no doubt that VR-based technology will be widely used in the training of various sports in the near future.
The analyses also included comparisons of the RT values obtained from MMA fighters in the tests. No statistically significant differences were observed between the results of the VR and computer-based tests, although the athletes reacted slightly faster during the computer keyboard tests. This may be due to the latency of the data transfer, as in the case of the computer tests, athletes pressed a key on a keyboard connected to the computer via a cable, while the tests in VR used controllers that connected to the HMD Oculus Quest 2 via wireless technology. The use of a wireless connection may have lengthened the transmission of signals, thus leading to longer reaction times recorded in VR. Vahle et al. [43], who compared the results of RT-based cognitive performance tests in VR and computer-based tests in real life (RL), also found no significant differences between the tests. However, a detailed analysis of the results showed significant differences in RT between standard ruler catching and the computer-based test or those in VR. Surprisingly, the reaction of MMA fighters was almost twice as fast in the conventional RDM test compared to the other tests. This was probably due to the aforementioned human factor present in this test, which could have affected measurement objectivity, resulting from the methodology used. While looking at the ruler, the athletes performing the test also watched the investigator. Therefore, they could, based on the investigator’s facial expressions and behavior, observe subtle body movements or muscular tension/relaxation preceding the start of the test and anticipate the moment the ruler was dropped. Such anticipation seems to have been confirmed by previous studies. Del Rossi [58], who compared the reaction time results using the RDM method with studies by previous authors [59,60,61,62], noted significant discrepancies due to methodological differences in conducting the test. One of them was the visualization of the ruler, which in Del Rossi’s study, was hidden from the participant by suspending it vertically in a polyvinyl chloride (PVC) tube to reduce the anticipation of the instant it would be released; in the other studies, the ruler was visible to the participants. There were also differences in the distance of the initial position of the ruler from the participants’ fingers, the thickness of the ruler, and the position taken by the participant. In a study by Del Rossi, participants achieved RTs of 241.6 ms, while in studies by other authors with a visible ruler, results ranged from 176.19 to 196.94 ms [63]. This illustrates how much the result is affected by the way the ruler is visualized.
The implementation of the RDM method within VR allows the user to be tested according to an objective and standardized test procedure, isolated from interfering external stimuli, and prevents the anticipation of the ruler movement. According to Neumann et al. [64], another advantage of VR technology is that the functions of the virtual environment can be controlled in a precise and reproducible way. The application used in our research is simple and intuitive for users. During the execution of the tests, the athletes did not report any problems with the operation of the testing program. They used the application’s interface without any difficulties, and no intervention from the investigator was required. Therefore, the measurements were completely independent of the investigator and can be considered objective. It should also be noted that the RT test in VR can always be reproduced under the same test conditions, while any deviation from the test procedure is detected and communicated to the user as an error. Furthermore, with the option of displaying VR on an external monitor, the investigator has constant control over the test and insight into the results obtained from the user. The test results are also saved in a database, which allows for detailed analysis after the test is completed. This makes it possible to monitor training progress, compare individual athletes, control the correct performance of the test, correct users, and repeat the test if anomalies are found.
It should also be emphasized that the MMA fighters, despite being right-handed, achieved better RT results by catching the ruler with their left hand. Although the observed differences in results were not statistically significant, the trend was repeated in each test. A study conducted by Jha et al. [63] that examined RDM in a group of non-athletes indicated that right-handed males scored better when performing the test with their right hand; but again, these were not statistically significant differences. Singh et al. [65] demonstrated higher motor conduction velocity in the dominant upper limb, i.e., the right hand in right-handed people. The faster reaction of the left hand in right-handed MMA fighters found in our study may be due to the asymmetrical position during fighting. In MMA fighters, the left hand is extended forward during fighting and plays a major role by initiating the attack, marking the blows, creating situations convenient for the actions of the right hand, and, if necessary, counteracting the opponent’s actions. The right hand, due to its higher strength and reach, is mainly used to deliver finishing blows. Similar results regarding RT of lower limbs in MMA fighters were obtained in a study by Ignatjeva et al. [13]. The authors showed small differences (statistically insignificant) in reaction times between the lead and rear legs during a leg press test performed on the Keizer pneumatic device (Keizer, Fresno, CA, USA) in MMA fighters of different sports skill levels. The results of their study may suggest that, like that of the upper limbs, RT of the lower limbs is affected by asymmetrical fighting positions. However, such conclusions should be approached with caution, as both our research and that by Ignatjeva et al. [13] found statistically insignificant correlations. Furthermore, Loturco et al. [66] showed that professional boxers were statistically significantly faster at delivering left jabs than right crosses when responding to a visual stimulus. It is likely that after many years of training, when fighters had performed attacks or counterattacks using jabs more often, a physical and neuromuscular adaptation developed, thus affecting RT.
There is no doubt that the present study has some limitations. VR is a new environment for study participants. Individuals may experience discomfort known as cybersickness [67]. Although the athletes did not report worrying symptoms, it is advisable to introduce the Simulation Sickness Questionnaire (SSQ) in the future to assess the comfort of using VR [68]. Another limitation is the data transfer latency between the wireless controller and the HMD. It would be advisable to take this parameter into account when planning future experiments. In addition, the relatively small number of study participants makes it necessary to adopt a careful approach to the obtained results.

5. Conclusions

In conclusion, it should be emphasized that the VR environment allows for designing and conducting valid reaction time tests reliably and objectively using standard testing procedures, while reducing the effect of human factors on the measurement results. Due to the innovative nature of our research, it is necessary to undertake similar research to confirm the potential of the virtual environment in RT diagnosis.

Author Contributions

Conceptualization, J.P. and A.L.; methodology, J.P. and A.L.; software, J.P.; validation, A.L. and J.P.; formal analysis, A.L., J.P., P.P. and J.C.; investigation, A.L. and J.P.; resources, J.P. and A.L.; data curation, J.P. and A.L.; writing—original draft preparation, A.L., J.P., P.P. and J.C.; writing—review and editing, A.L., J.P., P.P. and J.C.; visualization, A.L. and J.P.; supervision, J.P. and A.L.; project administration, A.L. and J.P.; funding acquisition, J.P. All authors have read and agreed to the published version of the manuscript.

Funding

The study was subsidized by the National Research and Development Center from European Union funds within the framework of project No. POIR.01.01.01-00-0365/20 “Developing study methods and shaping coordination performance with a dedicated set of training methods in boxing based on the use of virtual reality.”

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and reviewed and approved by the Research Ethics Committee of the Jerzy Kukuczka Academy of Physical Education in Katowice (protocol #9/2018).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Jensen, P.; Roman, J.; Shaft, B.; Wrisberg, C. In the Cage: MMA Fighters’ Experience of Competition. Sport Psychol. 2013, 27, 1–12. [Google Scholar] [CrossRef]
  2. Gauthier, J. Ethical and Social Issues in Combat Sports: Should Combat Sports Be Banned? In Combat Sports Medicine; Kordi, R., Maffulli, N., Wroble, R.R., Wallace, W.A., Eds.; Springer: London, UK, 2009; pp. 73–88. ISBN 978-1-84800-354-5. [Google Scholar]
  3. Follmer, B.; Vidal Andreato, L.; Silveira Coswig, V. Combat-Ending Submission Techinques in Modern Mixed Martial Arts. Ido Mov. Culture. J. Martial Arts Anthropol. 2021, 21, 6–10. [Google Scholar] [CrossRef]
  4. Yoon, J. Physiological Profiles of Elite Senior Wrestlers. Sport. Med. 2002, 32, 225–233. [Google Scholar] [CrossRef] [PubMed]
  5. Radovanovic, D.; Milovan, B.; Nurkic, M.; Stankovic, N. Recovery of Dynamic Lung Function in Elite Judoists after Short-Term High Intensity Exercise. Arch. Budo 2011, 7, 21–26. [Google Scholar]
  6. La Bounty, P.; Campbell, B.I.; Galvan, E.; Cooke, M.; Antonio, J. Strength and Conditioning Considerations for Mixed Martial Arts. Strength Cond. J. 2011, 33, 56–67. [Google Scholar] [CrossRef] [Green Version]
  7. Langer, A.; Ignatieva, A.; Fischerova, P.; Nitychoruk, M.; Golas, A.; Maszczak, A. Effect of Post-Activation Potentiation on the Force, Power and Rate of Power and Force Development of the Upper Limbs in Mixed Martial Arts (MMA) Fighters, Taking into Account Training Experience. BJHPA 2022, 14, Article2. [Google Scholar] [CrossRef]
  8. Gierczuk, D.; Bujak, Z.; Rowiński, J.; Dmitriyev, A. Selected Coordination Motor Abilities in Elite Wrestlers and Taekwon-Do Competitors. Pol. J. Sport Tour. 2012, 19, 230–234. [Google Scholar] [CrossRef] [Green Version]
  9. Mirzaei, B.; Curby, D.G.; Barbas, I.; Lotfi, N. Anthropometric and Physical Fitness Traits of Four-Time World Greco-Roman Wrestling Champion in Relation to National Norms: A Case Study. JHSE 2011, 6, 406–413. [Google Scholar] [CrossRef] [Green Version]
  10. Balasubramaniam, M.; Sivapalan, K.; Nishanthi, V.; Kinthusa, S.; Dilani, M. Effect of Dual-Tasking on Visual and Auditory Simple Reaction Times. Indian J. Physiol. Pharm. 2015, 59, 194–198. [Google Scholar]
  11. Balkó, Š.; Borysiuk, Z.; Simonek, J. The Influence of Different Performance Level of Fencers on Simple and Choice Reaction Time. Rev. Bras. De Cineantropometria E Desempenho Hum. 2016, 18, 391–400. [Google Scholar] [CrossRef] [Green Version]
  12. Pavelka, R.; Třebický, V.; Fialová, J.T.; Zdobinský, A.; Coufalová, K.; Havlíček, J.; Tufano, J.J. Acute Fatigue Affects Reaction Times and Reaction Consistency in Mixed Martial Arts Fighters. PLoS ONE 2020, 15, e0227675. [Google Scholar] [CrossRef]
  13. Ignatjeva, A.; Nitychoruk, M.; Terbalyan, A.; Langer, A.; Wacek, A.; Maszczyk, A. The Effect of the Dynamics of External Load Changes in the Aspect of the Lower Limit Response Time of the Competitors of Mixed Martial Arts (MMA) Taking into Account Weight Categories and Sport Experience. BJHPA 2021, 13, 1–10. [Google Scholar] [CrossRef]
  14. Collier, T.; Johnson, A.; Ruggiero, J. Aggression in Mixed Martial Arts: An Analysis of the Likelihood of Winning a Decision. In Violence and Aggression in Sporting Contests: Economics, History, and Policy; Springer: New York, NY, USA, 2011; Volume 4. [Google Scholar]
  15. del Vecchio, F.B.; Hirata, S.M.; Franchini, E. A Review of Time-Motion Analysis and Combat Development in Mixed Martial Arts Matches at Regional Level Tournaments. Percept. Mot. Ski. 2011, 112, 639–648. [Google Scholar] [CrossRef]
  16. Isaev, A.V.; Korshunov, A.V.; Leonov, S.V.; Sanoyan, T.R.; Veraksa, A.N. Quantitative and Qualitative Indicators of Developing Anticipation Skills in Junior Wrestling Athletes. Procedia-Soc. Behav. Sci. 2016, 233, 186–191. [Google Scholar] [CrossRef] [Green Version]
  17. Moriarity, J.; Collie, A.; Olson, D.; Buchanan, J.; Leary, P.; McStephen, M.; McCrory, P. A Prospective Controlled Study of Cognitive Function during an Amateur Boxing Tournament. Neurology 2004, 62, 1497–1502. [Google Scholar] [CrossRef] [PubMed]
  18. Akpina, S.; Zileli, R.; Senyüzlü, E.; Tunca, S.A. Anthropological and Perceptual Predictors Affecting the Ranking in Arm Wrestling Competition. Int. J. Morphol. 2013, 31, 832–838. [Google Scholar] [CrossRef] [Green Version]
  19. Chung, P.; Ng, G. Taekwondo Training Improves the Neuromotor Excitability and Reaction of Large and Small Muscles. Phys. Ther. Sport 2012, 13, 163–169. [Google Scholar] [CrossRef] [PubMed]
  20. Badau, D.; Baydil, B.; Badau, A. Differences among Three Measures of Reaction Time Based on Hand Laterality in Individual Sports. Sports 2018, 6, 45. [Google Scholar] [CrossRef] [Green Version]
  21. Gavkare, A.M.; Surdi, A.D.; Nanaware, N.L. Auditory Reaction Time, Visual Reaction Time and Whole Body Reaction Time in Athletes. Indian Med. Gaz. 2013, 147, 214–219. [Google Scholar]
  22. Aranha, V.P.; Sharma, K.; Samuel, A.J.; Joshi, R.; Kumar, S.P. Reaction Time in Children by Ruler Drop Method: A Cross-Sectional Study Protocol. PER 2015, 3, 61–66. [Google Scholar] [CrossRef]
  23. van Schooten, K.S.; Duran, L.; Visschedijk, M.; Pijnappels, M.; Lord, S.R.; Richardson, J.; Delbaere, K. Catch the Ruler: Concurrent Validity and Test–Retest Reliability of the ReacStick Measures of Reaction Time and Inhibitory Executive Function in Older People. Aging Clin. Exp. Res. 2019, 31, 1147–1154. [Google Scholar] [CrossRef] [PubMed]
  24. Ahir, K.; Govani, K.; Gajera, R.; Shah, M. Application on Virtual Reality for Enhanced Education Learning, Military Training and Sports. Augment. Hum. Res. 2020, 5, 7. [Google Scholar] [CrossRef]
  25. Cao, L.; Peng, C.; Dong, Y. Ellic’s Exercise Class: Promoting Physical Activities during Exergaming with Immersive Virtual Reality. Virtual Real. 2021, 25, 597–612. [Google Scholar] [CrossRef]
  26. Polechoński, J.; Nierwińska, K.; Kalita, B.; Wodarski, P. Can Physical Activity in Immersive Virtual Reality Be Attractive and Have Sufficient Intensity to Meet Health Recommendations for Obese Children? A Pilot Study. Int. J. Environ. Res. Public Health 2020, 17, 8051. [Google Scholar] [CrossRef]
  27. Zhou, S. Using Virtual Reality to Promote Physical Activity. J. Softw. Eng. Appl. 2020, 13, 312–326. [Google Scholar] [CrossRef]
  28. Polechoński, J.; Zwierzchowska, A.; Makioła, Ł.; Groffik, D.; Kostorz, K. Handheld Weights as an Effective and Comfortable Way To Increase Exercise Intensity of Physical Activity in Virtual Reality: Empirical Study. JMIR Serious Games 2022, 10, e39932. [Google Scholar] [CrossRef]
  29. Zhang, X.; Shi, Y.; Bai, H. Immersive Virtual Reality Physical Education Instructional Patterns on the Foundation of Vision Sensor. J. Sens. 2021, 2021, e7752447. [Google Scholar] [CrossRef]
  30. Bauer, A.C.M.; Andringa, G. The Potential of Immersive Virtual Reality for Cognitive Training in Elderly. Gerontology 2020, 66, 614–623. [Google Scholar] [CrossRef]
  31. Polechoński, J.; Langer, A. Assessment of the Relevance and Reliability of Reaction Time Tests Performed in Immersive Virtual Reality by Mixed Martial Arts Fighters. Sensors 2022, 22, 4762. [Google Scholar] [CrossRef]
  32. Wojciechowski, A.; Wiśniewska, A.; Pyszora, A.; Liberacka-Dwojak, M.; Juszczyk, K. Virtual Reality Immersive Environments for Motor and Cognitive Training of Elderly People—A Scoping Review. Hum. Technol. 2021, 17, 145–163. [Google Scholar] [CrossRef]
  33. Bideau, B.; Multon, F.; Kulpa, R.; Fradet, L.; Arnaldi, B.; Delamarche, P. Using Virtual Reality to Analyze Links between Handball Thrower Kinematics and Goalkeeper’s Reactions. Neurosci. Lett. 2004, 372, 119–122. [Google Scholar] [CrossRef]
  34. Qian, J.; McDonough, D.J.; Gao, Z. The Effectiveness of Virtual Reality Exercise on Individual’s Physiological, Psychological and Rehabilitative Outcomes: A Systematic Review. Int. J. Environ. Res. Public Health 2020, 17, 4133. [Google Scholar] [CrossRef] [PubMed]
  35. Todorov, K.; Manolova, A.; Chervendinev, G. Immersion in Virtual Reality Video Games for Improving Physical Performance Measures: A Review. In Proceedings of the 2019 27th National Conference with International Participation (℡ECOM), Sofia, Bulgaria, 30–31 October 2019; pp. 35–38. [Google Scholar]
  36. Bird, J.M.; Karageorghis, C.I.; Baker, S.J.; Brookes, D.A.; Nowicky, A.V. Ready Exerciser One: Effects of Music and Virtual Reality on Cycle Ergometer Exercise. Br. J. Health Psychol. 2021, 26, 15–32. [Google Scholar] [CrossRef] [PubMed]
  37. McClure, C.; Schofield, D. Running Virtual: The Effect of Virtual Reality on Exercise. J. Hum. Sport Exerc. 2019, 15, 816–870. [Google Scholar] [CrossRef]
  38. Shoib, N.A.; Sunar, M.S.; Nor, N.N.M.; Azman, A.; Jamaludin, M.N.; Latip, H.F.M. Rowing Simulation Using Rower Machine in Virtual Reality. In Proceedings of the 2020 6th International Conference on Interactive Digital Media (ICIDM), Bandung, Indonesia, 14–15 December 2020; pp. 1–6. [Google Scholar]
  39. Zeng, N.; Liu, W.; Pope, Z.C.; McDonough, D.J.; Gao, Z. Acute Effects of Virtual Reality Exercise Biking on College Students’ Physical Responses. Res. Q. Exerc. Sport 2021, 93, 633–639. [Google Scholar] [CrossRef] [PubMed]
  40. Jochymczyk-Woźniak, K.; Nowakowska, K.; Polechoński, J.; Sładczyk, S.; Michnik, R. Physiological Gait versus Gait in VR on Multidirectional Treadmill—Comparative Analysis. Medicina 2019, 55, 517. [Google Scholar] [CrossRef] [Green Version]
  41. Wodarski, P.; Jurkojć, J.; Bieniek, A.; Chrzan, M.; Michnik, R.; Polechoński, J.; Gzik, M. The Analysis of the Influence of Virtual Reality on Parameters of Gait on a Treadmill According to Adjusted and Non-Adjusted Pace of the Visual Scenery. In Information Technology in Biomedicine; Pietka, E., Badura, P., Kawa, J., Wieclawek, W., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 543–553. [Google Scholar]
  42. Wodarski, P.; Jurkojć, J.; Polechoński, J.; Bieniek, A.; Chrzan, M.; Michnik, R.; Gzik, M. Assessment of Gait Stability and Preferred Walking Speed in Virtual Reality. Acta Bioeng. Biomech. 2020, 22, 127–134. [Google Scholar] [CrossRef]
  43. Vahle, N.M.; Unger, S.; Tomasik, M. Reaction Time-Based Cognitive Assessments in Virtual Reality–A Feasibility Study with an Age Diverse Sample. In Studies in Health Technology and Informatics; Röhrig, R., Beißbarth, T., König, J., Ose, C., Rauch, G., Sax, U., Schreiweis, B., Sedlmayr, M., Eds.; IOS Press: Amsterdam, The Netherlands, 2021; ISBN 978-1-64368-206-8. [Google Scholar]
  44. Miarka, B.; Coswig, V.S.; Amtmann, J. Long MMA Fights Technical-Tactical Analysis of Mixed Martial Arts: Implications for Assessment and Training. Int. J. Perform. Anal. Sport 2019, 19, 153–166. [Google Scholar] [CrossRef]
  45. Anitha, M.; Samuel, V.V. Reaction Time in Sitting and Standing Postures among Typical Young Adults. Physiother.-J. Indian Assoc. Physiother. 2018, 12, 58. [Google Scholar] [CrossRef]
  46. Klocek, T.; Spieszny, M.; Szczepanik, M. Computer Tests of Coordination Abilities; COS: Warszawa, Poland, 2002; ISBN 83-86504-89-7. [Google Scholar]
  47. Eckner, J.T.; Kutcher, J.S.; Richardson, J.K. Between-Seasons Test-Retest Reliability of Clinically Measured Reaction Time in National Collegiate Athletic Association Division I Athletes. J. Athl. Train. 2011, 46, 409–414. [Google Scholar] [CrossRef] [Green Version]
  48. Shrout, P.E.; Fleiss, J.L. Intraclass Correlations: Uses in Assessing Rater Reliability. Psychol. Bull. 1979, 86, 420–428. [Google Scholar] [CrossRef] [PubMed]
  49. Koo, T.K.; Li, M.Y. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J. Chiropr. Med. 2016, 15, 155–163. [Google Scholar] [CrossRef] [Green Version]
  50. Cole, W.R.; Arrieux, J.P.; Schwab, K.; Ivins, B.J.; Qashu, F.M.; Lewis, S.C. Test-Retest Reliability of Four Computerized Neurocognitive Assessment Tools in an Active Duty Military Population. Arch. Clin. Neuropsychol. 2013, 28, 732–742. [Google Scholar] [CrossRef] [PubMed]
  51. Lezak, M.D.; Howieson, D.B.; Bigler, E.D.; Tranel, D. Neuropsychological Assessment; Oxford University Press: Oxford, UK, 2012; ISBN 978-0-19-539552-5. [Google Scholar]
  52. Warnick, J.E.; Warnick, K. Specification of Variables Predictive of Victories in the Sport of Boxing. Percept. Mot. Ski. 2007, 105, 153–158. [Google Scholar] [CrossRef]
  53. Warnick, J.E.; Warnick, K. Specification of Variables Predictive of Victories in the Sport of Boxing: II. Further Characterization of Previous Success. Percept. Mot. Ski. 2009, 108, 137–138. [Google Scholar] [CrossRef]
  54. Darby, D.; Moriarity, J.; Pietrzak, R.; Kutcher, J.; Mcaward, K.; Mccrory, P. Prediction of Winning Amateur Boxers Using Pretournament Reaction Times. J. Sport. Med. Phys. Fit. 2014, 54, 8. [Google Scholar]
  55. Holden, J.; Francisco, E.; Lensch, R.; Tommerdahl, A.; Kirsch, B.; Zai, L.; Dennis, R.; Tommerdahl, M. Accuracy of Different Modalities of Reaction Time Testing: Implications for Online Cognitive Assessment Tools. bioRxiv 2019, 726364. [Google Scholar] [CrossRef] [Green Version]
  56. Hinkle, D.E.; Wiersma, W.; Jurs, S.G. Applied Statistics for the Behavioral Sciences, 5th ed.; Houghton Mifflin: Boston, MA, USA, 2003; ISBN 978-0-618-12405-3. [Google Scholar]
  57. Shepherd, J.; Carter, L.; Pepping, G.-J.; Potter, L.-E. Towards an Operational Framework for Designing Training Based Sports Virtual Reality Performance Simulators. Proceedings 2018, 2, 214. [Google Scholar] [CrossRef] [Green Version]
  58. Del Rossi, G. Evaluating the Recovery Curve for Clinically Assessed Reaction Time After Concussion. J. Athl. Train. 2017, 52, 766–770. [Google Scholar] [CrossRef] [Green Version]
  59. Eckner, J.T.; Kutcher, J.S.; Richardson, J.K. Effect of Concussion on Clinically Measured Reaction Time in Nine NCAA Division I Collegiate Athletes: A Preliminary Study. PM&R 2011, 3, 212–218. [Google Scholar] [CrossRef] [Green Version]
  60. Eckner, J.T.; Kutcher, J.S.; Broglio, S.P.; Richardson, J.K. Effect of Sport-Related Concussion on Clinically Measured Simple Reaction Time. Br. J. Sport. Med. 2014, 48, 112–118. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  61. MacDonald, J.; Wilson, J.; Young, J.; Duerson, D.; Swisher, G.; Collins, C.L.; Meehan, W.P. Evaluation of a Simple Test of Reaction Time for Baseline Concussion Testing in a Population of High School Athletes. Clin. J. Sport. Med. 2015, 25, 43–48. [Google Scholar] [CrossRef] [PubMed]
  62. Eckner, J.T.; Chandran, S.; Richardson, J.K. Investigating the Role of Feedback and Motivation in Clinical Reaction Time Assessment. PM&R 2011, 3, 1092–1097. [Google Scholar] [CrossRef] [Green Version]
  63. Jha, R.K.; Thapa, S.; Kasti, R.; Nepal, O. Influence of Body Mass Index, Handedness and Gender on Ruler Drop Method Reaction Time among Adults. J. Nepal. Health Res. Counc. 2020, 18, 108–111. [Google Scholar] [CrossRef] [PubMed]
  64. Neumann, D.L.; Moffitt, R.L.; Thomas, P.R.; Loveday, K.; Watling, D.P.; Lombard, C.L.; Antonova, S.; Tremeer, M.A. A Systematic Review of the Application of Interactive Virtual Reality to Sport. Virtual Real. 2018, 22, 183–198. [Google Scholar] [CrossRef] [Green Version]
  65. Singh, P.I.; Maini, B.K.; Singh, I. Bilateral Asymmetry in Conduction Velocity in the Efferent Fibres of the Median Nerve Andits Relationship to Handedness. Indian J. Physiol. Pharm. 1977, 21, 364–368. [Google Scholar]
  66. Loturco, I.; Franchini, E.; Cal Abad, C.C. A Comparative Study of Specific Reaction Time in Elite Boxers: Differences between Jabs and Crosses. J. Athl. Enhanc. 2015, 4, 1–4. [Google Scholar] [CrossRef]
  67. Pastel, S.; Bürger, D.; Chen, C.H.; Petri, K.; Witte, K. Comparison of Spatial Orientation Skill between Real and Virtual Environment. Virtual Real. 2022, 26, 91–104. [Google Scholar] [CrossRef]
  68. Kennedy, R.S.; Lane, N.E.; Berbaum, K.S.; Lilienthal, M.G. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. Int. J. Aviat. Psychol. 1993, 3, 203–220. [Google Scholar] [CrossRef]
Sustainability 15 00648 g001 550
Figure 1. Visualization of the catch-the-ruler test performed by an MMA fighter in VR ((a) starting point, (b) end point).
Figure 1. Visualization of the catch-the-ruler test performed by an MMA fighter in VR ((a) starting point, (b) end point).
Sustainability 15 00648 g001
Sustainability 15 00648 g002 550
Figure 2. Conventional catch-the-ruler test performed by an MMA fighter ((a) starting point, (b) end point).
Figure 2. Conventional catch-the-ruler test performed by an MMA fighter ((a) starting point, (b) end point).
Sustainability 15 00648 g002
Sustainability 15 00648 g003 550
Figure 3. Comparison of reaction times of MMA fighters during different tests performed with the right and left hands. Notes: VR—virtual reality, RDM—ruler drop method.
Figure 3. Comparison of reaction times of MMA fighters during different tests performed with the right and left hands. Notes: VR—virtual reality, RDM—ruler drop method.
Sustainability 15 00648 g003
Table
Table 1. Correlation coefficients of the result of reaction time tests performed by MMA fighters.
Table 1. Correlation coefficients of the result of reaction time tests performed by MMA fighters.
Type of Test1.2.3.4.5.6.
1. VR RDM (right hand) 0.740 *0.767 *0.699 *0.468 *0.441 *
2. VR RDM (left hand) 0.676 *0.683 *0.374 *0.474 *
3. Computer-based test (right hand) 0.871 *0.471 *0.412 *
4. Computer-based test (left hand) 0.451 *0.402 *
5. RDM—(right hand) 0.634 *
6. RDM—(left hand)
Notes: VR—virtual reality, RDM—ruler drop method, *, p < 0.05.
Table
Table 2. Intraclass correlation coefficients (ICCs) for reaction time tests performed by MMA fighters.
Table 2. Intraclass correlation coefficients (ICCs) for reaction time tests performed by MMA fighters.
Type of TestICCp
VR RDM (right hand)0.7830.001
VR RDM (left hand)0.7100.001
Computer-based test (right hand)0.8000.001
Computer-based test (left hand)0.8450.001
RDM—(right hand)0.7340.001
RDM—(left hand)0.8520.001
Notes: VR—virtual reality, RDM—ruler drop method.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Langer, A.; Polechoński, J.; Polechoński, P.; Cholewa, J. Ruler Drop Method in Virtual Reality as an Accurate and Reliable Tool for Evaluation of Reaction Time of Mixed Martial Artists. Sustainability 2023, 15, 648. https://doi.org/10.3390/su15010648

AMA Style

Langer A, Polechoński J, Polechoński P, Cholewa J. Ruler Drop Method in Virtual Reality as an Accurate and Reliable Tool for Evaluation of Reaction Time of Mixed Martial Artists. Sustainability. 2023; 15(1):648. https://doi.org/10.3390/su15010648

Chicago/Turabian Style

Langer, Alan, Jacek Polechoński, Piotr Polechoński, and Jarosław Cholewa. 2023. "Ruler Drop Method in Virtual Reality as an Accurate and Reliable Tool for Evaluation of Reaction Time of Mixed Martial Artists" Sustainability 15, no. 1: 648. https://doi.org/10.3390/su15010648

APA Style

Langer, A., Polechoński, J., Polechoński, P., & Cholewa, J. (2023). Ruler Drop Method in Virtual Reality as an Accurate and Reliable Tool for Evaluation of Reaction Time of Mixed Martial Artists. Sustainability, 15(1), 648. https://doi.org/10.3390/su15010648

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop