Design and Evaluation for Immersive Virtual Reality Learning Environment: A Systematic Literature Review

Abstract

This systematic review broadly attempted to synthesize all relevant evidence residing in the Scopus, IEEE Xplore and MDPI databases, in order to inform the related Research Questions of this work. More precisely, the review protocol includes a broad and comprehensive search for eligible data sets from the Scopus, IEEE Xplore and MDPI databases, published from January 2017 to December 2022 by using inclusion/exclusion search criteria. Medical Education Research Study Quality Instrument (MERSQI) was commissioned to assess and analyze the quality of 69 quantitative studies. The findings generally received positive feedback and there was a discussion about the results. This work was an original contribution guided by pedagogical theory and the validity of the evaluation constitutes a proposal for future improvement.

1. Introduction

This systematic review broadly attempted to synthesize all relevant evidence residing in the Scopus and IEEE Xplore databases in accordance to review the protocol and to inform the related Research Questions of this work. Following this Introduction (Section 1) are Methods (Section 2), Results (Section 3), Discussion (Section 4), and, finally, Conclusion (Section 5).

1.1. Systematic Review

In general, the main objective of a systematic review is to perform high-quality reviews that are precise and transparent in each step of the review process, in order to secure the reproducibility and updatability as well as to present credible information that is free from commercial sponsorship and other conflicts of interest. In the review protocol, the research inquiries must be well-defined and include both inclusion and exclusion criteria. A critical analysis of the search results was reported after a wide and thorough search of the literature was carried out, and, ultimately, provided a current evidence-based awareness of the specific question [1,2,3]. Like most evidence-informed practices, this evidence-aware review never sought to provide an ‘answer’ to replace judgment and experience but instead attempted to inform decision-making and action [4]. There are, at least, five parallel studies to this work [5,6,7,8,9].

• Ryan et al. [5] located their data source from PubMed, EMBASE, Web of Science, CINHAL, and ERIC. Their latest search was conducted on 8 March 2021. The Medical Education Research Study Quality Instrument (MERSQI) was their preferred assessment tool.
• Tang et al. [6] conducted a search of reviews from the Web of Science database, restricted between 2000 and 2021 according to the guidelines of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).
• Radianti et al. [7] located their data source from IEEE Xplore, ProQuest, Scopus, and Web of Science.
• Rohwer et al. [8] performed their systematic review on the topic of E-learning of Evidence-based Health Care (EBHC) in Healthcare Professionals. Data was collected from Best Evidence Medical Education (BEME), CINAHL, CENTRAL, ERIC, EMBASE, MEDLINE, SCOPUS, PsycInfo, dissertation databases (ProQuest), and Web of Knowledge.
• Ghani et al. [9] conducted a bibliometric analysis of global research trends on Higher Education Internationalization based on 1412 publications retrieved from the Scopus database.

1.2. Immersive Virtual Reality

Since the arrival of Virtual Reality technology, it has been widely used in all walks of life and has brought great changes to the world, particularly within the field of education. The research has shown that a student’s motivation to learn will grow once they are supplied with pragmatic equipment that will help them to be more visually engaged in the classroom.

This goal can be achieved by developing useful equipment with Virtual Reality technology to visualize 3D models [10,11,12]. Users were not just passive observers but actually participated in an immersive virtual reality learning environment as active participants which allowed learning paradigm development with exploration as their basis [13]. In recent years, more and more researchers have used Virtual Reality technology to develop educational applications such as serious games or learning environments, hoping to trigger the motivation to engage in learning [14]. Immersion and scalability in the perception of the spatial aspect of virtual environments have made Immersive Virtual Reality (IVR) a significant and developing design tool [12,15,16,17,18]. Paes et al. [15] and Hou et al. [19] mention that spatial awareness in Virtual Reality systems allows the users to surmount cognitive limitations, resulting in more accurate spatial connection calculations in architectural design. The results from Alatta and Freewan’s [20] study indicates that virtual reality immersion in design contexts not only enhances the designer’s participation in the design, but also boosts their imagination and creativity, improves their performance, and expands the complexity of their design. The various methods of interaction in Immersive Virtual Reality systems lead to significant size differences in designed spaces and spatial relationship awareness [21,22].

Virtual Reality technology is characterized by immersion and interaction. It is generally believed that in an immersive virtual reality environment, participants are less intervened by the outside world and can focus more on the instructional content during the interactive process [23]. Research from Trunfino and Rossi [24] indicates that Metaverse is a combination of different emerging and existing technologies where Virtual Reality technology is one of them. The most critical value of Virtual Reality technology to the existing technology is being able to immerse in the virtual world constructed around the users [25]. The ability of the HMD system to offer self-motion feedback and to disconnect from the physical local environment by restricting eye contact with real-world objects makes it more immersive [26]. The learning environment created by Immersive Virtual Reality is less constrained by time, space, and other objective conditions, and is more suitable for those learning scenes that are difficult to be recreated or imagined [14,16,27]. For this reason, virtual worlds provide an exceptional platform for educators to apply learner-centered teachings where the students can participate more actively in order to carry out simulations and learn by joining in tasks that are usually difficult to be accessed in real life whether it is due to cost, cultural appropriation, and/or safety reasons [28,29,30,31]. At the same time, interactive experience is the preferred method of learning and training because it has a balanced cost, aids in the development of technology, and the possibility that such immersion would help the users in learning and improving their skills [20,32,33]. The results from Zhang et al. [34] suggest that Immersive Virtual Reality, rather than desktop virtual reality, has been increasingly valued by many researchers.

The studies prior to this displayed that those participants who were immersed in a virtual reality learning environment were normally more pleased regarding their experiences, were more prompt to learn, had better concentration, and had greater learning performance compared to the traditional method. Furthermore, a scientific measurement of learning performance with valid instruments and effective methods will not only lead to an accurate and objective evaluation, but will also promote and improve the design and development of an immersive virtual reality learning environment. Research on immersive virtual reality learning environments in recent years will be reviewed and analyzed with the focus being on combining the research design, method, process, and evaluation of outcome and an attempt to summarize all key factors related to learning performance. Worth re-emphasizing is the opinion of [35] who said that the application of educational technology had been discovered to carry the potential to improve self-efficacy, self-regulation, improve student engagement, and show growth in involvement and participation in curriculums and within the broader institutional community. If we make an assumption that disengagement shows detrimental effects on the outcomes of students’ learning ability and intellectual development, and is related to early dropout, it is important that we carry out an investigation into how technology has been utilized to develop more engagement. There were no fewer than four similar efforts to this work [36,37,38,39].

• Han et al. [36] presented their systematic review with comprehensive meta-analysis (CMA) software (version 3) for Augmented Reality in professional training. It was based on data made available in the Scopus database published between 2001 and 2020.
• Barteit et al. [37] focused their systematic review on examining Augmented, Mixed, and Virtual Reality-Based Head-Mounted Display (HMD) for Medical Education. Their source was built upon seven databases of Cochrane Library, Science Direct, PubMed, PsycINFO, Web of Science, ERIC, and Google Scholar. The MERSQI assessment tool was used.
• Muirhea et al. [38] conducted a systematic review to establish the effectiveness of Technology-enabled Dementia Education for Health and Social Care Practitioners. Web of Science, CINAHL, and MEDLINE were among eight bibliographic eligible databases searched. These data were published between the years 2005 to 2020. MERSQI was the assessment tool used.
• Watson et al. [39] conducted a study to recognize the best practices for undergraduate medical education learning environment interventions related to the improvement of students’ psychological well-being. The data source was restricted to biomedical electronic databases Ovid MEDLINE, the Cochrane Library, ERIC, and EMBASE from database inception dates to October 2016. MERSQI was the assessment tool used.

1.3. Pedagogical Theory

There were two reported current developments related to this work. They were (a) the contribution of Makransky and Peterson [40] for The Cognitive Affective Model of Immersive Learning (CAMIL), and (b) the contribution of Gorbanev et al. [41] in relation to the notion of gamification.

• Makransky and Peterson [40] proposed The Cognitive Affective Model of Immersive Learning (CAMIL): a theoretical Research-Based Model of Learning in Immersive Virtual Reality (CAMIL). CAMIL identified presence and agency as the general psychological affordances of learning in IVR and described how these affordances are facilitated by control factors, representational fidelity, and immersion.
• Gorbanev et al. [41] conducted a systematic review of serious games in medical education. It followed the Coachrane Collaboration Guidelines while evaluating the standard of affirmation using MERSQI. Games with a pedagogical intention were claimed seriously. Despite the game developers’ claims that the games were practical pedagogical tools, their suggested evidence of their effectiveness was deemed to be moderate. Cognitivism and behaviorism remained the main education strategies. Additionally, they claimed that the traditional medical teaching tools were not replaced but that the games were complementary. Moreover, they found that educators in the medical field generally prefer having simulations and tests focused on knowledge retention and the development of skills through reiteration. There was no request for these sophisticated games to be used in their classrooms.

1.4. Design

In relation to designing an immersive environment, there were at least four reported contributions in relation to this work [42,43,44,45].

• Blumenstein [42] proposed a Learning Analytics Learning Gain Design for actionable insight into the learning processes.
• Huang [43], in the researcher’s study, investigated the results of Head-Mounted Display (HMD) Virtual Reality on the Science Self-Efficacy of High Schoolers.
• Knobbout and van der Stappen [44] proposed a capability prototype to back the implementation and uptake of learning analytics. It was the researcher’s opinion that although uptake of LA showed promising effects, in the past decade, a lot of learning analytics practice was carried out at a minor scale with limited involvement from the learning institution. Consequently, higher education institutions also rarely apply large-scale learning analytic practices.
• Alhadad [45] defined data visualization as two domains: (1) as a means of communication: communication design, and (2) as a research methodology: data analysis for making inferences.

1.5. Process

There were three praiseworthy research efforts from immersive learning analytics professionals. They include the works of Ferguson et al. [46]; Skarbez et al. [47]; Kennedy et al. [48].

• Ferguson et al. [46] focused on learning analytics, or the measurement, collection, analysis, and reporting of data regarding the learners and their context.
• Skarbez et al. [47] provided Theory and Research Agenda on Immersive Analytics. Immersive Analytics looks into the ways to back analytical reasoning and decision-making with the new interaction and display technologies. The objective is to support collaboration and allow users to immerse themselves in their data by providing multi-sensory interfaces for analytics approaches.
• Kennedy et al. [48] focused their research attention specifically on using data mining techniques to obtain real-time feedback from users in a virtual learning domain.

1.6. Research Questions

In order to study the important points about the design as well as the assessment of the immersive virtual reality learning environment, we prepared research questions as below:

• What were the main research fields of immersive virtual reality learning environments?
• What was the situation of using pedagogical theory to guide the instructional design of an immersive virtual reality learning environment?
• What methodological research methods were noted, for example, sample size, statistical methods, and research design?
• What was the learning performance of using an immersive virtual reality learning environment?
• What were the main factors that may affect the learning performance of an immersive virtual reality learning environment?
• What are the potential problems of an immersive virtual reality learning environment that can be improved?

2. Methods

2.1. Searching and Screening

The process of searching and screening literature followed the three steps shown in Figure 1. Firstly, three (3) research databases including Scopus, IEEE Xplore, and MDPI were identified to ensure a comprehensive search. Each database was searched with keywords based on this Boolean search string: (Head-Mounted Display OR Immersive Virtual Reality OR HMD) AND (learn OR train OR education). In 2016, about 230 companies including Amazon, Apple, Facebook, Google, Microsoft, Sony, Samsung, etc. began to devote themselves to projects based on virtual reality. After that, Oculus Rift V1.0, PlayStation VR (Sony), and Hololens (Microsoft) were released successively. With the development of hardware equipment, Virtual Reality technology has begun to be widely used in various fields, and relevant research results have been published successively. Therefore, the papers published from January 2017 until December 2022 were mainly searched in this review. The search, after removing duplicates and not English literature, generated a sum of 4159 references to conference papers and journal articles. Secondly, through screening the topic, and reading the title and abstract, irrelevant data were excluded considering the purpose of this review. For example, some articles about the application of virtual reality in the medical field were removed, such as surgery simulators or recovery of patients. This is due to the fact that the group tested usually suffered from a certain disease, therefore making the sample not universal which made it focused on the evaluation of health indicators rather than learning performance. After they were removed, 196 records remained. Finally, the remaining articles were filtered by the conditions below: (1) Having the full-text version accessible and available; (2) Learning environment based on Immersive Virtual Reality (IVR); (3) Describing the evaluation of the outcome. Among the 196 records, 69 met all of the above conditions and were included in this analysis.

2.2. Quality of Research

Within the 48 studies that were identified, 42 used primarily quantitative methods. The remaining four were primarily qualitative [49,50,51,52]. The standard of the 42 quantitative studies was assessed using MERSQI [53,54,55]. As Jensen [54] stated, although MERSQI was developed for education in the medical field, the standard of non-medical education research can also be assessed. This is because it was neutral in practice and it had been strictly assessed to evaluate other quality assessment instruments. MERSQI consists of six domains, each of which has 3 as the maximum score, therefore a study will have the greatest sum of scores of 18. The completed MERSQI scores for each of the 42 quantitative studies are shown in Appendix A,

Table A1. The detailed MERSQI scores for each of the 42 quantitative studies.

Study	Design	Sampling		Data	Validity of Evaluation Instrument			Analysis		Outcome	Total
Reference	Study Design	No. of Institutions	Response Rate	Type of Data	Internal Structure	Content	Rel. to Other Variables	Appropriateness	Complexity	Outcome
[92]	1	0.5	1.5	1	0	1	0	1	2	1	9
[70]	3	0.5	1.5	3	0	1	0	1	2	1.5	13.5
[62]	1	0.5	1.5	3	1	1	0	1	2	1.5	12.5
[22]	3	1.5	1.5	3	0	1	0	1	2	1.5	14.5
[87]	1	0.5	1.5	1	0	1	0	1	2	1	9
[64]	1	0.5	1.5	1	0	0	0	1	2	1	8
[65]	2	0.5	1.5	1	1	0	0	1	2	1	10
[97]	1	0.5	1.5	1	0	1	0	1	2	1	9
[66]	1	0.5	1.5	3	0	1	0	1	2	1.5	11.5
[93]	3	0.5	1.5	1	1	1	0	1	2	1	12
[32]	3	0.5	1.5	3	1	1	0	1	2	1.5	14.5
[88]	3	0.5	1.5	3	0	1	0	1	2	1.5	13.5
[71]	3	0.5	1.5	3	1	1	1	1	2	1.5	15.5
[72]	3	0.5	1.5	3	1	0	0	1	2	1.5	13.5
[63]	1.5	0.5	1.5	3	1	0	0	1	2	1.5	12
[90]	1	0.5	1.5	1	0	1	0	1	2	1	9
[79]	1	0.5	1.5	3	1	1	0	1	2	1.5	12.5
[31]	1	0.5	1.5	3	1	1	0	1	2	1.5	12.5
[80]	3	0.5	1.5	3	0	0	1	1	2	1.5	13.5
[78]	3	0.5	1.5	3	1	1	0	1	2	2	15
[84]	1	0.5	1.5	3	1	1	0	1	2	1.5	12.5
[73]	3	0.5	1.5	3	0	1	0	1	2	1.5	13.5
[17]	3	1.5	1.5	3	0	1	0	1	2	2	15
[74]	1.5	0.5	1.5	3	0	1	0	1	2	1.5	12
[81]	3	0.5	1.5	3	1	1	0	1	2	1.5	14.5
[75]	3	0.5	1.5	3	0	0	0	1	2	2	13
[67]	3	0.5	1.5	3	0	0	0	1	2	1.5	12.5
[68]	3	0.5	1.5	3	0	1	0	1	2	1.5	13.5
[76]	3	0.5	1.5	3	1	1	0	1	2	1	14
[85]	3	0.5	1.5	3	0	1	0	1	2	1.5	13.5
[89]	3	0.5	1.5	3	0	1	0	1	2	1.5	13.5
[91]	1	0.5	0.5	1	0	1	0	1	1	1	7
[82]	3	0.5	1.5	3	1	1	0	1	2	1.5	14.5
[94]	1	1.5	1.5	1	0	1	0	1	2	1	10
[69]	1	0.5	1.5	3	0	1	0	1	2	1.5	11.5
[95]	3	0.5	1.5	3	0	0	0	1	2	1.5	12.5
[26]	3	0.5	1.5	3	1	1	0	1	2	1.5	14.5
[86]	3	0.5	1.5	3	0	1	0	1	2	1.5	13.5
[77]	3	0.5	1.5	3	0	1	0	1	2	1.5	13.5
[83]	3	0.5	1.5	3	0	1	0	1	2	1.5	13.5
[107]	2	1	1.5	3	0	0	0	1	2	1.5	12
[104]	1	0.5	1.5	1	0	0	0	1	2	1.5	8.5

.

The MERSQI score of these quantitative studies had a mean of 11.3, with a range from 7.0 to 15.5. It was similar to Cook and Reed’s study [53], which had a mean score of 11.3, and a range from 8.9 to 15.1. On the other hand, it is slightly higher than the scores in Jensen’s [54] study which has a mean score of 10.9, and a range from 6.0 to 14.5. The mean score of each domain was: (1) Study design: 2.0; (2) Sampling: 2.0; (3) Type of data: 2.9; (4) Validity evidence for evaluation instrument scores: 0.9; (5) Data analysis: 2.9; (6) Outcome: 1.3. Outcome was one of the domains with a lower score value, an important reason was that all of the included studies did not relate to the recovery of patients, which was one of the filtered conditions mentioned before. The highest score of this domain was 2.9 which would be the data analysis while the validity of the evaluation instrument had the lowest mean score of 0.9. This situation was not better than the conclusion of Jensen’s [54] study in 2017, and the effectiveness of evaluation tools was still the Achilles’ heel in the whole research.

2.3. Data Analyses

In this section, we will analyze the research design, methods, process, and outcome of the 69 studies in order to discuss the main points which can improve the design and evaluation of an immersive virtual reality learning environment.

2.3.1. General Overview

There was a wide variety of research fields among these analyzed studies. We tried to group them under specific topics as presented in Figure 2. It can be perceived that more than one-third of the studies targeted Natural Science (e.g., Geography, Biology, and Chemistry). Although more and more studies combined Archaeology or Cultural heritage with Virtual Reality technology in recent years [56,57,58,59,60,61,62], there were not a lot of cases for the purpose of education, as we can see that archeology has a lower value compared to others which is 5 and there were only 2 studies that were about cultural heritage out of these surveyed researches.

As Georgiou [63] stated, the intersection of evaluation of an immersive virtual reality learning environment, enactment, pedagogy, and technology integration was lacking in the amount of research carried out. We found that, although the main purpose of these 69 studies was to serve education, there were only a few cases that used pedagogical strategies to guide their design (n = 13), accounting for about 19% as shown in Figure 3.

2.3.2. Research Design

First, we analyzed the sample sizes of these studies. There were 4 studies that did not report the specific sample size, so they were not included in the calculation of the mean. Additionally, there were 7 studies that collected data from more than one sample: Boettcher [64] (n = 3), Bruno [65] (n = 4), Chan [66] (n = 2), Lugrin [67] (n = 2), Makransky, et al. [68] (n = 2), Puig, et al. [69] (n = 2). These studies were treated individually, so made up a total of 81 individual samples. The mean of these sample sizes was 56, with its range being from 2 to 262. As compared to the traditional teaching method, the cost of HMD is higher, so the sample size of the target audience is usually not very large. The most common sample sizes of these studies start from 10 to 30 (n = 27) and from 30 to 100 (n = 30), more details are shown in Figure 4.

The grouping designs of these studies are presented in Figure 5. Most of the studies prefer either setting up a single group with cross-sectional or only posttest (n = 33) or randomized controlled trial groups (n = 29). These two kinds of design have accounted for nearly 90% of all the grouping designs.

About one-sixth of the studies designed both pretest and posttest (n = 11), and only 6 studies set up a delayed posttest in order to study the connection between knowledge retention and Virtual Reality technology [70,71,72,73,74,75,76].

2.3.3. Methods and Statistics

After analysis and summary, the research methods utilized in these studies were mainly divided into four categories: questionnaire, data recording, interview, and observation. Figure 6 shows 51 studies that used a questionnaire for measurement, which was noted as the most common solution to measure the key factors. Twenty-one studies used more than one type of method for evaluation, among which the mixed methods of questionnaire and data recording was the most common type (n = 12). Furthermore, 3 studies used three research methods including a questionnaire, data recording, and interview [23,68,77]. Recorded data were usually used to evaluate knowledge acquisition in these studies (n = 18). Interviews with users were used in 12 of the studies to obtain opinions and suggestions on the use of an immersive virtual reality learning environment. Finally, only 2 studies used observation as one of the research methods, which is mainly used to analyze the behavior of participants to reflect on the experiment [50,78].

As the most common research method, the questionnaire is able to objectively understand users’ attitudes and also evaluate their learning performance. There were 21 studies that had identified the questionnaires that were used and, among them, the most used questionnaire was the System Usability Scale (SUS) (n = 9), followed by the Perceived Enjoyment Scale (PES) (n = 2) [68,76] and the Simulator Sickness Questionnaire (SSQ) (n = 2) [76,79]. Other studies mainly used custom questionnaires or did not mention what kind of scale they had used. More details are presented in Figure 7.

Through the analysis of these studies, most of the statistical methods adopt t-tests, Pearson’s correlation coefficient, Mann–Whitney U test, ANOVA, MANOVA, ANCOVA, and other quantitative data analysis methods, as shown in Figure 8. Among these studies, the t-test was the most commonly used statistical method (n = 24). Meyer [76] used not only the t-test but also ANOVA and ANCOVA to analyze his research results. Furthermore, ANOVA was also used in 5 other studies. Pearson correlation coefficient was used in 4 studies [80,81,82,83]. Mann–Whitney U test was used in 2 studies [84,85] and Miller [85] also used the t-test and ANOVA in his research. MANOVA was used in Vogt’s [83] and Taub’s [86] studies, and the Wilcoxon test was used in Gomez-Tone’s [31] study. Moreover, there were 16 studies that only used mean and standard deviation as the method to analyze the results, and 11 studies used other descriptive statistics such as percentages.

2.3.4. Environment and Process

Jensen [54] points out that laboratory-style experiments should be replaced in future research and educational virtual reality in an authentic environment should be evaluated and used instead as part of a real learning or training process. Therefore, we analyzed the experimental environment of these studies, and the result is shown in Figure 9. There were only 7 studies that set the experiments in the real instructional process [31,50,62,64,72,85,87] which can be interpreted, as only about 8% of the experimental environments are of an authentic setting if we put it into a percentile.

During the experiment, time limitations and familiarity with the Head-Mounted Display (HMD) deserve our utmost attention. Consequently, we analyzed the time limitation of these experiments and whether to provide support for participants before the experiment to let them familiarize themselves with the use of the HMD, as presented in Figure 10.

There were only 13 of these studies that did not limit the time of learning in the virtual reality environment. Whereas other studies that limit the time indicated that some users wanted to extend the time to continue the operation and time limitation had, in fact, brought a sense of pressure [63,88,89]. Furthermore, there were 24 studies that had set up the link to get familiar with the HMD. Some studies had explained or even demonstrated the use of the HMD, meanwhile some studies set aside time for participants to practice using the HMD. In the research without this link, some researchers from the user experience had also given feedback that the users from this research tend to spend more time because they were not familiar with the HMD and they required assistance from the others which greatly affected the learning effect. Therefore, they suggested including the links to get familiar with the HMD [51,77,88,90,91]. More details are presented in

Table 1. Feedback on Evaluation Procedure of Immersive Virtual Reality Learning Environment.

Reference	Statements
[92]	‘The main difficulty concerned the “time to get used to the technology”.’
[32]	‘Most users are not sufficiently familiar with the interfaces to benefit from the full potential for learning and training.’
[88]	Participants appeared to experience slightly lower levels of perceived competence, probably due to the inexperience with VR headset usage.
[90]	‘We propose allowing time to explore the environment prior to completing the specified testing tasks.’
[73]	‘A cable which is necessary in case of HTC Vive is somewhat problematic and can cause negative feelings.’
[68]	‘IVR adds complexity, which can distract the learner.’
[76]	‘IVR may lead to higher levels of cognitive load and thus a reduction in learning in settings where there is not enough pre-training.’
[91]	‘The absence of user experience in using VR applications affected the SUS score.’
[69]	‘Familiar with VR devices represents a time-consuming task and a cognitive overload. ’
[51]	‘Not all participants were familiarized with a motion controller. As a result, some of them required assistance to learn how to use the controller and conclude the tests.’
[77]	‘Feedback from the open-ended questionnaires indicated that some students were unfamiliar with the use of VR.’

.

3. Results

We analyzed the research outcome of these studies from two different perspectives. On the one hand, the feedback on attitude given by the participants through using these immersive virtual reality learning environments was mainly summarized into six aspects: satisfaction, immersion, controllability, usability, enjoyment, and discomfort. The exact numbers of studies for each of the items are shown in Figure 11. There were 52 studies involving at least one of the above items. Three of the studies reported four of these items [93,94,95], and 1 study reported five of these items [88]. Most of the studies gave positive results, but there were also some with negative feedback, especially for both “controllability” (n = 5) and “discomfort” (n = 9). As mentioned before, some of the participants had given negative feedback for controllability due to their lack of experience in handling the HMD. Besides this, physical differences of individuals also caused a small part of participants to report their discomfort in using the HMD.

On the other hand, the evaluation of learning performance will be: including the achievement of knowledge or skill, motivation, concentration, memory, and self-efficacy. The exact number of studies for each item are shown in Figure 12.

Firstly, we found out that there was both positive and negative feedback from the learning performance as 3 studies had reported worse or similar results compared with the controlled groups [70,80,88]. The feedback on concentration in some of the studies caught our attention. As we delved deeper into it, we realized that the negative aspects of the high immersion brought by HMD and the enjoyment brought by Virtual Reality technology may influence users’ concentration on the learning content. Some researchers pointed out that the unsatisfactory learning performance may have been related to these phenomena. More details from the references of these studies are shown in

Table 2. Feedback on Concentration of Using Immersive Virtual Reality Learning Environment.

Reference	Conclusion
[96]	The perception of disruptive outside sound while being inside a virtual reality simulation can break the experience of presence.
[97]	Given the total immersion in the virtual environment, the student focuses his attention completely on the activity, thus generating meaningful learning.
[90]	VR could be too distracting for some learners. At times participants and assistants were talking during the testing, breaking the immersion of the VR Classroom.
[54]	HMDs have no added value compared with cheaper and less immersive formats, because the immersive experience actually distracts from the learning task.
[81]	Participants in VR group, overwhelmed by the technology, paid less attention to the audio information, and this might explain lower increase in learning performance.
[98]	Fun or pleasurable experiences within immersive VR environments can lead users to disregard their instrumental value, and instead, concentrate on the entertainment value such systems offer.

.

4. Discussion

From the perspective of design and development, our previous analysis had shown the imbalance in the development of virtual reality that will assist educators in different research fields. As Oyelere [25] concluded in 2020, technology-mediated learning will be strengthened when society is more aware and has a better grasp of the application trend of technology in different fields of learning. Our analysis results indicated that the application of virtual reality in both Natural Science and Health Science had received the most attention. On the other hand, cultural and historical education such as Cultural Heritage and Archaeology had acquired the least amount of interest for the educators to apply Virtual Reality technology in their teachings. This bestowed upon us an opportunity to delve deeper into the application of Virtual Reality technology for educational purposes. If we are able to pay more attention to these fields, it will certainly be a large contribution to talent training.

Through our previous analysis, we conclude that although these analyzed studies were mainly to assist the instructional activities in specific fields, most of the instructional content design and development did not follow the pedagogical theory. In terms of pedagogy, instructional content design is one of the most crucial aspects of the transactions that go with the learning and teaching process, and the success of an immersive virtual reality learning environment largely depends on the instructional content design. An immersive virtual reality learning environment may be able to satisfy the needs of learners given that instructional tasks and activities are designed with an unfitting educational approach [30,98,99]. However, from previous studies, it showed that a scarcity of studies had a clear theoretical pedagogical model as a guideline for the instructional content’s design of virtual reality educational application [50,63,77,88,94,100,101,102,103]. Learning technologists and experts seem to show a lack of skills in regard to the ways to use virtual reality and to come up with learning solutions. In addition, learning performance may also be affected as there are hardly any studies on the potential and difficulties of Virtual Reality technology supported instructional design strategies. Hence, providing theoretical underpinnings that are based on pedagogical theories is necessary for those researchers who are interested in building educational virtual reality applications.

Kebritchi and Hirumi [104] developed the organizational pedagogical categories which can be used to assist the analyzing of the content from a variety of open sources and distinguish educational pedagogies of virtual reality applications. Based on this theoretical model, the pedagogical theories applied in the instructional content of virtual reality applications can be categorized as experiential learning, constructivism, discovery learning, situated cognition, unclassified approaches, or direct instruction [26]. Researchers can guide the design of instructional content through the application of these pedagogical theories which have been proven to be effective. Furthermore, in comparison to other pedagogical theories on the instructional content of virtual reality applications, constructivism acts as a close associate to experiential and discovery learning [101]. More importantly, based on discovery and experiential learning, constructivism further includes the final step with the construction of personal meaning by the learner [26]. At the same time, a growing body of literature [14,16,100,105,106,107] concludes that Virtual Reality technology could be utilized by researchers and instructional designers with the application of constructivist principles in order to engage with students mentally and improve their interest and learning performance.

Constructivist learning is heavily regarded in new environments. The new environment is shown to challenge the user’s current knowledge framework and allows them to experiment and explore the discrepancy within a period of time. An immersive virtual reality learning environment lays out ample teaching opportunities that can be used to aid the improvement of the learner’s problem-analyzing skills as well as allowing them to explore new hurdles. A well-designed immersive virtual reality may be utilized to help students to depend more on their biologically innate ability to understand perceptual phenomena and physical space with the assistance of multi-sensory learning environments [27]. Mikropoulos and Natsis [102] stated that even if immersion is shown to be a consequence of several perceptual channels being involved, which includes haptic, olfactory interactions, auditory and visual, with the visual representations predominate. This experiential feature and interaction of virtual reality provides valuable help for the traditional learning model [16]. Furthermore, the strength of this advantage can be explained by active learning from experience [108]. When users engulf themselves in an immersive virtual reality learning domain, they will need to reconstruct their cognition of the virtual world. Previous research by Keil et al. [109] found that distance estimations with Virtual Reality Locomotion Techniques were initially underestimated which is challenging in making metric distance estimations; however, distance estimation training can significantly enhance metric estimations. Findings by Lokka et al. [110] offer fresh perspectives on the creation of virtual environments and their potential application as tools for training visuospatial cognitive abilities for route learning, particularly among the elderly. Moreover, an immersive virtual reality learning environment can provide an opportunity for the learner to experience the world ignoring space and time, which would be out of reach otherwise due to several real-life restrictions in a classroom. The experience may motivate learners to explore without boundaries and achieve learning outcomes that are consistent with constructivist pedagogies. Fowler [101] pointed out that, even though some studies have adopted constructivism-based approaches, there was no clear pedagogical model that can act as the guideline for designing an immersive virtual reality learning environment. Therefore, it is urgent to carry out research regarding the compatibility of instructional content design theories as guidelines for facilitating immersive virtual reality learning environments and ways to utilize the model of constructivist learning principles as a new paradigm of design and development in the future.

In addition, in terms of implementation and evaluation, as mentioned in the previous analysis, not every outcome received positive feedback. In addition to using pedagogical theory to guide the design and development, several controversial issues in the implementation process also caught our attention. Firstly, it is about the balance between the immersion and concentration of learning in a virtual reality environment. Through the previous analysis, we discovered that Virtual Reality technology is indeed able to provide the participants with a satisfactory immersion experience. However, the learners may be excessively attracted by this experience and ignore the instructional content. In addition, interference from other participants or environmental sound may also distract the learner’s attention. Hence, during the implementation process, we should be guiding the participants to interact with the immersive virtual reality learning environment through professional instructional content design and at the same time try to create an undisturbed experimental environment.

Another problem that arises during the process of implementation is that most of the participants were not familiar with the operation of HMD. From the participants’ feedback, we found out that the evaluation result of learning performance was correlated to their ability to operate the HMD skillfully. This finding is in line with previous studies which indicated the challenges to widespread Virtual Reality Tools in education [14,30] and Aydin and Aktas’ [62] research mentioned that, due to prior knowledge, the virtual reality design experience could vary. In this case, the results of our previous analysis showed that this problem can be solved in two ways. First is to carry out effective pre-training to learn the operation of HMD and get familiar with the experimental environment under the guidance of professionals, and, secondly, by relaxing the restrictions on learning time we manage to reduce the time pressure for participants to learn in virtual reality environment, thus permitting the participants to construct knowledge freely in the immersive virtual reality learning environment according to their own rhythm. In the future, we can pre evaluate the participants’ familiarity with HMD and combine it with the aforementioned methods to help them overcome the difficulties in operation. This would aid the participants to learn more efficiently through the virtual reality environment.

The lack of evaluation data in the authentic instructional process is also a problem reflected in the previous analysis. In fact, most of the studies were based on the laboratory-style experiments, rather than applying the immersive virtual reality learning environment in real teaching activities. In addition to the setting of the implementation environment, unlike how the experiments are generally executed in a relatively brief period in a laboratory environment, educational programs usually last for a longer time. Therefore, even if the evaluation results are generally positive, it may be difficult to conclude whether it can be maintained for a long time [111,112]. However, among the analyzed studies, there were only a few evaluated “memories”. Most studies lacked long-term observation and evaluation of the transfer of knowledge or skills that the learners acquired from the immersive virtual reality learning environment. As Nersesian [89] pointed out in 2020, we should be paying more attention to the learner’s recall of the knowledge or skills over a specific period in an authentic instructional process in our future studies.

Valid evaluation is very crucial to accurately analyze the impact of an immersive virtual reality learning environment. Through our previous analysis, we found out that the questionnaire was the most commonly used research method in these studies. However, only about one-third of the studies have given clearly defined questionnaires and the reasoning for the validity of the learning performance evaluation instrument are also deemed to be lackluster in many studies. In addition, there are also some arguments regarding the statistical methods used to analyze the results. For example, the commonly used technique, ANCOVA requires each participant to be assigned to either the intervention group or control group randomly [113], which is very demanding for most of the school-based immersive virtual reality learning environment studies [112]. In short, ensuring the validity of evaluation instruments and adopting an appropriate scientific statistical analysis method will definitely help to complete the immersive virtual reality learning environment studies.

5. Conclusions

Our analysis has shown the imbalance in the development of virtual reality that will assist educators in different research fields. As Oyelere [25] concluded in 2020, technology-mediated learning will be strengthened when society is more aware and has a better grasp of the application trend of technology in different fields of learning. Our analysis results indicated that the application of virtual reality in both Natural Science and Health Science has received the most attention. Virtual Reality technology is more mature in these areas of education and can be combined with other hardware devices, such as sensors, so that better training results can be achieved. On the other hand, cultural and historical education such as Cultural Heritage and Archaeology has acquired the least amount of interest for the educators to apply Virtual Reality technology in their teachings. This may be related to the minor opportunity for researchers and educators in these fields to apply Virtual Reality technology and the lack of knowledge about the related technologies. However, this does not mean that Virtual Reality technology cannot provide suited technical support for these fields. Conversely, Virtual Reality technology can help learners enjoy an intense, immersive, and interactive experience in the content with a high historical and cultural value that needs to be protected. In some archaeological environments, due to the narrow space and the need to reduce the risk of the destruction of cultural relics, the only way to meet the learning needs of students is through looking at pictures and videos. Virtual Reality technology can be used to carry out equal proportion restoration and reconstruction of the scene, so as to create an immersive learning environment and help learners achieve better spatial cognition. In the future, we can combine Virtual Reality technology, especially spatial cognition, with the learning content in these fields, which will make great contributions to talent cultivation.