Analyzing the Use of Heuristics in a Virtual Reality Learning Context: A Literature Review

Abstract

With concepts such as immersion and presence, known as hedonistic qualities of importance in the virtual reality (VR) experience, the question arises whether the more pragmatic heuristics are effective in the evaluation of an artifact. However, despite the importance of the heuristics for artifact evaluations, the available studies do not provide a rigorous review of these heuristics and their efficiency. Thus, this review aims to look at how heuristics have been applied in various virtual learning environments (VLEs) that involve virtual reality learning activities, either for use in heuristic evaluations or as design principles. In addition, it examines how these heuristics support the evaluation of a more hedonistic quality, such as presence, and lastly, the aim is to gauge the estimated efficiency of heuristics as an evaluation method. This article is a systematic review of research investigating using heuristics in a virtual reality learning context. The review includes articles published from January 2017 to February 2022, and from the screened records, twelve articles were analyzed in full-text form. This review shows the versatility of heuristics and their applications as well as the key concepts that are vital to the user’s experience in a virtual reality learning context. This review indicates that heuristic evaluation is a valuable tool, as it provides a clear summary of what needs to be handled in the next iteration of the application.

1. Introduction

Since mid-1960, several definitions have been formulated for virtual reality (VR), for example, ref. [1,2] described VR as “real-time interactive graphics with 3D models, combined with a display technology that gives the user the immersion in the model world and direct manipulation”. Refs. [3,4,5] defined VR as “the illusion of participation in a synthetic environment rather than external observation of such an environment. VR relies on 3D, stereoscopic head-tracker displays, hand/body tracking, and binaural sound. VR is an immersive, multi-sensory experience”. According to these definitions, “virtual reality” implies experiencing the sense of a concrete existence without existing in the physical world, which is exactly what an effective virtual reality system provides. VR is also defined in terms of the technical hardware that “makes the dimensions of experience affording different levels of the vividness of interactivity in an immersive or para-reality environment” [3]. As we can see, these definitions, even though different, underline three common aspects of VR systems: “immersion, perception to be present in an environment, and interaction with that environment” [5].

The ability of VR to simulate something difficult to present directly in the real world makes VR widely applied in various sectors. VR has bloomed once more in the last decade, and there has been a surge of new, more affordable devices capable of bringing the technology into the education sector [6,7,8,9]. One of the main advantages that VR provides is the possibility to create a realistic experience in certain learning approaches, such as performing surgery or learning how to fly an airplane [10]. Furthermore, VR provides the opportunity to interact and visualize abstract concepts, such as the search algorithms posed in the article by [11] where the students had to learn complex concepts in a short amount of time. Such an environment is called a virtual environment (VE) [12]. A virtual learning environment (VLE) is a web-based communications system and tools that enable learners to utilize various learning tools, such as course materials, teacher support, discussion boards, file-sharing systems, and educational resources, at any time and from any location [5]. In addition, a VLE supports collaboration between a learner and an educator as well as among the learners themselves [12]. VLEs have become the new realm for learning and education, and most authors use the term “learning management system” (LMS) as a synonym for “virtual learning environment” [12].

VR systems have three common features: “immersion, perception to be present in an environment, and interaction with that environment” [5]. Immersion refers to “the number of senses stimulated, interactions, and the reality’s similarity of the stimuli used to simulate environments” [5]. The VR experience of the user could be revealed by measuring the presence, realism, and levels of reality. In VR, presence refers to the complex psychological feeling of “being there”, which comprises the sensation and impression of physical presence as well as the ability to interact and react as if the user were in the actual world [5]. Similarly, the level of realism corresponds to the level of expectation that the user has about stimuli and experience [5]. Other concepts that are vital for the user’s VR experience are the embodiment, empathy, and flow. Embodiment is a core concept in VR that often refers to “the sensation that our self is located inside a virtual body; we control this body and that this body belongs to us Hence, embodiment consists of three subcomponents, the self of presence (or self-location in the original paper), the sense of agency, and the sense of body ownership” [13]. Experienced embodiment aids the user in feeling as if they are a part of the virtual environment (VE) and feeling connected to the other agents in the world [3,14,15,16]. Being in the same space as another character makes the user strongly feel the character’s emotion in a situation. Users may view a virtual reality experience as more realistic and compassionate as a result of simulated empathy in VR [6]. The state in which the user is engaged in the task at hand also affects the experience; flow can be an experience of immersion into a certain user action. Users may experience flow when the task at hand is engaging and challenges the user to utilize their skills fully [14].

With the power and possibilities that VR can provide to VLEs, it is of the utmost importance that the nature of the system is adequate for its users and ultimately fits the curriculum. Therefore, it is very important to evaluate a system’s usability. Usability is a broad term that refers to the activities, principles, and procedures that support and promote the design of human–computer interfaces that take into account the needs of users [17]. However, usability problems are among the most common obstacles to the acceptance of artifacts, and they can lead to weariness and confusion of the user and, as a result, the withdrawal or rejection of artifacts. Therefore, the evaluation of usability seems to be necessary to make user interaction more effective [18]. There are several ways to evaluate usability [17,19,20], and they can be divided into expert-based and user-based methods [18]. Heuristic evaluations and cognitive walkthroughs are two well-known expert-based ways of detecting multiple issues with limited resources. A heuristic evaluation is guided by heuristic principles to identify user interface designs that violate principles that are commonsense rules or simplified principles [21] known as usability heuristics [17,21,22]. A cognitive walkthrough assesses the difficulty of performing tasks with the help of a system to identify the actions and objectives necessary to perform each task. Although there are many methods of usability evaluation [18], this review focuses on the heuristic evaluation method [17]. We have chosen this approach primarily because it is usually more affordable than conducting rigorous user tests, which can be too application-specific, and due to a rising trend in studies in this area, as shown, for example, in [17,19].

Heuristic evaluation is the most widely used form of usability inspection [23]. Because of its speed, low cost, and simplicity, it became common in the early 1990s. It was possible to uncover a high proportion of usability issues with only 4–5 evaluators and a restricted number of principles. Jakob Nielsen and Rolf Molich defined heuristic evaluation as a method for finding usability problems in a user interface design by having a small set of evaluators examine the interface and judge its compliance with recognized usability principles (the “heuristics”) [23]. To implement heuristic evaluations, evaluators generally evaluate sets of crucial tasks, where each task indicates a possible set of user interactions with a system or product. Throughout the evaluation, the evaluator matches the task stages to a set of already-defined usability principles called heuristics. In this way, professional evaluators utilize heuristic evaluation, in user experience (UX) design, to systematically determine a design or product’s usability, and the experts can uncover issues that design teams missed. Nielsen’s heuristics [23] have been the norm when conducting heuristic evaluations, though there are other ways to apply heuristics and reap a result, as was conducted by [24] where they designed the application based on the heuristics. However, in ref. [23], Nielsen’s heuristics were traditionally meant for examining graphical user interface (GUI) elements, and while plenty of user interface (UI) elements can exist in a VE, the whole experience is often more than examining UIs. Even though the heuristics of [23] have been reinterpreted for VR by [25], the question still exists whether they adequately support evaluating hedonic qualities such as embodiment, empathy, flow, immersion, and presence—concepts that are considered crucial for the virtual reality learning experience.

This study aims to look at how heuristics have been applied in various virtual learning environments VLEs that incorporate virtual reality learning activities, whether it is for using them in heuristic evaluations or as design principles. Additionally, the aim is to present the key user experience concepts in VLE as they have been discussed in the literature and, lastly, to investigate the efficiency of said heuristics as an evaluation method. The research question is the following:

RQ: How efficient is the use of heuristics in virtual learning environments (VLEs) that incorporate virtual reality learning activities?

This paper is organized in the following manner: In the Methods Section (Section 2), there is a detailed overview of the various stages of the study, and in the Results Section (Section 3), there are three central themes that arose from the thematic analysis. In the Discussion Section (Section 4), the results are compared with the existing literature and possible implications for future research. Section 5 presents the strengths and limitations of this review. Lastly, in the Conclusions Section (Section 6), there is a summary of the results that answers the research question.

2. Methods

This study set out to answer the research question. Therefore, we used this research question to determine the content and structure of the review, design strategies, locate and select primary studies, critically evaluate studies, and analyze their results. A systematic literature review was conducted following the PRISMA guideline [26]. PRISMA was used to identify, select, and critically evaluate research, reducing bias to improve the quality of reporting and make the review process more effective. The systematic literature reviews provide an opportunity to see and even evaluate the efficacy of using heuristics as an evaluation method as presented by researchers. Therefore, it can help in identifying knowledge gaps in this field. Furthermore, it helps researchers gain insight into how researchers apply heuristics and helps to build knowledge in the virtual learning field about important concepts, research methods, and experimental techniques that are used in this field.

2.1. Data Collection and Search Terms

The literature was acquired through three databases: ScienceDirect, SCOPUS, and Web of Science. These databases were chosen due to their relevance to the scope of the study and the research question. Additionally, the databases were chosen, as they were a good complement to each other, given their multidisciplinary nature. Google Scholar was readily available for conducting a backward search by scanning the references found in the articles and for conducting a forward search by examining newer articles that had been cited in the final articles. The search terms and their combinations were “heuristic*” AND (“virtual reality” OR “immersive computing”) AND (“learning” OR “Simulation-based training” OR “simulation training” OR “Learning environment” OR “Educational environment” OR “VLE” OR “Online learning” OR “immersive learning”) OR “presence”. The respective searches were exported as RIS files and later compiled in Zotero. Figure 1 shows the number of records identified, screened, and included at each stage of the selection process.

2.2. Inclusion/Exclusion Criteria

The inclusion criteria were the following: the article had to be either a journal article or a conference paper published in the last six years, meaning 2017 to February 2022, and it needed to be written in English. The articles needed to focus on VR and specifically focus on a learning context. The mention of heuristics and presence in a combination with either VR or a learning context became an inclusion criterion as well.

The exclusion criteria were the following: books, book chapters, and other records that were not published in peer-reviewed journals and conferences without proceedings were excluded from the search. Any work-in-progress articles or editorial articles were excluded. Articles that focused on VR but neglected the mention of VR in favor of, e.g., AR, were excluded. Articles that focused on comparing the hardware specifics of head-mounted displays (HMDs) were excluded as well.

2.3. Data Extraction

Figure 1 shows the filtering process through its various stages. The final twelve records were analyzed based on information concerning VR, heuristics, learning, and user experience. The records were screened in their entirety, though the focus was on the method, result, discussion, and conclusion sections of each paper. To determine eligibility and extract answers to the research question, the two authors independently reviewed each of the relevant articles. Any differences between the authors were resolved by discussion and agreement.

2.4. Analyzing the Articles

A thematic analysis was conducted, and the coding process happened in two iterations before they were categorized in a theme (see Appendix A). First, as the chosen records were reviewed, paragraphs and quotes of importance were highlighted. In the initial phase of the coding process, the highlighted parts were coded based on the words used in the text (i.e., immersion, engagement, user’s perception, etcetera). In the second coding phase, the highlighted parts received a single code (i.e., immersion), and then finally, in the third stage, the part was placed in a theme, such as key user experience concepts in VR. The coding was performed manually due to the limited number of articles, and while the process narrowed down the number of keywords significantly between coding phases one and two, all codes were kept to have a good overview of the highlighted parts and to synthesize the data more easily.

Any information related to how heuristics had been applied in a VR and learning context was coded alongside mentions of qualities valuable to the user experience. Information related to the results and conclusions were also noted. In the end, three themes were established that were believed to answer the research question: the first one gives an overview of the various ways heuristics can be applied, the second gives an overview of key user experience concepts important in a VR learning context, and the third gives an overview of the efficiency of heuristics.

3. Results

To answer the research question, the findings from the 12 reviewed articles have been grouped and aggregated under three key themes, as presented in the following subsections.

3.1. Overview of Various Applications of Heuristics

The literature shows the versatility of heuristics and that they can be applied differently depending on one’s goals (see Table 1). Heuristics were primarily used for the evaluation of a VR environment [3,14,17,27] or as design guidelines [24]. For example, ref. [24] created and used VLE to assess the awareness and the preference for applying virtual reality as a method of learning in Sri Lanka. Historic scenery was developed, where the learner was given the freedom to navigate through the scenery and learn from a virtual tutor. When designing this VLE, the purpose was to offer the user an immersive and satisfying experience where the user could navigate the landscape effortlessly [24]. To provide a better UX, the system was designed by employing Nielsen’s usability heuristics [23] and incorporating them for VR. When it comes to VR, the user interface serves as the whole VLE (visual) for the user. As a result, the heuristic principles of Nielsen were applied to the VR user interface as follows: anything that is too close to the viewer will make him/her cross-eyed and anything which is further may tend to blur the element. Therefore, interactive components and specific guidance information were set within a comfortable range of sight to offer a better user experience for the student. In addition, when positioning such interactive elements, the human eye’s comfort zone was considered [24]. The majority of the ‘call to action’ items were in the eye comfort zone, while any other essential interactive elements were positioned in the neck comfort zone [24]. However, based on Nielsen’s heuristics [23], the interface should offer identified functions where the user may perform his/her activities without causing any errors [24]. This means ‘User Control and Freedom & Recognition than recall’. The user should be able to undo or redo their actions if they make a mistake. However, though the authors of [24] explained that they had used Nielsen’s heuristics [23] and integrated them for VR, they did not go into depth regarding how the heuristics had been implemented in greater detail.

In two of the reviewed articles [17,28], the focus was to develop new heuristics, as both argued that there was a need for more domain-specific heuristics. While common Nielsen’s 10 usability heuristics [23] are used as the de facto standard in the process of heuristic evaluation, ref. [17] argued that there is a need for developing custom domain-specific heuristics. According to ref. [17], this is because Nielsen’s heuristics were not appropriately compliant with the certain features of VLEs systems. Therefore, ref [17] developed a set of usability heuristics, VLE heuristics, that is more applicable to the field of virtual learning environments [17]. In addition, the authors of [17] utilized the PROMETHEUS, which is a procedural methodology, to develop this set of heuristics for evaluating virtual reality learning environments. However, the PROMETHEUS methodology requires a foundation of established heuristics. Therefore, the authors of [17] performed a heuristics evaluation by using both VLEs and Nielsen’s heuristics [23] to validate the results. According to ref. [17], the VLEs perform better than Nielsen’s heuristics, discovering more problems, which are also more related to the field. In addition, compared to evaluators who used Nielsen’s heuristics, VLE heuristics evaluators indicated increased satisfaction with the utility, clarity, ease of use, and need for extra components [17]. Other sets of heuristics, such as Benson et al.’s heuristics [29] and Mtebe and Kissaka’s heuristics [30], were used as a comparison to the newly developed set of heuristics by [3] to highlight the fact that the various sets carry a high level of resemblance. For example, [17,29,30] all have heuristics that concern the visibility of system status, the match between the system and the real world, consistency and standards, error prevention, and help and documentation.

As mentioned, the authors of [17] were not the only ones to develop a new set of heuristics to evaluate usability. In the study by [28], the authors first evaluated students’ experiences with virtual reality learning, and the students expressed an overall positive response to the technology, both regarding the hedonistic and pragmatic effects. Based on the results of the evaluation, literature, and previous research, the authors developed a framework specifically meant for teachers as to how to create student-centered lesson plans that incorporate VR in the classroom. This framework contained eight heuristics: focus, provocation, simulation, collaboration, control, digital life, skills, and multimodal experience [28]. Similarly, the authors of [31] created their own set of heuristics that focused on evaluating the technological and pedagogical knowledge that the teacher implements plus the content knowledge of the material being delivered through the VR experience. To evaluate the usability of the VE, two expert review boards aided in assessing the VE. The first board was computer science experts, and the second was health science experts. Each board assessed the environment and the situation using heuristics evaluation and cognitive walkthroughs. The suggestions made during each of the expert reviews were applied to improve the VE, thus enabling learners to feel an accurate virtual situation that could positively affect their learning capability [31]. However, in essence, the heuristics were created to help teachers decide whether VR technology is fit to use as a tool to reach the learning goals.

In a more classical approach, given heuristics and usability evaluations, the authors of [14,18,32] all performed heuristic evaluations to discover potential usability issues with their artifacts. For example, the authors of [32] conducted a cognitive walkthrough of the educational VR environment where the students practiced a scenario where they had to treat a patient with a foreign object stuck in their airway. In addition, heuristics were used to decide if VR fit with the learning objectives. However, the methodology and the set of heuristics used were not identified. Ref. [14] is another study that used heuristics as an evaluation method, where the authors explored VR teaching and the learning of abstract concepts in software engineering using an application called OO Game VR, and the usability of the application was evaluated through a heuristic evaluation using Sutcliffe and Gault’s VR heuristics [20]. In addition, the authors of [14] referred to Nielsen’s [23] heuristics in their study and argued that, while [23] “formed the basis on which custom VR heuristics were developed”, they still chose the VR-specific set of heuristics. Interestingly enough, the authors of [17] argued in a similar vein and stated that Nielsen’s [23] heuristics were not optimal for evaluating virtual reality learning environments. However, to circle back to the study of [14], the result of their heuristic evaluation found problems related to the following aspects: a lack of realism in both the avatar and the surrounding world, a lack of feedback when acting, a lack of learning support, and some functionality was perceived as limited. Still, the evaluation yielded a so-called backlog—a list of identified issues in the application—and by prioritizing the most acute usability issues, the authors now have a summary of what needs to be rectified for the next iteration.

In the study by [33], the authors examined a virtual reality learning environment as a complement to traditional advanced life support (ALS) training by determining specific user needs in the developed application ALS-Sim VR. The usability was evaluated through semi-structured interviews with the supplementation of a heuristic evaluation where they too used the following 12 of Sutcliffe and Gault’s VR heuristics [34]: “natural engagement, compatibility with the user’s tasks and domain, natural expression of action, close coordination of actions and representation, realistic feedback, faithful viewpoints, navigation, and orientation support, clear entry and exit points, consistent departures, support for learning, clear turn-taking, and sense of presence” [33]. The combined results from the interviews and the heuristic evaluation led to five areas of design considerations when developing VR applications of similar contexts: affordances, agency, diverse input modalities, mental models, and advanced roles [33].

In the study by [27], the authors developed Communica-Enf 3D, a serious game to help students learn how to employ verbal and nonverbal communication with a virtual patient and how to apply clinical reasoning and decision making. Furthermore, it was created to help students learn how to recognize conflict and handle it professionally. Ref. [27] described the heuristic evaluation of the Communica-Enf 3D game for the improvement of communication ability. The application was evaluated with the Heuristic Evaluation for Digital Educational Games (HEDEG) because of the opportunity to use HEDEG by non-expert evaluators. In addition, it used the common Nielsen heuristic principles for technology evaluation. The application of Nielsen’s heuristic [23] to evaluate the Comunica-Enf serious game ensures ”player motivation, pedagogical quality, and technical effectiveness of this digital educational technology” [27]. In addition, it discovered issues related to six heuristics: the interface, educational element, content, gameplay, and multimedia [27]. Lastly, in the study by [3], the authors examined the impact of VR in classrooms by deploying three various devices ranging from low-end to high-end. The modified heuristic [3,35] was used to structure the analysis of each device, though the reasoning for choosing that specific heuristic was not explained.

Table 1. Summary of how heuristics have been applied through the various studies.

Refs	Heuristics Mentioned	Application of Heuristics
[3]	- Modified heuristic [3,35] (the authors improved two heuristic evaluations, one developed by Sutcliff and Gault [34] and the other by Rusu et al. [36])	The modified heuristic was used to structure the analysis of each device, though the reasoning for choosing that specific heuristic was not explained.
[14]	- Nielsen’s heuristics [23] - Sutcliffe and Gault’s VR heuristics [34]	The authors performed a heuristic evaluation with Sutcliffe and Gault’s VR heuristics [34]; the evaluation yielded a backlog of what needed to be rectified in the next iteration.
[17]	- Nielsen’s heuristics [23] - The Benson et al.’s heuristics [29] - Mtebe and Kissaka’s heuristics [30] - The VLE heuristics of [17] that were developed through the PROMETHEUS method	The authors used the PROMETHEUS methodology to create a new set of domain-specific heuristics for VLE learning environments.
[24]	- Nielsen’s heuristics [23]	The authors used heuristics to design the VR application and its overall interaction.
[27]	- Heuristic Evaluation for Digital Educational Games (HEDEG), which is allegedly based on Nielsen’s heuristics [23]	The authors evaluated their system using the HEDEG method, which was based on Nielsen’s heuristics [23]
[28]	- Nielsen’s heuristics [23] - Shneiderman and Plaisant’s heuristics [37]	The authors created a framework for teachers planning a VR-inclusive curriculum and developed eight heuristics of their own.
[31]	- Heuristics developed by the authors (The authors developed their own set of heuristics that were utilized by experts to assess the VE. These heuristics were derived from Nielsen’s initial heuristics and supplemented with existing literature and practical experience.)	The authors used a set of heuristics to help decide if VR technology is fit to use to reach the learning objectives.
[32]	- Heuristics were developed by the authors based on Nielsen’s heuristics [8].	The authors performed a heuristic evaluation and a cognitive walkthrough by deriving their own set of heuristics but based them on [23] and later enriched them with existing literature and practical experience.
[33]	- Sutcliffe and Gault’s VR heuristics [34]	The authors used a mixed-method approach, using both interviews and a heuristic evaluation that yielded five areas of design considerations when developing VR applications of similar contexts.

Table 1. Summary of how heuristics have been applied through the various studies.

Refs	Heuristics Mentioned	Application of Heuristics
[3]	- Modified heuristic [3,35] (the authors improved two heuristic evaluations, one developed by Sutcliff and Gault [34] and the other by Rusu et al. [36])	The modified heuristic was used to structure the analysis of each device, though the reasoning for choosing that specific heuristic was not explained.
[14]	- Nielsen’s heuristics [23] - Sutcliffe and Gault’s VR heuristics [34]	The authors performed a heuristic evaluation with Sutcliffe and Gault’s VR heuristics [34]; the evaluation yielded a backlog of what needed to be rectified in the next iteration.
[17]	- Nielsen’s heuristics [23] - The Benson et al.’s heuristics [29] - Mtebe and Kissaka’s heuristics [30] - The VLE heuristics of [17] that were developed through the PROMETHEUS method	The authors used the PROMETHEUS methodology to create a new set of domain-specific heuristics for VLE learning environments.
[24]	- Nielsen’s heuristics [23]	The authors used heuristics to design the VR application and its overall interaction.
[27]	- Heuristic Evaluation for Digital Educational Games (HEDEG), which is allegedly based on Nielsen’s heuristics [23]	The authors evaluated their system using the HEDEG method, which was based on Nielsen’s heuristics [23]
[28]	- Nielsen’s heuristics [23] - Shneiderman and Plaisant’s heuristics [37]	The authors created a framework for teachers planning a VR-inclusive curriculum and developed eight heuristics of their own.
[31]	- Heuristics developed by the authors (The authors developed their own set of heuristics that were utilized by experts to assess the VE. These heuristics were derived from Nielsen’s initial heuristics and supplemented with existing literature and practical experience.)	The authors used a set of heuristics to help decide if VR technology is fit to use to reach the learning objectives.
[32]	- Heuristics were developed by the authors based on Nielsen’s heuristics [8].	The authors performed a heuristic evaluation and a cognitive walkthrough by deriving their own set of heuristics but based them on [23] and later enriched them with existing literature and practical experience.
[33]	- Sutcliffe and Gault’s VR heuristics [34]	The authors used a mixed-method approach, using both interviews and a heuristic evaluation that yielded five areas of design considerations when developing VR applications of similar contexts.

3.2. Overview of the Efficiency of Using Heuristics

The authors of [6,33] performed a heuristic evaluation using Sutcliffe and Gault’s VR heuristics [34], though in slightly different ways. In the study by [14], the researchers thoroughly prepared the evaluation by training the participants beforehand. Upon the completion of the evaluation, findings that were issues believed to negatively affect the experience were categorized into nine problems that were later ranked by severity on a 4-point scale [14]. Consequently, the ranked problems generated a priority list of what needed to be completed for the next iteration of the application. However, the level of efficiency of using heuristics to evaluate the VR applications in this study was related to the participant’s expertise in using Sutcliffe and Gault’s heuristics [18] and the selection and identification of problems [14]. Additionally, given that each evaluator only completed one training session, their experience can be viewed as a risk [14]. Furthermore, evaluator skills in the specific domain of the study, such as object-oriented programming, would have a better ability to evaluate the VR application in that domain [14].

Similarly in the study performed by [33], they recruited ten participants and had them answer a series of usability questions after completing the scenario in VR. Upon completion of the evaluation, the heuristic-based usability testing method revealed a higher score for the natural expression of action, natural engagement, and sense of presence, meaning that it should be addressed in the next iteration of the application [33]. Additionally, the application received lower scores for consistent departures, realistic feedback, close coordination of action, representation, and faithful viewpoints, meaning that the application performed well on those aspects [33].

In the study by [31], two expert panels used two different methods—the heuristic evaluation and the cognitive walkthrough—and used heuristics derived from [23]. The result yielded a backlog for the next iteration of the VR environment, and the conclusion that both panels could draw was that the VR technology was a suitable and affordable way to conduct learning if the issues found were corrected [31].

Ref. [6] used a modified heuristic approach to structure the analysis of the three different devices analyzed in the study, though in what way it was modified was not detailed explicitly. Similarly, in the study conducted by [14], the authors were equally ambiguous about how they used heuristics to shape the design of their application.

In the study by [26], the authors explained how they had used the Heuristic Evaluation for Digital Educational Games (HEDEG) method, which allegedly has its roots in Nielsen’s recommendations for technology assessments. The HEDEG method consists of 30 statements that evaluate the following heuristics: interface, educational elements, content, gameplay, and multimedia [27]. Similar to the heuristic evaluations carried out by [14,33], the HEDEG method gauges the severity of the findings on a 4-point scale. The result indicated that the heuristics were considered adequate, and neither of the findings rated on the higher end of the severity scale was above 25% [27].

Refs. [17,28,31] developed new sets of heuristics for different purposes. In the study by [17], they conducted a heuristic evaluation with two groups, one used Nielsen’s heuristics [21] and the other used the heuristics developed through the PROMETHEUS method, which were intended for virtual reality learning contexts. The sets of heuristics identified 93 problems using Nielsen’s heuristics [21] and 172 problems with the newly developed VR-specific heuristics. It should be said, though, that the aim of the study was two-fold in the sense that [17] set out to test the PROMETHEUS method and explore whether there existed a need to develop more fine-grained heuristics. Both [14,33] performed a heuristic evaluation to assess an application, meaning that their study goals differed from [17]. Ref. [28] developed a set of eight heuristics based on a questionnaire, broader observations of student VR users from the literature, and previous research. Since the heuristics were not tested as in the study by [17], the effectiveness of the new heuristics is still in question. The same can be said for the heuristics developed by ref. [32].

3.3. Overview of Key User Experience Concepts in VLE

The literature shows that the user experience (UX) has been a key focus in the various articles, regardless of whether the study aimed to evaluate a VR application, a design for one, or research new heuristics. Five key concepts vital for the user experience in a virtual reality learning context were identified throughout the literature: embodiment, empathy, flow, immersion, and presence (see

Table 2. Key identified concepts that are considered vital in a VLE context.

Concept	Definition	Refs
Embodiment	A core concept in VR that often refers to the experienced embodiment a user feels in a VE; ultimately generates a sense of presence in the virtual world. Experienced embodiment aids the user in feeling as if they are a part of the VE and feel connected to the other agents in the world.	[3,14,15,16]
Empathy	Being in the same space as another character makes the user strongly feel the character’s emotion in a situation. Users may view a virtual reality experience as more realistic and compassionate as a result of simulated empathy in VR.	[16]
Flow	The state in which the user is engaged in the task at hand; flow can be an experience of immersion into a certain user action. Users may experience flow when the task at hand is engaging and challenges the user to utilize their skills fully.	[16]
Immersion	An ambiguous term, often used synonymously with presence, though the literature states that it could be either the level of fidelity of the VE or the feeling the user has while immersed in the environment.	[3,16,31,32,33]
Presence	Generally refers to the user’s experience in the virtual world and how they act and react as if they are physically there.	[3,31,33]

).

Even though concepts such as embodiment, immersion, and presence are key components when developing a virtual reality learning experience, several articles such as [17] do not present any heuristics that focus specifically on those concepts. Indirectly, by not addressing concepts such as embodiment, presence, and immersion, the approach of [17] aligns with the argument in [30] that both immersion and presence should not be a feature of the technology. Furthermore, in other studies, such as [28], which developed a new set of domain-specific heuristics, the focus was on evaluating whether virtual experiences are appropriate to apply in the classroom. The heuristics were ultimately grounded in either hedonic or pragmatic effects. However, that was not the case with the heuristics developed by [16].

Immersion and the immersive experience are often discussed concerning presence. For example, in interviews with experts, scholars, and industry representatives conducted by Shin [16], the concept of presence was linked to immersion, and respondents stated presence as the degree to which two VR users feel like they are together. They also described it as being aware of someone else in a VR environment. In addition, respondents intertwined presence with immersion and used the words “absorption”, “concentration”, and “engrossment”. However, they all referred to immersion [16]. Empathy and embodied cognition are two terms that often arise in the articles that discuss VLE. Users can “realize and empathize when they comprehend another user’s subjective experience and environment” [16]. Stimulated empathy in virtual reality can make users understand a VE to be a more realistic and empathic experience [16]. VR can help viewers understand the thoughts and feelings of another individual. By being in the same place and close to a character in VLE, viewers may strongly feel another person’s feelings or circumstances. Engagement in VR can increase empathy [16]. By empathy, users may experience embodied cognition or a sensation of embodiment [16]. In addition, respondents claimed that “while using VR, an avatar-like virtual body is created inside the immersive virtual environment as an analog of their biological body” [16]. However, based on the result by [16], the immersion should be conceptualized, assessed, and evaluated throughout users’ interactions with technology instead of relying on technical features. The authors in [3] did what [16] states should not be done, meaning that they used immersion to describe the level of fidelity of the VE. The authors of ref. [16] argue in their article that “immersion exists in a dormant state and becomes concrete when the user experiences it; thus, instead of seeking immersion from technology, it should be sought within the user’s in-situ contexts: their cognition, interaction, and experience”.

In the article by [33], the authors conducted a heuristic evaluation where they used the VR-specific heuristics developed by [34]; out of the twelve heuristics, one focuses specifically on the sense of presence. The result indicated that their application—ALS-SimVR—was successful in replicating a realistic experience and that it theoretically could result in an enhanced sense of presence in the virtual reality world. Though it is difficult to pinpoint the qualities of a realistic experience, the claim is partly supported by [16], which states that empathy can heighten the sense of presence in a VR environment, something that is reasonable given the fact that the ALS-SimVR application is intended for advanced life support training in the healthcare field.

In the article by [11], the authors concluded in their results that virtual reality learning is a very efficient way to increase students’ interest, motivation, and most of all, their knowledge construction. In addition, VR provides an opportunity for students connected in the virtual world to apply what they have learned [27]. However, as [28] pointed out, a limitation exists to incorporating VR in the classroom, as a small number of users can experience the phenomenon known as simulation sickness. This includes symptoms such as nausea, dizziness, motion sickness, and headaches. However, none of the other articles mentioned simulation sickness as a possible downside to VR. Though the concept of embodiment was mentioned by [3,15,16], and [14], the embodiment is closely linked with the sense of presence and immersion in a virtual world, as implied by [6]. Two other concepts that ultimately affect the sense of presence are empathy and flow [15,16]. The concept of flow is ultimately about creating an engaging experience. However, it should be said that all five concepts that were presented above are deeply interlinked. According to [7], using an avatar in the virtual world offers a sense of presence and awareness and improves your capacity to interact with the 3D constructions in the world while being able to communicate and collaborate with others. In addition, the act of embodying a virtual character is vital since the core of the immersive experience is the presence of the user as an avatar in the virtual world [5].

The literature suggests that heuristic evaluation is a valuable tool, as it summarizes what needs to be worked on in the next iteration of an application. However, with concepts such as presence and immersion in mind, the hedonic qualities of a user experience disappear in favor of more pragmatic qualities. This can be seen, especially in the article by [17] where the developed VR heuristics for a learning environment focused on aesthetics, feedback, functionality between multiple devices, interactivity, and measuring learning. Now, with their set of finalized heuristics, it could be a case where the whole is greater than the sum of its parts. An application may be able to be developed and evaluated using their heuristics to reach a high level of perceived embodiment, empathy, flow, immersion, and presence. As previously stated, these concepts have been identified as key concepts of the user experience when it comes to learning in VR (see Figure 2).

The concept of presence has been heavily debated over the years. Ref. [38] summarized the challenge with presence in the following way: “If immersive virtual environment systems were able to deliver the perfect illusion of being and acting in a virtual world then probably the issue of ‘presence’ would never have arisen”. It is a complex concept, though the definition established by [16,18] correlates with the definition of presence in [39]: “… a psychological state in which virtual objects are experienced as actual objects in either sensory or non-sensory ways”. Presence can be categorized into three types: physical presence, social presence, and self-presence [39]. Both [6,28] defined the sense of presence as something physical in their respective articles.

In their literature review, [40] suggested a theory of presence based on the research conducted by [39,41,42]. According to the theory, the perceived sense of presence in a VR-based learning environment enhances learners’ motivations, learning engagement, and learning outcomes by allowing focused and naturalistic interactions with learning materials and activities [40,41,42]. If the theory could be verified further, it would strengthen the claim that presence is indeed a key concept when it comes to learning in VR. Another key concept that was identified was the sense of embodiment. Similar to how presence can be defined in multiple layers, the sense of embodiment can be sectionalized as well. Ref. [43] suggests that embodiment can be divided into the sense of self-location, sense of agency, and sense of body ownership. By understanding how the user embodies the virtual reality learning experience, one can unlock ways to design for a more accessible and meaningful session, which can ultimately increase the perceived flow. As previously stated, immersion and presence are terms that tend to be used interchangeably.

According to [38], immersion is a term used to describe the overall fidelity of the display and interaction systems concerning the physical reality. Their definition is on par with the stance [3] provided in their article, stating that immersion is a technological feature. Ref. [16] argued for the opposite, though [38] grounded their argument by fixating immersion as something technical; it makes presence research possible.

If immersion is a variable that can be manipulated, one can create an equation where presence is on the left-hand side and the factors of immersion are on the right-hand side [38]. This, in an incredibly simplified manner, sheds light on the still-abstract nature of these concepts and their immense complexity. To continue regarding the article by [17], one can make the argument that since Nielsen’s heuristics [23] were developed for traditional GUIs it focuses on more pragmatic aspects such as information hierarchy, icons, symbols, etcetera. Furthermore, by applying the PROMETHEUS method when developing the new heuristics, the method demands the foundation of preceding heuristics [17]. However, despite mentioning heuristics from both [29,30] that were compared to their newly developed VR learning heuristics, [17] fails to mention the VR-specific heuristics developed by [34].

Furthermore, while a better method would have been to use Sutcliffe and Gault’s heuristics [34] as a control variable in [17], there is still a reinterpretation of Nielsen’s heuristics for VR by [19]. Both [14,33] used the heuristics coined by [34], and in the set, there is one heuristic that focuses specifically on the sense of presence. Ref. [17] argued for the need for finer-grained heuristics, and perhaps it is true that there is a demand for heuristics specifically meant for a VR learning context, though the question arises if a new set of heuristics should solely focus on the pragmatic side of the experience or if it should also evaluate hedonic qualities.

The authors of [3,14,15,16,20,32,34,35,37,44] framed the five key concepts of the user experience for a VR learning context (see Figure 2). One can argue that empathy is a redundant quality when it comes to the VR experience, though one must understand that there is a range of various fields that use VR in some way to make the students learn more efficiently, easier, and safer. In the study by [14], where the study aimed to help the students learn complex algorithms quicker, empathy might not be a key component in enhancing the user experience. However, in the applications created by [27,28,29,30,31,32,33], where the aim was to help the students provide better care for patients, empathy is crucial.

4. Discussion and Future Directions

The purpose of this review was to review published articles using heuristics in a virtual reality learning context. This review showed how heuristics have been applied in various virtual learning environments (VLEs) that incorporate virtual reality learning activities, whether it is for using them in heuristic evaluations or as design principles. Additionally, it investigated the efficiency of said heuristics as an evaluation method, and lastly, it presented the key user experience concepts in VLEs as they have been discussed in the literature.

Usability heuristics were initially intended to ensure that a system’s user interface is simple to use. However, this review showed its use might extend further than a system’s user interface, for example, as design guidelines. This review showed the versatility of heuristics and that they can be applied differently depending on different aspects. While most of the reviewed articles (eleven articles) used heuristics for the evaluation of a VR environment, one article used them as design guidelines [24]. In both cases of using heuristics, already existing heuristics such as Nielsen’s 10 usability heuristics are used as the de facto standard in the process of heuristic evaluation or as design guidelines. However, this review showed that there is a need to develop custom domain-specific heuristics, and this is because Nielsen’s heuristics were not appropriately compliant with certain features of VLEs systems. In this review, two articles developed a set of usability heuristics that is more applicable to specific fields such as virtual learning environments. These studies also built their new set of heuristics based on already existing heuristics such as Nielsen’s heuristics and Sutcliff and Gault’s heuristics. However, further research is needed to develop specially tailored heuristics for VLEs. In the same context, there is a need to develop a clear methodology to develop and validate such domain-specific heuristics.

This review showed that the efficiency of the heuristics evaluation method used in the reviewed studies depends on several aspects, such as the experience and characteristics of the evaluators, the tools and techniques used, and the application settings, among others. However, the majority of the reviewed articles did not consider all these factors in the methodology used or in the discussion of the validity of the results. Although evaluators in some studies have completed training, their low level of experience can be viewed as a threat. In addition, the heuristics used were too general, limited, and extremely dependent on the evaluators. Furthermore, although two articles developed and validated a modified set of heuristics for VLEs, some methodological shortcomings still exist that can affect the results. Therefore, further research to improve the effectiveness of heuristics evaluation is needed.

This review showed that the user experience (UX) has been a key focus in the reviewed articles, regardless of whether the study aimed to evaluate a VR application, design for one, or develop new heuristics. Five key concepts vital for the user experience in a virtual reality learning context were identified throughout the literature—embodiment, empathy, flow, immersion, and presence. It is important to note that, while these key concepts are deemed important based on the analyzed literature, one should bear in mind that while the concepts are interlinked, the concepts can exist as their entities. It is all a matter of context and the specific goal of the application that is either under development or under evaluation. Similarly, rather than trying to rank the key concepts in a hierarchy, it is better to treat them equally and understand the way they are interconnected. By understanding that the concepts are features of the experience rather than features of the technology, one can begin to understand the dilemma that comes with both developing and choosing heuristics that support these hedonic qualities.

One can make the argument that the articles by [15,16] should have been excluded according to the exclusion criteria, as they focused on storytelling rather than learning. While [14,23,33] showed the diversity of how VR can be applied in a learning context, the scope is still broader. Ref. [24] developed the historic scenery of the Chola Dynasty and, while it might be overreaching, what is history if not storytelling written down? The pedagogic aspect is not something that should be forgotten, as the experiences that can be accessed through VR can not only aid learning but also accelerate it. However, considering the scope of this study, how various pedagogical elements are fair concerning the use of heuristics has not been deeply researched. It is evident, though, in the studies by [28,32] that the authors wanted to determine using their new set of heuristics whether VR could be a suitable tool to enhance learning, raising the focus on the pedagogical aspect of VR and its many possibilities.

5. Strengths and Limitations of This Review

A strength of the current review is its innovation. This study provides a comprehensive review of the scientific literature concerning the various ways heuristics have been applied in a virtual reality learning context. In addition, the search for relevant studies occurred across several sources and databases. However, only published studies were included in this review, resulting in publication biases. These biases occurred throughout the selection and evaluation of the papers as well as during the synthesis and analysis of the data. Therefore, there is a possibility of subjectivity in the interpretation of research, which could influence the conclusion.

The literature search was carried out through the following databases: ScienceDirect, SCOPUS, and Web of Science. Although these databases cover several areas and cover many individual databases, this decision may have influenced the number of relevant articles obtained. The use of other databases might have increased the number of articles analyzed and could have contributed to an improvement in the overall analysis. In addition, the research strategy was considered to limit the number of irrelevant articles (articles published many years ago, articles that are too general, or articles that do not focus on research goals). In addition, only articles in English were included. These options may have ruled out relevant articles, such as articles written in languages other than English. These restrictions may have had a significant impact on the number of records obtained and may have had some effect on the retrieval of relevant papers. As a result, the small number of papers reviewed and the eligibility of varied studies constrained our study. They may also have influenced the data extraction and analysis. However, these constraints had no significant impact on the discussion or conclusions.

6. Conclusions

Heuristics are a powerful tool that can guide a design process or aid a decision-making process. The key contribution of this review is to provide a clear picture that summarizes what has already been written about applying heuristics in a virtual learning context. The review identified the most important and relevant studies in the field, providing details on the topics that have promoted more academic attention and detailing various uses of heuristics in a VR learning environment. The methodology chosen to answer the research questions was a literature review.

To answer the research question (“How efficient is the use of heuristics in a virtual learning environment (VLEs) that incorporate virtual reality activities?”), this review showed the versatility of heuristics and that they can be applied differently depending on different aspects. While most of the reviewed articles used heuristics for the evaluation of a VR environment, a few articles used them as design guidelines. In addition, this review showed that the efficiency of the heuristics evaluation method used in the reviewed studies depends on several aspects such as the experience and characteristics of the evaluators, the tools and techniques used, and the application settings among others. Furthermore, five key concepts that are vital for the user experience in a virtual reality learning context were identified throughout the literature: embodiment, empathy, flow, immersion, and presence.

However, this study demonstrates that pragmatic heuristics are more than useful, and one should not disregard them. Ultimately, many of them were created for the sake of usability, meaning that they still put the user experience at the forefront. By combining both a hedonic and pragmatic approach, one is well on the way to ensuring a positive, engaging experience, when it comes to virtual reality learning, that ultimately benefits both the teacher and the student. Still, with concepts such as immersion and presence still being abstract to a certain degree, it might be easier said than done. After all, one can never design an experience; one can only design for it. A possible path forward after this study would be to further investigate the various sets of existing heuristics and empirically examine their hedonic versus pragmatic nature.