Learning Abstract Chemistry Concepts with Virtual Reality: An Experimental Study Using a VR Chemistry Lab and Molecule Simulation

Abstract

In this 2 × 2 between-subject experimental study, a virtual reality (VR) laboratory simulation is paired with a VR molecular world intervention to teach chemistry concepts. The independent variables are the implementation timing of the molecular world intervention (Pre-lab vs. Integrated) and the level of embodiment (Traditional vs. VR). Eighty students (N = 80), ages 11–18 years old, from a community center in New York City participated, completing a pretest, a laboratory simulation, a molecular intervention, and a post-test. The pre- and post-test measures included multiple-choice, free-response, and drawing questions. A key finding was that integrating the intervention within the lab simulation, no matter which level of embodiment, led to significantly higher gains in learning. The combination of using physical manipulatives and integrating them within the lab exercise (Integrated Traditional condition) demonstrated the greatest gains overall. On drawing measures, the Integrated VR condition showed significant improvement in three out of the four drawing categories (i.e., molecule shape, atom quantity, and relative sizes). The implications are that even though using a VR molecular world intervention can lead to significant learning of abstract chemistry content, the use of physical manipulatives is still a more effective tool.

1. Introduction

1.1. Research Significance

Chemistry is a notoriously difficult subject to learn for many primary and secondary-level students. The combination of abstract concepts, specialized vocabulary with nuanced meanings, unique mathematical explanations, and few obvious connections with daily life often confuses novice chemistry students [1,2,3]. Teaching strategies with technologies like virtual and augmented realities (VR, AR) can potentially be tools for new motivating and effective learning experiences.

1.2. Background

Virtual reality is broadly understood as some form of technology that creates an interface of a simulated environment between the user and the real, physical environment [4]. Augmented reality is a similar technology where an interface is used to simulate objects upon a background of the real, physical environment. These systems always involve some form of visual stimulation and possibly audio, haptic, or olfactory feedback to increase the illusion of realism. Educationally speaking, all of these features of VR and AR can be employed to immerse users in highly interactive, designed learning experiences [5]. VR research into its educational affordances has highlighted its ability to increase student motivation, allow for constructivist learning, provide authentic experiences for identity and empathy, provide creative opportunities, and visualize models and concepts [6].

1.3. Research Questions

This study utilizes a laboratory simulation that has been observed to provide a similar experience to a real-life laboratory [7] and a molecular world intervention that uses either physical manipulatives or a VR experience. A pre–post-test design is utilized, including assessments with written and drawing modalities. An analysis of different categories of chemistry knowledge is also conducted. This study seeks to explore the following research questions:

• How does the timing of a VR sub-micro experience in a laboratory exercise affect a student’s ability to learn sub-micro and macroscopic chemistry knowledge?
• How does the level of embodiment in a VR sub-micro experience affect a student’s ability to learn sub-micro and macroscopic chemistry knowledge?

1.4. Key Contributions

This study is a 2 × 2 between-subject experimental study where a VR laboratory simulation is utilized in conjunction with a VR molecular world intervention to teach chemistry concepts. The timing of the molecular world intervention (Pre-lab vs. Integrated into the lab simulation) and the level of embodiment (physical manipulatives vs. VR) are the two independent variables. The results showed that the timely integration of a sub-micro intervention into a VR lab exercise, no matter what modality, is more effective than presenting the same material as a pre-lab. Also, the immersive and interactive capabilities of the current state of VR are not as effective for learning sub-micro chemistry content when compared with physical manipulatives. However, the use of VR can lead to significant improvements in learning certain aspects of sub-micro chemistry content. The key contributions of this paper are:

• A pedagogy for effective implementation of VR technology in chemistry visualization;
• Design elements and processes for the development of VR science visualization learning environments;
• Empirical evidence of learning advantages and disadvantages for chemistry knowledge through the use of VR.

2. Literature Review

2.1. Chemistry’s Importance to Education

Chemistry is a critical field when considering science, technology, engineering, and mathematics (STEM) subjects. The difficulties that students encounter when learning chemistry can be long-lasting and far-reaching. They can often lead to discouragement and eventual withdrawal from not just chemistry but related fields in STEM [2,8]. This is due to many of chemistry’s topics being foundational for understanding other disciplines, such as biology, and advantageous for learning others, like physics [9,10]. Difficulties in learning chemistry also have the added consequence of disproportionately deterring underrepresented minoritized students and women from pursuing STEM degrees and careers [11]. If chemistry education can be improved, it can have a ripple effect in improving many other areas of STEM education.

Promising chemistry teaching strategies center on using better pedagogy to teach abstract concepts within the subject. Many of these effective pedagogies are founded upon Alex Johnstone’s [12] framework of the triplet nature of chemistry knowledge. He viewed the topics in chemistry as split into three different categories: the macro, symbolic, and sub-micro. The macro (or macroscopic)-level knowledge refers to aspects of chemistry that can be seen and measured empirically, the symbolic level being abstract and containing the representations using equations and notation, and the sub-micro level being the abstract mental models and images of the molecular world. Since his initial explanation of the triplet relationship of chemistry knowledge, many researchers have observed greater learning by students when the macro-symbolic–sub-micro relationships are directly addressed [1,3,12,13,14].

2.2. The Chemistry Laboratory as a Connector of Macro and Sub-Micro Knowledge

The chemistry laboratory, or practical work, is an experience where students gain hands-on experience conducting chemistry experiments. It is an experience where the combination of macro and sub-micro knowledge is presented in many ways but also where students have many difficulties [3,15]. Laboratory work is primarily an activity centered on macro-level knowledge, where students have hands-on interactions with substances, observe laws, and measure changes. Tsaparlis [15] speculates that the level of stimuli and complication of laboratory experiments can potentially overload a student’s capacity to process information, keeping them from properly analyzing the experiment and its underlying mechanisms in the sub-micro realm. Tan, Goh, Chia, and Treagust [16] have observed similar behaviors where students blindly follow laboratory instructions without thinking of the sub-micro level. Others have found that novices generally think at the macro level when confronted with new information and need to be prompted to think at the sub-micro level [17]. It could be argued that considering the sub-micro realm is not even a natural instinct for many novice chemistry students. These factors contribute to a high chance of missing the important relationship between chemistry’s macro and sub-micro levels, undermining the potential of the laboratory experience.

2.3. Virtual Laboratories and Their Benefits to Chemistry Learning

Virtual laboratories can also provide great benefits for students in science education. Their ease of use, repeatability for experiments, timely feedback, and efficient means of demonstrating multiple concepts are some of the most beneficial aspects for learners [18]. Olympiou and Zacharia [19] also identified many other benefits unique to virtual labs: providing capabilities for altering natural time scales, simplifying real-world models, making phenomena more visible, allowing changes to variables that would be impossible naturally, immediate feedback, directly focusing on targeted phenomena, and enabling performance in a safe environment. A virtual lab’s ability to make unseen phenomena visible is one of the most valuable affordances for chemistry education. It assists students in visualizing the sub-micro world and eases their cognitive load, making the environment much more conducive to learning [18].

2.4. Virtual Reality as a Tool for Learning Abstract Chemistry Knowledge

VR is a tool that has the potential to address this weakness due to its ability to simulate immersive, macro-level chemistry experiences while also enabling interactions with sub-micro models. These VR lab simulations are capable of enabling similar levels of learning when compared to real-life lab experiences [7]. Regarding learning at the sub-micro level, VR poses an interesting use case where immersive, interactive digital environments can be used to study these abstract concepts. Past research shows VR to be an effective tool for visualizing and simplifying complex abstract concepts [5,20,21,22]. Current research efforts have been used for immersive exploration of potential energy landscapes in chemical reactivity [23], as well as microscopic dynamical behaviors of molecules [24]. In addition, conducting a laboratory experiment in VR allows one to control the timing of processes and bring attention to sub-micro mechanisms at appropriate times, an ability that is rarely available in real-life laboratories [7].

These previously mentioned affordances of VR technology demonstrate a promising tool for exploring the macro–sub-micro relationship in chemistry. Explorations in augmented reality, a related technology, have demonstrated that chemical visualization tools can already prove useful to students [25]. Previous studies involving VR and chemistry have centered on simulations of outside environments [26], replicated laboratory environments for training purposes [27], or just polled stakeholders in the field without actually conducting an experiment [28]. In one similar research study at a higher education institution, Ferrell et al. [21] tested an immersive, microscopic VR world with an organic chemistry class. They showed that the intervention led to an increase in learning but did not mention if the control group was given a comparable learning activity. So, it is possible the students learned more in VR because they simply were able to have more time exposed to the content. Thus, this study seeks to fill this gap in knowledge surrounding pedagogy and VR’s efficacy to assist students in learning abstract chemistry concepts.

2.5. Learning Theory Framework

The possible differences between conditions are framed by certain aspects of different theories of cognition and learning. Differences in learning results based on the level of embodiment could provide evidence for its effects on understanding abstract chemistry content. For instance, “immersion” is a frequently cited affordance of VR that has little clarity in learning theory. It most closely resembles Csikszentmihalyi’s flow theory [29], in that the user is so immersed in an experience that they are in a state of extreme concentration and oblivious to any outside stimuli. Supporting research proposes that a high level of immersion allows the students’ minds to focus solely on the content, which may lead to greater understanding [30].

Barsalou’s grounded cognition theory [31] also provides the conceptual framework for the physical gestures and environment involved in our VR intervention and its potential to clarify specific abstract concepts. For instance, having the user move their hands towards each other to connect the digital representations of molecules should reinforce the idea of atoms combining to create a molecule. Furthermore, positioning the user as a microscopic object in a molecular world environment should create a bodily “simulation” (p. 618). By doing this, a user gains a multimodal experience that can reinforce the abstract concept they are focused on. Utilizing physical manipulatives does this as well but to a lesser extent. The varying levels of embodied movements could demonstrate nuances between these levels and their link to creating knowledge. The use of physical manipulatives with fine motor motions of finger gestures is much more intricate but less expansive and fits into what Johnson-Glenberg and Megowan-Romanowicz [32] call the second level of “embodied education taxonomy” (p. 2). The use of a VR headset and controllers with high levels of sensorimotor engagement, gestural congruency, and immersion fits into the fourth level, meaning that there is a difference in embodiment that can be used to analyze learning results.

Finally, the importance of visuospatial cues for learning abstract knowledge [33] supports our efforts in the VR simulation to reify atoms, molecules, and their unseen behaviors. Our utilization of different colors, shapes, and sizes for atoms are spatial representations that ease the user’s mental load for recognizing reactive properties (a common strategy among chemical experts) [34]. The VR modality allows designers to have greater flexibility over the appearance of artifacts, allowing designers to match colors, scale shapes more accurately, and even scale weights to more clearly signify specific characteristics. Physical manipulatives are more limited, being uniform in size, weight, and randomly colored, restricting their efficacy in representing multiple aspects of atomic content.

3. Methodology

This 2 × 2 between-subject study examined the learning results from manipulating the chronological order and level of embodiment of a VR chemistry intervention and laboratory exercise. Quantitative methodology utilizing statistical analysis on pre- and post-test measures was used to determine if there were significant differences in learning gains between conditions. Each condition involved an interactive VR laboratory simulation and either a session of playing with a set of traditional molecular physical manipulatives (i.e., ball and stick models) or interacting with an immersive VR molecule modeling program.

3.1. Research Sample

The data collection of the study, Teachers College IRB #20-434, took place from March to November 2022. The research sample was 80 students, ages 11–18 years old, randomly selected from a local community center in a multicultural, socio–economically diverse area of New York City. The study was conducted during the after-school and summer camp sessions. A total of 96% of the students were of Black, Hispanic, or both descent. Gender balance was 42 females and 38 males. Over 80% of the students were between 11 and 14 years old. Rationale for conducting the study outside of schooltime was to minimize lost instructional time due to the study participation. Participants completed the study in a single session, taking from 80 to 95 min. Students were compensated with a USD 10 Amazon gift card at the completion of their session. Most students had already been introduced to the basic concepts of atoms and molecules in their schools. About half of the students had experienced VR in some form prior to this study. More detailed demographic statistics for the sample population are represented in Appendix A.

3.2. Study Design & Procedure

There are four conditions into which students were randomly divided, 20 students per condition (see

Table 1. The 2 × 2 between-subject design for this study.

		Timing Placement of Sub-Micro Experience
		Pre-Lab	Integrated
Embodiment Level: Mode of manipulatives experience	Traditional (imaginative experience with physical manipulatives)	1. Physical manipulatives experience before the lab	3. Physical manipulatives experience within the lab
	Virtual Reality (immersive embodied experience with virtual manipulatives)	2. VR experience before the lab	4. VR experience within the lab

). The control condition used physical manipulatives as the intervention for the pre-lab exercise (Pre-Lab Traditional). Condition 2 used the VR intervention as the pre-lab exercise (Pre-Lab VR). Condition 3 used the physical manipulatives at a specific time point during the lab exercise (Integrated Traditional). Condition 4 used the VR intervention at a specific time point during the lab exercise (Integrated VR).

The procedure of the research study had commonalities between the conditions but differed in the timing and modality of the intervention. Each condition began with the demographic survey, pretest, and viewing of the Lab Safety Video. After this, each condition diverged with the Integrated and Pre-lab VR conditions having VR practice time while the control condition started with the physical manipulatives intervention. The Pre-lab conditions conducted the intervention and Manipulatives Assessment prior to conducting the laboratory exercise. The Integrated conditions conducted the laboratory exercise and participated in the intervention immediately after the chemical reaction began. The detailed procedure for each condition is detailed in Figure 1.

3.3. The Design of the VR Chemistry Lab

The VR lab simulation used in this study is called the VR Chemistry Lab and was designed to mimic the traditional American laboratory classroom. The entire simulation was created and designed by the principal investigator and is available in the Meta App Lab. The development platform used to create the VR chemistry laboratory exercise and the virtual manipulatives lesson was Unity 3D [35]. It was chosen for its ease of use, robust capabilities, strong user community, and expansive learning ecosystem. Supporting tools were also needed to create the objects and environment in the VR application. Three-dimensional models were first searched for on pre-made model sites like CGTrader [36], Turbosquid [37], or Free3D.com [38]. Then, if no satisfactory free models were found, they were created using Blender [39], Autodesk Fusion 360 [40], or Tinkercad [41]. Sounds and audio sources were sourced from free websites such as Freesound [42] and SoundBible [43]. The VR hardware used in this study was the Oculus Quest and Meta Quest 2 [44].

As per recommendations by the American Chemistry Society [45] for physical laboratory spaces, the virtual classroom contains a demonstration table with a sink as well as extra tables to support student workstations. Safety equipment such as eye goggles, gloves, and an apron are interactable and can be worn by the player. Safety stations, such as a shower, eyewash station, and fire extinguisher, are all functional 3D models. A virtual sink provides water, and virtual gas hookups also exist in the room. Glassware such as beakers, Erlenmeyer flasks, and test tubes are accurately modeled and will realistically break if dropped to the floor. A working thermometer was also created to enable the student to make accurate measurements during the experiment (See Figure 2 and Figure 3). However, not all typical lab equipment was included. Examples that were excluded due to non-involvement in the experiment were a fume hood and computers. Rationale for not including these pieces of equipment is based on Sweller’s [46] cognitive load theory. By excluding them, the student’s attention is directed only to the main procedures.

The intent behind the design of the virtual environment is founded upon Lave and Wenger’s [47] situated learning theory. The socio–cultural emphasis of this learning theory is important in the design of the virtual environment, not because of the social interactions (of which, there are none) but because of the implied learning atmosphere due to the cultural cues of the room. The cultural simulation of a traditional laboratory classroom is achieved through the use of black countertops, wood frames, whiteboards on plain walls, and a large wall-mounted periodic table poster. The intent is to mimic the learning atmosphere of traditional school lab classrooms, one of “conversation, collaboration, and discovery” [45] (p. 14).

3.4. The Design of the VR Chemistry Lab Experiment

The VR laboratory exercise is centered on the Next-Generation Science Standard MS-PS1–2: Matter and its Interactions [48]. The laboratory exercise involves mixing anhydrous copper (II) chloride with water and then introducing a piece of aluminum metal to the aqueous solution. The aluminum then combines with the chlorine atoms, replacing the copper in a single-replacement reaction. This lab experiment was chosen because it is recommended for middle to high school chemistry students by the American Association of Chemistry Teachers [49].

The visual and interactive design of The VR Chemistry Laboratory experiment was intended to simulate the real experiment while introducing effects that have the potential to increase its effectiveness for learning (See Figure 4). Models used for laboratory objects like chemical containers, beakers, and flasks were chosen for accuracy to their real-life counterparts. Interactions with objects were governed by intuitive gestures designed to mimic similar real-life motions. This was designed to minimize the cognitive load exerted by the player on these tasks [32]. Water effects were designed to be as realistic as possible while being constrained by the computational limitations of the hardware. Physical movement in the VR space was achieved in three ways: by naturally walking around in the space, the movement of thumb sticks, and by digital teleportation.

The dynamics of the experiment (i.e., reaction times, color changes, sounds, gas release) were determined by observing the actual reaction in real-life. This empirical determination of effects was necessary but not ideal for a robust and realistic simulation. Ideally, mathematical computations of the chemical reaction rates and the various interacting variables would be used but were too difficult or time-consuming to compute. Broadly, the interactions between chemicals and different substances hold true to their real-life examples and maintain the ability to simulate accurate effects in the experiment.

3.5. The Sub-Micro Intervention

3.5.1. Traditional Version (Physical Manipulatives)

The physical manipulatives version of the sub-micro intervention was an introductory lesson on atoms, molecules, and atomic behavior. It utilized a “ball-and-stick” molecule set, physically manipulable objects that are used traditionally in chemistry classes (see Figure 5). The lesson that incorporated these manipulatives was designed by the primary investigator and verified as appropriate by two subject matter experts.

3.5.2. Virtual Reality Version

The virtual manipulatives lesson was a VR experience designed and developed by the principal investigator (Figure 6 and Figure 7). It utilizes the advantages of VR to immerse the user in the “molecular world” where atoms and molecules are at the same scale as the participant. The experience takes place inside a beaker as if the user is witnessing the chemical reaction at the molecular level. Within the lesson, the user is guided verbally by an outside experimenter. Almost all of the procedure of this intervention mimics what was conducted in the traditional physical manipulatives version. The script used is identical to the physical manipulatives script except for a few references to the controls in VR. Both lessons are aligned with the underlying chemical reaction present in the macroscopic lab exercise.

3.6. Data Collection Measures

The internal reliability of the measures was assessed by looking at their Cronbach’s alpha value. This checks the consistency of the measurement instruments to ensure all questions are accurately measuring the same concept. SPSS software was used to compute Cronbach’s Alpha for the pretest, manipulative assessment, and post-test. Their values were all greatly above the “good acceptance” value of α > 0.7 [50] except for the “acceptable” value for the manipulative assessment (α = 0.656).

Since the pretest questions all had parallel questions on the post-test, reliability for the measure was similar to the item difficulty distribution as seen for the post-test (see Appendix B). Development and validation of this instrument, as well as the manipulative assessment and post-test, followed Adams and Wieman’s [51] framework for instrument design. Questions were developed by the principal investigator in collaboration with a subject-matter expert. Each question was constructed to align with Johnstone’s [12] categories of chemistry knowledge: sub-micro, macroscopic, and symbolic. In this manner, analysis could illuminate any resulting growth in each category.

Each measure was administered on a laptop using Qualtrics software. Some questions required drawing, so a pencil and paper were also provided. A short description of each is below.

3.6.1. Pretest

The pretest was conducted at the beginning of the session. The students individually completed the 35-question pretest: nineteen multiple-choice (MC), twelve free-response (FR), and four drawing (D) questions. Examples of a few of the questions are below:

MULTIPLE CHOICE (Sub-micro): The bond between hydrogen and oxygen in water is strong compared to the bond between sodium and chlorine in salt. Which molecule’s bonds would take more energy to break?

(A) Water (B) Salt (C) The same amount of energy is needed for either

FREE RESPONSE (Sub-micro): What is the difference between an atom and a molecule?

3.6.2. Manipulatives Assessment

The manipulative assessment was designed to measure any learning gains strictly from the pre-lab versions of the sub-micro intervention. It was intended to show whether the VR version of the sub-micro intervention has any benefits over the traditional, physical manipulatives version without any interference by the VR lab simulation. This assessment had 13 questions (six MC, six FR, one D) that pertain to the concepts covered in the manipulatives lesson. The item difficulty distribution for this measure is shown in Appendix B as well.

MULTIPLE CHOICE (Macroscopic): What did you observe happen to the temperature of the mixture when the copper chloride was placed into the water?

(A) It increased (B) It stayed the same (C) It decreased

FREE-RESPONSE (Symbolic): What should be the formula for the newly created molecule involved in this reaction?

Al + Cl + Cl + Cl → ____________

3.6.3. Post-Test

The post-test had 22 questions (thirteen MC, six FR, three D) that covered similar concepts to the pretest, but some questions utilized information specific to the laboratory experiment. For the Integrated conditions, the Manipulatives Assessment was combined with the post-test to be one summative assessment. This was designed to be able to match the content of every pretest question.

3.6.4. Drawing Measures

Two questions on each of the pre- and post-tests required the participants to draw a “ball-and-stick” model of certain molecules. Special attention was given to these questions as they represented students’ attempts at visualizing abstract content. The grading rubric was adapted from a subject matter expert’s publicly available “molecular model construction” rubric [52]. Drawings were scored based on the accuracy of the elements used, the number of specific atoms, the relative angle between atoms, and the relative size between atoms.

Two drawing questions were also designed to measure students’ ability to connect macroscopic chemistry information to sub-micro knowledge. These questions utilized an image of some object (e.g., solid pieces of metal floating in a flask of clear liquid, cup of orange juice) and asked students to draw what the molecules of objects in the picture might look like and how they are arranged. A successful drawing would demonstrate an understanding of individual particles, their physical proximity to each other, and their position or structure in relation to other particles. An example of a drawing question and two students’ responses is below (see Figure 8).

DRAWING Q: A carbon atom will combine with four hydrogen atoms and create the stable molecule of methane, CH₄. If you had the objects below (see Figure 8) how would you create a model of this molecule? Draw a picture of it. Use balls to represent atoms and the sticks to be bonds. Label the names of the atoms and where the bonds are.

3.7. Data Analysis and Synthesis

Pretest—Post-Test Measures

An inter-rater reliability (IRR) process was conducted for the free-response and drawing questions [53]; five cases were randomly chosen and five raters independently scored all of them. The scores were compared, a Krippendorff α value was computed [54], and if the α < 0.80, any discrepancies were discussed and the common rubric was edited. Then, five new cases would be chosen and the process repeated until the value of α > 0.80. Once this satisfactory value was achieved, the remaining cases were divided equally amongst the raters and individually evaluated.

Paired sample t-tests were used on each condition to analyze the score differences between the pretest and post-test. Questions in each measure were categorized into topics of symbolic, sub-micro, and macro and compared from pretest to post-test. For the 2 × 2 conditions, a two-way ANOVA was used to determine if there were any main and interaction effects due to the two independent variables. If a statistically significant difference between the groups was found, then a Tukey post hoc test was used to determine which groups were significantly different.

4. Results

4.1. Overall Score Results for Pre-Lab vs. Integrated Conditions

A paired samples t-test was conducted to determine if any significant changes occurred within the symbolic, sub-micro, or macroscopic categories from the pretest scores to the post-test. The two opposing conditions were the Pre-Lab and Integrated conditions. The results showed both conditions scored significantly better on the post-test in the symbolic and sub-micro categories (Pre-Lab Symbolic: M = 14.6%, t = 3.820, p < 0.001; Sub-micro: M = 7.6%, t = 2.617, p = 0.013; Integrated Symbolic: M = 16.4%, t = 4.344, p < 0.001; Sub-micro: M = 13.7%, t = 4.515, p < 0.001). However, both conditions declined significantly in the macroscopic category (Pre-Lab: M = −19.4%, t = −6.76, p < 0.001; Integrated: M = −10.6%, t = −4.227, p < 0.001) (see

Table 2. Paired sample t-test for SSM categories for timing conditions.

Condition	Pretest			Post-Test			ΔM	t(39)	p	Cohen’s d
	M	SD	SE	M	SD	SE
1. Pre-lab (control)
Symbolic	31.3	22.4	3.5	46.0	24.8	3.9	14.7	3.820	<0.001 *	0.604
Sub-micro	31.7	23.1	3.7	39.4	16.7	2.6	7.6	2.617	0.013 *	0.414
Macroscopic	43.3	19.9	3.1	23.8	12.8	2.0	−19.4	−6.757	<0.001 *	−1.068
2. Integrated
Symbolic	26.9	21.3	3.4	43.3	26.6	4.2	16.4	4.344	<0.001 *	0.687
Sub-micro	28.9	22.2	3.5	42.5	15.2	2.4	13.7	4.515	<0.001 *	0.714
Macroscopic	39.7	19.9	3.1	29.1	13.5	2.1	−10.6	−4.227	<0.001 *	−0.668

* p < 0.05.

).

4.2. Overall Score Changes for Embodiment Level Conditions

A paired samples t-test was conducted to determine if there were any significant changes in symbolic, sub-micro, and macroscopic scores based on embodiment level. The results showed that the Traditional condition significantly improved in the symbolic (ΔM = 24.8%, t = 7.358, p < 0.001) and sub-micro categories (ΔM = 13.9%, t = 4.242, p < 0.001) but decreased in macroscopic (ΔM = −15.3%, t = −5.048, p < 0.001). The VR condition significantly improved in sub-micro (ΔM = 7.44%, t = 2.828, p = 0.007) but decreased in macroscopic knowledge (ΔM = −14.7%, t = −5.839, p < 0.001) (see Appendix C).

4.3. Manipulative Assessment—Comparison of Embodiment Levels

A paired sample t-test showed that neither condition changed scores significantly from the pretest to the manipulative assessment. A one-way ANOVA determined that the differences between conditions were not significant (F(1) = 0.595, p = 0.445, η² = 0.015).

4.4. Overall Score Changes for 2 × 2 Conditions

Data analysis was also conducted to compare each of the four conditions within the 2 × 2 testing matrix. The goal was to determine if there was any interaction between the variables that significantly affected their scores from the pretest to post-test. Overall score changes, scores on the chemistry triplet of knowledge categories, molecule drawings, and macro–sub-micro drawings were analyzed for significant differences between the four conditions.

A paired samples t-test was conducted on each condition to determine whether they changed significantly from the pretest to the post-test. It was found that only condition 3 improved significantly (ΔM = 8.35%, SD = 15.3%, SE = 3.4%, t = 2.438, p = 0.025) (see

Table 3. Paired sample t-test for overall score changes for 2 × 2 conditions.

Condition	Pretest			Post-Test			Diff	t(19)	p	Cohen’s d
	M	SD	SE	M	SD	SE
1. Traditional Pre-lab (control)	36.5	19.7	4.4	38.4	13.9	3.1	1.94	0.656	0.520	0.147
2. Virtual Reality Pre-lab	34.2	18.5	4.1	29.6	13.7	3.1	−4.60	−1.674	0.111	−0.374
3. Traditional Integrated	34.5	17.9	4.0	42.8	11.3	2.5	8.35	2.438	0.025 *	0.545
4. Virtual Reality Integrated	29.3	19.7	4.4	31.8	15.3	3.4	2.45	1.108	0.282	0.248

* p < 0.05.

).

A two-way ANOVA was then conducted to determine if any of these conditions or the interactions between conditions had significant differences between each other. There was no statistically significant effect of the interaction between the timing variable and the embodiment variable (F(1) = 0.012, p = 0.913). However, a significant difference was identified within the timing conditions (F(1) = 5.513, p = 0.021) and the embodiment conditions (F(1) = 4.701, p = 0.033). No post hoc test was necessary since each condition had only two levels. The Integrated condition increased significantly more than the Pre-Lab condition and the Traditional condition increased significantly more than the VR condition (See Appendix D).

4.5. Molecule Drawings Analysis—Scores on Specific Categories for 2 × 2 Conditions

A two-way ANOVA was used to search for significant differences between conditions and their interaction for each specific drawing criterion. There was a significant interaction revealed between the timing and embodiment variables for the atom quantity category (F(1) = 5.808, p = 0.018). There were also significant main effects for embodiment conditions for the names of elements (F(1) = 6.774, p = 0.011) and the relative sizes of atoms (F(1) = 4.057, p = 0.048) (see

Table 4. ANOVA results for molecule drawing category scores for 2 × 2 conditions.

Source	Sum of Squares	df	Mean Square	F(1)	p	η2
Timing Condition
Element Names	0.800	1	0.800	1.084	0.301	0.014
Atom Quantity	0.313	1	0.313	0.317	0.575	0.004
Molecular Shape	2.113	1	2.113	2.142	0.147	0.027
Relative Sizes of Atoms	1.80	1	1.800	2.980	0.088	0.038
Embodiment Condition
Element Names	5.00	1	5.00	6.774	0.011 *	0.082
Atom Quantity	0.313	1	0.313	0.402	0.528	0.005
Molecular Shape	0.313	1	0.313	0.317	0.575	0.004
Relative Sizes of Atoms	2.45	1	2.45	4.057	0.048 *	0.051
Interaction
Element Names	0.050	1	0.050	0.068	0.795	0.001
Atom Quantity	4.513	1	4.513	5.808	0.018 *	0.071
Molecular Shape	0.313	1	0.313	0.317	0.575	0.004
Relative Sizes of Atoms	1.80	1	1.80	2.980	0.088	0.038

* p < 0.05.

). The Tukey HSD test for pairwise comparisons was then used to distinguish which conditions were actually significantly different (see Appendix E). There was a significant difference between the Traditional Integrated and the VR Integrated conditions in three categories. Traditional was significantly greater in recalling element names (ΔM = 0.550, p = 0.046), whereas, VR was significantly greater in recalling atom quantity (ΔM = 0.600, p = 0.035) and relative sizes of atoms (ΔM = 0.650, p = 0.010).

5. Discussion

5.1. Integrating a Sub-Micro Intervention within the Lab Exercise Is Significantly Better for Learning

Research question 1 explored how changing the timing of intervention implementation would affect sub-micro knowledge gains. Both the Pre-lab and Integrated conditions showed significant growth in symbolic and sub-micro knowledge scores. This is a positive sign that a sub-micro-focused intervention either before or during a lab exercise will improve students’ learning of abstract, sub-micro chemistry content.

The differences between the learning outcomes due to the timing conditions could be due to the lower amount of time between engaging with the intervention content and experiencing the related observable phenomena in the lab exercise. Traditionally, pre-labs are used in classroom laboratory settings to prime the students for learning specific topics and ease their cognitive load as they conduct the experiments [55]. While pre-labs have been proven to be helpful for learning [56], there could be a recency effect at work here. This effect, where more recently observed objects or concepts are more easily retrieved from memory [57], could explain why the Integrated condition scored better on the post-test.

Pedagogically, these results reinforce the idea that “just-in-time” learning could be a more effective learning method for laboratory experiences [58] (p. 341). With “just-in-time” learning, a student can pause the situation they are in and quickly research the content they are confused or curious about. For a science lab, when students reach a point in the exercise where they encounter an exciting development and are curious about the underlying reasons, that is the best time for explicit information to be presented to the student [59]. Rather than rely on the student remembering an explanation that was encountered earlier or hoping that an instructor is nearby, the timely explanation is now coupled with the instigating event, and a stronger connection is made.

The ability to pause time and allow the student to delve deeper into the underlying scientific principles was one of the primary reasons for the implementation of the lab exercise in VR. Doing the lab exercise with real chemicals and equipment is valuable but would not afford the student the ability to pause the reaction and immediately seek out relevant information. The VR simulation allows explanations to be available exactly when the student encounters the most stimulating part of the reaction. They can also have the time to reflect and think about the hidden mechanisms behind what they have just observed. This advantage of VR, its ability to control time in an immersive context, can provide valuable instances of learning that would normally be inaccessible.

5.2. Traditional Hands-On Embodiment Has Advantages over Virtual Reality Embodiment

Research question 2 focused on the effects of the level of embodiment for the participant during the intervention. It was hypothesized that participants would make better connections between sub-micro and macroscopic knowledge in the VR conditions due to the immersive model environment. However, this was not confirmed by the results. It is possible that the learning affordances of the physical manipulatives in the Traditional conditions are strong enough to overcome the affordances presented by immersive VR. As seen in other STEM education research, physical manipulatives are sometimes preferred by students over virtual ones and can lead to greater motivation towards learning [60].

Another explanation is that immersive VR is too distracting for students. This is consistent with findings from [61], who found that a highly immersive VR experience led to less fact-learning for students than a low-immersion desktop experience. Their explanation was that the high level of immersion led to cognitive overload for the learners and distracted them from the material they were supposed to learn. For many students, especially those who were experiencing VR for the first time, being in an immersive VR experience was quite novel and distracting. Perhaps with more experience in VR, they would be able to focus more on the tasks at hand.

It is possible that the physical manipulatives experience was a stronger grounding mechanism for cognition than the VR experience, leading to a better ability to extrapolate the information to other situations. The participants who used the physical manipulatives were able to feel the textures and physical weights of the objects representing atoms, whereas the VR participants may have been affected by the tactile gap between visually seeing certain objects and physically only holding the hand controllers. From an embodied cognition perspective, this is not entirely surprising. Though VR enthusiasts champion embodied cognition as a supportive theory for comparable learning in VR, there must be a limit to its support when the learning tool is unable to fully replicate the sensorial experience. The current state of VR hand controllers is this limitation.

It is also possible that the disparity in the presence of the experimenter in each condition could have contributed to the participants’ academic outcomes. In the case of the Traditional condition, where the experimenter was in-person and sat near the participant, this level of human presence was more tangible than in the VR condition, where the experimenter spoke instructions from outside the VR headset and did not have a digital avatar in the VR experience. A comparable situation is the different experiences of in-person learning vs. remote learning for students. Tomasik, Helbling, and Moser [62] found that the majority of students ages 9–15 years old learned more than twice as fast in-person when compared to remote learning. This seems to be an age-specific effect, as studies comparing in-person and online learning for students over 18 years old observed little difference in conceptual understanding between the two groups [63]. Thus, an argument could be made against VR as a primary educational tool for young people or that the designs of educational VR environments need to contain avatars or other teacher-like entities to assist young learners.

5.3. Benefits of VR for Abstract Sub-Micro Knowledge Learning

One important point from this comparison is that even though the Traditional condition showed greater learning than VR in some categories, both conditions showed significant improvement in the sub-micro knowledge realm on the post-test. Research supports this result, as students have been observed to remember more information by experiencing it in VR [64]. The gestures in both modes of the intervention provided a concrete situation with which the abstract concept could then be associated [65]. The VR intervention also led to significant improvements in identifying the correct sizes and atom quantity of different molecules in the drawing measures, which were helpful assessments of the students’ visuospatial abilities. The VR environment allowed digital objects to be rendered at various sizes, whereas the traditional ball-and-stick models were all manufactured to be equal sizes.

5.4. Limitations

5.4.1. Sample Size & Variation

There are a few limitations that affect the breadth of the results from this study. First, the sample size was not at the preferred number to attain a large effect size (0.80) for significant results. A conservative pre-study power analysis found that to achieve a FWER significance level appropriate to the standard α = 0.05, each condition would need N = 39 participants per condition. The broad variation in ages also limited the conclusions one could make for specific age groups.

5.4.2. Alignment of the Macro–Sub-Micro Connection in Measures

From the poor results in macroscopic knowledge of students in all conditions, it is clear that addressing macroscopic and sub-micro chemistry concepts together is difficult. One potential fault in this study may lie in the alignment of the measures with the design of the sub-micro intervention. The low values for item discrimination for some questions in the measures allude to this. For example, the intervention presented relatively little modeling of the inner atomic structure of solids. However, most of the questions connecting macroscopic and sub-micro content involved visualizing the inner structure of solid objects, which led to poor performance on the post-test measures by students. In addition, reminders of the macroscopic changes after the reaction were not robust enough in the sub-micro intervention. It was hoped that the return to the lab simulation for the Integrated conditions would implicitly create that connection. However, a more overt scaffold may be more effective since guided scaffolding activities are often necessary in situations where students need to reflect on the connections between these worlds [66]. Effective methods of teaching the macro-sub-micro connection in chemistry continue to be a pedagogical research need.

6. Implications

The results of this study are meant to inform practitioners of the usefulness of VR in chemistry education. It is safe to say that VR holds a motivational aspect for most students that can provide a boost to their interest. While it can be debated if this is compelling enough to warrant the use of VR in formal education settings, the confirmation of physical manipulatives as an effective learning tool is heartening. As most teachers have easier access to physical manipulatives, VR use can continue to inspire as an extracurricular tool and allow the traditional items to remain as everyday learning objects.

Implications for educational technology research is that there were shown to be significant differences in learning chemistry content using different levels of the embodied education taxonomy [32]. Specifically, content about atoms, molecules, and chemical reactions was remembered better by students who engaged in using physical manipulatives (level 2) than those who used immersive VR (level 4). This is not the only factor that could have affected the learning results but it is interesting that in this case, a lower level of embodiment resulted in greater learning outcomes. More research is needed in this area.

Additionally, researchers designing learning experiences for VR must agree that it is difficult to do well. Due to the almost limitless abilities one can provide the user in VR, the lack of boundaries can cause an instructional designer to lose focus. Innocuous details in the VR world can easily become distractions that keep the learner from focusing on the learning objective. For chemistry education, this study represents a step in the direction of effective design of VR learning experiences. Knowing effective learning designs can lead to better chemistry experiences for students and more students pursuing this important field.

7. Conclusions

Virtual reality is a technological tool that is gradually moving from its place in educational research to actual educational use. As a novel technology, it provides great motivation to students to try learning experiences on subject matter that they otherwise would not notice. It also provides learning advantages over traditional pedagogies for some facets of abstract knowledge. This study found that integrating a targeted intervention into a VR laboratory experience was effective in helping students learn abstract chemistry content, whether the intervention was a hands-on or VR experience. However, integrating the VR intervention was especially useful in helping students have accurate visual models of atoms and molecules. VR’s ability to host experiences that manipulate certain aspects of reality (e.g., the passing of time, geographic location, visual perspectives) was demonstrated to be effective for learning sub-micro chemistry knowledge but not macroscopic knowledge. STEM and other academic subjects stand to make great strides in learning possibilities as effective pedagogy for VR advances. This study moves us in that direction, clarifying the areas where VR is lacking, where it is comparable, and where it can advance the future of education.