Research on Equipment and Algorithm of a Multimodal Perception Gameplay Virtual and Real Fusion Intelligent Experiment

Featured Application

Authors are encouraged to provide a concise description of the specific application or a potential application of the work. This section is not mandatory.

Abstract

Chemistry experiments are an important part of chemistry learning, and the development and application of virtual experiments have greatly enriched experimental teaching. However, in the existing virtual experiments, there are problems such as low human–computer interaction efficiency, poor user sense of reality and operation, and a boring experimental process. Therefore, this paper designs a multimodal perception gameplay virtual and real fusion intelligence laboratory (GVRFL). GVRFL uses virtual and real fusion methods to interactively complete chemical experiments, which greatly improves the user’s sense of reality and operation. This method proposes a multimodal intention active understanding algorithm to improve the efficiency of human–computer interaction and user experience and proposes a novel game-based virtual–real fusion intelligent experimental mode that adds gameplay to the process of virtual–real fusion experiments. The experimental results show that this method improves the efficiency of human–computer interaction and reduces the user’s operating load. At the same time, the interaction between the real experimental equipment and the virtual experimental scene greatly improves the user’s sense of reality and operation. The introduction of game elements into the process of virtual and real fusion experiments stimulates students’ interest in and enthusiasm for learning.

1. Introduction

Chemistry is an experiment-based discipline. Chemistry experiments are an important way to learn chemistry knowledge. Chemistry experiments can help students quickly understand and learn from chemical experiments, thus gaining knowledge beyond what they read in textbooks. The task of teaching chemical experiments in a traditional laboratory has some limitations. For example, many phenomena in chemical experiments are unexplainable or unobservable, some chemical experiments are dangerous, and some chemicals are expensive [1].

For these reasons, the virtual laboratory has become increasingly important. Virtual laboratories provide integration of laboratory and computer simulations [2]. In addition, the virtual lab provides a safe learning environment for students. In the virtual environment, students can pause, continue, and repeat experiments many times. They waste no time or chemicals and can observe phenomena that are not easily observable. However, existing virtual laboratories often use devices such as a computer mouse [3], fingers [4], and handles [5] for interaction, and users lack the sense of operation and realism gained from using experimental equipment in a physical laboratory environment. This is clearly an important missing link for experimental courses that exercise students’ hands-on skills. Therefore, virtual labs that combine software and hardware can better meet the teaching objectives. However, there is a problem of a low experimental rate because the existing virtual labs lack timely guidance from the system on the user’s operation process [6]. In addition, learning is a boring process and proper entertainment is important for students to maintain their interest in learning [7]. Therefore, it is necessary to design an intelligent experimental system that can intelligently guide experimenters and provide them with some appropriate game elements and incentives to improve students’ learning interest. To achieve these functions, the system needs to understand the experimenter’s operations and thus provide timely responses. Unimodal information is one-sided and experiments using unimodal information are less interactive and less accurate [8].

To solve these problems, we designed a multimodal perception gameplay virtual and real fusion intelligence laboratory (GVRFL). GVRFL uses behavioral information from two channels, touch and voice channels, for fusion to obtain the user’s operational intent for natural multimodal human–computer interaction [9]. We proposed a multimodal intention active understanding algorithm to improve the efficiency of human–computer interaction and user experience and designed a set of new experimental equipment to sense the user’s wrong behavior during the experiment. We adopted the method of the fusion of virtual and real to enhance the user’s sense of reality and operation in virtual experiments. At the same time, we proposed a novel game-based intelligent experimental mode combining virtual and real elements. We added gameplay to the process of the virtual and real fusion experiment. In this way, students’ interest in and motivation for learning are stimulated during the experiment, their interest in virtual experiments is increased, and their learning efficiency is improved.

This article makes the following three contributions:

• Propose a multimodal intention active understanding algorithm suitable for chemical experiments and construct the core engine of virtual and real fusion experiments.
• Propose a novel intelligent experimental mode of fusion of virtual and real gameplay and introduce gameplay into the experiment.
• Complete the prototype of a multimodal perception gameplay virtual–real fusion experimental platform, and prove the usability of the platform through experiments.

The structure of this article is as follows: Part 2 briefly discusses related work, Part 3 outlines understanding the user’s intention in the gameplay virtual and real fusion experimental mode, Part 4 presents the design strategy of the gameplay virtual and real fusion experimental mode, Part 5 describes the experiment and evaluations, and Part 6 discusses the of conclusion and prospects.

2. Related Work

With the development of technology, the application of artificial intelligence in the field of education has become increasingly more extensive, and the virtual laboratory is a typical application. The concept of virtual experiments was first proposed by Professor William Wolf of the University of Virginia in 1989. Due to the lack of laboratories or inadequate laboratory equipment, few hands-on chemistry experiments are conducted in Turkish schools. Therefore, Tysz et al. [10] developed a 2D virtual environment for school chemistry education where students perform experiments in a virtual environment, and the results showed that the virtual laboratory had a positive impact on students’ academic performance and learning attitudes. Tsovaltzi [11] and others developed a web-based virtual experiment platform “Vlab”, where students could use menus and dialog boxes to control chemical instruments to complete chemical experiments. The above two virtual experimental platforms are both 2D experimental environments, and authenticity is lacking in the experimental environment. In the experiment, the students use a keyboard and mouse to interact with the system, thus lacking a sense of operation.

Ali et al. [5] developed a 3D interactive multimodal virtual chemistry laboratory (MMVCL). In MMVCL, the user completes the experiment with a virtual hand controlled by Wiimote. The information in the experimental process is received through audio and text information. The experimental results showed that MMVCL improved students’ learning skills and grades. Wu et al. [12] developed a virtual reality chemistry laboratory that uses Leap Motion to detect users’ gestures for operation. Users wear a head-mounted display and use gestures to interact with virtual objects. The results showed that at an appropriate learning intensity, the virtual reality chemistry laboratory could improve and enhance users’ learning confidence. Lam et al. [13] developed an augmented reality chemical experiment platform that uses augmented reality (AR) markers to control chemical experiment instruments and realize movement such as shaking, and dumping. All the above experimental platforms are 3D experimental environments that enhance the immersive sense and sense of reality of virtual experiments. However, these experimental platforms have a single mode of interaction due to the limitations of the input behavioral information methods, such as just using handles or gestures.

Human–computer collaboration is essentially a process of communication and understanding between humans and machines. If the platform can understand the user’s intentions and communicate with the user during the virtual experiment, then the user’s human–computer interaction process is natural and comfortable. In the process of human–computer interaction, researchers have been attempting to provide a natural and harmonious way of interaction. Multimodal human–computer interaction (MMHCI) is an important way to solve the problem of human–computer interaction. It captures the user’s interaction intention by integrating accurate and imprecise input from multiple channels and improves the naturalness and efficiency of human–computer interaction. Kaiser et al. [14] proposed a multimodal interactive system that fuses symbolic and statistical information from a set of 3D gestures, spoken language, and referential agents. The experimental results showed that multimodal input could effectively eliminate ambiguity and reduce uncertainty. Hui et al. [15] fused the user’s voice and pen input information. This method enabled the platform to understand the user’s intention and improve robustness. Liang et al. [16] proposed an augmented approach to help a robot understand human motion behaviors based on human kinematics and human postural impedance adaptation. The results showed that the proposed approach to human–robot collaboration (HRC) is intuitive, stable, efficient, and compliant; thus, it may have various applications in human–robot collaboration scenarios. Gavril et al. [17] also proposed a multimodal interface that makes use of hand tracking and speech interactions for ambient intelligent and ambient assisted living environments. The system has a spoken language interaction module and a hand tracking interaction module. The results showed that because of the user’s interaction with the multimodal interface, the ability to understand the user’s intention was improved. Zhao et al. [18] published a prototype of a human–computer interaction system that integrates face, gesture, and voice to better understand the user’s intention. Jung et al. [19] proposed a combination of voice and tactile interaction in a driving environment. The experimental results showed that the interactive mode of voice and touch improved the efficiency of interaction and user experience.

If the system is able to understand the user’s intent and communicate with the user in a virtual experiment, the human–computer interaction process for the user is natural and comfortable. Isabel et al. [20] proposed a multimodal interaction method, adding vision, hearing, and kinesthesia to virtual experiments. The experimental results showed that this method improved students’ interest in learning. Ismail et al. [21] proposed a method combining gesture and voice input for multimodal interaction in AR, with experimenters controlling virtual objects by a combination of gestures and voice. Wolski et al. [22] developed a virtual chemistry laboratory based on a hand movement system, with users using actions and gestures to perform the experiment. The experimental results showed that students’ learning performance was better in the virtual laboratory. Edwards et al. [23] developed a virtual reality multisensory classroom (VRMC), with users using haptic gloves and Leap Motion to complete the experiment in the VR multisensory laboratory (VRML). The abovementioned experimental platform uses multichannel interactions. Multichannel interaction in the virtual experiment can improve the efficiency and naturalness of human–computer interaction.

Many scholars have also explored the field of smart chemistry labs. Song et al. [24] proposed a chemical experiment system, CheMO, which uses a stereo camera to detect the users’ gestures and the positions of the instruments, mixed object (MO) beakers and an MO eyedropper as instruments for the chemistry experiments, and a digital workbench. When the user uses an MO instrument to experiment, he or she can obtain the sense of operation of the real experiment. The experimental results showed that MO enhanced the realism of virtual experiments, and users could learn effectively during the experiments. In 2019, Hartmann et al. [25] allowed users to communicate with people and objects in physical space in a virtual reality system, so that users would not lose their sense of presence in the virtual world. The experiments showed that the user experience is enhanced by the system. Amador et al. [26] designed a physical tactile burette to embed the physical sense of the laboratory in a virtual experiment. Students can control the physical tactile burette to operate the virtual burette in virtual reality. Zeng et al. [27] also proposed an intelligent dropper and built a multimodal intelligent interactive virtual experiment platform (MIIVEP). The system integrates three modalities, speech, touch, and gesture, for interaction. The experimental results showed that this method improved students’ learning efficiency. Yuan et al. [28] designed an intelligent beaker with multimodal sensing capabilities. The intelligent beaker is equipped with multiple sensors. To obtain the real intention of the user during the experiment more accurately, a multimodal fusion algorithm is proposed to fuse the user’s voice information and tactile information. The user uses the intelligent beaker to complete a variety of experiments under the guidance of teaching navigation. Xiao et al. [29] also designed an intelligent beaker and proposed another multimodal fusion algorithm to fuse the user’s voice information and tactile information. The experimental results showed that users could feel the sense of operation in a traditional laboratory when using the above two intelligent beakers. Wang et al. [30] proposed a multimodal fusion algorithm (MFA) that integrates multi-channel data such as speech, vision, and sensors to capture the user’s experimental intention and also navigate, guide, or warn the user’s operation behavior, and the experimental results showed that the smart glove was applied to teach chemistry experiments based on virtual–real fusion with a good teaching effect. Xie et al. [31] proposed a virtual reality elementary school mathematics teaching system based on GIS data fusion, which applied a virtual experiment teaching system, intelligent assisted teaching system, and virtual classroom teaching system analysis-related technologies to teaching. The experimental data and investigation results showed that the system could help students simulate operation experience and understand the principles, and at the same time, improve users’ learning interest. Wang et al. [32] proposed a smart glove-based scene perception algorithm to capture users’ experimental behaviors more accurately. Students are allowed to conduct exploratory experiments on a virtual experimental platform to guide and monitor users’ behavior in a targeted manner. The experiments showed that the smart glove can also infer the experimental intention of the operator and provide timely feedback and guidance to the user’s experimental behavior. Pan et al. [33] proposed a new demand model for virtual experiments by combining human demand theory. An integrated MR system called MagicChem was also designed and developed to support realistic visual interaction, tangible interaction, gesture and touch interaction, voice interaction, temperature interaction, olfactory interaction, and avatar interaction. User studies have shown that MagicChem satisfies the requirements model better than other MR experimental environments that partially satisfy the requirements model. The above experimental platforms propose the use of instruments and devices in real scenes to operate objects in virtual environments, which greatly retains the user’s sense of operation and realism in traditional experiments.

The combination of virtual experiments and education is still in the trial stage, and many researchers have carried out different explorations of virtual experiments. Ullah et al. [34] added teaching guidance to the MMVCL experiment. The experimental results showed that teaching guidance could improve students’ performance in the virtual laboratory. Su et al. [35] designed and developed a virtual reality chemistry laboratory simulation game and proposed a sustainable innovative experiential learning model to verify the learning effect. The experimental results showed that the virtual chemistry lab had a significant effect on academic performance. Students using the sustainable development innovative experiential learning model gain a better understanding of chemical concepts. Oberdrfer et al. [36] combined learning, games, and VR technology. The experiment showed that this approach improved the quality and effect of learning and enhanced the presentation of abstract knowledge. Rodrigues D et al. [37] used an experimental study of artificial intelligence techniques to customize the educational game interface based on the player’s profile, and the empirical results show that the game components adjusted in real time for a better player experience and, at the same time, improved the correctness rate of the players participating in the study. Marín D et al. [38] proposed a multimodal digital teaching approach based on the VARK (visual, auditory, reading/writing, kinesthetic) model that matches different learning modes to different students’ styles. The experimental results show that the use of the VARK multimodal digital teaching approach has proven to be effective, efficient, and beneficial in HCI instruction, with significant improvements in users’ learning scores and satisfaction. Hong et al. [39] designed a gamification platform called TipOn that allows students to ask and gamify questions based on different game modes in order to facilitate students’ practice of English grammar. The focus was on designing a reward system to gamify students’ learning content. The results suggest that teachers can use this gamification system in the flipped classroom to motivate students to tap into their cognitive curiosity in order to increase their content learning and improve learning outcomes.

In summary, the multimodal interaction method can make it easier for computers to understand the user’s intentions and can effectively improve the efficiency of human–computer interaction and user experience. To improve the understanding of user intention in the virtual–real fusion experiment in this article, we propose a multimodal intention active understanding algorithm. The algorithm integrates the user’s tactile information and voice information during the experiment to improve the efficiency of human–computer interaction and reduce the user’s interaction burden.

Absorbing the above research results and experience, this paper proposes a game-based intelligent experiment based on the fusion of virtuality and reality. Through real experimental equipment and a virtual experimental platform, students can complete chemical experiments that combine virtual and real elements. At the same time, to eliminate the boring nature of virtual experiments, game elements are added to the virtual experiments. By adding gameplay to the experimental mode of fusion of virtual and reality, students’ enthusiasm for learning is stimulated, and their inquiry ability and practice level are improved.

3. Understanding User Intentions in the Experimental Mode of Gameplay

To simulate the chemistry experiment scene and operation in the traditional laboratory, we use the method of virtual and real fusion to perform the chemistry experiment. Real refers to the intelligent beaker, and virtual refers to the virtual experimental platform. At the same time, to improve the ability of the virtual and real fusion experimental platform to understand the user’s intention, we have added a multimodal intention active understanding algorithm. On this basis, we learn from the existing game platforms and add game elements to the virtual and real fusion experimental platform. The above methods can increase the exploratory and interesting nature of the experiment, improve the sense of immersion and participation in the experiment, and stimulate students’ learning enthusiasm. The overall framework of the multimodal perception gameplay virtual and real fusion experimental mode is shown in Figure 1.

The experimental platform obtains tactile information and voice information through the intelligent beaker and microphone and recognizes the obtained tactile information and voice information. The tactile recognition results and voice recognition results are input into multimodal intention active understanding algorithms to obtain the user intention. Finally, the user intention is input into the plot of the virtual and real fusion experimental scene. In the experimental plot, the platform analyzes the user’s intention information and operational information and combines game elements to present the experimental process to the user through a multichannel method.

3.1. Design and Interaction of Intelligent Devices

We use the method of fusion of virtual and real for chemical experiments. Real refers to the new intelligent device that senses tactile information. The intelligent device (Figure 2) is a 3D-printed beaker model equipped with two touch sensors, a posture sensor, and a Bluetooth module. To facilitate users’ use of intelligent devices, we use Bluetooth to output tactile information. The intelligent beaker inputs sensor information into the computer through the Bluetooth module.

The interaction of intelligent devices occurs through touch sensors and posture sensors. The interactive mode of the intelligent beaker is as follows:

• The No.1 touch sensor has the function of selecting chemical reagents. Users can choose different chemical substances through the touch sensor on the intelligent beaker.
• The function of the No.2 touch sensor is to detect whether the way the user holds the intelligent beaker is standard. The No.2 touch sensor is set at the lowest end of the beaker, a position that cannot be touched when the intelligent beaker is held normally. If the user touches the No.2 touch sensor when holding the intelligent beaker, it will prompt the user to hold the beaker properly.
• The posture sensor on the intelligent beaker is used to sense the beaker’s own information, including the angle, direction, and speed.
• The posture sensor gives the beaker a rough dumping quantification function. The dumping quantification function is determined by the dumping time, angle, and speed. As shown in

  Table 1. Dumping quantification.

  Grade	Dumping Angle θ	Dumping Volume mL/s	Dumping Speed v	Dumping Time t/s/s
  Not dumped	$\mathit{\theta }\mathit{ }\mathit{>}\mathit{ }\mathit{50}$	$\mathit{0}$	$\mathit{v}\mathit{ }\mathit{=}\mathit{ }\mathit{0}$	$\mathit{0}$
  Dump slowly	$\mathit{30}\mathit{ }\mathit{<}\mathit{ }\mathit{\theta }\mathit{ }\mathit{\le }\mathit{ }\mathit{50}$	${\mathit{\eta }}_{\mathit{1}}$	$\mathit{0}\mathit{ }\mathit{<}\mathit{ }\mathit{v}\mathit{ }\mathit{\le }\mathit{ }\mathit{30}$	$\mathit{0.8}\mathit{ }\mathit{t}$
  Normal dumping	$\mathit{10}\mathit{ }\mathit{<}\mathit{ }\mathit{\theta }\mathit{ }\mathit{\le }\mathit{ }\mathit{30}$	${\mathit{\eta }}_{\mathit{2}}$	$\mathit{30}\mathit{ }\mathit{<}\mathit{ }\mathit{v}\mathit{ }\mathit{\le }\mathit{ }\mathit{60}$	$\mathit{t}$
  Quick dump	$\mathit{-}\mathit{ }\mathit{10}\mathit{ }\mathit{<}\mathit{ }\mathit{\theta }\mathit{ }\mathit{\le }\mathit{ }\mathit{10}$	${\mathit{\eta }}_{\mathit{3}}$	$\mathit{v}\mathit{> }\mathit{\ge }\mathit{ }\mathit{60}$	$\mathit{1.2}\mathit{ }\mathit{t}$

  , angle $\theta$ is the angle of the current tipping direction of the intelligent beaker, the speed v is $v=\sqrt{(x^2+y^2+z^2 })$, and $x, y, z$ are the angular velocities of the coordinate axes of the intelligent beaker. The rough dumping quantification is divided into three levels: slow, normal, and fast. ${\eta }_1$, ${\eta }_2$, and ${\eta }_3$ are an estimate of the amount of liquid poured in one moment t when the pouring angle ($\theta$) matches the range of the four hierarchical classifications in Table 1, respectively, and in this paper, one moment represents 1 s. Since the values of pouring quantification are roughly estimated, we also present them in hierarchical levels in terms of animation presentation. For example, the values in the interval (30,50) are presented with the same effect in the virtual experimental scenario, where the same amount of liquid is viewed. For the quantification of the pouring speed ($v$), we reflect it by the pouring time, when the pouring speed is less than 30. Based on experience, we believe that the amount of 1-s pouring at this time can be calculated according to 0.8 times the normal speed. Similarly, when the dumping speed is greater than 60, we believe that the amount of dumping in 1 s can be calculated according to 1.2 times of the normal speed. Finally, the total amount of dumping is correlated with the dumping volume and dumping time, and the calculation of the total amount depends on the product of the dumping volume and dumping time for different levels.

Intelligent beakers have the common requirements of chemical experiment beakers and can be used to perform different chemical experiments.

3.2. Multimodal Information Perception

First, we define a set of intents in the chemical experiment. Since we use a combination of tactile and speech information to obtain user intent, the set of chemical experiment intents corresponding to tactile and speech information is the same. We set up multiple paths and obstacles in the experimental process so that the chemical experiment is no longer a single operational process. Since chemical experiments are relatively independent processes, different sets of intentions are set up for different experiments. We divide the whole experimental process into several experimental steps $P_n=\left\{P_1,P_2,\dots ,P_i\right\}$, where $P_n$ denotes all steps in the nth experimental process. To complete a step requires multiple operations. We set one or more paths $R_n=\left\{R_1,R_2,\dots ,R_j\right\}$ in each step according to the rules of chemical experimental knowledge points, where $R_n$ denotes all the paths in the nth experimental step, i.e., the set of all the operation behaviors required to achieve this step. Finally, a different set of intentions $I_n=\left\{I_1,I_2,\dots ,I_m\right\}$ is set according to the combination of multiple paths in each step, where $I_n$ denotes the set of all intentions for the nth experiment and $m$ denotes the number of experimental intentions. Behavior and intent are a one-to-many relationship, and the user’s operational intent is predicted by behavior. We assign the intention corresponding to each step according to the steps of the experiment and the path. For example, an experimental intention set is $I_n$ = $\left\{I_1, I_2, I_3, I_4,I_5\right\}$. According to the three steps $P_i$ = {$P_1$, $P_2$, $P_3$} of the experiment, the intention set assigns intention$I_n$ = {($I_1$,$I_2$|$P_1$), ($I_3$, $I_4$|$P_2$), ($I_5$|$P_3$)} according to the step. Among them, two intentions $I_1$,$I_2$ are set in step $P_1$ to reflect the concept of multipath and obstacles in the experimental process. A concrete example is shown in Figure 3.

After entering the experimental scene, the user uses the intelligent beaker to perform experimental operations. The intelligent beaker perceives the user’s tactile information $Tac$ under the current experimental step $P_i$. User tactile information, the current experimental step, and the intention set input the tactile information conversion function $TicF\left(Tac,P_i, I_n\right).$ Finally, the tactile information is recognized by the tactile information conversion function to obtain the recognition result $It$:

$$It=TicF\left(Tac,P_i, I_n\right)=Tac \cap ( I_n|P_i),$$

(1)

where $Tac$ is tactile information, $I_n$ is the tactile intention set of the current experiment, and $P_i$ is the current experimental step.

In the process of the virtual and real fusion experiment, users can not only use tactile information but also voice information to express intention. First, we build a voice command database $V_n=\left\{V_1, V_2,\cdots , V_m\right\}$based on different experimental intention sets. Among them, the speech database has a one-to-one correspondence with the intention set$I_n$ of the chemical experiments, $n$ represents the type of experiment, and $m$ represents the number of intentions of the experiment. We interviewed 20 students and asked how they would use language to express the intentions in the intention set, and finally, each intention had five expressions. Using the laboratory preparation of carbon dioxide as an example, the set of intents is $I_n$ = $\{$react limestone with dilute hydrochloric acid, react limestone with dilute sulfuric acid, react limestone with concentrated sulfuric acid, react calcium carbonate with dilute hydrochloric acid, react calcium carbonate with dilute sulfuric acid, react calcium carbonate with concentrated sulfuric acid$\}$. The voice command database corresponding to the intent $I_1$-React limestone with dilute hydrochloric acid is $\{$mix limestone with dilute hydrochloric acid, add dilute hydrochloric acid to limestone, react limestone with dilute hydrochloric acid, pour dilute hydrochloric acid into a conical flask containing limestone, put limestone with dilute hydrochloric acid into a reaction reagent flask$\}$. Second, we found that the commands issued by users are all shorter voice commands. Therefore, we use word vectors to calculate the similarity between the speech information and the command set in the speech intention database. We use word2vec to train the word vector model and then use the word vector model to convert the user’s voice information Aud and voice commands in the voice database $V_n$ of the current step $P_i$ into word vectors. The maximum cosine similarity between the two is the result of speech information recognition:

$$I_a=SaF\left(Aud, P_i ,V_n\right)=arg max\left(cos\left(Aud · \left(V_n|P_i \right)\right)\right),$$

(2)

where Aud is voice information, $V_n$ is the speech intention set of the current experiment, and $P_i$ is the current experimental step.

3.3. Active Understanding of Multimodal Intentions

The proposed algorithm focuses on the basic purpose that the experimental system can understand the user’s operation intention based on the multimodal fusion. After obtaining the user’s tactile results and voice results, the multimodal fusion algorithm fuses multimodal information to obtain the user’s intentions. To improve the accuracy of intention understanding, an active understanding process is added after multimodal fusion. The specific algorithm framework is shown in Figure 4.

As the figure shows, the user inputs voice information and tactile information. We use a multimodal information fusion model to fuse the recognition results of speech and touch. After obtaining the fusion results, we preprocess the fusion results and analyze the fusion results with lower confidence. The tactile information and voice information of the fusion result are input into a single-modal evaluation function, and the evaluation function analyzes which channel of the input information is missing or incomplete. The system requests that the user enhances or supplements the information of a certain channel according to the result of the single-modal evaluation function. The fusion result with higher confidence is considered to be the user’s current intention. The multimodal intention active understanding algorithm uses the evaluation of the fusion result and the single-modal information evaluation to request that the user enhances the input information of a certain channel to improve the intention understanding ability.

We choose to fuse the multimodal information in the experimental process at the decision-making level and input the multimodal information into the multimodal information fusion function $Miff\left(It, Ia\right)$:

$$Miff\left(It, Ia\right)=\left\{\begin{array}{c} It, Ia=\varnothing ,It\ne \varnothing \\ \alpha It+\left(1-\alpha \right), Ia\ne \varnothing ,It\ne \varnothing \\ Ia, Ia\ne \varnothing ,It=\varnothing \end{array}\right.,$$

(3)

In terms of multimodal information fusion, we consider three situations: (1) only tactile information, (2) only voice information, and (3) tactile information and voice information at the same time. When tactile information and voice information exist at the same time, the average weighting method is used for fusion: $\alpha =0.5$.

After the fusion result is obtained, the system preprocesses the fusion result.

If $Miff\left(It, Ia\right)>\beta$, the system will input the obtained user intention $B$ into the experimental scene. If $Miff\left(It, Ia\right)<\beta$, the system inputs the tactile and auditory recognition results into the single-channel evaluation function $SEva\left(It, Ia\right)$:

$$SEva\left(It, Ia\right)=f\left(x\right)=\left\{\begin{array}{c}1, It<{\lambda }_1\\ 2, Ia<{\lambda }_2\end{array}\right.,$$

(4)

where 1 and 2 represent touch and sound, respectively, and ${\lambda }_1$, ${\lambda }_2$ are the thresholds of the tactile and speech channels, respectively.

According to the results of $SEva\left(It, Ia\right)$, we analyze which channel has low information quality. According to the evaluation results, the system reminds the user to enhance the information of a certain channel and combines the context to obtain new user intentions. Finally, the real intention of the user is input into the current experimental scene.

The multimodal intention active understanding Algorithm 1 is as follows:

Algorithm 1 Multimodal intention active understanding algorithm (MIAUA)

Input: Tactile recognition results $It$. Speech recognition result $Ia$;

Output: User intention $B$;

1: while $It, Ia$is not empty do

2: $\hspace{1em}\hspace{1em} It, Ia$ input multimodal information fusion function $Miff\left(It, Ia\right)$;

3:   while $Miff\left(It, Ia\right)$ is not empty do

4:    if $Miff\left(It, Ia\right)>\beta$ then

5:     Get user intention $B$;

6:    end if

7:    if $Miff\left(It, Ia\right)<\beta$ then

8:     Get user intention $B$;

9:      When the fusion result is less than the threshold, $It. Ia$ are input to the single-channel evaluation function $SEva\left(It, Ia\right)$;

10:     if $SEva\left(It, Ia\right)=1$ then

11:     Remind users to enhance tactile channel information;

12:     end if

13:     if $SEva\left(It, Ia\right)=2$ then

14:       Remind users to enhance auditory channel information;

15:     end if

16:     if  $SEva\left(It, Ia\right)!=1 \&\& SEva\left(It, Ia\right)=!2$ then

17:      Remind users that they cannot obtain accurate intentions and continue to predict user intentions;

18:     end if

19:    end if

20:   end while

21: end while

3.4. Algorithm Analysis

The intention capture result of the fusion layer does not necessarily reflect the real intention of the user, so its credibility needs to be evaluated. Under a specific algorithm framework, one of the key factors affecting credibility is the quality of multichannel information at the perceptual level. For example, if the user’s voice information is unclear, incomplete, or cannot be captured by voice recognition, the credibility of the intention perception based on the voice channel is low. In this case, the system can actively ask the user to input voice information again or change to a voice that can express the same intention. Therefore, this article evaluates the results on two levels: one to evaluate the result of fusion intention, and the other to evaluate the credibility of each channel.

The MFPA algorithm in the literature and the MMNI algorithm in the literature are both multimodal fusion algorithms in virtual and real fusion experiments. The MIAUA algorithm in this paper similarly fuses voice and tactile information at the decision-making level. The difference is that MIAUA verifies the fusion result, unlike the MFPA algorithm and the MMNI algorithm, which directly confirm or deny whether the fusion result is the real intention of the user. The MIAUA algorithm evaluates the fusion result, and the fusion result obtained through the evaluation is considered the real intention. For fusion results with low credibility, the MIAUA algorithm evaluates the information of each channel. For channels with low credibility, the system actively requires users to supplement and strengthen the information of the current channel. We use the method of active understanding of the intention to accurately identify the user’s intention. The ability of the MIAUA algorithm to recognize user intentions is stronger than that of the MFPA algorithm and MMNI algorithm. During the experiment, the user’s interaction burden will be lower, and the human–computer interaction will be more natural and coordinated.

4. Gameplay Design of Virtual and Real Fusion Experimental Mode

People always experience a series of psychological changes when they study or play games. “Flow” is one of the most representative positive psychological experiences. Mind flow is a phenomenon in which participants focus on the goal and task at hand, concentrate so intensely that they forget about the passage of time, and filter out all external influences [40]. In terms of guiding teaching activities, Liao et al. [41] created an evaluation model to analyze students’ motivation and behavior by combining the practice of mind flow theory for guiding distance learning and concluded that when students enter a state of flow while learning, the learning effect is greatly improved. If students enter a state of flow when studying, the learning effect will be greatly improved. With the help of this mechanism, we incorporate game elements into the virtual experiment. In the game state, it is easier for students to enter the state of flow, which improves their learning interest and motivation to a great extent. Therefore, we combine the characteristics of existing game platforms to introduce gameplay into the experimental interactive mode of virtual and real fusion. The introduction of gameplay stimulates students’ interest and motivation in the experimental process and improves their learning efficiency.

The addition of game elements to virtual experiments can make learning a means of passing game levels and eliminate students’ negative emotions about learning. Since different chemical experiments have their own operating points and knowledge points, we design game rules in the virtual and real fusion experiment. We take the learning path as the experimental plot and learn chemical experiments by guiding the main line. At the same time, real-time feedback is realized in the form of points, levels, and rewards during the experiment so that users can quickly and intuitively understand the situation of their experiments. We design the experimental mode of the virtual and real fusion experiment as a breakthrough game mode and design the number of experimental levels according to the complexity of the chemical experiments. According to the chemical experiment steps, we set the plot $P_i$ at each experimental level. We add a multipath strategy, obstacle setting strategy, and incentive strategy to the experimental mode of gameplay fusion of virtual and reality. The level rules are established on the basis of the multipath strategy and the obstacle setting strategy. The experimental levels are formed through individual experimental steps, so that the entire experimental process is consistent; at the same time, setting obstacles during the experiment increases the difficulty of the experiment and enhances the interest and playability of the experiment. In psychology, reward is a universal state of mind that causes pleasant feelings and leads those who experience it to expect to be recognized and appreciated by others. Therefore, an incentive strategy is added during the experiment. When the learner completes an expected goal or passes the checkpoint, he or she will be rewarded according to the performance of the experiment.

4.1. Multipath Strategy

Chemistry is a natural subject based on experiments. Chemistry experiments are an important part of chemistry teaching and can cultivate students’ scientific inquiry ability and practical ability. Different chemical experiments have corresponding experimental steps, as do certain exploratory experiments. We propose a multipath strategy in the game-based virtual and real fusion experiment and add multiple learning paths to the experimental plot. The multipath approach is defined as the design of multiple paths $R_j$ for users to choose based on the knowledge points of the chemical experiment in a certain plot in the game process, increasing the exploratory nature of the virtual and real fusion experiment. For example, the method of producing carbon dioxide in a traditional laboratory is the reaction of limestone with dilute hydrochloric acid. In exploratory experiments, limestone can be replaced with marble or sodium carbonate, dilute hydrochloric acid can be replaced with dilute sulfuric acid, the amount of limestone or the concentration of dilute hydrochloric acid can be changed, etc.

4.2. Set Obstacle Strategy

We use three methods to set obstacles in the experiment. Method 1: When the user wants to obtain nonexploratory experimental results, he or she can observe only the desired experimental phenomenon by selecting the correct path in the multipath format. Method 2: We set up game levels in complex chemistry experiments, and users can enter the next level after passing the first level. For example, in the experiment of preparing carbon dioxide from limestone and dilute hydrochloric acid, the carbon dioxide produced is not pure carbon dioxide (containing water and hydrogen chloride gas). We set up two levels according to the degree of difficulty: the first level is to produce carbon dioxide, and the second level is to remove impurities from carbon dioxide to obtain pure carbon dioxide. Method 3: During the experiment, we add a question-and-answer session related to knowledge points. The user can enter the next step only if the answers are correct. If the user answers incorrectly, the system will feed the user several answer options for the user to choose until the user chooses the correct option. In addition, in the observation of experimental phenomena, part of the experimental phenomenon is obscured by a mosaic. The user can observe the experimental phenomenon after passing the knowledge points of the question-and-answer session.

4.3. Incentive Strategy

The incentive strategy uses real-time feedback in the form of scores and rankings, so that users can quickly and intuitively understand their situation. The score consists of three parts: (1) During the experiment, the question-and-answer session about knowledge points. Set the score according to the number of times the user answers a question. (2) Operational behavior during the experiment. Set the score according to the user’s operational behavior. Users use intelligent beakers for the experiments. The intelligent beaker is equipped with a sensor that can sense the state of the beaker and detect the dumping function of the beaker. In the course of the experiment, two kinds of incorrect operations can be detected: the user dumping an empty beaker or dumping a breaker too quickly. Different path combinations correspond to different experimental results and experimental phenomena and are scored according to the path scores set in the game rules.

5. Experiment and Evaluations

5.1. System Settings

Our system is built using Unity 2018.3.8f1 (64-bit) version. Our computer is equipped with an Intel(R)Core(TM)i7-7700HQ CPU with a 2.80 GHz processor and 8 GB of memory. The computer’s operating system is Windows 10. The intelligent beakers are constructed with 3D printing technology. The development board uses the improved version of ATMEGA328P Pro Mini. The posture sensor is JY61 from WitMotion. The touch sensor is the TTP223 touch sensor module. The Bluetooth module is the model HC-05.

The intelligent beaker is shown in Figure 5. The beaker is made with 3D printing technology, and the beaker is small to facilitate user experimentation. The red mark is the posture sensor, the blue mark is the touch sensor, and the Bluetooth module is placed in the beaker. The Bluetooth module, posture sensor, and touch bed sensor are connected to the development board. The sensor information of the intelligent beaker is input to the computer through the Bluetooth module.

5.2. System Implementation

5.2.1. Laboratory Preparation of Carbon Dioxide Experiments

In the process of chemical experiments, there are problems such as the danger of the experimental process and unobvious experimental phenomena. A virtual and real fusion experiment can not only avoid the danger in the process of chemical experiments but can also use information enhancement technology to enhance the effect of chemical reactions to facilitate students in observing the phenomena. At the same time, the intelligent beaker (physical object) is used for interaction in the virtual and real fusion experiment to give students a sense of operating in a real environment.

The reaction between limestone and dilute hydrochloric acid is as follows: the limestone gradually dissolves, bubbles are generated, and a colorless gas is generated. Since we want to obtain pure carbon dioxide gas, impurities need to be removed from the gas, so we set up two experimental levels.

Due to the use of multipath strategies to increase the exploratory nature of chemical experiments during the gameplay virtual and real fusion experimental process, we show only the path of using limestone and dilute hydrochloric acid to produce carbon dioxide.

After the user selects the limestone and dilute hydrochloric acid experiment, he or she enters the first level of the experiment (Figure 6a), where the red mark is the user’s current level score, and the green mark is the user’s current level score ranking. The content is displayed after the user completes the experiment of the first level. As shown in Figure 6b, the yellow part is the experimental equipment for the reaction of limestone with dilute hydrochloric acid. The virtual experiment platform first broadcasts the current level task by voice, and then asks a question about the name of the chemical instrument indicated by the blue arrow marked by the red marker. As shown in Figure 6c, the green mark shows the chemical reagents that the current user can choose: limestone, marble, and sodium carbonate, and the blue part is the current score. The virtual reality fusion experiment platform uses the MIAUA algorithm to fuse the user’s touch and voice information to obtain the user’s true intention. After the user selects limestone, the chemical reagent selected in the green mark turns red. Next, the user can select the amount of limestone, as shown in Figure 6d. When the appropriate amount of limestone is selected, the amount selected in the green mark above the blackboard will turn red. At the same time, an appropriate amount of limestone can be seen in the green-marked conical flask. The blue mark in the upper right corner shows the change in the total score during the experiment. Throughout the experiment, the system reminds the user of the chemical experiment steps (the reminders refer only to the experimental process and do not stipulate which chemical reagent the user must choose) and provide feedback regarding whether the user’s answers to the questions are right or wrong. Users can obtain voice feedback during the operation of the experiment.

Next, as shown in Figure 7a, the user selects another chemical reagent, and the green mark is the optional chemical reagent (dilute hydrochloric acid or dilute sulfuric acid). After the user selects 20% dilute hydrochloric acid using the MIAUA algorithm, the red text reminder can be seen from the green mark on the blackboard. As shown in Figure 7b, the user uses the intelligent beaker to pour the dilute hydrochloric acid in the beaker in the virtual platform into the separatory funnel. At the same time, the liquid in the conical flask (marked in green) rises and the amount of dilute hydrochloric acid is added (the yellow mark). The reaction starts after the addition of dilute hydrochloric acid, as shown in Figure 7c, and the green and blue marks are experimental phenomena. At the beginning of the reaction, we use animation effects to zoom in the lens for the users to observe. The bubbles produced by the reaction can be observed on the limestone in the conical flask (marked in green), and bubbles can also be observed when the gas produced is passed into the clarified lime water (marked in blue). Here, information technology is used to enhance the chemical reaction phenomena, allowing users to observe experimental phenomena that are not clear in real experiments. In Figure 7d, users can observe the experimental phenomena after the reaction is complete. From the conical flask (marked in green) we can see that the limestone disappears, many white bubbles are produced, and the clarified lime water (marked in blue) becomes turbid. At the same time, we save pictures (yellow marks) of the experimental phenomena on the blackboard for users to observe the differences in the contrasting experimental phenomena during the exploration of the experiment.

When the user finishes the first-level experiment, as shown in Figure 8a, the red marked part is the user’s current score, the stars are lit according to the score, and the current score ranking can be seen in the green-marked part. The user then enters the second level, informed by voice broadcast that the current experimental task is to obtain pure carbon dioxide gas. Since the carbon dioxide gas produced by the reaction of limestone and dilute hydrochloric acid contains water and hydrogen chloride gas, saturated sodium bicarbonate and concentrated sulfuric acid are used to remove impurities, as shown in the blue mark in Figure 8b. To detect whether impurities have been removed, we use silver nitrate and anhydrous copper sulfate to check whether the solution contains hydrogen chloride and water (yellow mark). As shown in Figure 8c, after the user adds saturated sodium bicarbonate and concentrated sulfuric acid (marked in red) to beakers #1 and #2 in order, the inspection device (marked in blue) is covered by a mosaic. At this time, the system asks a knowledge question. After the user answers correctly, the experimental phenomena can be observed. As shown in Figure 8d, the experimental phenomena can be observed with the blue mark, and the final experimental results are presented on the blackboard.

The following will introduce possible user errors in the path of limestone and dilute hydrochloric acid. As shown in Figure 9a, incorrect operations may occur during the experiment, and the intelligent beaker can sense the user’s dumping behavior. If the user intends to pour chemical reagents into the beaker, the system will prompt the user that no chemical reagents have been added. The empty beaker marked in blue appears to have been dumped. During the user experiment, there may be a problem of the chemical reagents being added in the wrong order (Figure 9b). In the second stage of removing impurities, the gas should be introduced into the saturated sodium bicarbonate and then the concentrated sulfuric acid. If the selected order is wrong, the anhydrous copper sulfate will turn blue (blue mark).

Let us explore the influence of the amount of limestone and dilute hydrochloric acid on the chemical reaction during the reaction of limestone and dilute hydrochloric acid. We use the controlled variable method to present different experimental phenomena. First, we do not change the amount of limestone or the concentration of dilute hydrochloric acid. As shown in Figure 10a, the amount of diluted hydrochloric acid added at this time is estimated to be roughly 55. The reaction phenomenon is shown in Figure 10b. Compared with Figure 7c, we observe a conical flask (marked in green). The amount of dilute hydrochloric acid is small, and there are few bubbles during the reaction (green mark and blue mark). The experimental phenomena of the completion of the reaction (Figure 10c) are compared with those shown in Figure 7d. The liquid in the conical flask (marked in green) is lighter in color and produces fewer white bubbles. Next, we change only the concentration of dilute hydrochloric acid. The experimental phenomena of the completion of the reaction (Figure 10d) are compared with those shown in Figure 7d. The liquid in the conical flask (marked in green) is lighter in color and produces fewer white bubbles. There is more dilute hydrochloric acid in the conical flask in Figure 10d than in Figure 10c. The pictures of the experimental results are saved on the blackboard for comparison and observation. It is apparent that the high-concentration dilute hydrochloric acid reacts violently and there are many white bubbles on the liquid surface.

Next, we change only the amount of limestone, and choose a small amount of limestone and a large amount of limestone to compare the experimental phenomena. Figure 11a,b shows the reverse phenomena of a small amount of limestone. Figure 11c,d shows the reverse phenomenon of a large amount of limestone. The experimental results in the yellow marked area are in the following order: appropriate amount, small amount, and large amount. A large amount of limestone causes the most violent reaction in the conical flask: the most bubbles, the darkest color in the liquid, the most carbon dioxide, and the most turbid clarified lime water.

5.2.2. Reaction of Copper Sulfate with Sodium Hydroxide Solution

We added an additional chemical experiment to prove that our algorithms and intelligent devices are universal. The phenomenon of the reaction between copper sulfate solution and sodium hydroxide solution is as follows: the solution becomes colorless, blue precipitates are formed, and the blue precipitates turn black when heated. Since the reaction experiment between copper sulfate solution and sodium hydroxide solution is relatively simple, we set up a single experiment level.

In the experiment of the reaction between copper sulfate solution and sodium hydroxide solution, the user needs to mix two chemical reagents. As shown in Figure 12a, the blue markers are user-selectable chemical reagents: copper sulfate and sodium hydroxide. The user can choose to add copper sulfate or sodium hydroxide solution first. As shown in Figure 12b, the user chooses to add copper sulfate solution (blue solution) through the MIAUA algorithm. The green mark shows that the chemical reagent in the beaker is copper sulfate solution. The blue mark in Figure 12c shows that the user is pouring sodium hydroxide solution into the beaker containing copper sulfate solution. As shown by the blue mark in Figure 12d, the reaction beaker is blocked by a mosaic image.

The user needs to answer knowledge questions to observe the experimental phenomena. After the user has answered the question, he or she can see that the solution in the beaker in Figure 13a has become colorless, and a blue precipitate has been produced. Next, we explore what happens when the blue product is heated. As shown in Figure 13b, the user heats the blue precipitate. At the same time, the experimental phenomena are blocked (Figure 13c). After the user answers the question, he or she can observe that the blue precipitate in the solution has turned black (Figure 13d).

5.2.3. Experimental Prototype System

Our experiment system currently integrates 10 representative experiments. Users can enter the system, voice select the name of the experiment they want to perform, and the system automatically jumps to the experiment of our choice. The main interface of the prototype system is shown in Figure 14.

Figure 15 shows screenshots from other experimental procedures. As shown in Figure 15, (a) is the experiment of ammonia preparation; (b) is the experiment of investigating the properties of burning charcoal; (c) is the experiment of investigating the combustion of red phosphorus; and (d) is the experiment of investigating the properties of concentrated sulfuric acid.

5.3. Experimental Results and Analysis

We invited 15 graduate volunteers (never been exposed to these three chemistry lab platforms, computer majors, with less chemistry knowledge) to participate in the experiment. All volunteers were required to have no experience in conducting virtual chemistry experiments in GVRFL. We chose to let the volunteers complete the reaction experiment of limestone and dilute hydrochloric acid. First, we taught the volunteers the experimental process of this experiment. Then, we trained them in the NOBOOK, MSNVRFL, and GVRFL operation methods. The experimental process was divided into two parts: (1) the volunteers verified the feasibility of the algorithm and the platform in GVRFL. (2) Comparative experiments were conducted in NOBOOK, MSNVRFL, and GVRFL.

5.3.1. Algorithm Analysis

To verify the availability of the MIAUA algorithm and intelligent equipment, we first counted the success rates of seven operational intentions during the reaction experiment of limestone and dilute hydrochloric acid. We asked 15 volunteers to perform 20 experiments for each operational intention. The seven experimental operations were as follows: choose limestone (CL), choose a small amount of limestone (CSAL), choose an appropriate amount of limestone (CAAL), choose an excessive amount of limestone (CEL), choose dilute hydrochloric acid (CDHA), choose saturated hydrogen carbonate sodium (CSHCS), or choose concentrated sulfuric acid (CCSA). The volunteers used tactile information and voice information to conduct the experiments, and user intentions were obtained through the MIAUA algorithm. The experimental results are shown in Figure 16. The success rates of the 7 intentions using the MIAUA algorithm were all above 98.3%, and the average success rate was 99.01%. The success rate of CL, CDHA, and CCSA was 99.7%. Thus, the MIAUA algorithm can understand user intentions well and improve the efficiency of human–computer interaction.

After simple training, the 15 volunteers were asked to start the limestone and dilute hydrochloric acid reaction experiments, and we then counted the number of attempts made by the 15 volunteers to successfully complete the experiment. The results of the experiment are shown in Figure 17. Four students succeed in one experiment, accounting for 26.7%. Nine students succeed in two experiments, accounting for 60%. One student succeeded after trying three experiments, and the proportion was 13.3%. We observed the students who tried two to three times. Most of them made multiple attempts because they were not familiar with the virtual and real fusion experiment platform, which led to errors in the experimental process. A small part of the problem was that the system does not detect the user’s operation. Of the students who had not been exposed to the virtual reality fusion experimental platform, 86.7% successfully completed the reaction of limestone and dilute hydrochloric acid after two attempts. The experimental results reflect the ease of use of the intelligent beaker and the experimental platform.

After the experiment, the 15 volunteers were asked to answer questions for knowledge point inspection. The content of the test paper was the question-and-answer session from the experiment. During the experiment, there were a total of five questions and answers on knowledge points. The results of the experiment are shown in Figure 18. Of the respondents, 12 answered all the questions correctly, and 3 answered 4 questions correctly. Thus, 80% of the students answered all the questions correctly. The experimental results show that it is effective to incorporate game elements into GVRFL. In observing the experimental process, we found that the students were very engaged in the experimental process and were very interested in our experimental platform. The experimental results show that the introduction of gameplay can stimulate students’ interest in and motivation for the experimental process and improve their learning efficiency. Figure 19 shows the volunteers doing the experiment.

5.3.2. Comparison Experiment

At present, some mature virtual experiment platforms are being promoted in schools. We chose the NOBOOK experimental platform for comparison. GVRFL uses the fusion of virtual and real for chemical experiments, and we chose MSNVRFL for comparison. We compared the three experiments on six aspects. In order to ensure the uniformity of the experimental variables, we selected 15 students who had never been exposed to the GVRFL, NOBOOK, and MSNVRFL experimental platforms to perform the experiments and fill out the forms. Finally, we conducted the statistics and concluded that the platform could achieve this basic performance if more than 75% of the students thought that the platform had this performance; otherwise, they thought that the platform could not achieve this performance. The final results are shown in

Table 2. Performance comparison.

Virtual Laboratory	NOBOOK	MSNVRFL	GVRFL
Interactivity	$\surd$	$\surd$	$\surd$
Operability	$\times$	$\surd$	$\surd$
Effect observation	$\surd$	$\surd$	$\surd$
Intelligence	$\times$	$\surd$	$\surd$
Inquiry	$\surd$	$\surd$	$\surd$
Interestingness	$\times$	$\times$	$\surd$

, $\surd$means that the basic performance can be achieved and $\times$ means that the basic performance cannot be achieved. In terms of interactivity, all three experimental platforms can conduct virtual experiments through interaction. In terms of operability, NOBOOK uses a mouse or finger to drag to complete the experiment, and therefore does not create the sense of operation of a traditional experiment. MSNVRFL and GVRFL use intelligent beakers for experiments and thus have a certain sense of real experimental operation. In terms of effect observation, all three experimental platforms can observe the effect of chemical reaction experiments, and the NOBOOK simulation effect is relatively better. In terms of intelligence, NOBOOK is not an intelligent platform. In terms of inquiry, all three experimental platforms can carry out exploratory experiments. In terms of interestingness, GVRFL adds game elements while teaching knowledge, which improves students’ interest in learning during the experiment.

GVRFL uses multichannel interaction to conduct virtual experiments. The MIAUA algorithm combines tactile information and voice information to obtain user intentions. Some articles also proposed a multimodal fusion algorithm (MFPA) that fused tactile information and voice information and tested the success rate of two intentions in the experimental part: choosing concentrated sulfuric acid (CCSA) and choosing water (CW). We used the MIAUA algorithm to test these two intentions. In addition, we used the MFPA algorithm to test operational intention during the limestone and dilute hydrochloric acid experiment: CL, CSAL, CAAL, CEL, CDHA, and CSHCS. The volunteers performed each experiment 20 times. The results are shown in Figure 20, which shows that the correct rate of intention understanding of the MIAUA algorithm is higher than that of MFPA. The success rate of CCSA using the MIAUA algorithm is 99.7%, and the success rate of CW is 100%. The average correctness rate of MFPA’s intention understanding is 95.91%, the average correct rate of MIAUA’s intention understanding is 99.13%, and MIAUA’s correctness improves by 3.22% because the MIAUA algorithm has an active understanding process. This method will interact according to the user’s input information, request the user to enhance or supplement the information of a certain channel when the user’s true intention is unclear, and obtain the user’s true intention through multiple interactions.

Next, we invited the volunteers to score GVRFL. The indicators are interactivity, operability, effect observation, intelligence, inquiry, and interestingness. The statistical results are shown in Figure 21. GVRFL scores higher than NOBOOK and MSNVRFL in terms of interactivity. The MIAUA algorithm can better understand the real intentions of users and improve the interactivity of the experimental platform. In terms of operability and intelligence, GVRFL scores much higher than NOBOOK and basically the same as MSNVRFL. In terms of effect observation, GVRFL scores slightly higher than MSNVRFL and lower than NOBOOK. The presentation of experimental phenomena needs to be optimized. In terms of inquiry, the three experimental platforms are basically the same. In terms of interestingness, GVRFL is much higher than NOBOOK and MSNVRFL because GVRFL adds game elements and students are more engaged in the experimental process, which improves the fun of the virtual experiments.

Meanwhile, we conducted a single-factor comparative analysis of the ratings of these 15 students on these 3 platforms, and the data were analyzed separately for interactivity, operability, effectiveness observation, intelligence, inquiry, and interestingness using ANOVA analysis, and the results are shown in

Table 3. Single-factor analysis of user evaluation.

	GVRFL and NOBOOK		GVRFL and MSNVRFL
	p	S	p	S
Interactivity	<0.001	Y	0.0011	Y
Operability	<0.001	Y	0.2136	N
Effect	0.1546	N	0.1987	N
Intelligence	<0.001	Y	0.2045	N
Inquiry	0.2315	N	0.2513	N
Interestingness	<0.001	Y	<0.001	Y

, Where, Y means significant difference, and N means relatively small significant difference. It can be seen that the scores of GVRFL are significantly higher than NOBOOK in the aspects of interactivity, operability, intelligence, and interestingness, and the scores of GVRFL are significantly higher than MSNVRFL in the aspects of interactivity and interestingness. This shows that our system has more advantages in terms of interactivity and interestingness for the experimenter, which also verifies that our proposed multimodal fusion intention understanding algorithm is more instructive for the user’s operation during the experiment and allows the user to interact with the system in a more natural way. In addition, the game mode we designed also increases the fun of the experiment and improves users’ interest in learning.

5.3.3. User Study

We used the SUS questionnaire (Appendix A) to evaluate GVRFL. Due to the specificity of the application of the smart experiment system, we summarized and improved the assessment content of the SUS scale to make it more relevant to the user’s assessment of the application needs of the system. Therefore, in addition to the usability and ease of operation assessments previously included in the scale, we also added fun and experimental immersion assessments. The questionnaire contains 10 questions, and each question is scored using a 5-point system, with 1 meaning disagree and 5 meaning strongly agree.

The results of the experiment are shown in Figure 22. The mean scores for questions 1 and 2 were 4.42 and 4.95, respectively, indicating that students preferred to use GVRFL. The mean score for question 3 was 4.37. Students felt that using GVRFL would help them focus on learning chemistry. The average score for question 4 was 4.31. Students felt that the combination of virtual and realistic manipulation of the smart beaker would improve their hands-on skills. The average score for question 5 was 4.35. The students felt that they were highly engaged in the experiment. Therefore, the addition of game elements to the virtual experiment can increase students’ interest and motivation to learn. The mean score for question 6 was 4.32. Students thought that the use of GVRFL for chemistry experiments was better than swiping the mouse on the computer. The average score for question 7 was 4.39. Some students still prefer to conduct experiments in a traditional laboratory, but most of them believe that the use of GVRFL is effective in avoiding the dangers of traditional experimental procedures and observing less obvious phenomena. The average score for question 8 was 4.71, indicating that the system is already well integrated with its functions. The average score of question 9 was 4.65, which indicates that users think the system is still easy to get started and use, and will not be too much of a burden to use. The average score of question 10 was 4.53, indicating that users engaged in using the system with confidence that the system’s intent-understanding algorithm could help students overcome the difficulties encountered during the experiment and complete it successfully.

Finally, according to NASA-TLX, all participants were divided into mental demand (MD), physical demand (PD), performance (P), effort (E), and frustration (F). The mental requirements describe the user’s operational memory load, the physical requirements describe the user’s effortlessness in the operation, the operational performance and the smoothness of the user’s operation, and the effort level describes whether the user feels relaxed during the operation. The degree of frustration describes the users’ degree of negativity during the operation. NASAs evaluation indicators use a 5-point system, and each indicator is divided into 5 levels: 0~1 points indicates a low cognitive burden, 1~2 points indicates a relatively small cognitive burden, 2~3 points indicates that the cognitive burden is average, 3~4 points indicates that the cognitive burden is relatively large, and 4~5 points indicates a large cognitive burden. Figure 23 shows that the score of GVRFL is generally lower than that of NOBOOK. GVRFL scores are slightly higher than MSNVRFL scores in MD, and for the other four indicators, GVRFL scores are generally lower than MSNVRFL scores. The results indicate that the user’s sense of frustration and fatigue is very low during the experimental process of GVRFL. The experimental process is relatively smooth and relaxed, the user’s cognitive load is low, and the user has a high evaluation.

At the same time, we further analyzed the NASA measurement results of the 15 students, using ANOVA analysis, and carried out data analysis on the five indicators of GVRFL and NOBOOK and GVRFL and MSNVRFL, respectively, and the results are shown in

Table 4. Single factor analysis of NASA user evaluation.

	GVRFL and NOBOOK		GVRFL and MSNVRFL
	p	S	p	S
MD	<0.001	Y	0.0871	N
PD	<0.001	Y	0.2077	N
P	<0.001	Y	<0.001	Y
E	<0.001	Y	0.0015	Y
F	<0.001	Y	<0.001	Y

, Where, Y means significant difference, and N means relatively small significant difference. It can be seen that GVRFL has a significant reduction in five indicators in MD, PD, P, E, and F compared to NOBOOK, and the changes in two indicators in MD and PD are not very significant compared to MSNVRFL, but there is also a significant reduction for the other three indicators. It indicates that through our active intention understanding algorithm, the active guidance of the system to the operator can effectively reduce the user’s brain power and cognitive conformation while the added game elements can also improve the user’s pleasure during the experiment and reduce the degree of psychological frustration, thus increasing the students’ enthusiasm for learning.

6. Conclusions and Prospects

The existing virtual laboratory has low interaction efficiency, the user experience is poor, and the experimental process lacks interest and realism. We designed a multimodal perception gameplay virtual and real fusion intelligence laboratory (GVRFL). This scheme proposes a multimodal intention active understanding algorithm (MIAUA) to improve the efficiency of human–computer interaction and user experience, proposes a novel gameplay intelligent experimental mode of virtual and real fusion, and introduces gameplay into the experimental interactive mode of virtual and real fusion.

The proposed MIAUA gives GVRFL the ability to understand user intentions and improves the efficiency of human–computer interaction. At the same time, the experimental method of fusion of virtual and reality enhances the user’s sense of reality and operation. The introduction of game elements into the virtual experiment stimulated students’ interest and motivation in the experimental process, increased their interest in the virtual experiment, and improved their learning efficiency. The experimental results show that students’ frustration and fatigue are relatively low during the experiment, which reduces the burden of human–computer interaction and simultaneously increases students’ interest in learning. Students like to use GVRFL for chemical experiments. Compared with the cost problem of traditional experiments, our equipment uses inexpensive sensors, which will not cause too much burden on the school and can be used repeatedly to save resources and avoid unnecessary waste. In the future, we will continue to optimize the presentation of the experimental reaction effects and add more chemical experiments.