Serious Game for VR Road Crossing in Special Needs Education

Abstract

Autism spectrum disorder (ASD) is a developmental disability that can impair communication, social skills, living skills, and learning capabilities. Learning approaches usually differ between mainstream schools and special needs schools, to cater for the different learning processes of children with ASD. Besides the traditional classroom-based education, alternative technology and methods are explored for special needs education. One method is to train children with ASD using Virtual Reality (VR) technologies. Many prior works show the effectiveness of VR-based learning with varying degrees of success. Some children with ASD may face challenges to gain independent living skills. Their parents or guardians have to expend a significant amount of effort in taking care of children with ASD. It will be very helpful if they can have a learning opportunity to gain such living skills. In this research, we develop a VR serious game to train children with ASD one of the basic living skills for road crossing safely. The VR serious game operates on multiple types of platforms, with various user interaction inputs including the Microsoft Kinect sensor, keyboard, mouse, and touch screen. The game design and methodology will be described in this paper. Experiments have been conducted to evaluate the learning effectiveness of the road crossing game, with very positive results achieved in the quiz and survey questionnaire after the gameplay.

1. Introduction

Autism spectrum disorder (ASD) is referred to as a range of conditions [1]. Children with ASD may be associated with a wide spectrum of symptoms. While some need less help in their daily lives, others need more. For those with medium and severe symptoms, much attention is needed to care for them [2]. These children encounter more difficulties in the training and learning of living skills, including daily activities and basic living skills. They need more help and support in their learning journey. There are some treatments and learning technologies that can improve children’s development. For example, special needs education is conducted by professional educators to train children with ASD. Traditional classroom education and textbooks guide their learning. Occupational therapy (OT) is referred to as the art and science of helping people do day-to-day activities. It is important and meaningful to their health and well-being through engagement in valued occupations [3]. These traditional methods have a certain level of effectiveness with positive results varying from person to person.

Nowadays, special needs schools have provided more technology-enhanced learning approaches using tablets or iPads. Tablets and iPads are mobile platforms with powerful computation capabilities. Their touch screen user interfaces provide great convenience to users. They are easy to operate for users of various age groups, including children with ASD. Many teaching materials in the form of mobile applications (apps) running on tablets or iPads have been adopted in special needs schools. Serious games are educational and training games to embed the learning contents into the gamifications. Such serious games can be designed as mobile apps. Mobile phones have also become more popular. The mobile phone used to be for communication purposes between people. Now, it functions as a computer. Some teaching material apps for tablets or iPads are also able to run on mobile phones. These mobile platforms enable users to learn in more interactive ways.

Learning living skills is important to every child, as it develops their independent capabilities to take care of themselves. Usually, the living skills training of children is provided starting from kindergarten and primary school. Special needs children may have short attention spans in learning [4]. It requires different educational approaches to better help their learning of living skills. Virtual reality (VR) has been reported as an effective technology-enhanced learning approach [5,6,7]. However, there is a learning curve and long adaptation time for children with special needs, due to the different learning experience and environment compared to the real learning environment. Sometimes, children with special needs may feel that correlating the virtual and real scenes is confusing. Players may also need to memorize the control instructions and methods in the gameplay. It is not an easy task to understand and memorize these controls for them. Some help and guidance from teachers or parents is required in playing VR serious games. Good design of user interfaces and user experiences is needed to be more engaging and attractive to children.

The co-existence of pedestrians and cars is a common daily scenario. Road safety is necessary and important to every person, with the increasing number of vehicles in most countries. Road crossing skill is one of the important daily living skills to be gained for children who live in cities and towns. Such skills can be trained in real road junctions with the help of parents or guardians of children, as children are able to learn from the observations. However, it may present a challenge for children with special needs, as they may need longer training and practice time. This means more effort and time are required for their parents or guardians to accompany them in training at real road junctions. It will be very helpful if there is a way to mimic the road junctions in a realistic way, such that children with special needs are then able to have learning opportunities to try and practice road crossing by themselves in a safe environment, before accompanying them for road crossing in the real world. A VR serious game to train road crossings is a good candidate to help children with special needs to learn in a controlled environment without safety concern.

There are two types of traffic systems in the world: left-hand and right-hand traffic systems. For example, the UK and Singapore adopt the left-hand traffic system, while United States and China adopt the right-hand traffic system. Children with special needs in different countries will learn the traffic systems differently. In prior work [8], a road crossing serial game was reported for special education with the left-hand traffic system. It would be beneficial and attract players from different countries if the serious game can simulate both left-hand and right-hand traffic systems.

The objective of this research is to explore the metaverse applications in special needs education fields, to train road crossing skills, as a part of living skills for children with special needs such as ASD. It aims to address the challenges of the VR serious games on road crossing training mentioned in the previous three paragraphs. It aims to provide more engaging features, functionality, multi-modals models, and operating on multiple computation platforms for the metaverse application in special needs education. Unity3D tool will be used to create the game scenes with multiple input devices for user interactions with the virtual game world, including a Microsoft Kinect motion sensor, keyboard, mouse, and touch screen. The VR serious game can be operated on multiple platforms such as PCs, laptops, tablets, iPads, and mobile phones. It will provide multiple options for special needs education. The components of stealth assessment which incorporates performance assessment features in VR serious games can help players to measure their learning progression toward the goals [9]. The developed road-crossing serious game in this paper embeds the scoring system, to help players benchmark their performances in the serious games.

The main contributions of the paper are as follows:

• The learning effectiveness of metaverse application on special needs education is studied. An educational VR serious game is analyzed to train the living skills of children with special needs. It simulates real-world traffic junctions to give children realistic learning environments for road crossing.
• Multiple electronic sensing devices and various computation platforms are provided in the VR serious game, which train children in different sensory systems, including arms with gesture interactions, fingers with touch screen interaction, and hands with keyboard and mouse interactions.
• The VR serious game supports both left-handed and right-handed traffic systems. It has flexible in training children from countries with different traffic rules. To the best of our knowledge, it is the first road crossing VR serious game for special needs education which supports both types of traffic systems.

The remaining parts of the paper are organized as follows. Section 2 provides a brief discussion of related works. Section 3 describes the game flow and game design. Preliminary experiments have been conducted and discussed in Section 4. Conclusions are discussed in Section 5.

2. Related Works

Under the scope of applications in Metaverse and Digital Twins, VR for Education is one of the technology enhanced approaches for learning and scholarship research. It provides a useful alternative type of education with better visualization and immersive experiences. The supports for multiple sensory inputs, including 3D rendering display, visual outputs, auditory instructions, etc., will be positive to technology enhanced special needs education.

A successful development of educational VR serious games involves both software and hardware electronic devices. The design of educational content and pedagogies embedded in VR serious games mainly relies on software tools, including Unity3D, Blender, Photoshop, etc., while user interactions, sensing, and user interfaces of VR serious games are provided by electronic devices and sensors. Both designs in software and choices of electronic devices play important roles. With the current technology advancement in high performance computation devices such as graphics processing unit (GPU) at affordable prices, software processing and 3D graphics rendering have become smoother and more realistic. There are multiple types of electronic hardware input/output devices of metaverse applications, including Cave Automatic Virtual Environment (CAVE), PC-connected head-mounted display (HMD), standalone HMD, PC, tablets, iPad, mobile phones, etc. Each of these approaches has its own advantages and drawbacks. CAVE is room-based immersive environment with multiple projectors and surrounding wall display. However, it is usually very expensive. Nowadays, the quality improvement of HMD allows better immersive visual experiences [10]. PC-connected HMD devices, such as HTC Vive PRO used in [11], have good computational power drawn from the PC. However, there are some cables connecting between the HMD and the PC, which is inconvenient in use. Standalone HMD devices, such as Oculus Quest 2, is within the affordable price ranges of $200–$400. The downside is the battery life, and some players may suffer 3D sickness. To address the 3D sickness issue, a workaround strategy is to operate the 2D version games on PC and laptops. With the tablet, iPad, and mobile phones being widely equipped by end users, many serious games running on these platforms are popular and reported in the literature. For example, a 3D visualization application is introduced to display node-link diagrams with mobile phones [12]. The effectiveness of four different VR/AR approaches including operating on HMD and mobile devices in Management Education are studied and compared [13].

In this section, the prior works in the literature related to VR serious games will be discussed.

2.1. VR Effectiveness in Treating ASD

VR is supported by high-performance computer graphics, hardware, and graphics rendering to create a simulated environment [14]. VR has been used in many fields including education and training, with limitless potential in various applications [15]. In a study in which a group of new managers participated in the same training [16], they were divided into three groups: the first group took the classroom-based approach, the second group took the e-learning approach, and the last group took the VR approach. The results show that VR learners achieved the best training performance compared to the classroom-based and e-learning approaches. Other related works on VR treatment for children with special needs for vocational training and communication skills training are reported in [4,5].

The effectiveness of using VR as a method for treating ASD has been analyzed in prior works. The potential health risks have been also studied, which are associated with prolonged use of wearing the VR headset. According to a systematic review conducted by Mesa-Gresa et al. across 31 studies [17], it is reported that VR-based treatments mostly have a positive effect on children with ASD. Out of the 31 studies, only one was found to have inconclusive results. There is a study to learn nonverbal communication through hand gestures with virtual dolphins, which provides another way of training children with ASD [18].

The work reported in [19] shows that the VR-based training is effective among young children. The study is conducted using VR to teach children with ASD air travel skills. The device used for the project is Google Cardboard with relatively low cost. It shows that VR technologies have large price ranges. It could be built from a low cost device to high-end equipment. The prior results show that most children obtain improvements through the VR game learning. The effects of VR-based exercises are presented in 15 types of studies on physiological, psychological, and rehabilitative analysis in various users [20]. Compared to traditional approaches, it indicates that VR-based learning has potentially exerted positive outcomes and impacts. A study is conducted with 15 randomized controlled research with a total of 816 learners [21]. It shows that VR has the potential to improve the teaching effectiveness of anatomy when compared to traditional methods.

With the evidence above, it is demonstrated that VR-based training and learning are beneficial, although learning ability differs from child to child. Overall, it shows the potential of VR approaches in various training and learning programs.

2.2. VR Serious Games for Training of Road Crossing Skills

There are certain risks of experiencing an injury at road crossings for young children of school age [22]. Virtual learning environments are developed for road safety education to teach children about traffic lights, safe cross codes, virtual objects, and avatars in traffic scenes, using the HMD and gaze tracking interactions [23]. Several mobile applications of VR road crossing games are available in apps stores. For example, an Android mobile app named Cross the Road 3D supports both VR and non-VR modes [24]. A Street Champ VR game chargeables at $0.99 is operated on Oculus Rift platform [25]. A game, Street Ranger VR road crossing, at $0.99 is presented supporting both iOS and Android platforms [26]. A road safety VR game is introduced where players act as a traffic officer to coordinate traffic when the traffic light is malfunctioning [27]. A game, VR Real Traffic Road Crossing for Virtual Glasses is developed for players to cross the road without getting hit by cars [28]. These games do not target only child players for road crossing learning.

A virtual environment is presented for road safety training to students in primary and middle-level education, with the Kinect sensor as the user interaction interface [29]. It uses the projector display on a living room setup using a Kinect sensor, or a simple desktop PC display using a keyboard and mouse. A virtual simulation is developed to train young children in primary schools to correctly cross a road, abide by traffic lights rules, learning road signs, and safely ride a bicycle [30]. A virtual environment is adopted to assess how children cross the road and analyze potential risks [22]. A simulated environment with VR is presented to teach road safety knowledge to children aged 7–9 years in UK schools, which is an iPad-based game to improve their behavior in real traffic environments [31]. However, these road crossing serious games are not designed and catered for special needs education.

The effectiveness of VR based therapeutic interventions for individuals with ASD is discussed with various games, including the street crossing games [32]. A 3D VR serious game is introduced to train children with ASD with video instructions embedded by special needs teachers [8]. A desktop street-crossing virtual environment is designed to examine the capability of children with ASD in learning skills to cross a street safely [33]. An immersive virtual reality game is developed on an HMD platform using Arabic language to help children with ASD improve their street-crossing skills [34]. With an immersive VR safety street crossing training environment, a study is conducted to evaluate the learning effectiveness in teaching three children with ASD in the experiments [35]. A virtual city environment with the Kinect input device for street crossing is used to be explored with seven adult participants in the experiment [36].

2.3. Health Risks and Side Effects of Using VR Headsets

The physical, cognitive, and psychosocial development of children may be affected either negatively or positively when experiencing VR applications at different ages and stages [37], with the main concern being the overuse of screens and HMD. Although some children with ASD enjoy the VR experiences with HMD [38], there are a couple of health risks associated using VR headsets. Such risks include loss of spatial awareness, dizziness, seizure, and nausea [39]. Once players have delved into the virtual world in VR, they will be immersed in it. With the headset on, the players may not be aware of their surroundings. If they are not in a safe environment, there is a chance they will trip on something or hit a wall, causing injury. Dizziness which is caused by motion sickness occurs differently among players. Some might not experience it. It will be beneficial to provide multiple user interfaces and options to players, to reduce the negative impact of the side effects.

3. Game Flow and Design

In this research, a VR serious game was designed to provide the training of road crossing skills to children with ASD in both left-hand and right-hand traffic systems. The designed VR serious game supports multiple platforms and user interaction methods, including the Kinect sensor, touch screen, keyboard, and mouse. The Kinect sensor will be used to detect gestures produced by the player. It is used as the primary mode of user interactions. In the serious game, children will learn the traffic rules and how to cross the road safely. Road scenarios include traffic light crossing and zebra crossing. There are tutorial sessions where children will be taught the basics of the traffic rules, such as Do Not Cross when the pedestrian light is red or blinking green, and Only Cross when the pedestrian light turns green, etc. After completing the tutorials, children will proceed to play the serious game under four game levels of traffic scenarios with limited help. The hints will be given if children are taking too long to complete the tasks.

Unity3D tool will be used for the development of the serious game. It is a game engine with a software development environment. Unity3D has a user-friendly development environment and good community support. Unity has a wide range of 3D objects such as the cube, sphere, cylinder, etc. Even though it may not have sophisticated features like other 3D modeling software such as Blender, Unity does its job decently for the game design. Game objects and game scenes are developed in Unity3D. Unity3D allows scripts that are written in C# to be attached to game objects. In this section, the road traffic systems and various game scenes will be presented, including traffic lights, background scenes, moving vehicles, avatars, control flows, and interactions.

3.1. Traffic Light System

The traffic light system is the main aspect of the VR serious game. In depicting a real traffic light system, the sequence goes as follows:

(1)

green light for vehicles (when red pedestrian light)

(2)

orange light for vehicles (when red pedestrian light)

(3)

red light for vehicles (when green pedestrian light)

(4)

red light for vehicles (when blinking green pedestrian light)

(5)

iterative from (1) to (4) again.

The timing of each traffic light differs from one another depending on the road junctions and other factors such as the level of traffic and pedestrians. For the design of the traffic light system in this paper, the timing for vehicles from the green light to the orange light is set as 20 s, and 3 s from the orange light to the red light is set. At most road junctions in the Singapore context, when pedestrians cross a road junction, they need to press the button located at the traffic light post first. After the button is pressed, the pedestrian lights will change to the green man according to the traffic light settings. The timing from the green pedestrian light to the blinking green pedestrian light is 20 s, and from the blinking green pedestrian light to the red pedestrian light is 15 s. The total time to the road crossing of pedestrians is 35 s.

The reason for using such timings is partly due to the design of the level scenario. For each level, there will be instructional captions and audios to guide the player through. The timing of 15 s to 20 s should be sufficient for the player to follow. Another issue to consider is the timer from the green light to the red light. Usually, it is much longer in a real traffic light system (1 to 3 min). However, the waiting time is too long in a serious game. To give a better depiction of the traffic light system, the flowchart for its behavior is shown in Figure 1.

The design of the traffic light is created in Unity3D, using 3D objects such as the cube, cylinder, and sphere. The green man and red man are drawn using Microsoft Paint. These 3D objects, green man, and red man are then pieced together into the designed traffic light game object, as shown in Figure 2. It has four main components consisting of the vertical pole, pedestrian light, pedestrian button, and traffic light. Subcomponents for the pedestrian light and the traffic light include the rectangular boxes and the spherical objects attached representing the traffic indicators (red man, green light, etc.). Multiple light sources including the green light, orange light, and red light are incorporated into the game. Each respective colored light source corresponds to its traffic light panel. Lights will be switched on and off depending on the status of the traffic light.

3.2. Design of Game Scenes

Game objects are placed across the map around the game scenes. The purpose is to beautify the game scenery to mimic the real world. Three-dimensional models of high-rise apartments are added in the game scenes. The buildings are created in Unity3D using countless blocks (cubes) being placed and joined together. Greenery is added into the game and trees are planted on the side of the road in the background complementing the overall scene.

There are three main game scenes: road crossing with traffic lights, zebra crossing, and combining traffic lights and zebra crossing.

3.2.1. Traffic Light Crossing Scene

A screenshot of the traffic light crossing scenes in the game is shown in Figure 3. Moving cars are included in the traffic light game scenes with both oncoming and outgoing lanes. Depending on the state of the traffic light, cars will behave in a certain way. During a green light, cars will move, while during a red light, cars will stop. To simulate the real-life traffic light scenario, cars will honk when the traffic light is green while the pedestrian is still on the road (indicating it is dangerous to be on the road). Its flow chart is shown in Figure 4.

3.2.2. Zebra Crossing Scene

Besides the usual traffic light crossing, there is another type of crossing called the zebra crossing. Differing from traffic light crossing, the pedestrian has the right-of-way to cross the road ahead of the motorists. Motorists are required to stop when a pedestrian is waiting to cross the road, although there is no traffic light. Due to its black and white stripes resembling a zebra’s coating, it is called a zebra crossing. A game scene of a zebra crossing is shown in Figure 5.

Compared with the traffic light crossing, the zebra crossing is simpler to design. Walk towards the zebra crossing, wait for the car to stop, and cross. These are the basic steps to cross a zebra crossing. One might ask the question of how to trigger the car to stop? In designing this level, a panel is added on the ground covering the zebra crossing which is invisible with the mesh off. Once the player steps on the panel, it will trigger cars to prepare to stop. Once all cars stop in front of the zebra crossing, it allows the player to start the road crossing. The flowchart of car behaviors in the zebra crossing is shown in Figure 6.

3.2.3. Variation of Car Behaviors in Game Scenes

In reality, it is not uncommon to see irresponsible motorists not stop and give way to the pedestrian at the zebra crossing. As such, we shall add random behaviors for cars that ignore traffic rules. The CarBadBehavior() functions are added in the game scenes for car random bad behaviors for not stopping and giving way to pedestrians at road junctions.

The random number generator is utilized to obtain a random number in different game scenes. The idea is to set a 20% probability rate for the car to have the bad behavior trait. If the car rolls a “0” or “1”, it will ignore traffic rules. However, if the car rolled between “2” to “9”, it would stop for the player. The dice roll occurs every time the car resets its position. In a way, this gives randomness to the gameplay as the behavior of the car is different each time. Lastly, the CarGoodBehavior() function is the same code from the previous section, except that I have changed the name to better reflect the overall code.

3.2.4. Player Controls with First-Person and Third-Person View

In most game scenes of the original design plan, an avatar would move in four directions—forward, backward, left, and right, using the key “W/A/S/D or Up/Down/Left/Right”. For most players, there would be no issue controlling the avatars in this way. However, for some special needs children, this might pose an issue. They might not know how to corelate the directions with the controls by keyboards. To simplify the player controls, the number of inputs is reduced. Only the mouse or touch screen is required to move the avatars. Right mouse button to move the player forward, left mouse button to interact with the game objects, and mouse movement to look left or right. Players can also use the Microsoft Kinect sensor and gesture to control the avatars of the serious game.

Furthermore, the control with the touch screen is also designed for operating the serious game on the tablet or iPad platform. According to the feedback of our collaborating special needs schools, learning with the iPad platform has been adopted in the curriculum for the special needs children. The children are familiar with the control and interaction using the touch screen of the iPad. As such, the serious game is improved to support the touch screen graphical user interface (GUI).

By default, the serious game is played from the first-person perspective. Some players might prefer to be in third-person view. To accommodate this, there is a choice in the serious game to switch between first-person and third-person, and vice versa.

3.3. Design of Game Levels

The serious game supports both left-hand and right-hand traffic systems. Some countries, such as the United States, use the right-hand driving system, while other countries, such as Singapore, adopt the left-hand driving system. This option allows the player to choose which system they are familiar with. The player can change the setting in the menu of Controls. By default, the left-hand driving system is selected, as shown in Figure 7.

There are two modes of gameplay in the serious game. One is the “Tutorials” mode where the player will follow the onscreen instructions to cross the road. Another is the “Play” mode where the player will cross the road on their own without any instructional assistance. The entire game flow of the serious game is shown in Figure 8.

In the “Tutorials” mode, there are four levels as follows.

• Level 1 is a traffic light scene, where the player starts with red pedestrian light on.
• Level 2 is a traffic light scene as well, but the player starts with a blinking green man light on.
• Level 3 is the zebra crossing scene.
• Level 4 is a combination of zebra crossing and traffic light crossing.

The objective of all these four levels is to cross over to the other side of the road. There will be instructional captions and audio to guide the player if he or she is unfamiliar with the laws of the traffic. To further simulate a real traffic crossing, ambient noises such as car engine throttling and cars zooming past the traffic are added to the game. Figure 9 shows the screen when the “Tutorials” mode is selected. The “Level Select” screen will show four different levels. Pressing “Enter” for each of the levels will bring you to the tutorial section of that level.

In the “Play” mode, the player will practice the road crossing through all four levels as mentioned above. An example scene at Level 4, i.e., the combination of zebra crossing and traffic light crossing in the serious game is shown in Figure 10.

3.4. Score or Points System

A scoring system or points system is implemented in this game mode. The player starts with 10 points. Points are carried over through all four levels. Point(s) is/are deducted whenever the player breaks traffic rules. The objective is to train players to follow traffic rules and cross the road safely through all four levels. The points system is implemented according to the actions as follows.

(1)

Traffic Light Crossing

(i)

Deduct 1 point when starting to cross with the blinking green man light.

(ii)

No point will be deducted when the avatar is halfway through crossing the road while the green man light starts to blink.

(iii)

Deduct 2 points when crossing during the red light.

(iv)

Deduct 2 points every 5 s if the avatar is still on the road.

(v)

Deduct 5 points when a car crashes onto the avatar. The avatar will respawn back to the starting point if hit by a car.

(2)

Zebra Crossing

(i)

Deduct 1 point for crossing recklessly, if the avatar does not wait for cars to stop.

(ii)

Deduct 5 points when a car crashes into the avatar. The avatar will respawn back to the starting point if hit by a car.

At the end of the gameplay, a rating is given to the player on how well the player performs in the serious game.

3.5. Gesture Control with Microsoft Kinect Sensor

In the developed serious game, there are multiple types of GUI for human–machine interactions, including the keyboard, mouse, touch screen, and gesture recognition using Microsoft Kinect sensor. Kinect is a motion-sensing device that is developed by Microsoft. To be able to use Kinect on computers, the Kinect software development toolkit (SDK) needs to be installed first. Next, the Kinect SDK can be imported into the Unity Asset Store for the serious game. It allows the Kinect to detect and track remotely the joints of the player. The joints of the body are tracked and are outputted in a 3D-coordinate system, shown in Figure 11. When the player faces the Kinect sensor, the x-axis is the horizontal axis, the y-axis is the vertical axis, and the z-axis is the axis pointing towards the camera.

There is a total of 20 joints that are tracked by the Kinect, such as hands, elbows, shoulders, knees, etc. For this project, only hand motion and gesture movements will be used to interact with the avatars in the serious game. There is a total of eight gestures to control the serious game, configured as follows, which is also shown in Figure 12, captured from the camera point of view.

• Right hand pointing to the right (Turn right or Next Selection)
• Left hand pointing to the left (Turn left or Previous selection)
• Raise right hand (Move forward or Confirm selection)
• Raise left hand (Move backward)
• Right hand forward (Press button on traffic light posts)
• Left hand forward (Toggle between First-Person and Third-Person view)
• Right hand on the waist (Toggle instructions for game controls)
• Left hand on the waist (Toggle instructions for gestures)

As there are eight different hand gesture inputs, some children might not be able to remember them all in the gameplay. Therefore, the instructions for gesture control commands are added in the game and can be toggled on and off as players wish.

4. Experiments

With the completion of the road crossing serious game, preliminary experiments are conducted to evaluate the effectiveness of learning road crossing for children. According to the experiment plan, young participants at a special needs school will be trained using the serious game of learning traffic rules and road crossing. However, the experiment plan was not executed successfully, due to the tightening of school activity policies, which was caused by the COVID-19 pandemic. For safety considerations, group learning activities in schools in Singapore have been carefully restricted and external visitors are not allowed in schools.

After several months waiting while the school activity policies have not been revised, and we had to change to an alternative experiment plan. Eventually, only a small group with five students ranging from 5 to 12 years old participating from other schools were engaged individually at different times in the preliminary experiments.

Each participant played the serious game, with any of the interaction inputs from the Kinect, keyboard, mouse, and touch screen. Each participant first took turns to go through the four levels of the game in the Tutorial mode to learn the rules and controls of the serious game individually. Each participant then started to practice in the Play mode without instructions being provided.

After the gameplay, each participant was tested using some simple quiz questions to evaluate their learning. These quiz questions were crafted by incorporating the comments from two teachers at a special needs school, Katong School, Singapore, where quiz questions were specifically designed in a simple way for easy understandability. The quiz questions are shown in

Table 1. Example quiz questions for participants.

No	Questions
1	What are the steps that you need to do BEFORE crossing the road?
	(A) Stop look and talk.	(B) Close your eyes.
	(C) Stop, look both sides before crossing.	(D) Keep walking.
2	What must you do when you are crossing the road?
	(A) Run.	(B) Listen to music.
	(C) Talk to friends.	(D) Keep looking and listening traffic.
3	When is it safe to cross at a zebra crossing?
	(A) When the traffic has slowed down.	(B) When the traffic has stopped.
	(C) Any time it is always safe just to cross and not look.
4	When the traffic light is the red man, what must you do?
	(A) Stop.	(B) Cross the road with care.
	(C) Sit.	(D) Run.
5	When the traffic light is the green man, what must you do?
	(A) Stop.	(B) Cross the road with care.
	(C) Sit.	(D) Run.
6	When the traffic light turns the flashing green man while you have not started the crossing yet, you must?
	(A) Stop crossing.	(B) Start crossing the road.
	(C) Sit.	(D) Run.
7	When the traffic light turns the flashing green man while you are half way on the crossing, you must?
	(A) Stop crossing.	(B) Start crossing the road.
	(C) Sit.	(D) Run.

.

The quiz results are shown in

Table 2. Summary of the quiz results.

Quiz Results
Participant No.	Q1	Q2	Q3	Q4	Q5	Q6	Q7
1	√	√	√	√	√	√	√
2	√	√	√	√	√	√	√
3	√	√	√	√	√	×	√
4	√	√	√	√	√	√	√
5	√	√	√	√	√	√	×

. It is observed from the results of the quiz that most children scored well for the quiz. Three out of five participants achieved perfect scores, with two participants only having one mistake in the quiz. The reason is likely that those two participants may not differentiate and understand the two questions clearly, as questions Q6 and Q7 are longer and look similar.

The survey questionnaires were also conducted to gather the feedback of participants. Again, the survey questionnaires were drafted by incorporating the comments from two teachers at the special needs school, Katong School, to avoid complicated questions. According to the suggestions of the special needs educators, children with ASD have short focus time span. Fewer number of survey questions were recommended. As such, only five survey questions were chosen in the experiment. The questionnaires are shown in

Table 3. Simple survey questionnaires after gameplay.

No	Survey Questions
1	Compared to classroom-based learning of road-crossing skills learning, to what extent you like the virtual serious game to learn road-crossing? (1): Do not like. (2): Below average. (3): Neutral. (4): Above average. (5): Like very much.
2	After playing the virtual serious game, to what extent you have gained the road-crossing skills? (1): No Effect. (2): Below Effect. (3): Moderate Effect. (4): Above Effect. (5): Substantial Effect.
3	To what extent you can recognize a few common traffic signs, such as zebra cross, traffic lights, red man, green man, flashing green man, after learning the road-crossing skills through the virtual serious game? (1): No Effect. (2): Below Effect. (3): Moderate Effect. (4): Above Effect. (5): Substantial Effect.
4	To what extent you gain self-confidences to cross real road junctions without prompts and instructions of parents or guardians, after learning the road-crossing skills through the virtual serious game? (1): Strongly disagree. (2): Disagree. (3): Neutral. (4): Agree. (5): Strongly agree.
5	To what extent you agree that more living skills can be learnt in the form of the virtual serious games? (1): Strongly disagree. (2): Disagree. (3): Neutral. (4): Agree. (5): Strongly agree.

.

The average results of the survey from all participants are shown in Figure 13. It is observed that participants expressed positive feedback in 80% of the survey questions towards the learning with the road crossing serious game. Overall survey responses show that most participants gained road crossing skills and could recognize traffic signs after playing the game. Most participants are supportive of learning more skills with serious games. Most of them prefer learning through VR serious games to classroom-based learning. This shows the appeal of VR-based serious game learning to children.

5. Conclusions

In this research, the learning effectiveness of the metaverse application in special needs education was discussed. We combined the VR and serious games with interactive user interfaces for children with ASD to learn the daily living skill of crossing roads safely, with knowledge and stealth assessment components embedded in the metaverse application. The road-crossing serious game contains two game modes: the Tutorial and the Play. The Tutorial mode provides the learning phase to players with four levels in various scenarios of road junctions, while in the Play mode, gamers can practice their skills in the four levels. The points systems in the serious game have been implemented according to the time spent and the number of mistakes made by the players.

The serious game can be operated on various computation platforms such as computers, iPads, tablets, or mobile phones. Players can interact with the serious game through different electronic input/output devices, including the Microsoft Kinect motion sensor, keyboard, mouse, and touch screen. Players’ control skills can be trained with multiple user interfaces. The serious game can recognize different gestures and control the gameplay with the Microsoft Kinect sensor. It provides unique training to children with ASD to interact with avatars remotely from a certain distance from the display screen. With the wider adoption of mobile platforms in special needs education, serious games need to support the interactions with touch screens.

The preliminary experiments have been conducted. Participants engaged in learning with the gameplay of the serious game. Each participant was tested in a quiz after the gameplay. It is observed that positive quiz results have been obtained. The feedback has also been collected from the survey questionnaires after the gameplay, with promising feedback obtained. However, due to the impact of COVID-19, only five children could participate in the preliminary experiments. The sample size is too small to determine the learning effectiveness with the road crossing serious game. As part of future work on the topic, a larger number of participants is required.