**An Interactive Serious Mobile Game for Supporting the Learning of Programming in JavaScript in the Context of Eco-Friendly City Management**

**Rytis Maskeliunas ¯ 1 , Audrius Kulikajevas 1 , Tomas Blažauskas 2 , Robertas Damaševiˇcius 2, \* and Jakub Swacha 3**


Received: 13 November 2020; Accepted: 15 December 2020; Published: 17 December 2020

**Abstract:** In the pedagogical process, a serious game acts as a method of teaching and upbringing, the transfer of accumulated experience and knowledge. In this paper, we describe an interactive serious programming game based on game-based learning for teaching JavaScript programming in an introductory course at university. The game was developed by adopting the gamification pattern-based approach. The game is based on visualizations of different types of algorithms, which are interpreted in the context of city life. The game encourages interactivity and pursues deeper learning of programming concepts. The results of the evaluation of the game using pre-test and post-test knowledge assessment, the Technology Acceptance Model (TAM), and the Technology-Enhanced Training Effectiveness Model (TETEM) are presented.

**Keywords:** serious game; gamification; game-based learning; programming teaching; sustainability teaching; mobile app

#### **1. Introduction**

Ubiquitous learning is an emerging educational paradigm that is based on digital content, mobile devices, smart environments, and wireless communication to provide teaching–learning experiences to learners at anytime, anywhere, and in any way [1]. Ubiquitous learning allows students to break beyond the traditional classroom-based setting of formal education and to interact with different computing devices and digital technologies in a blended way. An important enabler of ubiquitous learning is serious games, i.e., games whose primary purpose transcends pure entertainment [2]. Specifically, educational serious games primarily aim at education rather than entertainment [3]. They have been shown to promote critical thinking and strategic and logical skills in computer-supported learning environments [4].

In pedagogy, serious games are used as an attractive way to transfer experience and knowledge to a learner [5]. The implementation of serious games as a part of teaching curricula leads to the gamification of education and instruction, which is considered to be appealing and engaging among the learners [6,7]. Gamification is designed to make the necessary routine fascinating, whether it is the study of a large amount of information, but at the same time leaving the person in his/her reality, by himself/herself, allowing him/her to improve the skills necessary for this particular subject. The game and game technologies in pedagogical practice depend on the creation of certain conditions for achieving goals, the modeling of a special game reality with its internal laws (role-playing games, business games, organizational activity games, etc.). Games complement traditional forms of education and contribute to the activation of the learning process and the successful implementation of collaboration-based learning [8] in practice. Unlike in traditional teaching resources and digital media, educational games provide a virtual space for learners, in which learners can practice and actively engage in the subject of learning without being subject to the stress typically associated with formal learning [9].

Games have a strong emotional impact on students and can help form many skills and abilities: first of all, communication skills, the ability to work in a group, to make decisions, and to take responsibility for oneself. Serious games combined with other educational technologies increase the effectiveness of programming education. Recent studies claim that learning outcomes and learner engagement are improved when using digital game-based learning versus traditional teaching methods [10]. In contrast to entertainment-only games in general, pedagogy-oriented games have an essential feature—a clearly defined goal of teaching and corresponding pedagogical results that can be substantiated, singled out in an explicit form, and are characterized by an educational and cognitive orientation [11]. The reasons why students love computer games can be summarized as follows:


The multi-aspectual nature of computer games creates an opportunity to develop creativity, technical skills, and collaborative work experience [16]. Moreover, they have been demonstrated to positively affect students' learning outcomes [17,18].

Several studies claim that students usually prefer playing a serious game over traditional pedagogical methods in several areas of science, technology, engineering, and mathematics (STEM) education (see, e.g., [19] for a comprehensive case study for mathematics). Certain qualities of educational games can help increase student interest: a challenge that encourages the learner to explore a specific topic and bring knowledge and skills to perfection because only in this case can one go to the next level. In particular, programming education for non-STEM students should use enjoyable game-based tools to overcome their anxiety, increase their engagement and motivation [20,21], and support computational thinking [22]. There is no single definition of serious games. The disagreements and discrepancies among definitions of serious games lead educators, tutors, and mentors into confusion when they try to figure out which games should be used for effective teaching. The general point of view is that:


Educational serious games provide educational materials, by choosing which, the student himself/herself chooses the pace of learning. In most educational serious games, the cognitive and visual load of students is realized through the computer (or smartphone) screen. The game is an interface in which one or more loads are constantly increasing, each time increasing the level of difficulty and thus keeping the participants in suspense. In some games, there are high cognitive loads, since during the game the student must understand how the storyline develops and analyze the situation [23]. Such examples of mobile app-based serious games for education include math games [24], musical games for preschool children [25], digital games for teaching young children about programming [26], a gamified informatics course [27], improving children's procedural abstraction thinking skills in Scratch [28], a collaborative gamified quiz [29], game for fighting child obesity [30], or a serious game for carers of dementia patients [31]. For a systematic review of open educational games, see [32,33]. Each game is characterized primarily by a specific game context (the inner "world" of the game), which is built and maintained using special means, and assumes the presence of a set of positions and roles of the participants and, therefore, a controlled communication system, as well as special mechanisms that allow for generating game actions.

Our goal is to design, implement and launch an application designed for mobile devices, which will simultaneously visualize: the course of action of an algorithm (input data, change of data state, output data) and the source code responsible for these changes. The contribution of this paper is as follows: (1) providing a design of a mobile app implementing a serious game aimed at teaching how to solve common algorithmic problems in the JavaScript programming language, and (2) the evaluation of the game using the Technology Acceptance Model (TAM) and Technology-Enhanced Training Effectiveness Model (TETEM).

The organization of the remaining sections of this paper is as follows. Section 2 discusses the pedagogical and methodological backgrounds. Section 3 describes the game scenario, the design of gamification, and the implementation of the app. Section 4 presents the evaluation results. Section 5 presents the discussion. Finally, Section 6 presents conclusions and discusses future work.

#### **2. Pedagogical and Methodological Backgrounds**

In the traditional model of teaching programming at university [34], the leading role in the lecture belongs to the teacher. From him/her is required not only good knowledge of the training material but also the ability to put it to the audience: to present it in an interesting, figurative, and clear way. The most common way to visualize lectures is to prepare demonstration materials (presentations in MS PowerPoint). Using MS PowerPoint presentation materials allows for visually presenting the studied material in the form of static text or graphic information. Interest in the subject increases if, along with the systematic presentation of the material, including in the discipline program, the lecturer shows his understanding of the perspective on the development of the subject, shares the experience of his scientific developments, and can refer to the history and reasons that prompted the study of this or a different phenomenon.

The methodological backgrounds for the application of serious games for education are Game-Based Learning (GBL) [35,36], simulation gaming [37], and mobile microlearning [38]. Here, mobile microlearning means that the educational content is created for the small screens of smartphones, and is structured in small, self-contained bits of knowledge, which can be assimilated by students in no more than five minutes. The digitalization of information directly affects and modifies human mental activity in the process of education. The problem of the influence of a computer on human mental activity can be considered based on three main approaches established in psychology [39]: substitution theory, complement theory, and transformation theory. Substitution theory identifies the work of a computer program with the process of human mental activity. Complement theory is based on the theory of thinking, according to which a computer significantly increases a person's ability to process and perceive information. Transformation theory claims that the computer transforms human mental activity, and contributes to the emergence of new forms of mediation. The interactivity of a program is its ability to conduct a "dialogue" with the user, i.e., to respond to user-entered requests or commands. A feature of human–machine interaction is its belonging to a special type of communication called interaction. Interaction involves not only the exchange of information between the participants in the communication but also their joint activity. The system immediately reacts to commands and user requests (feedback), and allows the latter to determine and, if necessary, adjust their further actions.

Current approaches to enriching the traditional lecture with elements of entertainment and games include embedding computation in physical objects like cubes, etc. [40], developing tangible interfaces for teaching children about robot programming [41], or gamified quizzes and puzzles [42]. However, these games usually focus on young school children and teens, whereas they are considered as too simplistic by young adults such as university students. Our novelty is the development of a

serious game for mobile devices, which is aimed at teaching university students how to solve common algorithmic problems in the JavaScript programming language, while simultaneously visualizing: the course of action of an algorithm (input data, change of data state, output data) and the source code responsible for these changes.

#### **3. Materials and Methods**

#### *3.1. Pedagogical Models*

In this paper, we have adopted a model of game-based learning from [43] (see Figure 1). When developing an interactive game, we followed the guidelines for integrating gamified learning in the classroom [44] as follows. First, we clearly define the pedagogical objectives, which are to acquaint the students with introductory concepts in programming. Next, we determine the technological competency of our target students. Finally, we identify the content to teach.

**Figure 1.** Model of game-based learning (adapted from [34]).

Our pedagogical model of the learning process based on the serious game (Figure 2) is based on 5 key elements, which interact in achieving a sustainable serious outcome while keeping the students engaged and motivated. The model starts with the element representing the pedagogical aims, i.e., the learning of the JavaScript (JS) programming language. Other elements such as adaptive learning [45] are used to support the achievement of this aim. Adaptive learning ensures the adaptiveness of learning through feedback, which constructs an individual learning path for each learner. As a result, gaps in knowledge are closed and the difficult topics are encouraged to be repeated for rooting the acquired knowledge. At the same time, adaptive learning ensures that the well-understood topics can be covered faster and without unnecessary repetition. Game mechanics are used to support such reinforcement of desired learning behavior in an engaging and motivating way. The effectiveness of the pedagogical process is evaluated using the TAM and TETEM, which allow for evaluating the effectiveness of the game-based learning process. Finally, in line with the current trends for interdisciplinarity and woke pedagogy [46], the digital game is placed within the context of ecology and sustainability issues, which promotes and encourages green thinking [47] to raise awareness of societal issues and provide diversity in education that transcends the direct needs of the programmer profession. Furthermore, the game, which can be played remotely and anytime on a smart user device (smartphone), fits well into the concept of 21st century online education [48], which is going to dominate the landscape of education in the post-COVID-19 world.

#### *3.2. Design of Game Scenarios and Game Implementation*

The game was developed by adopting the gamification pattern-based approach [49]. The designed interactive game has four major structural blocks: an authorization block (entering the game, choosing a topic), a game mechanics block (instructions, game rules), a learning block (user interface), and a game situation assessment block (analysis and evaluation). A schematic representation of the game flow is given in Figure 3.

game.

**Figure 2.** The pedagogical model of the learning process-based on serious game. TAM is Technology Acceptance Model. TETEM is Technology-Enhanced Training Effectiveness Model.

〇 ◎ is the end of the **Figure 3.** Schematic representation of the game flow. # is the start of the game. ⊚ is the end of the game.

The story of the game is based on environmental awareness of ecological problems occurring in everyday life, while the direct (not-serious) aim is the prevention of city pollution in daily activities. The user acts as a commercial advisor having to solve puzzles to build an industrial quarter for the city to start generating income while avoiding excessive pollution, which decreases the game score. Successful implementation of programming-related tasks allows for increasing city revenue and the game score. The game is available online at http://algo-js.usz.edu.pl/.

#### *3.3. Gamification of Programming Algorithms*

Many studies [50,51] claim that problems in programming education arise due to the complexity of abstractions and concepts of programming such as variables, arrays, functions, or loops. To overcome this barrier for learning programming, the principles of visual programming, which focus on the use of visual abstractions corresponding to programming abstractions [52], are adopted. The selection of topics follows the list of suggested topics of an undergraduate computer science course by ACM Computing Curricula [53].

The concepts that our system supports are common programming algorithms: linear algorithms, branching (conditional) algorithms, iterative algorithms, search and sorting algorithms, recursion, tree raversal, and graph algorithms. We describe the gamification of these algorithms in more detail in the following subsections.

#### 3.3.1. Linear Algorithm

Linear algorithms are algorithms which do not involve branching, i.e., there are no conditional statements. Examples of such algorithms are recipes, which describe how to complete a task by executing several steps. The case for such an algorithm is finding an exit through a maze of streets in a city. The idea is represented by the following image from the game's interface (Figure 4). Each command is evaluated and visualized in a loop. The algorithm finishes the execution if: the client is found (or the exit is reached), or the car object makes an invalid move. The visualization includes showing the truck position with a rotation related to the last command. Each step of a truck should be visualized as well. Several levels could be implemented. The difficulty of the task is managed through the maze complexity. The introductory level includes only one command and a very simple maze. Additional levels include using an increasing number of commands. In the highest levels, a maze is generated randomly, so that a student has to make a correct algorithm in one go.

**Figure 4.** Illustration of a linear style algorithm by finding an exit through a maze of streets in a city.

#### 3.3.2. Branching (Conditional) Algorithms

Branching refers to conditional statements, conditional expressions, and conditional constructs, which are dedicated to performing some computations or actions depending on the evaluation of the programmer-specified Boolean condition. To solve the task, a student has to write conditional statements including equals, not equals, less than, more than, less than or equal, and more than or equal, which are visualized as turning decisions at each city intersection (Figure 5). To reach the aim, the branching conditions must be written correctly.

**Figure 5.** Illustration of a conditional algorithm—finding a correct route in the city streets.

A student will have to implement a function that will be tested with several sets of input data. Each command except the conditional statement is evaluated and visualized in a loop. The algorithm finishes if all test cases were successfully executed, or there was a mistake in a test case. There are three containers of different sizes and the task is to sort out several types of color-coded waste (green—bio, blue—plastic, red—construction waste, etc.) trucks to the appropriate waste dumps. Visualization includes moving a person to a container and dropping an item. Each step of a person's movement is visualized as well.

#### 3.3.3. Iterative Sum Algorithm

A simple summation task which does not include arrays could be formulated like this: given a number n, find the sum of digits in all numbers from 1 to *n*. The sum algorithm normally involves a variable where the sum is accumulated (sometimes called an accumulator) and a loop that iterates within a given interval. The general case for such an algorithm is has an increasing number of objects which are moved to a container of a specific type and the sum of the objects in the container is displayed. The narrative of this case is that there are several parking places and the parking place number denotes how many cars it can contain. The task is to calculate the sum of the cars which could be placed in a given number of parking places.

Each command is evaluated and visualized in a loop. If the sum variable is equal to the previous sum plus the expected number of trucks, then the animation should start moving trucks to the big waste depot. The algorithm finishes if all trucks are moved to the empty waste deposit place, or there is a mistake and the sum variable is not equal to any possible combination. Visualization includes moving trucks to the waste deposit lot and showing the value of the sum variable.

#### 3.3.4. Iterative Search Algorithm

The simple search task is to find a specific value of a specific item in a given range of values. In this case, we could use the waste truck scenario. Given the array of random truck numbers assigned to waste deposit places, we need to find a maximum (or a minimum) number of trucks in a waste depot (Figure 6). A student will have to implement the minimum or maximum finding functions. That function receives an array of numbers called trucks which has to be investigated and the minimum or the maximum number of cars is found.

**Figure 6.** Illustration of an iterative search algorithm—finding the correct waste deposit lot.

Each command is evaluated and visualized in a loop. Each item from the cars array is evaluated against the current min or max variable. The algorithm finishes if the loop finishes successfully and the min/max value is found, or there is a mistake in the provided solution and the min/max value is not found. The visualization includes comparing the current state of the truck array to the minimum or maximum value found before that iteration.

#### 3.3.5. Iterative Sorting Algorithm

Sorting consists of ordering a list of objects according to the value of a specified property. We start familiarizing the student with this concept from the simplest (and less efficient) algorithm called *bubble sort*. In this case, we could use the building scenario. Given a set of city buildings, we need to sort them according to their heights (number of floors) as larger buildings pollute more.

A student will have to implement the building sorting function. That function receives an array of heights called buildings which have to be sorted in increasing order (see Figure 7). Each command is evaluated and visualized in a nested loop. The users' data array is investigated in each iteration, and if there is a change in an array, then the animation of swapping two players starts. The algorithm finishes if the loop finishes successfully and the data are sorted in increasing order, or there is a mistake and the returned data do not match the sorted data. Variations of the implemented algorithm may include changing the sorting direction.

The recursion is implemented by using functions that call themselves from within their code. The Fibonacci recursive algorithm, which generates the Fibonacci number sequence, is used to visualize the recursion. The task for this scenario is to implement a function that takes one parameter (*n*) and returns the corresponding Fibonacci number. The scenario narrative includes trucks and waste deposit places. The idea is that every newly built waste depot should be able to contain the waste trucks from the waste deposit places built before. A student has to implement the Fibonacci sequence calculation function. The calls to the Fibonacci function are visualized as a call stack. Numbers are visualized as

trucks that appear after the result of the function call is returned. The algorithm finishes if all test cases were successfully executed, or there was a mistake in a test case.

**Figure 7.** Illustration of sorting algorithm using the building ordering scenario (the higher building is a more polluting building).

#### 3.3.6. Recursion

#### 3.3.7. Tree Traversal Algorithms

Traversal is visiting all the nodes of a tree data structure and executing the associated actions. Generally, we traverse a tree to find and access an item stored in the tree data structure. To visualize tree traversal algorithms, we use the city maze scenario. The maze is organized as a tree that has several levels. Given the picture, the student will have to create a tree data structure and to implement the chosen tree traversal algorithm. Thus, the picture needs to be transformed into a tree. The city maze root node is a maze itself. It has two children—greenery and structures. Greenery contains parks, lakes, and fields. Structures contain buildings, roads, and parking places. Each node has a title, coordinates, and children. A student needs to create a tree data structure and construct the specific tree of a given city maze. Initially, the game app shows a semi-hidden maze (a semi-transparent layer is shown over the maze). According to the data structure and the tree construction code created by a student, the app opens specified tiles. The task will be completed if all tiles are opened.

#### 3.3.8. Graphs—Shortest Pathfinding Algorithm

A graph is an abstract data type used to model a set of connections. To find the shortest path, the city maze needs to be transformed into a weighted graph with the weights corresponding to the distance between intersections. The most important/well known algorithm to find the shortest path in a weighted graph is Dijkstra's algorithm. The task is to find an exit through the maze of streets in a city. That image needs to be transformed into a graph. The game narrative is a truck driving through the city to get some building materials or to pick up waste. To solve this task, a student needs to create a graph structure. For that purpose, the street intersections need to be named and visualized in a city street maze. According to the graph created by a student, the game visualizes the connections as lines and numbers assigned to these connections as weights. The connections are drawn in a green color if the connections exist in the reference graph. Otherwise, the student graph connections will be drawn in red. The app draws the weights in a green color if the weights are accurate compared to the reference graph. Otherwise, the student graph weights will be drawn in red.

#### *3.4. Evaluation Measures*

To assess the effectiveness of the game in teaching the introductory concepts and the construct of programming logic and thinking, a pre-test and the post-test were performed and analyzed. All tests were created by computer science teachers. Each test (pre-test and post-test) included the same topics of programming and covered the same content of materials and degree of complexity.

The pre-test assessment was done before the start of the course to find the knowledge of programming constructs and abstractions among students before the course, while the post-test was taken by the students after the course. Each group was given 10 exercises to solve within a time limit of 90 min. After the end of the course, the students were given a post-test, which was delivered in the same way as the pre-test. Finally, the overall assessment was done by comparing the evaluation scores obtained during the pre-test and post-test to find the improvement in the student knowledge.

To validate the developed game, a survey was conducted using the Technology Acceptance Model (TAM) [54]. The TAM was already used for evaluating the use of gamification for serious purposes [55]. The questionnaire had 10 questions that address five elements (two questions per each element of the TAM) to validate: perceived usefulness (PU), perceived ease of use (PEU), attitude towards use (ATU), intention to use (IU), and perceived enjoyment (PE) as an external factor. PU evaluates how useful gamification is for improving professional competencies. PEU is an indicator that measures users' behavioral attitudes towards the ease of technology use. ATU is the degree to which a student perceived desirable feelings related to the use of a serious game for learning. IU measures the intention to use a serious game for learning. PE evaluates the hedonic enjoyment of students while using the serious game. Each item is assessed with a 7-point Likert-type scale from 1 (strongly disagree) to 7 (strongly agree). For the evaluation of the internal consistency of survey results, we use Cronbach's alpha, which verifies the reliability of the measurement scale used.

We also adopted the Technology-Enhanced Training Effectiveness Model (TETEM), which was created to analyze the adoption of virtual worlds in organizational training [56], but later adopted to evaluate the effects of gamification [57]. The model evaluates experience with videogames (EVG), attitudes toward game-based learning (AGBL), control valence (CV), and gamified valence (GV), and can be used to assess the relationship between gamification and training. EGV refers to the amount of time spent playing different games, and identification with gaming culture. Valence refers to the anticipation of gain from learning, which in turn influences student reactions, outcomes, and the degree of knowledge transfer. Each item is assessed with a 5-point Likert-type scale from 1 (strongly disagree) to 5 (strongly agree).

The results of the survey are evaluated using statistical testing (Shapiro–Wilk normality test, Mann–Whitney U test) and correlation analysis (Pearson correlation coefficient). The Shapiro–Wilk normality test is used to test the hypothesis that the data are normally distributed. The Mann–Whitney U test is a nonparametric test commonly used to compare outcomes between two independent groups.

Pearson correlation is a statistical measure that assesses the linear correlation between two variables. We used it to check the relationship between the elements of the TAM and TETEM.

#### **4. Results**

#### *4.1. Participants and Educational Setting*

The participants consisted of 54 undergraduate students at Kaunas University of Technology (Lithuania). The sample was 83.4% male and 16.6% female (mean age = 19.43, SD = 1.17). The survey was approved by the Institutional Review Board at the Faculty of Informatics, Kaunas University of Technology. All participants voluntarily agreed to take part in the survey. Answers were fully anonymous, and no personal data were collected, while only aggregate information was later used.

Based on the prior academic results, the participants were randomly split into two groups (game group and control group) to maintain academic balance. Both groups attended a traditionally delivered (with PowerPoint-based lectures) programming course, but the game group was introduced to the game and encouraged to play it.

For the TAM survey, we used the original TAM questionnaire as in [58]. For the TETEM survey, we used the questionnaire presented in [59], in which we only replaced "video games" with "computer games", and "work training" with "programming training".

#### *4.2. Results of Performance Evaluation*

The results of the pre-test evaluation were explored using skewness and kurtosis values, which satisfied the normality requirement, suggesting that any diversity in the students' background knowledge originated from a normally distributed population. To support this assumption, a Shapiro–Wilk normality test was performed to examine the distribution of scores achieved by students. The results indicated that the scores may have been normally distributed (W = 0.9263, *p*-value = 0.2158; W = 0.9366, *p*-value = 0.2267, for both groups, respectively).

In our case, the Mann–Whitney U test was used to compare the background knowledge between students from both groups, showing that there was not a statistically significant difference (*p* = 0.08) between students from the game group (M = 6.56, SD = 1.24) and the control group (M = 6.65, SD = 1.38) in the pre-test evaluation. The control group achieved slightly better results than the game group in the pre-test evaluation. In the post-test, there was a statistically significant (*p* < 0.001) difference between students from the experimental (M = 8.41, SD = 1.29) and control (M = 7.62, SD = 1.16) groups according to the Mann–Whitney U test (Figure 8).

**Figure 8.** Summary of pre-test and post-test evaluation of game group and control group. \*\*\*—statistically significant (*p* < 0.001). n.s.—not significant (*p* > 0.5).

Moreover, the game group has a learning gain (MD = 1.855) greater than the control group (MD = 0.969) between the two evaluations, while the statistical significance of this difference was confirmed by the Mann–Whitney U test (*p* < 0.001). The results are summarized visually in Figure 9.

**Figure 9.** Change in evaluation scores of the game group and control group between pre-test and post-test. \*\*\*—statistically significant (*p* < 0.001).

α

#### *4.3. TAM*

For the evaluation of the internal consistency of TAM responses, we use Cronbach's α, which is equal to 0.89. Note that α ≥ 0.80 is considered good. α ≥

α α ≥ Pearson correlation was calculated to check the relationship between the analyzed TAM elements (see Figure 10). A positive correlation was observed between PEU and PU (*R* = 0.516, *p* < 0.001), between PU and ATU (*R* = 0.798, *p* < 0.001), between PU and IU (*R* = 0.417, *p* < 0.01), and between ATU and IU (*R* = 0.446, *p* < 0.01).

**Figure 10.** Pearson correlations between the elements of the TAM.

#### *4.4. TETEM*

For the evaluation of the internal consistency of TETEM responses, we use Cronbach's alpha. The result is α = 0.96. Note that α ≥ 0.80 is considered good. α α ≥

Pearson correlation was calculated to check the relationship between the analyzed TETEM elements (see Figure 11). The statistically significant correlation was noticed between EVG and GV (*R* = 0.60; *p* < 0.001), and AGBL and GV (*R* = −0.38, *p* < 0.01). The results of the survey support the claim that a gamified learning experience leads to higher valence, which provides for a causal relationship between the use of gamification and the learning outcomes. −

**Figure 11.** Pearson correlation between the elements of the TETEM.

#### **5. Discussion**

Game-based learning is a process both unpredictable and difficult to manage. Its course is affected by some factors, which are very difficult to calculate, and even more so to assess their mutual influences. However, the systematic and purposeful use of game methods can give certain results, both in changing the basic qualities of a person and in the effectiveness of educational activities. So, we can discuss gamification as a new way of organizing training, which has a huge pedagogical potential. Inherent mechanics of gamification allow you to run the very highest level activity, which is the root cause, the source of the child's activity, which has a creative character [59], with how to stimulate subjective activity, but does not remove a learner from the reality.

Elements of gamification make the standard university course more interesting. For example, game mechanics can motivate a student to do homework and solve tests, and if the topic is too complex, then simple programming examples with gamification will help to better understand and learn the material for the future. A large programming course, in which students risk getting lost, can be supplemented with a serious game—such a tool provides an incentive for the student to complete the course. Incentive badges or points for different actions when completing practical programming tasks will motivate the student to pass the course on time. Among other things, game mechanics make the learning activities themselves more attractive, which was confirmed by the results of the TETEM survey (item "gamification valence"). This study is the first study that had adopted the TETEM questionnaire for evaluating a serious game for teaching programming.

On the other hand, the TAM model is well known and has been used for evaluating games several times. Wang et al. [60] used the TAM to evaluate a mobile game, which supported students in learning color mixing in design education. They found a significant correlation between PEU and PU, between PU and ATU, and between ATU and IU. Onashoga et al. [61] developed a 3D game-based learning approach for increasing awareness of phishing attacks. The study has found that all relationships between PEU, PU, ATU, and actual usage of the system (AUOS) were significant and positive. Giannakoulas and Xinogalos [62] developed an educational game for teaching simple programming concepts to primary school children. They used the TAM to evaluate the game but did not perform an analysis of relationships between the TAM concepts. Therefore, our results of evaluating the developed game using the TAM model are largely in agreement with the results achieved by other studies.

Thus, the main principle of gamification is to ensure that constant, measurable feedback from the student is received, which provides the possibility of dynamic adjustment of his/her learning behavior to achieve adaptive learning, and, as a result, the rapid development of all the functionalities of the serious game application and the gradual mastering of educational material. The latter aspect is especially relevant in the era of COVID-19 lockdowns, which have disrupted the usual flow of teaching in schools and universities, while current modes of remote learning, which were hastily enacted, lack feedback mechanisms to monitor the learning process effectively, raising new challenges for educators and serious game designers [63,64]. The mobile game, such as the one presented in this paper, can be used to sustain student–student and student–teacher ties in the learning process by putting the student in the learning-oriented gaming community. The emergence of educational serious games also changes the pedagogical position of the teacher and creates conditions for the development of new modes of remote education for the COVID-19 world. These new modes based on mobile applications, including mobile games, can be used to bridge the digital divide in teaching information technologies [65]. However, the usefulness of a serious game for improving the learning performance among disenfranchised groups of learners still requires further research.

The adoption of serious games also has limitations. Our approach is limited to a single course in the programming study curriculum at the university level. Developing games for a different study course (especially for a course attended by non-technologically oriented students) may require adopting a different set of solutions. Designing a game for school or high school students also may require adjustment of the pedagogical model. Finally, when evaluating our game, we followed a closed-world assumption, as the students did not use any games for learning in other courses. If a serious game-based learning approach were adopted more widely, for example, at the study program level, and the students would use the games in many courses, and the attractiveness of a serious game as an educational tool per se may decrease.

Following our experience of using the game in the teaching process, we outline the following limitations of the application of serious games in education as follows. External rewards, such as points, are certainly necessary, but the internal motivation of students to study is more important [66]. The student must clearly understand that it is the educational achievements for which the awards (badges, points, etc.) are given. There is a risk that gamification can undermine behavior psychologically as some students may focus solely on receiving awards rather than on the educational process itself.

#### **6. Conclusions**

In this paper, we described the use of our developed interactive mobile application as a serious game for learning how to solve common algorithmic problems in the JavaScript programming language. The game was validated using the TAM and TETEM. The results of the TETEM survey supported our assumption that gamified learning experience leads to higher valence, which supports the positive relationship between the use of gamification and the learning outcomes. On the other hand, the TAM survey revealed positive relationships between perceived usefulness (PU) and perceived ease of use (PEU), as well as between attitude towards use (ATU) and intention to use (IU), which support the use of gamification as a tool for improving professional competencies, and the student intention to use the developed serious game for learning.

Future work will focus on the repurposing of our approach for other fields of education beyond programming and computer science.

**Author Contributions:** Conceptualization, J.S. and R.M.; methodology, J.S., R.M., T.B., and R.D.; software, A.K.; validation, J.S., R.M., T.B., and R.D.; formal analysis, J.S., R.M., and R.D.; investigation, J.S., R.M., A.K., T.B., and R.D.; resources, J.S., R.M., and T.B.; writing—original draft preparation, J.S., R.M., T.B., and R.D.; writing—review and editing, R.M. and R.D.; visualization, R.M. and A.K.; supervision, J.S. and R.M.; project administration, J.S. and R.M.; funding acquisition, J.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** Supported by a grant from the Cooperation Fund Foundation within the "Paths of cooperation—support for entities implementing international cooperation" project co-financed from the European Social Fund under the Operational Programme Knowledge Education Development.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


66. Ašeriškis, D.; Damaševiˇcius, R. Computational evaluation of effects of motivation reinforcement on player retention. *J. Univers. Comput. Sci.* **2017**, *23*, 432–453.

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

**Angeliki Leonardou 1, \*, Maria Rigou 2, \* , Aliki Panagiotarou <sup>2</sup> and John Garofalakis 1**


**Abstract:** Educational games and digital game-based learning (DGBL) provide pupils interactive, engaging, intelligent, and motivating learning environments. According to research, digital games can support students' learning and enhance their motivation to learn. Given the central role teachers play in the learning process, their perceptions of DGBL play a significant role in the usage and effectiveness of game-based learning. This paper presents the main findings of an online research on primary school teachers' attitudes toward DGBL. Furthermore, the research investigates teachers' opinions about the functionalities provided by the implemented Multiplication Game (MG) and the newly incorporated teacher dashboard. The MG is an assessment and skills improvement tool that integrates an adaptation mechanism that identifies student weaknesses on the multiplication tables and in its latest version also supports a strong social parameter. Students can be informed about their own progress as well as the progress of their peers in an effort to examine if social interaction or competition can increase players' motivation, which is a subject that raised some concerns in the teaching community. The paper describes the functional options offered by the MG dashboard and documents the outcomes of an online survey conducted with the participation of 182 primary school teachers. The survey indicated the potential usefulness of MG and the benefits it can offer as a learning tool to improve pupil multiplication skills and help teachers identify individual pupil skills and difficulties and adapt their teaching accordingly. The analysis applied has found a correlation between teachers' perceptions about MG and their view on using digital games in general.

**Keywords:** digital game-based learning; media in education; multiplication game; digital games usefulness

#### **1. Introduction**

According to Prensky [1], what allures nowadays children to participate in video and computer games is neither violence nor their subject but the provided learning. Children similar to all humans enjoy learning when the notion of obligation is missing. Through modern computer and video games, game players not only become familiar with ways to use games and act inside the game theme and plot but are also offered opportunities of metacognitive learning (p. 2):

"to take in information from many sources and make decisions quickly; to deduce a game's rules from playing rather than by being told; to create strategies for overcoming obstacles; to understand complex systems through experimentation"

Furthermore, players learn to interact and cooperate with others while developing a social consciousness.

Today's game-players at their mid-school age are already capable of comprehending and possess a remarkable fluency in doing many complex things (e.g., reasoning, building, flying); therefore, the typical school curriculum is considered rather unattractive and disengaging. Consequently, it is crucial that teachers try hard to keep up with their students'

**Citation:** Leonardou, A.; Rigou, M.; Panagiotarou, A.; Garofalakis, J. The Case of a Multiplication Skills Game: Teachers' Viewpoint on MG's Dashboard and OSLM Features. *Computers* **2021**, *10*, 65. https:// doi.org/10.3390/computers10050065

Academic Editors: Carlos Vaz de Carvalho and Antonio Coelho

Received: 4 April 2021 Accepted: 13 May 2021 Published: 17 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

digital pace and even to embrace their online capabilities through designing appropriately the whole teaching form [1]. Furthermore, during the COVID-19 pandemic, digital educational games have earned a significant role through distance learning settings (either synchronous or asynchronous) deployed for considerable periods of time in educational systems around the world. Educational games and digital game-based learning (DGBL) offer pupils learning environments that are interesting, engaging, intelligent, adaptive, motivating, and interactive, where pupils can communicate their knowledge, experiences, feelings, and thoughts [2–4].

This paper, starting from game-based learning theory attempts to offer an insight of teacher perception and acceptance of digital games in general, and the Multiplication Game (MG) in particular. MG is a digital assessment tool, and part of its usefulness results from the incorporated logic of DGBL. Furthermore, it is designed to promote flow experience that fosters learners' enjoyment and concentration. Student motivation is developed by incorporating the Open Learner Model notion, as parts of the underlying user model are exposed to pupils in a graphically simplified form. Moreover, MG shares a social aspect as the learner model does not open exclusively for the particular pupil but also for the teacher and class pupils through adequately designed views. This study focuses on revealing pupil's information to the teacher via a dashboard where the teacher can monitor the progress of individual pupils and the whole class on multiplication skills. The objective of the conducted survey was to investigate the perceptions teachers have toward the opening of pupil progress data to the pupil they concern and to peers, and also their attitude toward the benefits and support MG can provide to their teaching. Moreover, the survey recorded the teachers' views of the MG in relate with their views on DGBL.

#### **2. Digital Games and DGBL**

Undoubtedly, modern pupils impose high demands on the technological aspects of their learning as they do not consider traditional educational approaches as interesting and require up-to-date learning environments and tools. Game-based learning is the pedagogical approach that makes use of games offering pupils the opportunity to take part in the educational process and material in an active and enjoyable way [5]. Game-based learning comprises the design and development of game activities targeting interactive learning and supporting pupils to gradually apprehend concepts and be guided toward a final goal [6]. At the same time, pupils are experiencing feelings of achievement, reward, and progression. Game-based learning can be utilized as a teaching method because through game content and plot, pupils can explore the various game parts, using them as ways of knowledge acquisition and skills enhancement [7,8]. Based on similar observations, Prensky [1] supported that educational software design should take its shape using game design methods and techniques. This belief is widely respected, and its popularity is growing [4]. Nevertheless, educators should be cautious about the frequency of deploying digital games in their teaching practice, as many of today's children already make excessive use of digital games, which leads to problematic habits such as lower physical activity, fewer social interactions, poor sleep patterns [9], or even substance addiction symptoms e.g., craving, mood changes, tolerance, and salience [10,11]. Still, the potential benefits of digital educational games outnumber their side effects and cannot be considered as a reason to exclude digital games from the classrooms.

Digital games span a wide range of categories, use a variety of digital technologies (e.g., computers, (handheld) consoles and mobile devices) [12], and their popularity is steadily rising [13,14]. Digital educational games are defined as "computer-assisted instructional tools and techniques in which skills and chance are combined and implemented on previously acquired information and experiences developing thus, engaging and immersive learning experiences in order to achieve specific learning goals, outcomes and experiences" (p. 120) [7]. Moreover, digital educational games support adaptability and foster situated learning environments where students through playing obtain and exchange knowledge and skills [2,15]. As digital games provide a virtual environment, they support

students to overcome the limitation of physical space and offer hands-on access to learning materials [16]. According to Li [17], digital game creation and the engagement in them can result in active students, which will therefore lead in developing 21st century skills and supporting more effective and thorough curriculum understanding. Furthermore, according to [18], digital games help players develop skills such as "critical thinking and problem solving, teamwork and communication, creativity and innovation, and technology proficiency" (p. 421). Through digital gaming, pupils that initially possessed only the ability of applying pre-arranged solution strategies are gradually guided to develop the ability to comprehend original solution strategies [19]. Digital games can offer "opportunities for investigating and understanding real-world situations" (p. 24) [14]. According to [4], they provide an effective and motivational approach to support pupils' learning, while they can significantly improve knowledge transfer, increasing student enjoyment and interest on the particular subject.

According to [7,20], digital educational games when used as a teaching and learning tool can lead to:


Research studies support that game-based learning can be even superior to traditional classroom instruction as it enhances the motivation to learn, while offering opportunities to explore and acquire new knowledge and skills [8,30,31]. DGBL is a student-centered learning approach that unites digital games with educational material to provoke pupils' interest, while they are given the opportunity to empower their learning efficacy. As a result, pupils face knowledge acquisition and education in general, positively. Games designed according to these principles give pupils the chance to practice their skills in a virtual and safe environment, enhance collaboration, promote communication, and develop cognitive and soft skills [7].

#### **3. Teachers' Attitude toward Digital Games**

Shifting from the traditional form of teaching to teaching that includes using digital educational games in the classroom is considered to be a significant change [32]. According to Fullan [33], the implementation of an educational change is determined by three factors: (1) the characteristics of the change (e.g., if the change is useful, practical, and not complicated or long), (2) local characteristics (the district, the community, the principal, the teachers, etc.); and (3) external factors (government, other agencies, etc.) [32]. The adoption and the effectiveness of game-based learning depend largely on the grade of acceptance by classroom teachers [34,35], as they can be considered the true change agents of the schools [36]. Therefore, it is crucial to obtain an insight on teachers' perceptions and beliefs that guide their decision-making process. If teachers have negative perceptions about using DGBL, this can proved a significant obstacle against technology integration and against using digital games for learning [37,38].

Among models that examine and predict teachers' behavior stands out the Technology Acceptance Model (TAM) [39]. According to the TAM model, the acceptance of any technology can be predicted by (a) its perceived usefulness and (b) its ease of use. Furthermore, the TAM model highlights the correlation between these two factors: a technology is considered to be more useful if it is easier to use. In the field of educational research, it is observed that teachers will use a technology in the classroom, only if they are convinced about the advantages (on an administrative and teaching level) this technology can offer [40].

In an effort to summarize the main findings from studies such as [37,41–43] on digital games' adaptation in formal education, major points can be categorized under the following three axes:

	- To enhance student motivation [3,42,44–47]
	- To support students' acquisition of knowledge and cognitive skills [42,44,45]. These beliefs regarding learning opportunities have the strongest direct effect on teachers' intentions to use games [38,48]
	- To offer students a safe learning environment where the consequences of failure are smoother [49]
	- To empower students' activeness [48],
	- To offer students feedback on their learning skills and actions [48],
	- To visualize students' progress for them to watch [48],
	- To propose additional learning material or a reward [37],
	- To entertain students [37],
	- To support evolvement as digital games are considered 'the future' (teachers support the belief that the adoption rate of game-based learning will continue to speed up in the very immediate future [37]).
	- Teachers who experience playing games in their spare time are interested in the idea of digital games in their teaching process [45,50–52], while teachers' ability to effectively deal with new technologies does not necessarily imply that they support the idea of digital games in the classroom [37,45]
	- Degree of relevance (according to teachers) games have to their educational practice [34,41,44,51]
	- Usefulness and learning opportunities offered by the game [37],
	- Aspects in the social environment of teachers (students, colleagues, principal) [37,50,53],
	- Teachers' own experience, which has convinced them about the positive consequences of technology [48],
	- Pupil competition during game play [47], as this can be a reason for using games in the classroom [48,54].
	- Lack of time and technical issues [34,47,52],
	- Inflexibility of the curriculum or fixed class schedules [32,52], that makes teachers feel restricted and unwilling to try non-conservative ways of teaching,
	- Perceived negative effects of gaming e.g., addiction, emotional suppression, repetitive stress injuries, relationship issues, social disconnection [32],
	- Unprepared students [32] that delay teaching, as without adequate preparation at home, students cannot cope with the subject and concepts, and therefore, they deprive the class of the opportunity to play games or the teacher needs to consume teaching time for repeating the same teaching material,
	- Absence of supportive materials to help teachers find the suitable digital game that is compatible with the subject and the class level and also offers proper usage instructions [29,32,52],
	- Limited budgets [32] that lead to poorly equipped computer school labs, etc.,
	- Teaching experience may affect the type of limitations teachers consider when they think about using games in the classroom [32,34], as older teachers are less willing to try non-traditional ways of learning due to their poor or incomplete technology-related skills [29,52],
	- Classroom management issues [41] that can vary but take away the opportunity from the teacher to deal with issues such as the adoption of digital games,

#### **4. Adaptiveness and Open Learner Modeling**

In the game world, it is crucial for a player to face challenges that are in close correspondence with individual skills' level. If the challenge is higher than the player can handle according to own skill level, then the player can feel anxiety or be discouraged. On the other hand, if the challenge is much lower than the player's skills level, it is possible that the player will feel bored and not engaged [55]. Adaptive are the games that possess an internal mechanism that stores data for every player individually and therefore can make inferences that match the needs and preferences of each player. If apart from maintaining data about the learners and their interactions, some of these elements are also exposed in a suitable way, a connection is set up between adaptive educational gaming and open learner modeling (OLM). OLM was introduced as a notion in the research field of intelligent tutoring systems and adaptive learning environments with the aim to support personalized instruction to learners. At first, adaptive systems did not give learners the chance to access the data stored in their learner model, but things changed when researchers and educators supported through experiments the educational gain that derives from such a revealing [56]. It was proved that by offering learners an insight into specific parts of their own user model, they could use this feedback to self-assess and also reflect on their current skills and competences, organize more effectively their learning, apprehend the system's adaptation decisions, and also demonstrate greater motivation to learn and improve [57–59].

Social OLM (OSLM or OSSM) is an extension of OLM [57,60], and its basic idea is to expose elements from a learner's model also to others, apart from the particular user the data refer to, e.g., instructors, peers, and parents. OSLMs can intensify OLMs' cognitive aspects via social aspects, as learners are given the opportunity to explore other individuals' model or summative information of a peers' group and support them through suitable content topics [61].

Different studies [62,63] support the opinion that when accessing peers' models, learners achieve a wider coverage of topics in the system and higher rates in self-assessment problems. In fact, the OSLM notion is in agreement with past research in the domain of social navigation, which can be utilized in order to guide users through the learning content by revealing other learners' paths and therefore replace knowledge-based guidance [64]. Another contribution of OLM and OSLM is that they contribute in promoting engagement to learning environment and content [61,65].

The choice of which information the user will access and in what way it will be represented are both central issues, as OLM and OSLM will not meet its practical value if the intuitiveness and ease of perception of the offered data are not assured. Therefore, visualization has a central role that determines the effectiveness of offered information such as the level of assessed knowledge and skills, or the difficulties/misconceptions encountered by the user [66]. Basically, all types of learner models can be opened to users, and the reason for this access will determine the technique of illustrating the model. Moreover, model data visualization depends on who will access these data (educators, parents, or peers) in combination with the purpose for using the information (i.e., context and tasks) [67].

#### **5. OLM Visualization Options**

OLMs are able to take over the responsibility for presenting learner model data in an understandable form, to allow for appropriate actions and decision making. Usually, internal learner model mechanisms and inference logic are too complicated to be presented to learners, peers, teachers, or parents without being processed or filtered; therefore, the simplification of learner model data is necessary through visual presentation [67]. Due to the many different usages and potential users accessing the model, various visualizations could be deployed to serve adequately the OLM purposes [68]. OLM visualizations may include data about the learner, data that compare the learner with peers (individuals or a group, e.g., best scoring classmates), data about other individual learner(s), or the average/summative performance of a complete class. A search in the related literature identifies a variety of available visualizations, namely:


Visualization plays a crucial role in representing data from a learner model, as the appropriate match between data and type of visualization would result in users' better apprehension of the information. According to Bull and Kay [67], it is important to go through a phase simplification on learner model data when deciding the form of visualization being used for pupils, teachers, and parents, omitting complex details such as user monitoring details, inference logic deployed by the adaptation mechanism, etc. Different ways of visualization are recommended for different viewer roles, as different information is necessary in each case and should serve different purposes and user tasks [68]. When visually representing data, variations in fill, color position, and/or size can indicate differences in level of understanding, competencies, skills, and curricula coverage [92]. Most typical visualization types are bar charts, pie charts, radar plots, scatterplots, tables, timelines, network diagrams, and skill meters [93]. There are OLM-based systems that support multiple representations, as research has supported that users enjoy having control over the choice of visualization type, although some visualizations are more preferable [87,94,95].

When visualizing learner models in digital educational games, simple quantized representations are recommended, such as target–arrow, where the number of arrows depicts the level of knowledge for the specific topic (typically up to four levels) [64,96], smilies (smiley faces), where a smiley (or not) face with scalar variations represents knowledge level or contrast a learner's level with the level of peers [97], stars, where the number of stars presented or filled with color (from a fixed number of total stars typically 4–5) depicts learner skill level [77], liquid in a cup or container, where the amount of filled liquid depicts learner's skill level [81], growth of tree, where the different growth stages of the tree depict the different skill levels of a learner [98], and more.

For OSLMs, user model visualizations are selected on the basis of how well they can show social comparison (two individuals, one individual and a group, etc.) [60,61,92,99,100].

#### **6. Multiplication Game Description**

The MG is a web-based practice and progress monitoring game that serves educational and scientific purposes. It started out as an adaptive mobile application (v.1) [101], and it was extended as a desktop application with incorporated OLM elements (v.2) [102]. Followingly, it was revised by adding social characteristics to the OLM, as the learner model is now open apart from the learner, also to teachers and peers (v.3) [103]. The design

and implementation of the MG is based on the belief that multiplication tables fluency is a very significant skill; therefore, it should be supported by engaging and motivational assessment tool. Since in this paper, the focus is placed on teacher support aspects, the game play is briefly described to provide an overview picture of the game, while the functionalities that are offered to teachers are presented in more detail.

#### *6.1. Brief Description of MG*

After the player logs into the game, (s)he can choose which number(s) to practice. The player goes through four levels with different types of questions in each one: (a) right or wrong, (b) multiple choice, (c) matching question with result, and (d) fill in the blank. With the completion of each level, the player can be informed about their own progress. Furthermore, the MG incorporates an adaptive algorithm that undertakes to make a diagnosis (after completing each level) of the player's main weakness and tries to "treat" this difficulty by supporting the player on this particular number in the next level. Upon the game finishing, the player has the option to access the overall progress in the specific game as well as compare the overall progress of the three most recent games. Student can also compare own progress with the average progress of their classmates and see the 5 high scorers in the class.

#### *6.2. Description of MG's Teacher Daskboard*

The MG offers teachers insight into their pupils' multiplication skills (progress/evaluation and history). By using the MG, a teacher can be discharged from the time-consuming correction of pupils' schoolwork on paper (or significantly reduce this effort) using instead the detailed data and progress record of each student maintained by the MG. Depending on teacher's selections, the MG collects selected data from the respective learner model (in the case of an individual student) or from multiple learner models (in the case of all the classmates), and they are presented in an adequate graphical form [104]. As already mentioned, visualization enables comprehension and communication [105], and it depicts a much clearer image for the human brain compared to words or numbers [106]. In this approach, data are presented mainly in the form of tables and barcharts [107]. The information presented via the dashboard is selected according to the criteria of supporting teachers to easily access and assess their pupils, self-assessing their teaching practice outcomes, expose common mistakes of the particular student group, identify low achievers (in order to be more supported) and also help to (re-)arrange teaching process in the special needs of the pupil group (based on the feedback by the above visualizations).

Through the MG dashboard, the teacher can monitor the progress of the pupils—the progress of each individual student either overall or for a selected training number—and keep track of the student's game dates, the selected numbers for each game, the overall success rate and mistakes, as well as the frequency of each mistake. Figure 1 depicts a dashboard instance where the teacher chose to see the overall activity of student named "Zω*η*´ N.".

**Figure 1.** Individual student progress.

"Ζωή Ν.".

When selecting a specific number, the teacher can see on a bar chart the student's success rates in each type of question (different game level). In Figure 2 for instance, number 4 was selected for tracking the activity of the student named «Aιµιλ*ι* ´ α Φ.». Αιμιλία Φ

ber 4 was selected for tracking the activity of the student named «Αιμιλία Φ.»

'

**Figure 2.** Individual student progress for a selected number.

In addition, the teacher can monitor the activity of all students on a specific date or time period (Figure 3).


**Figure 3.** Class progress on a selected date.

After selecting a specific date or a period of time (e.g., month), the teacher can see the students that played the game (Figure 4), their success rate for each number, and the summative success of all students on this particular multiplication number.


**Figure 4.** Class progress on a selected month.

#### **7. Survey Methodology and Analysis**

In the current study, we examined how this approach (i.e., the implemented MG and the dashboard recently incorporated) could facilitate the learning and teaching of multiplication skills. To reach teachers, a call for participation was sent through the official e-mail lists of primary schools. The 182 teachers that took part in the survey used the game (both the student view and the teacher dashboard) and were asked to fill in an anonymous online questionnaire with 37 questions and an optional comments field for general remarks about the game.

For the student view of the game, teachers were simply asked to play the game two to three times, make intentionally some mistakes, observe the score, see their progress per level, and see the progress of other students at the end of each game. For the teacher view, teachers were encouraged to explore the dashboard by connecting to a virtual test class with pre-assigned test students and readily available progress data. For additional support and to make sure that the teachers will try all major functionalities, a set of optional tasks were also provided: (1) See the progress record of student "Zoe N." for all multiplication tables she has selected. How many times did she answer 4 × 6 wrong? (2) Access the success rates for student "Aimilia F." for the multiplication table of number 4. Is 9×4 a multiplication she has answered wrong more than 3 times? Which type of questions seem to trouble her more?, and (3) See the activities of all your students on a specific date (4/2/21). Compare the score of student "Zoe N." with the average class score for the multiplication table of number 6.

This survey was conducted in order to investigate potential correlation between teachers' acceptance and positive attitude toward the notion of DGBL and their acceptance and perception about MG. Furthermore, it is very significant to have educators' opinion on the innovations MG shares with teachers, as they are given access to a dashboard that reveals detailed and summative aspects of their pupils' progress. More analytically, the purposes of the study included the following:


*7.1. Method*

7.1.1. Hypothesis Testing

**Hypothesis 1 (H1).** *Teacher's perception about MG's usefulness (MGU) is related to attitudes toward using digital games (ADG).*

**Hypothesis 2 (H2).** *Teacher's acceptance of MG (TA) and MG's usefulness (MGU) is related to gender, age, and teaching experience.*

**Hypothesis 3 (H3).** *Teacher's acceptance of using MG (TA) is related to perceived barriers in using digital games (BD).*

**Hypothesis 4 (H4).** *Teacher's acceptance of using MG (TA) is related to teacher's attitude toward digital games (ADG).*

More specifically, in order to examine teachers' beliefs about MG usefulness (MGU), two statements were given to be assessed on a 6-point Likert scale regarding how they consider the opportunity for students benefit in multiplication tables fluency when using MG, and MG's potential to support the traditional teaching process. Teachers' acceptance of MG (TA) was tested by asking them to assess on a 6-point Likert scale the likelihood of using MG systematically in their classroom and their intention of recommending MG to their students to use during their free time.

To examine teachers' attitude toward digital games (ADG), eleven statements were given to be assessed on a 6-point Likert scale regarding teachers' own attitude toward digital educational games, their own opinion on the usefulness of digital educational games in the teaching process, whether digital games can have a firm place in the educational practice, and if digital games are a strong current trend in education (and they expect that they will be used more widely in the near future). Regarding teachers' opinion on barriers (BD), statements with five significant barriers identified in the related literature (i.e., lack of time, technical problems, lack of educational curriculum flexibility, and lack of information about suitable and available digital games) were assessed in terms of their perceived significancy on a 6-point Likert scale.

#### 7.1.2. Research Participants

The sample used in this research included primary education teachers in Greece from the general public education. Responses for analysis were collected by distributing the questionnaire online. Hence, we gathered a total of 182 responses.

#### 7.1.3. Design of the Instrument

The questionnaire used in this study was composed of three sections. The first section records the demographics of participants including gender, age, teaching experience, and teacher's frequency of playing privately digital games. The second section refers to teachers' attitudes toward digital tools in their teaching, and the third section concerns MG's usefulness. The assessment tool contained 6 factors and 37 questions. Within them, factors represented by questions were created, aiming to assess the teachers' attitudes. Each subscale comprised 2–11 items. The score of each item ranged from 1 to 6 based on a 6-point Likert scale design. The last question (38th) was an optional open-ended one where respondents could fill in in free-form text any comment they had concerning MG.

#### 7.1.4. Methodology

Cronbach's alpha was calculated to determine the instrument's internal consistency. As concerns hypothesis testing, mean scores for MGU, ADG, TA, and BD were used to establish association between the study variables. Comparisons of means of the construct's distribution TA and MGU was desired for gender, age, and experience groups. Due to the non-normality of the variables' nonparametric tests, Mann–Whitney test and Kruskal– Wallis test were carried out. Spearman's rho identified the associations between the teacher's acceptance of MG (TA) scores and teachers' attitude toward digital games (ADG), the teacher's acceptance of MG (TA) scores and barriers in using digital games (BD), as well as MG's usefulness (MGU) and attitudes toward using digital games (ADG).

#### 7.1.5. Data Analysis

Out of 182 teachers, 159 (76.8%) were female, with the majority (96, 46.4%) in the age group of 30–45 years. The mean of teaching experience of the study was 15.4 ± 9.4 years. A total of 113 teachers (54.6%) work in urban schools, with 69 (34%) working in a provincial area (Figure 5 and Table 1).

ers' attitudes toward digital tools in their teaching and the third section concerns MG's

Cronbach's alpha was calculated to determine the instrument's internal consistency.

– st were carried out. Spearman's rho identified the associations between

ll as MG's usefulness (MGU) and attitudes toward using digital games (ADG).

'

the teacher's acceptance of MG (TA) scores and teachers' attitude toward digital games (ADG), the teacher's acceptance of MG (TA) scores and barriers in using digital games

created, aiming to assess the teachers' attitudes. Each sub-

–

**Figure 5.** Demographics charts.

**Table 1.** Demographics of data.

teacher's frequency of playing privately digi

struct's distribution TA and MGU was desired for

–

–


Table 2 displays the mean scores for the key constructs, which are Teacher's attitude (perception) toward digital games (ADG)—3.97(0.672), Teacher perception about MG's usefulness (MGU)—5.32(0.78), Teachers' acceptance of MG (TA)—5.16(0.927), Barriers in digital games (BD)—4.52(1.066), Interface of MG (I)—5.10(0.759) and Social Opening of MG (SG)—5.07(0.890). Most of them present high mean scores (up to 5).



The constructs consist of several variables as depicted in Table 3, and a Cronbach test was used to assess their internal consistency. Cronbach's alpha values were higher than 0.7, which indicated the reliability of the proposed instrument.


**Table 3.** The construction of instrument.

7.1.6. Results of Hypothesis Testing

Spearman's rho correlation revealed that there is a significant correlation between MGU-ADG (r = 0.569, *p* < 0.01), TA-BD (r = 0.374, *p* < 0.01), and TA-ADG (r = 0.594, *p* < 0.01) (Table 4).

**Table 4.** Correlation between MGU, ADG, BD, and TA.


\*\* Correlation is significant at the 0.01 level (two-tailed).

As concerns H2 (Hypothesis 2), a Kruskal–Wallis test was conducted to examine the differences on TA and MGU according to the age groups and experience groups. No significant differences on TA (χ <sup>2</sup> = 1.907, *p* = 0.385, df = 2) and MGU (χ <sup>2</sup> = 2.656, *p* = 0.265, df = 2) were found among the three categories of age. Additionally, no significant

differences on TA (χ <sup>2</sup> = 0.432, *p* = 0.806, df = 2) and MGU (χ <sup>2</sup> = 0.263, *p* = 0.877, df = 2) were found among the three categories of experience.

In addition, the Mann–Whitney U test showed that the distribution of TA (U = 1.454, *p* = 0.1) and MGU (U = 1.632, *p* = 0.382) is the same across categories of gender.

#### **8. Discussion**

This game of multiplication offers teachers the possibility to monitor the progress of their pupils. Specifically, the teacher can follow the progress of an individual pupil either for the overall activity or for a selected number. Through a properly configured dashboard, the teacher can keep track of pupil playing dates, the overall success rate, and the wrong multiplications as well as the frequency of each mistake. In the case of selecting a specific number, the teacher can see through the bar chart the student's success rates in each type of questions (which correspond to a different level in the game). Finally, the teacher can monitor the activity of all students on a specific date or time period (month). On a properly configured chart, the teacher can see which students were active at that time, each student's success rate on a training number (multiplication table), and the average success rate of all pupils in the classroom for a given number. The main purpose of this research was to investigate teachers' perception toward the MG in general and in relation to their attitude toward digital games. It was very important to find out teachers' willingness to use our tool and understand which are considered the main barriers that will possibly discourage them from utilizing the MG in their teaching process, as well as to reflect on their reactions regarding the social opening of the learner model supported by the MG.

According to the analysis of collected data in the previous section, the majority of respondents stated that it is very likely that they will use MG in the classroom and will recommend it to their students to use it at home as well, while they had a very positive opinion regarding the usefulness of the MG. More specifically, teachers stated that they strongly believe their students can benefit from using the MG and that it can significantly support them in their teaching.

Teachers rated positively the game interface usability and child-friendliness. The social opening of learner data to teachers was also assessed positively, as it allows them to plan more efficiently their teaching, making the appropriate adaptations to respond to individual pupil needs. Moreover, the fact that teachers can see the specific mistakes and their frequencies is considered as a very important feedback for improving their pupils' skills. Teachers also appreciated access to information about a pupil's progress on a selected number and the bar chart display of a pupil's progress in each game level (which reflects pupil scores for different question types in successive game plays).

Further statistical analysis revealed that teachers' positive attitude (perception) toward digital games leads to highly positive perception of MG's usefulness and to acceptance of the MG (H1 and H4). Furthermore, it was demonstrated that despite the severity teachers assign to barriers against using digital games, they are not becoming less willing to use MG in the classroom (H3). Finally, it is quite encouraging that teachers regardless of their gender, age, and teaching experience accept MG's educational value and usefulness (H2).

There were some quite interesting remarks made by teachers in the comments field of the questionnaire. Indicatively, there were remarks that a game such as the MG could greatly help students with attention deficit and other disorders, and that the game can help teachers save a lot of time they would typically spend examining each student in the classroom. In addition, some teachers stated that even though it is very important for the teacher to see details about the progress of each student, it is necessary to verify that it is the student alone that plays the game, and this cannot be guaranteed during off-school hours. This is a realistic issue and a limitation of the approach that can only be tackled if pupils play the game only at school, but this limits the opportunities for pupils to practice and improve their skills. It is a trade-off left to the teacher to decide and cannot be resolved by the game and the way it is implemented. In fact, it is a limitation faced by almost all remotely executed applications.

Another observation was that comparing the last pupil score with previous ones is a helpful indication of personal progress, but the social comparison (seeing the class average and the top scoring pupils) is not necessary and may discourage low-achieving pupils. Social comparison can be considered as a competition increase factor, which in turn is recognized as a strong motivation for improvement in educational settings [48,65,108–110]. The degree of competition and the details of implementing such features in the digital domain is a controversial subject. Our planned large-scale experiments with pupils will hopefully provide more insight on the topic from the pupils' standpoint.

Based on the feedback received by teachers and the analysis of collected data, teachers have positive views toward this approach for teaching multiplication and consider MG a useful learning tool from different points of view. Thus, there is strong evidence that MG could effectively support teaching and learning multiplication facts. This argument though needs to be verified and supported by large-scale experiments of the game with students and their teachers in the real-life setting of primary schools, which will be the next step of this effort.

#### **9. Conclusions**

The MG is a web-based assessment tool that supports pupils in acquiring and establishing multiplication facts skills. In the MG, learning and teaching goals are met, as it not only provides an engaging and motivating environment for pupils to play but also maintains a record of their activities in order to adapt to individual learner needs, to offer social comparison with peers, and to support informed decision making by the teachers. This paper describes the functional options offered to teachers by the MG dashboard and documents the outcomes of an online survey conducted with the participation of 182 primary school teachers. The purpose of the survey was to investigate teachers' attitude toward the benefits and support MG can provide to their teaching. To this end, the survey also recorded teachers' opinion on DGBL in general to allow for investigating potential correlation with their attitude toward MG.

According to existing bibliography [32,34,111], factors such as gender, age, and teaching experience influence teachers' beliefs about digital games. In our study, we found no evidence of gender, age, or teaching experience effect on teacher's acceptance of MG and their opinion about MG's usefulness. In addition, related bibliography [29,32,34,38,41,47,49,50,52] has identified many barriers that distract teachers from using digital games in the classroom. Evidence from the current study suggests that although teachers acknowledge the seriousness of four identified barriers, they did not affect their acceptance of MG. According to other researchers, factors such as the degree of relevance a digital game has to the educational context [34,41,44,51], its usefulness, and the learning opportunities it offers [37] can lead to adapting it in the educational process. Since MG satisfies these factors, it was expected and proved by the survey that MG is positively perceived by teachers. Furthermore, features such as students' support in knowledge acquisition [38,42,44,45,48], students' progress visualization [48], student rewarding, and entertainment [37] lead to the positive perception of a digital game. MG also possesses such characteristics, which contribute to its usefulness as perceived by teachers and their intention of using it.

The findings offer a promising basis for further exploration of the integration of gamebased approaches to multiplication learning to promote active participation and interaction. These findings will be further investigated by planned extended studies that will involve a larger sample of participants comprising both teachers and pupils to lead to observations of learning effects and comparative analysis. As a next step, MG will be used a learning tool through several activities to study its effectiveness in the classroom with the participation of an adequately large number of pupils per grade of interest (second to fourth). The comparative testing of different MG versions (adaptive game, adaptive game with OLM features and adaptive game with OLM and OSLM features) is expected to reveal interesting findings in terms of learning outcomes, meta-cognition and motivation, and thus support

teachers' positive opinion regarding the educational value of the MG as recorded in the presented study.

**Author Contributions:** Conceptualization, A.L. and M.R.; methodology, A.L.; software, A.L.; validation, A.L. and A.P.; evaluation of MG, A.L.; writing—original draft preparation, A.L.; writing—review and editing, M.R.; supervision, M.R. and J.G.; funding acquisition, A.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Hellenic Foundation for Research and Innovation (HFRI) under the HFRI PhD Fellowship grant number 267.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The questionnaire used for the survey is available (in Greek) at https:// docs.google.com/forms/d/e/1FAIpQLSd3gYoK\_c7wjGWz6j\_K0lPWGKqOnVP36MGl7mt6cf5Zfsdamg/ viewform?vc=0&c=0&w=1&flr=0&usp=mail\_form\_link (accessed on 18 March 2021) and the survey collected data can be acquired from https://docs.google.com/spreadsheets/d/1mS8nqKVW\_2SuS617 FDftxKStescnQuGTUcmS02sHFTM/edit?usp=sharing (accessed on 15 May 2021).

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


## *Article* **A VR-Enhanced Rollover Car Simulator and Edutainment Application for Increasing Seat Belt Use Awareness**

**José V. Riera <sup>1</sup> , Sergio Casas 1, \* , Francisco Alonso <sup>2</sup> and Marcos Fernández 1**


**Abstract:** Most countries have active road safety policies that seek the objective of reducing deaths in traffic accidents. One of the main factors in this regard is the awareness of the safety measures, one of the most important being the correct usage of the seat belt, a device that is known to save thousands of lives every year. The presented work shows a VR-enhanced edutainment application designed to increase awareness on the use of seat belts. For this goal, a motorized rollover system was developed that, synchronized with a VR application (shown in a head-mounted display for each user inside a real car), rolls over this car with up to four passengers inside. This way, users feel the sensations of a real overturn and therefore they realize the consequences and the results of not wearing a seat belt. The system was tested for a month in the context of a road safety exhibition in Dammam, Saudi Arabia, one of the leading countries in car accidents per capita. More than 500 users tested and assessed the usefulness of the system. We measured, before and after the rollover experience, the perception of risk of not using the seat belt. Results show that awareness regarding the use of seat belts increases very significantly after using the presented edutainment tool.

**Keywords:** edutainment; serious game; gamification; virtual reality; traffic safety; rollover simulator; seat belt; awareness

#### **1. Introduction**

The three-point seat belt was invented by Nils Ivar Bohlin, a Swedish Volvo mechanical engineer, in 1959. Prior to that date, some cars included passenger retention systems, but not as we know them today. It was recently the 60th anniversary of the invention of the seat belt and Volvo claims that its invention has saved more than one million lives [1]. Studies on the matter, such as the one presented in [2], assume that the seat belt has an effectiveness of around 45% (best estimate) in saving a person's life in a car accident.

The most advanced countries have included, in their legislations, the mandatory use of seat belts, both front and rear seat belts. Despite the legislation, seat belt usage rates are very different, depending on the country. In 2003, the European Union reported that, in its member states, only 76% of front seat occupants and 46% of rear seat occupants used seat belts [3]. In Australia, however, set belt wearing rates are much higher (95% in 2004) [4].

Since the use of seat belts became mandatory, it has become common for countries to develop campaigns that aim to increase the rate of seat belt use. Technical solutions have also been implemented in order to enforce its use. One of the first attempts in this regard was carried out by the United States of America, which in 1973 introduced in its legislation that cars should not be allowed to start unless seat belts were fastened. However, the measure was met with strong public opposition and was withdrawn only six months later.

Current techniques to encourage seat belt usage include smart seat belt reminders (SBRs) in cars; these systems trigger warning lights or sounds to remind passengers to buckle up their seat belts. This technique is quite successful, as reported in [5], where 82.3%

**Citation:** Riera, J.V.; Casas, S.; Alonso, F.; Fernández, M. A VR-Enhanced Rollover Car Simulator and Edutainment Application for Increasing Seat Belt Use Awareness. *Computers* **2021**, *10*, 55. https://doi.org/10.3390/ computers10050055

Academic Editors: Carlos Vaz de Carvalho, Antonio Coelho and Paolo Bellavista

Received: 3 March 2021 Accepted: 16 April 2021 Published: 21 April 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

of car users without SBRs were reported to use seat belts; this number increased to 98.9% in cars with SBRs. In addition, car drivers with mild reminders used seat belts 93% of the time.

Technological evolution has also brought about new methods to develop campaigns with the objective to increase awareness on seat belt usage (with aims to achieve 100% usage). In this regard, serious games and educational entertainment (*edutainment*) applications have become very effective tools at raising awareness in various areas, not just for road safety [6]. Serious games are frequently used for educational, training, and health purposes, but are especially useful at increasing social awareness [7]. These are the so-called "games for good" that are characterized by addressing public, relevant social issues, such as epidemics, sexism and racism, climate change, etc.

This article presents an edutainment system developed by the Institute of Robotics and Information and Communication Technologies of the University of Valencia (IRTIC-UV), within the framework of a road safety campaign in Dammam, Saudi Arabia, to help increase awareness on seat belt usage. Saudi Arabia is one of the leading countries in traffic accidents per capita, with one traffic accident every minute, causing up to 7000 deaths and over 39,000 injuries annually [8]. In fact, the use of the front passenger seat belt only became mandatory in December 2000 in Saudi Arabia [9]. Thus, these kinds of campaigns are very important, since social awareness regarding the use of seat belts is rather low [10].

The developed system includes a virtual reality (VR)-enhanced rollover car simulator, where each of the car's four passengers has a different view, provided by a head-mounted display (HMD). The objective is to show how a virtual car experiences an accident and overturns, while the real car, where the passengers are mounted, turns around driven by a powerful motor synchronized with the VR scene. Although the application has a playful appearance, and the virtual scene is setup as an entertainment car simulator, the rollover simulator gives car occupants the unpleasant sensation of being in an overturned vehicle. This is expected to cause that users immediately understand that wearing a seat belt is a serious matter, increasing their awareness in the use of the seat belt, which is the objective of the application. In fact, our initial hypothesis was that the use of the rollover car simulator and edutainment application would provide a significant increase in seat belt use awareness. This hypothesis is confirmed by the datasets collected with the use of the application.

The rest of the paper is organized as follows. Section 2 reviews the related work. Section 3 describes both the design and the system's architecture. Section 4 describes and discusses the experiments and their results. Finally, Section 5 draws the conclusions and outlines future improvements.

#### **2. Related Work**

VR could be defined as "the process, means and technologies by which one or several individuals experience the sensation of belonging to an alternative reality that is not the one they are actually living in" [11]. Although this alternative synthetic reality needs to be believable enough so that it is accepted by the participants, accurate recreations of all the perceptual stimuli are still impossible. In fact, this is often undesirable because the simulated actions could be, in some occasions, harmful. Thus, it is acceptable that the perception of belonging to the virtual world be only partial. The simulation of accidents is a perfect example of this situation.

A simulator is a system (it does not need to be computer-based!) that replicates a process, natural phenomenon, or experience. Simulators often use computers to solve internal models and/or provide visual outputs, but there are also simulators in which no visual output—or computers—are strictly necessary. Rollover simulators are an example of this. A rollover simulator is an engineering application in which the cab of a vehicle—or even a complete vehicle—is mounted on top of a motorized encasement/structure. The motor rolls over the vehicle, simulating a rollover accident [12].

There are many types of VR applications, including of course, entertainment, educational applications, and serious games. These applications can also be provided with other visualization and interaction paradigms, such as mixed reality (MR), augmented reality (AR), etc. Although many VR applications and simulators are designed as games, they do not have to be games. Thus, it is important not to confuse these terms.

There are many definitions of *edutainment*, but one of the most accepted is provided by Corona in several of his works [13,14]. He defines it as "the combination of education and entertainment in a learning process". Makarius [15] states in her work that "the process of educating in an entertaining way has been greatly facilitated for educators thanks to new technologies". Buckingham and Scanlon [16] state that "edutainment is based on attracting and maintaining the attention of students through the use of displays or animations to ensure that learning is fun". In fact, it has been shown that, thanks to the use of these new technologies, which usually include various stimuli (e.g., images, sounds, or videos), students are more likely to pay attention to the content, transferring it from their shortterm memories to their long-term memories [17]; thus, it essentially becomes knowledge to them.

There are many educators, in several areas, who are increasingly betting on the use of edutainment for the transfer of knowledge. Ma published a book [18] from which the close proximity between edutainment and the so-called *serious game* was extracted. A serious game (SG) is usually defined as "a computer-based game with a particular learning purpose" [19]. The term *gamification* refers to a slightly different concept. Gamification is the introduction of game-like mechanisms in learning applications, whereas a serious game is usually a full game with a learning purpose. As an example of the increase in the use of these educational systems, Zhonggen [20] analyzed the number of publications made between 2009 and 2019 related to serious game assisted education, using the search tool Web of Science. The results show that the number increased from about 20 publications in 2009 to more than 200 in 2017.

Aksakal [21] and Simon [22], in their works, include the concept of using edutainment for awareness. In this sense, serious games, gamification, and edutainment, have historically been used to raise awareness about good environmental practices, social behavior, cultural heritage, nutritional health, etc. [6,23–27]. Safety and health are two areas in which other types of IT-based applications can provided important benefits, using paradigms such as big data or artificial intelligence [28,29].

Focusing now on the use of serious games, gamification and edutainment to raise awareness about good driving practices, Riaz [30] conducted a study to evaluate the use of gamified e-learning to improve road safety in elementary school students. The study concluded—similarly to Klawe [31]—that strategies based on gamification are very positive for motivation in learning. Vera [32] developed a serious game based on a hybrid system with virtual reality and augmented reality, with the aim of raising awareness about road safety, concluding that the serious game was an effective tool to increase driving safety awareness, especially for younger people. This is good news for the future.

In a very recent publication, Gounaridou [33] proposed an SG in which a virtual character moves around a virtual city to complete a mission. The character needs to follow road safety rules as a pedestrian or as a vehicle driver. The results show that the SG could enhance road safety awareness and social responsibility. The use of the seat belt is not specifically targeted or analyzed. Other similar works [34–36] studied the use and benefits of using edutainment or SGs for traffic safety. It is generally accepted that gaming approaches can provide safe environments where users can practice and learn traffic rules and also recognize and manage dangerous situations [37].

However, the academic literature is very limited in relation to the use of these types of applications in raising awareness on seat belt usage. In aviation, Chittaro and Buttussi [38] evaluated the use of a serious game in relation to safety in airplane cabins, which included the use of seat belts. Although some road safety applications also include the use of seat belts in their learning goals, there is a lack of edutainment applications dedicated specifically to increase seat belt use awareness.

In regards to rollover simulators, several works exist in the academic literature where accidents are simulated, including simulated turnovers (i.e., simulated tests of rollover maneuvers), but no documented real rollover simulators (where a real car is physically turned upside down) have been found. Some rollover simulators can be found in the commercial field [12,39,40], but, to the best of our knowledge, no scholars have ever proposed an edutainment VR-based application synchronized with a rollover simulation, with the intention of being used to raise awareness on road safety. Thus, our proposal is innovative, original, and intends to target this research gap. The use of VR helps increase the immersion and presence of the system, complementing the rollover experience. Immersion and presence have an impact on memory [41]; therefore, it is expected that they help users remember the rollover experience, contributing to the fulfillment of the objectives of the proposed application.

#### **3. System Design**

This section describes the mechanical system, the simulation software, and the communication architecture between the different parts of the system.

#### *3.1. Mechanical System*

The rollover simulation system is based on a steel structure that is divided into three parts (see Figure 1). The first one (Figure 1a) is anchored to the ground and includes a powerful electrical motor, which will be used to roll the car over. The second part (Figure 1b) is supported by two large bearings. This is the moving (rotating) part of the system onto which the real car is placed and clamped. It is responsible for the generation of the motion cues of the simulator, although no motion cueing algorithm (MCA) [42,43] was needed in this application.

**Figure 1.** Hardware parts in which the main structure of the rollover simulator is divided; (**a**): supporting structure with the electrical motor (shown in blue color); (**b**): rotating structure onto which the real car is placed and clamped; (**c**): auxiliary structure to ease access to the car.

To convey a correct aesthetic appearance and facilitate the entry and exit of users to the car, the entire steel structure is surrounded by an aluminum structure (Figure 1c). The moving part integrates the car's clamping mechanism. Specifically, this mechanism was designed to anchor a 2016 *Kia Picanto*. This car model has been selected based on its dimensions, weight, ease of access to the chassis fixing bolts, and because it is a mid-range utility vehicle. Thus, it perfectly exemplifies much of the existing vehicle market.

The moving part of the system is driven by a 1.5 kW electrical motor and a 1:750 gearbox (in fact, there were two gearboxes connected in series, making a 1:750 gearbox system). To turn the moving part and the car with this electrical motor, very precise calculations were made to take into account the resulting center of mass with four users inside the vehicle and, thus, place the motor and the rotation axis in the optimal location.

The nominal angular speed of the engine is 1500 revolutions per minute (rpm). Thus, a 1:750 gearbox basically amplifies the output torque by a factor of 750, at the cost of reducing the output angular speed to a maximum of 1500/750 = 2 rpm. This means that it takes 30 s to turn the car completely (360◦ ), and 15 s to turn the car upside down (180◦ ). Of course, this is slower than the angular speed at which a real car overturns when it suffers an accident. However, for safety reasons, rotational speed should be kept slow in the simulator.

The motor is controlled by a frequency inverter using amplified analog signals provided by an Arduino Uno, which includes an Ethernet connection module that, through a router, allows commands to be sent using UDP sockets. In addition, for cases where manual control of the motor rotation is necessary, the Arduino Uno has a control system with two buttons (forwards and backwards), and an emergency stop pushbutton (mushroom-type button) connected directly to the motor break system. This way, the rollover car can be stopped immediately in case of emergency. The communication architecture will be described in Section 3.3, once the application is described in detail.

#### *3.2. VR-Based Edutainment Application*

To increase the immersion of the users, in addition to the actual rollover motion that they suffer thanks to the mechanical system, each person wears a VR HMD (VR glasses). Specifically, we used a Samsung Gear VR with an LG G4 smartphone inside. Given the high power consumption needed by the VR application, the smartphones remained always connected to the power supply. They were also numbered according to the seat they correspond to in the vehicle, since the visual perspective is different for each of the seats.

The VR application includes a recreation of the interior of a Kia Picanto, positioning the virtual camera according to the seat that each user occupies. A virtual road and landscape were also created, so that the virtual car is shown travelling through a two-lane road. The car is driven by an autopilot. Thus, the person in the driver's seat experiences the same sensations as the rest of the passengers.

At a certain random moment, the application produces a highly directional sound (suggesting a puncture or the breakage of a mechanical part) that causes confusion in the users. They usually stop looking forward and try to locate the source of the sound, as if it was a game of discovery. At this point, after a few seconds, the vehicle leaves the road, falls into an uneven sand embankment and overturns. As soon as the virtual vehicle leaves the road, the actual rollover of the car occurs. This is accomplished by synchronizing the virtual car with the electrical motor of the rollover simulator. Figure 2 shows a snapshot of the VR application.

When the (real) car has turned 180◦ , the electrical motor stops, thus holding the users for 15 s upside down (see Figure 3), so that they can experience through the VR-based edutainment application what happens to the objects in the car. In a real car, there are usually objects in the interior of the vehicle and they fall and cause injuries in the event of a rollover accident. VR is a safe way to simulate the fall of these objects without causing damage to users. Thus, all the virtual objects placed inside the vehicle interior (a pair of sunglasses, a pack of tissues, a soft drink, and a backpack) fall down. In addition, if the users have small objects in their (real) pockets, such as coins, tissues or candy, these objects usually fall down as well, increasing the perceived danger. Nevertheless, users are not allowed to test the rollover simulator with heavy or dangerous objects that can fall down and cause injuries.

After 15 s of holding the passengers upside down, the virtual simulation ends, and a command is sent to the Arduino Uno to turn the car another 180◦ . Therefore, the car returns to its original position, finishing the seat belt awareness experience.

**Figure 2.** Snapshot of the simulation as seen from the front passenger's position. The two images correspond to the right and left eye, since the VR application is a stereoscopic software.

**Figure 3.** The rollover simulator with the car turned upside down (180◦ turn).

Even though the real motion cues generated by the rollover simulator provide an uncanny feeling, the application is setup with an edutainment perspective. The visuals of the VR application are created using a cartoon look (see Figure 2). In addition, the mascot of the road safety campaign, called Hakeem, is shown on the dashboard of the car (see Figure 4). Hakeem also falls down when the car turns over. This fictional character is used throughout the road safety campaign to provide advices and warnings about road safety. Mascots and virtual assistants are common in SGs and edutainment applications [44,45]. Their playful look is essential to engage the younger audience [46,47], which is very important in the case of road safety. For this reason, the use of mascots is also very common in marketing campaigns of products targeted for children [47].

**Figure 4.** Hakeem, the virtual mascot of the road safety campaign, appears in the VR application.

#### *3.3. Communication Architecture*

The communication between the mechanical system and the mobile phones showing the VR application is achieved through a router to which both the mobiles phones (via Wi-Fi) and the Arduino Uno (using the Arduino Ethernet shield) that controls the electrical motors, are connected. There is also a control tablet, used by a system operator, which is connected via Wi-Fi as well. This tablet controls the simulation, allowing starting and resetting the edutainment application. It also allows performing an emergency stop and the direct control of the rotation of the electrical motor. Figure 5 shows the communication architecture.

**Figure 5.** Communication architecture. The system is composed of a control tablet connected to a router via Wi-Fi, four mobile phones connected also to the router via Wi-Fi, an Arduino Uno connected to the router with an Ethernet cable, and an electrical motor controlled by the Arduino Uno through a motor driver. This motor rolls over the real car. The users wear Samsung Gear VR glasses, where the cell phones are inserted.

The control operator is the person who controls the rollover experience. Once the users are in the car and with the seat belt properly fastened, he/she launches the simulation. In the unlikely event that any person suffers any problem, he/she can immediately realize (since the operator has direct vision of the rollover car and its passengers), abort the simulation and bring the car back to the rest position.

#### **4. Experiments and Results**

The system was installed in a road safety awareness campaign in Dammam, Saudi Arabia, organized by the state oil company Saudi Aramco. This campaign was oriented and organized so that the drivers came with their whole family. Thus, both old and young people could increase their road safety awareness. Young people are essential, since they will become the main drivers in the next decade and are prone to accidents. This type of campaign is common in the Kingdom of Saudi Arabia (KSA) (e.g., [48]), given the high accident rate and annual driving fatalities that the country suffers [49].

The campaign was organized in a circuit of five edutainment activities, one of them being two rollover simulators, such as the one described above. This attraction was the only one of the five activities focused on raising awareness about the use of seat belts. Figure 6 shows the two rollover simulators installed. The other four IT-based activities prepared for the road safety campaign were an AR road safety game shown in [32], a 5D interactive theater ([50] shows a preliminary version), an AR application for vehicle maintenance tips, and an interactive e-game with tablets for theoretical and practical driving support.

**Figure 6.** Two rollover simulators installed inside the road safety campaign in Dammam, KSA.

When the families arrived at the venue, they were first registered at a welcome desk (see Figure 7). Then, they were required to fill out a questionnaire on one of the 10 tablets enabled for this purpose. Once registered, they were provided with a tracking code that they used during the various activities. They also used this code to fill out a second questionnaire before leaving the venue. Since both questionnaires, pre and post, are known to belong to the same person thanks to the generated code, it is easy to see the differences between the *before* (pre) questionnaire and the *after* (post) questionnaire.

**Figure 7.** Register/welcome desk (**a**) and tablet used to fill out the questionnaires (**b**). The welcome desk was the entry and exit point of the campaign and users were prompted to fill the questionnaires upon entering and leaving the venue.

Table 1 shows the questions asked about the use of the seat belt before (pre) and after (post) going through the set of five activities. The complete questionnaire included other questions related to road safety. We only show in Table 1 the questions related to the seat belt. It is important to point out that none of the other four activities of the campaign was related to the seat belt. Therefore, it is expected that all the differences regarding awareness on the use of the seat belt be caused by the use of the presented VR-enhanced edutainment application.


**Table 1.** Pre (left) and post (right) questionnaires related to the use of the seat belt.

The campaign ran for four weeks in a row, being open to the public six days a week. In total, more than 5000 people passed through this road safety venue—not counting those under 8 years of age, who did not answer the questionnaires—. However, of all the users who tested the VR-enhanced rollover simulator and answered the questionnaires, some of them did not fill them completely or did it in an inconsistent way. For instance, some users always chose the first or the last option shown to them. Others filled all the answers with "Yes" or "No" answers. Others provided contradictory answers for similar questions. Finally, some people did not fill either the pre or the post questionnaire correctly. Thus, those users for which it was clear that either of the questionnaires was not seriously answered were discarded. Finally, 561 pairs of questionnaires were considered consistent, the results of which can be seen in Table 2.


**Table 2.** Results of the questionnaires.

As can be seen in the questionnaire, the desired answer for each of the questions was "Yes". As can be seen in Figure 8 and Table 2, in all of them a considerable increase in positive responses is observed when comparing the pre and post questionnaires.

**Figure 8.** Pre and post responses in the questionnaire. Blue bars represent the amount of positive responses before the use of the application (pre). Red bars represent the amount of positive responses after the use of the application (post).

Regarding questions A, B, and C, which were very much oriented to the use of the seat belt in the different seats of the car, it should be noted that, initially, only 23.17% of users perceived that it was important to wear the seat belt in the rear seats of the vehicle. After using the rollover simulator and edutainment application, this figure increased to 86.63%. Despite the large increase in awareness, there is still a significant variation between awareness of seat belt use in the front and rear seats. In the post questionnaire, 93.40% (driver seat) and 93.05% (passenger seat) of the users perceived the use of the seat belt in the front seats as important, compared to the 86.63% already mentioned for the rear seats.

On the other hand, questions D and F reflect two important concepts: whether the use of the seat belt is considered important and whether or not its use depends on the seat in the car. These questions show an increase from 62.57% and 53.12% in the pre questionnaire to a 97.33% and 92.69%, respectively, in the post questionnaire. These are very significant increases. Given the large amount of answers obtained (N = 561), these increases are statistically significant.

#### **5. Conclusions and Further Work**

The use of the seat belt saves thousands of lives every year. For this reason, the presented work shows a VR-enhanced rollover and edutainment application designed to increase awareness on the use of the seat belt. The use of VR provides three important benefits in this context: (i) it allows simulating the fall of objects safely; (ii) it provides a way to control the visual output of the simulator and modulate its playful look; (iii) it provides context for the rollover simulator, making clear that a rollover is not just a fabricated experience and can happen quite easily in a car accident.

Our hypothesis is that the use of this edutainment application would provide a significant increase in seat belt use awareness. The system has been tested for a month in the context of a road safety exhibition. More than 500 users tested and assessed the usefulness of the system. We measured—before and after the rollover experience—the perception of risk of not using the seat belt.

From the results obtained, it can be seen that users had a very low initial awareness regarding the need to use seat belts in cars, especially in the rear seats. However, after the experience in the edutainment rollover simulator, it is observed that this awareness increased very significantly, not only in relation to the rear seats, but also to the use of the seat belt in general. Therefore, the presented edutainment application, combining VR, a playful look, and real motion, is able to fulfil its goals, as it provides a significant increase in the awareness of the use of the seat belt, showing the potential that these kinds of applications can offer, and confirming our research hypothesis.

It should be noted that this increase in awareness occurs after the use of a simulator that only rotates at 2 rpm, which is far from the angular velocity that users would feel in a real accident. Real accident accelerations and angular velocities are several orders of magnitude higher. Of course, we did not want to hurt anyone. Therefore, a slow motion system was implemented. Despite this apparently important limitation, the application is able to generate a significant impact on the users. Indeed, the feeling of being upside down trapped in a car that has overturned is as real as it can be in a real accident (blood and injuries aside). The authors have tried the simulator themselves, and although the rollover motion is fun at first, the feeling of being upside down in a car, even for a few seconds, is unpleasant. In that position, you realize that the seat belt can prevent you from hitting the interior of the car when you suffer an accident.

In the future, we intend to introduce interaction with users in the system, so that instead of being an autopilot simulation, the users themselves would drive the vehicle inside the VR system. Thus, if they lose control of the car and suffer an accident, the rollover simulator will provide the motion cues so that users feel how the car overturns. It is possible that this new setup will increase the concentration of the driver and, therefore, the surprise when losing control, further increasing awareness on the need to wear a seat belt. We also plan to test the system under different configurations (angular speed, duration, virtual content - including also the possibility of not using VR at all -, etc.), so that we can evaluate the influence of these factors in the increase of seat belt use awareness.

**Author Contributions:** Conceptualization, S.C. and F.A.; methodology, S.C. and F.A.; software, S.C. and J.V.R.; validation, J.V.R., F.A., and M.F.; formal analysis, F.A.; investigation, J.V.R. and M.F.; resources, J.V.R. and M.F.; data curation, J.V.R.; writing—original draft preparation, J.V.R.; writing review and editing, J.V.R. and S.C.; visualization, J.V.R. and S.C.; supervision, F.A. and M.F.; project administration, M.F. and F.A.; funding acquisition, M.F. and F.A. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the R&D contract "Development Installation and Support of an Educational Road Safety Campaign in the Framework of the 'House of Knowledge' Campaigns in the Scope of the 'Traffic Safety Signature Program' of Saudi Aramco" between the University of Valencia and Sahara Consultancy.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** All subjects gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki.

**Data Availability Statement:** Data sharing not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Tracy Arner <sup>1</sup> , Kathryn S. McCarthy <sup>2</sup> and Danielle S. McNamara 1, \***


**\*** Correspondence: dsmcnama@asu.edu

**Abstract:** Literacy skills are critical for future success, yet over 60% of high school seniors lack proficient reading skills according to standardized tests. The focus on high stakes, standardized test performance may lead educators to "teach-to-the-test" rather than supporting transferable comprehension strategies that students need. StairStepper can fill this gap by blending necessary test prep and reading comprehension strategy practice in a fun, game-based environment. StairStepper is an adaptive literacy skill training game within Interactive Strategy Training for Active Reading and Thinking (iSTART) intelligent tutoring system. StairStepper is unique in that it models text passages and multiple-choice questions of high-stakes assessments, iteratively supporting skill acquisition through self-explanation prompts and scaffolded, adaptive feedback based on performance and self-explanations. This paper describes an experimental study employing a delayed-treatment control design to evaluate users' perceptions of the StairStepper game and its influence on reading comprehension scores. Results indicate that participants enjoyed the visual aspects of the game environment, wanted to perform well, and considered the game feedback helpful. Reading comprehension scores of students in the treatment condition did not increase. However, the comprehension scores of the control group decreased. Collectively, these results indicate that the StairStepper game may fill the intended gap in instruction by providing enjoyable practice of essential reading comprehension skills and test preparation, potentially increasing students' practice persistence while decreasing teacher workload.

**Keywords:** reading comprehension; strategy training; game-based learning; intelligent tutoring system; feedback

#### **1. Introduction**

Literacy refers to "the ability to understand, evaluate, use, and engage with written texts to participate in society, to achieve one's goals, and to develop one's knowledge and potential" [1] (p. 61). Literacy skills are not only critical for educational and career success, but the ability to read and comprehend various text types across multiple subjects is necessary to function in everyday life. However, national reading assessment data suggests that many students struggle with reading comprehension. The most recent National Assessment of Educational Progress [2] on reading skills found that 63% of twelfth-grade students were below proficient in reading. Similarly, 66% of eighth-graders and 65% of fourth-graders were also below proficiency. These numbers suggest that additional instructional support is needed to improve students' reading achievement as they progress through grade levels.

The emphasis placed on standardized testing has increased in the last few decades to ensure that all students (i.e., non-White, lower-income) are receiving equal, high-quality instruction and to monitor adequate yearly progress (AYP) [3,4]. These goals are admirable but have fallen short of the intended targets as many students have failed to learn essential skills and strategies necessary to become part of the global workforce while preparing for

**Citation:** Arner, T.; McCarthy, K.S.; McNamara, D.S. iSTART StairStepper—Using Comprehension Strategy Training to Game the Test. *Computers* **2021**, *10*, 48. https:// doi.org/10.3390/computers10040048

Academic Editors: Carlos Vaz de Carvalho and Antonio Coelho

Received: 16 March 2021 Accepted: 6 April 2021 Published: 9 April 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

standardized assessments [4,5]. Amidst increasing pressure and limited instructional hours, teachers may resort to "teaching to the test" in an effort to demonstrate learning gains on these standardized assessments [3,5,6]. Unfortunately, this practice leads to inaccurate inferences about the knowledge and skills that students have acquired and unreliable inflation in scores on state-level standardized assessments that are not achieved on the NAEP [5,7], nor do these types of tests reflect the types of literacy tasks that the student will encounter outside of the testing room [8,9]. The result is that students fail to develop the comprehension strategies that will better serve them outside of a standardized test [10].

There are multiple barriers to building students' reading comprehension strategies. Developing comprehension strategies requires ample opportunity for cycles of deliberate practice and targeted feedback. One issue is that providing feedback on ill-structured tasks like reading comprehension is both time- and resource-intensive for instructors and students alike. Thus, students have few opportunities to practice with one-on-one support. A second issue is that students may become disengaged from the strategy-building activities before they have mastered the skills [11].

With these issues in mind, we developed a game-based module, StairStepper, to implement an intelligent tutoring system, iSTART. StairStepper was designed to support and reward the use of reading comprehension strategies in the context of a mock standardized reading comprehension test. Our aim was to leverage the power of automated evaluation and game-based principles to offer a scalable, efficient, and fun way for students to develop their reading comprehension skills that can transfer to high-stakes testing environments.

To contextualize StairStepper, we first provide a brief background on the theoretical and educational motivation for reading comprehension strategies as well as an overview of the broader intelligent tutoring system, iSTART, in which StairStepper was built.

#### *1.1. Reading Comprehension Strategies*

Theories of discourse comprehension suggest that as learners read, they construct mental models or mental representations of the text information [12]. The mental representation that readers construct as part of comprehension is comprised of multiple levels or layers, which include the (1) surface code, (2) textbase and, (3) situation, model. The surface code includes specific words and syntax but is unlikely to be retained except in cases of rote memorization. This immediate textual information gives rise to the textbase or gist meaning of the text. In the situation model, the reader goes beyond the present text to make inferences and to elaborate from prior knowledge [12–14]. Readers must develop a coherent mental model, beyond the surface level, for text comprehension and knowledge transfer [12,15].

There is ample evidence to suggest that prompting and training students' comprehension strategies improves reading comprehension [16–20]. One such strategy shown to benefit comprehension is self-explanation [21–24]. Generating an explanation to oneself aids in the integration of new information with prior knowledge; these connections support the construction of a more elaborated and durable mental representation of the content [22]. Despite the benefit of using reading comprehension strategies, students may not adopt and use these strategies on their own [10,17,23]. However, there is ample evidence of the benefit of strategy training and practice, specifically self-explanation training [17,23–26], particularly with low-knowledge or struggling students [18,27].

Self-explanation reading training (SERT) improves students' reading comprehension through instruction on five active reading strategies (comprehension monitoring, paraphrasing, predicting, bridging, and elaborating) that lead to generating high-quality self-explanations, which in turn, improves text comprehension [17,18]. The comprehension monitoring strategy encourages the reader to continuously evaluate whether or not they understand what they just read [22]. Comprehension monitoring is an inherent feature of generating self-explanations because if the student is not able to successfully explain what they just read, it is an indication that there is a breakdown in understanding. Thus, comprehension monitoring provides an indicator that the reader needs to employ strategies

to repair the gap in knowledge [28]. Skilled readers are more likely to engage in comprehension monitoring, notice inconsistencies in the text, gaps in understanding, and use strategies to repair gaps when they do not understand [28,29]. Paraphrasing is a frequently used strategy in which the reader restates the content of the text in their own words [17]. Although this strategy focuses mainly on developing a textbase, putting the text into one's own words is an important step toward more meaningful processing. A prediction is when the reader speculates about what they think might happen next in the text [17]. While predictions are relatively infrequent during reading, they support comprehension by encouraging the reader to consider more global aspects of the text [30,31]. The last two strategies are similar in that they are both generations of inferences, bridging and elaborative. Bridging inferences are those that connect a statement to a prior sentence or passage in the text. In contrast, an elaborative inference occurs when the reader connects the current text to prior knowledge [32,33]. Generating inferences is an essential component of reading comprehension.

Helping students to use active reading strategies is effective for elementary [34,35], middle [23,26], and high [17,18,36] school students as well as young adults (i.e., college students) [17,37]. However, simply prompting students to use these strategies or providing direct instruction about the strategies is only part of the process of improving students' reading skills. That is, in addition to instruction, students also need ample time to engage in deliberate practice where they are able to use the strategies while receiving feedback on how to improve [38,39]. Although strategy instruction and practice have pronounced benefits for literacy skills [40–42], it is sometimes difficult to keep students engaged and motivated so that they keep practicing. One potential method to encourage the training and practice of these beneficial strategies is through the use of automated intelligent tutoring systems (ITSs) that provide a mechanism for more engaging, game-based practice [39,43,44]. Given the ample evidence of the benefit of strategy training and teachers' limited time to teach strategies, intelligent tutoring systems may be useful in filling that gap as they can provide adaptive feedback inside an engaging, game-based activity that may increase students' motivation to engage in deliberate practice of reading comprehension strategies.

#### *1.2. Intelligent Tutoring Systems*

Computers have been used to support learning for the last few decades [45,46], initially in the form of computer-assisted instruction and, more recently, as intelligent tutoring systems (ITSs) [47]. Meta-analyses suggest that computer-assisted instruction (CAI) and ITSs have positive impacts on learning [45,47,48]. ITSs differ slightly from other CAI systems in that they attempt to emulate the one-on-one tutoring experience through adaptive instruction and more granular feedback [46,47,49]. For example, students may receive stepwise feedback (i.e., correct/incorrect, solution hint) during problem-solving (i.e., error detection), and they may also be able to engage in natural dialog with the system emulating a human tutor [37,47]. Comparisons of learning outcomes between human, CAI, and ITS systems suggest that, while the more sophisticated ITSs may be more beneficial to learning than some CAI systems, they are still not quite as effective as human, one-on-one tutoring, which is considered the "gold standard" of instruction [45,47,50]. One feature that may make an ITS system more similar to one-on-one tutoring while also providing actionable feedback and increasing student motivation is the addition of a pedagogical agent [51,52].

Pedagogical agents are characters in technology-based instructional applications designed to facilitate learning [52,53]. The interactions that the agent has with the learner may serve to provide instruction, feedback, or motivation [37,54–57]. The addition of a pedagogical agent may facilitate or increase interaction between the learner and the intelligent tutoring system [58–60]. Pedagogical agents may be a "talking head" that provides information via text or audio comments, or they may be full-body characters who have animated gestures that can be used for additional learning supports, such as signaling [58,59,61,62]. Anthropomorphizing an intelligent tutoring system with a pedagogical agent that has a

human-like figure, voice, or both (i.e., *persona effect*) [63] may further increase students' motivation to engage with the intelligent tutoring systems [64–67]. This *persona effect* can lead students to view the engagement with the pedagogical agent as a social interaction similar to what would occur with a human tutor [68]. Thus, students have more positive perceptions of the learning environment and are more accepting of instructions or feedback from the pedagogical agent, which may aid in learning or motivation to persist [59,63,65].

Feedback, broadly defined, is information provided about one's performance. It may also include the difference between one's performance and the learning objective or goal [69,70]. The influence of feedback on student learning outcomes has myriad evidence evaluating its efficacy across task type, subject area and grade levels [71–75]. The general consensus is that feedback has a positive effect on student learning outcomes through the benefit may be moderated by learners' prior knowledge, context, timing, and type of feedback [71,76–79]. Feedback provided by a pedagogical agent in an ITS might be goal-driven (i.e., response correctness), instructional (i.e., hint or strategy suggestion), or affective (i.e., positive reinforcement to continue), which may motivate the learner to continue with the task or practice in the ITS [80,81]. For example, learners with low priorknowledge experience a greater benefit from explanatory feedback (e.g., "That answer is incorrect because...") than basic corrective feedback (e.g., right or wrong) [77,82].

Students' motivation to engage in a task or persist through struggle is positively related to their achievement [83–86]. The more motivated a student is to engage with a learning task, the more likely they are to complete the task, thereby achieving the learning goal [87]. Motivation to persist in the practice necessary to improve reading comprehension skills may be bolstered by the affordances of ITSs [48], particularly those with anthropomorphized feedback mechanisms (e.g., pedagogical agents) [68] and gamebased learning and assessment [39,40,88].

#### *1.3. iSTART*

Interactive strategy training for active reading and thinking (iSTART) is an intelligent tutoring system (ITS) based on SERT. iSTART provides self-explanation followed by gamebased practice. The iSTART system first provides overview lessons on each of the selfexplanation strategies (i.e., paraphrasing, bridging and elaborative inferences, prediction and comprehension monitoring) using video instruction and modeling [89]. During the generative practice, students are given passages to read and then asked to self-explain target sentences. Students' responses are evaluated using natural language processing algorithms that detect evidence of the different comprehension strategies. This algorithm is used to provide a summative score (0–3) as well as formative feedback indicating ways to improve their self-explanation. For example, when responses are too short or too long, Mr. Evans, a pedagogical agent, provides various types of feedback that can help the student to write higher quality self-explanations [26,89,90].

Although the original system demonstrated positive impacts on learner's self-explanation and reading comprehension, it was difficult to keep students motivated in repeated rounds of guided practice. Thus, iSTART-motivationally enhanced (ME) [39,91] introduced additional motivational features via game-based practice. iSTART includes both generative and identification games. In *generative* games, students practice writing self-explanations. For example, students can play "Self-Explanation Showdown", in which they play against a CPU in a head-to-head competition. In *identification* games, students view example self-explanations and need to correctly identify the strategy. Reaching new high scores or levels earns trophies as well as additional "iBucks", the system currency units, which can be used to open and play more games or customize their player avatar. These game-based features support learning in that they may encourage students to engage in prolonged practice, which is critical for developing reading comprehension skills [39,92].

#### *1.4. StairStepper*

Building upon iSTART's tradition of game-based literacy practice, StairStepper was designed to provide engaging, a game-based practice that closely approximates question types that students experience in standardized assessments. More specifically, StairStepper gamifies the use of scaffolding to challenge the student to read increasingly difficult texts. Thus, the goal of StairStepper was two-fold; to (1) provide students generative practice of self-explanation strategies that will benefit their reading comprehension skills while simultaneously (2) preparing them for the standardized assessment texts and questions that they will see throughout their educational careers.

#### 1.4.1. Reading Comprehension Strategies for Standardized Testing

Traditional standardized reading assessments are designed to isolate and evaluate reading comprehension skills. For example, The National Assessment of Educational Progress [2] reading assessment is used ubiquitously in K-12 education. The assessment for grade four consists of two texts that students read and then respond to approximately 20 questions that are either selected response (i.e., multiple-choice) or constructed response (i.e., open-ended text entry). The questions are written to assess three types of *cognitive targets* or the kinds of thinking that underlie reading comprehension: locate and recall, integrate and interpret, and critique and evaluate.

The "locate and recall" cognitive target requires students to recall content from the text to answer the question. While students do have the option to refer back to the text, data show that students frequently do not do so [2]. Each of the reading comprehension strategies that students learn about and practice in iSTART can support performance on these types of tests. In StairStepper, like the assessment, students must decide if they *definitely know* the answer to the question, *definitely do not know*, or *might know* the answer. Practicing the comprehension monitoring strategy in the StairStepper game can help students be better prepared to make a clear decision about what they do and do not know when responding to questions. The second target, "integrate and interpret," requires students to make complex inferences within and across texts to derive meaning, explain a character's motivation or action, or uncover the theme of the text. The bridging and elaboration strategies that are practiced in StairStepper are the same strategies that are used to make these complex inferences when responding to the "integrate and interpret" target questions. The third question type, "critique and evaluate", requires students to think critically about text and evaluate aspects of it using a variety of perspectives based on their knowledge of the world. The paraphrasing and elaboration strategies encourage students to think about the text in ways as if they were going to explain it to another while also using their knowledge of the world to make sense of the content. Despite the benefit of reading comprehension strategies, they are often put to the side as teachers focus on preparation for standardized tests when, in fact, these strategies can and should be leveraged in standardized testing environments.

#### 1.4.2. Text Set and Questions

The first step in designing the StairStepper game was to develop and evaluate a corpus of texts and corresponding questions that would emulate these standardized assessments. The texts and their accompanying questions were retrieved from publicly available educational resources. The text topics span multiple domains, including knowledge gained in school (i.e., science and social science) and knowledge gained in daily life (i.e., sports and pop culture). Texts range from seven to 80 sentences in length. Rather than relying on shallow measures of readability, the texts were leveled through comparative judgments made by independent raters (for description, see [93]). The initial set of 172 texts was separated into 12 levels of increasing difficulty. These rater judgments of difficulty were correlated with both Flesch–Kincaid grade level (*r =* 0.79) and Dale–Chall readability (*r* = 0.77) [94]. After inspection and piloting, the final text set was reduced to 162 leveled texts chosen to

mimic those students may see in reading comprehension assessments taken in classrooms every year.

The full corpus of multiple-choice questions from these texts was piloted to check for floor and ceiling effects. Some items were removed or slightly edited for clarity. The remaining items were then categorized by question type based on the source of the knowledge required to answer the questions correctly. Questions categorized as *textbase* (*N* = 677) can be answered from information found in a single sentence in the text. *Bridging inference* questions (*N* = 160) require the reader to combine information from two or more sentences in the text. Finally, *elaboration* questions (*N* = 144) require the reader to use the information found in the text and prior knowledge to answer correctly. Lower level texts (below level 8) have a higher percentage of textbase questions (>70%), whereas the higher-level texts (level 8 and above) had fewer textbase questions (35–45%) and more bridging inference (45–55%) and elaboration (8–10%) questions.

The texts and question types used in StairStepper are representative of various standardized assessments that students are likely to encounter in their educational careers. Therefore, these questions may be useful in helping students prepare for standardized assessments. Furthermore, StairStepper also provides students a more engaging way to practice reading comprehension strategies that are supported by evidence from numerous studies [17,18,90]. The underlying benefits of practicing these strategies inside the StairStepper game are that students may be more motivated to persist in practice, and they may be more likely to transfer the strategy used to the standardized assessments taken in the future.

#### 1.4.3. Game Play

The goal of StairStepper is to ascend to the top stair by answering comprehension questions about increasingly difficult texts. Students begin the game with instructions on their task and a reminder of the strategies (comprehension monitoring, paraphrasing, prediction, bridging, elaboration) that they can use when writing self-explanations.

The game begins with the student's avatar on step five of twelve, where they are presented with a text of low-moderate difficulty (iSTART's default setting begins at level 5, but this is an adjustable feature). In the first text, they are not prompted to self-explain. At the end of the passage, they are asked to answer a series of multiple-choice questions about the text. Students who meet the correct response threshold (75%) on the multiple-choice questions are promoted to the next step and begin a new, slightly more difficult passage. In contrast, students who answer less than 75% of the question incorrectly receive another text at the same level. In this second text, the student is prompted to self-explain at various target sentences. In the first phase of scaffolding, students receive a score of the quality of their self-explanation on a color-coded, four-point scale ranging from Poor to Great (See Figure 1).

As depicted above, the StairStepper game includes iSTART's Mr. Evans, who serves as a guide through the game-based practice. He provides three types of information to students during gameplay; task instructions, feedback, and progress messages. The task instruction messages let students know what they need to do or what will happen next. For example, when the student begins StairStepper, Mr. Evans tells them that they will read the text and answer the questions (see Figure 2). Statements like this are provided any time there is a change in procedure, such as when a student moves between scaffolded levels.

Second, Mr. Evans provides feedback to the students. One type of beneficial feedback is motivational (i.e., praise) [47,49,52], such as telling the student, "Great, you got that one right." when they answer a multiple-choice question correctly (Figure 3). Mr. Evans will also provide metacognitive prompts that require students to think about and identify what self-explanation strategy they used (bridging, elaboration, paraphrasing) [17].

**Figure 1.** Players receive feedback on their self-explanation quality from the system.

**Figure 2.** Mr. Evans gives the player instruction on what will happen next in the game.

**Figure 3.** Mr. Evans provides feedback on comprehension practice. Player avatar moving up to the next level.

Students' After submitting their self-explanations for the entire text, students are given between 5 and 20 multiple-choice questions, depending on the text. If the student again receives a score below the threshold (75%), the next text includes prompting for self-explanation and feedback on the quality of the self-explanations with an opportunity to revise. Thus, students can receive three support levels (no SE, SE, SE + feedback; see Figure 4). If the student continues to struggle, the text difficulty is decreased, and as their comprehension improves, the subsequent texts become more challenging. Students' progress through the

game on students' perceptions and motivations

game follows this same cycle of assessing comprehension at each level of text difficulty, and when the minimum is not met, students are provided scaffolded strategy training and feedback to aid in text comprehension. Students'

**Figure 4.** Scaffolded support process in the StairStepper game.

#### *1.5. Present Study*

game on students' perceptions and motivations The iSTART research team continues to refine and evolve the types of game-based activities available in the ITS. The purpose of the present study was to investigate the effects of the new game-based adaptive literacy module, StairStepper. More specifically, we examined the potential benefit of the scaffolded support design of the StairStepper game on students' perceptions and motivations, as well as the effects of short-term practice with StairStepper on reading comprehension skills. We sought to answer three research questions with the present study.


College students (*n* = 51) completed the iSTART lesson videos and a round of Coached Practice. They then engaged in 90 min of StairStepper practice. Students were asked to complete a questionnaire about their experiences to measure their enjoyment and interest in the game-based practice and their self-reported sense of learning. To explore the efficacy of StairStepper, half of the participants (*n* = 25) were assigned to a 3-day treatment condition that received a pretest, iSTART/StairStepper, and then a post-test. The other half (*n* = 26) were assigned to a delayed treatment control in which they completed a pretest, a posttest, and then the iSTART/StairStepper training. We hypothesized that students in the StairStepper treatment condition would show pretest to post-test improvement on the proximal outcome of standardized Gates–MacGinitie reading test (GMRT) score and the more distal comprehension scores.

#### **2. Results**

#### *2.1. Perceptions of the StairStepper Game*

Our first question regarded students' enjoyment of StairStepper. Our purpose for building StairStepper was to include a fun and motivating test prep module in a way that aligned with the purpose of iSTART (reading comprehension strategy training). Thus, it was important to investigate the extent to which participants enjoyed the game-based features of StairStepper. To this end, we asked students to answer survey questions regarding their experiences and perceptions of StairStepper. These analyses include 48 students, including those in the delayed treatment control, who played StairStepper after their post-test assessment. Three students did not complete the perceptions portion of the study.

As shown in Figure 5, participants rated their experience with the game interface (e.g., objects in the game) and game features (e.g., visual appearance) as well as personal attributes as they related to learning in game environments (e.g., goal setting). Overall, participants had positive attitudes about the StairStepper game as a method to practice reading comprehension strategies (see Figure 5).

**Figure 5.** Participant responses on 5-point Likert scale questions on perceptions of the StairStepper game. Three participants did not complete the perceptions survey.

" " " " " " We conducted Wilcoxon signed-rank tests to evaluate whether or not participant responses were significantly different from neutral. Results revealed that three items were indeed significantly positive: "Objects were easy to control" (*p* < 0.000); "Environment responded accurately" (*p* = 0.02); and "I wanted to perform well" (*p* < 0.000). The other perceptions items were not significantly different from neutral, suggesting that the students did not have negative opinions of StairStepper.

We further analyzed participants' perceptions of th influence of reading skill on participants' perceptions of the StairStepper game (see Table n participants' agreement with " " " " " " We further analyzed participants' perceptions of the StairStepper game as a function of reading skill using a median split on the GMRT pretest scores to determine the influence of reading skill on participants' perceptions of the StairStepper game (see Table 1). Results indicated a significant difference on participants' agreement with "Enjoyed the practice environment" (*t*(1, 45) = 3.45, *p =* 0.001), "Interface had game-like features" (*t*(1, 45) = 2.16, *p =* 0.04), and "I would use for other skills" (*t*(1, 45) = 1.98, *p =* 0.005). These results suggest that participants who had lower reading comprehension skills found the StairStepper game more enjoyable than those who were more proficient in reading. These results may stem from proficient participants not believing that they were benefiting from the StairStepper practice module.

**g** 

72) \*\*

2) 938 353


**Table 1.** Participant perceptions of the StairStepper game in iSTART.

Note: \* *p* < 0.05, \*\* *p* < 0.01.

#### *2.2. System Data*

Our second question regarded students' traverse through the system in terms of whether they descended or ascended the "stairs" or text difficulty. To this end, we conducted a visual inspection of the log data. Figure 6 shows each participants' trajectory through StairStepper with text number along the *x*-axis and text difficulty along the *y*axis. The participants are ordered based on their pretest GMRT score and color-coded accordingly. This visual inspection demonstrated two important findings. First, the game architecture was responsive to participants' reading skills. Participants with lower GMRT scores were given less difficult texts; more skilled readers ascended to the most difficult texts more quickly. Second, these graphs also demonstrate that many of the participants showed some decreases and increases in text difficulty, suggesting that the different amounts of scaffolding (self-explanation, feedback, text leveling) were effectively providing just-in-time support.

#### *2.3. Reading Comprehension*

Our third question regarded the impact of StairStepper on reading comprehension skills. Reading skills are generally impervious to relatively brief treatments, as in this study. For example, observed increases in self-explanation and comprehension skills generally have required at least 4 to 8 h of instruction and practice [17,18,39]. Yet, given that students in the StairStepper treatment condition received explicit instruction and practice on selfexplanation and comprehension strategies, one of our objectives was to examine the extent to which this brief game-based practice impacted their ability to comprehend challenging science texts as well as their performance on the GMRT and comprehension of a science text. The GMRT texts are similar to the practice texts in StairStepper, whereas the science text included textbase and open-ended inference questions. Descriptive data and correlations between the measures are presented in Table 2.

Our second question regarded students' traverse through the system in terms of " " ed a visual inspection of the log data. Figure 6 shows each participants' trajectory

−0

−0

Participants' log data shows their progression through the StairStepper texts (number sues: (1) the "jump" in participant 5' should have "won" the game after completing two level **Figure 6.** Participants' log data shows their progression through the StairStepper texts (number along the *x*-axis) as a function of text difficulty at each level (*y*-axis). These data revealed two system issues: (1) the "jump" in participant 15's data reveals a system crash, and (2) several of the more skilled readers should have "won" the game after completing two levels 12 texts (e.g., 45). However, system settings prevented the game from ending. These issues were reported to the programmer and addressed.


**Table 2.** Descriptive statistics for comprehension measures.

architecture was responsive to participants' reading skill

Note: pretest text is *Red Blood Cells,* and post-test text is *Cell Repair* to assess comprehension of challenging science texts. \*\* *p* < 0.01.

We conducted preliminary analyses to examine whether there was a significant difference in reading comprehension skills between groups. Results of an independent samples t-test conducted on participants' GMRT pretest scores indicated that there was a significant difference (*t*(1, 53) = −2.59, *p* = 0.012) in pretest means between the delayed treatment control (*M* = 0.44, *SD* = 0.22) and the StairStepper training condition (*M* = 0.58, *SD* = 0.19). Similarly, results of independent samples t-test conducted on students' science comprehension (i.e., *Red Blood Cells*) pretest mean scores indicated a significant difference (*t*(1, 53) = −2.13, *p* = 0.038) in mean scores between the delayed-treatment control (*M* = 0.30, *SD* = 0.19) and the StairStepper training condition (*M* = 0.42, *SD* = 0.22). As such, pretest scores were included as covariates in the analyses to control for prior reading skills.

#### 2.3.1. GMRT

We examined the extent to which 90 min of StairStepper practice impacted performance on a standardized reading comprehension measure. To account for group differences, we conducted a t-test on pretest to post-test change scores to evaluate the impact of

the StairStepper practice game on students' reading comprehension. Results indicated that there was a significant difference in pretest to posttest change (*t*(1,49) = −2.72, *p* = 0.009; Figure 7) between the StairStepper training group (*M* = 0.04, *SD* = 0.12) and the delayed treatment control (*M* = −0.04, *SD =* 0.09). While we did not have hopes of observing substantial gains from such a short training session and single, 90 min practice session on a standardized test, such as GMRT, these results suggest that the StairStepper practice game has strong promise in helping students to improve their reading skills and performance on similar tests. − −0

**Figure 7.** Gates-MacGinitie reading test (GMRT) pretest to post-test change as a function of condition. Error bars indicate standard error.

#### 2.3.2. Science Comprehension

A 2 (question type: textbase, bridging inference) by 2 (condition: control, StairStepper training) analysis of covariance (ANCOVA) was conducted to examine the effect of StairStepper training on the two different question types. Item type was included as the within-subjects factor, condition as a between-subject factor, and performance on the pretest comprehension test was included as a covariate. There was no significant effect of question type, *F*(1, 48) = 0.580, *p* = 0.45, nor was there any effect of StairStepper training *F*(1, 48) = 1.28, *p* = 0.26). There was also no interaction effect, *F*(1, 48) = 1.21, *p* = 0.27 (Figure 8).

**Figure 8.** Comprehension question scores as a function of question type and condition.

#### **3. Discussion**

participants' perceptions of the new game and to examine the possible benefits of The present study investigated a new game designed to provide students with an engaging environment to practice reading comprehension strategies while simultaneously preparing for standardized reading comprehension assessments. StairStepper, housed in the iSTART intelligent tutoring system, uses adaptive text and scaffolded feedback support to guide students through self-explaining and answering questions about increasingly challenging texts. The goal of this study was to evaluate participants' perceptions of the new game and to examine the possible benefits of StairStepper practice as measured by lab-designed comprehension measures and standardized (GMRT) performance.

" " students' motivation and engagement [39]. Results indicated that participants had positive attitudes about the StairStepper game as a way to practice the reading comprehension strategies. Specifically, participants considered the objects in the environment to be easy to control and that the game provided an accurate reflection of their performance. In addition, of note is that a significant number of participants reported "wanting to do well" while engaging with the system. Indeed, these results align with prior work on the use of the game-based practice to support students' motivation and engagement [39]. Interestingly, the participants, who had lower reading comprehension skills, had more positive attitudes about the game environment and the game features. Furthermore, they indicated that they would use this game to practice different types of skills. One explanation for these results is that the benefit of gameplay may have been more salient to those participants who had lower reading comprehension skills. These results align with prior research indicating that the students who have lower reading comprehension scores garner a greater benefit from self-explanation reading training [18,36] and strategy training in iSTART [95]. While these studies investigated self-explanation reading training and reading comprehension training in iSTART for longer durations, the participants' perceptions of StairStepper in the present study are promising.

training in iSTART for longer durations, the participants' perceptions of StairStepper in participants' scores on a standardized reading assessment (i.e., Gates– We evaluated the influence of iSTART training and StairStepper practice on participants' scores on a standardized reading assessment (i.e., Gates–MacGinitie reading test, GMRT). Preliminary results indicated that there was a significant difference between groups at the pretest. To account for differences between groups, we analyzed change scores from pretest to post-test on the GMRT and found that the participants in the StairStepper game condition maintained their reading comprehension score, while those in the delayed treatment control experienced a significant decrease in comprehension score from pretest to post-test. These results suggest that the StairStepper practice game benefited participant

maintenance and use of the reading comprehension strategies. This aligns with prior research suggesting that reading comprehension strategies need to be practiced in order for students to consistently adopt and use them [40,41].

This study also investigated the influence of the short iSTART training session followed by 90 min of practice on the StairStepper game on participants' reading comprehension scores. The results indicate no significant effects of training on open-ended comprehension scores. These results may reflect the need for larger sample sizes to detect effects as a function of pre-training differences. That is, pretest scores on comprehension and GMRT were strongly predictive of post-test scores. Given that less-skilled readers found the game more valuable, it may be that these students would benefit more from StairStepper and from extended practice. Indeed, these results suggest that students may need more training and practice than occurred in this study (i.e., one session of training and one session of practice). Evidence from prior studies indicates that consistent adoption of strategy use requires extended, deliberate practice [95]. Therefore, additional work is needed to investigate the number of practice sessions that may result in participants' efficient use of different types of reading strategies and the extent to which this supports performance on textbase and bridging question performance.

Taken together, these results suggest that the students who received self-explanation strategy training and StairStepper game-based practice did benefit in that their reading comprehension scores remained stable. Conversely, participants in the delayed-treatment control group experienced a significant decrease over the course of the three-day study. While we did not expect to see an increase in reading comprehension skills after a short training and practice session, these results indicate that there is a benefit to students' motivation to perform well on the test. Further work is needed to evaluate the practice dosage (i.e., number of sessions) and duration (i.e., length of sessions) that may lead to long-term improvement in reading comprehension skills. Additionally, larger studies will also allow us to more rigorously investigate how StairStepper training varies across different individual differences, such as reading skills.

The positive attitudes that participants reported about the StairStepper game and the maintenance of reading comprehension scores on a standardized assessment are promising. The goal of this work was to develop a game-based module in the iSTART intelligent tutoring system that would be engaging for students to practice using self-explanation strategies while also preparing them for the standardized assessments that they will experience throughout their educational career. Additional work is needed to investigate the dosage (i.e., how many practice sessions) and the durability (i.e., how long will strategy adoption last) that is most beneficial for this type of game-based practice. In sum, the StairStepper game-based practice module may serve an important role in students' acquisition of and long-term adoption of self-explanation strategies that contribute to reading comprehension and literacy skills.

#### *Limitations and Future Directions*

While the results of this study are promising, we acknowledge some limitations that should be considered in future work. First, the StairStepper game was designed as an engaging way for high school students to practice self-explanation strategy use while preparing for standardized assessments that are common throughout K12 education. However, our sample comprised undergraduate students who were earning course credit as part of the participant pool. As demonstrated by some ceiling effects in our data, several of our participants were skilled readers. These students are less likely to substantially benefit from this practice game in this context. However, there were also a number of undergraduates in our sample who did not immediately reach the highest level in the game. Thus, we will continue to explore the student characteristics and contexts under which StairStepper practice could be most beneficial. To this end, future work will broaden the scope of the participant pool to include a diverse sample of secondary students to evaluate the efficacy of the intervention with the target population.

Second, the current study relied on a small sample completing only 90 min of practice. We are developing additional studies in which larger, more diverse samples of students complete extended training and practice. Such studies will allow us to better detect and articulate the effects of self-explanation training and deliberate strategy practice using the StairStepper game in iSTART.

#### **4. Materials and Methods**

#### *4.1. Participants*

The participants in this study were 55 undergraduate students from a large university in the southwest. A demographic questionnaire indicated the sample was predominantly male (female = 38.2%, male = 61.8%, *M*age = 19.83 years) and the sample was 1.8% African American, 36.4% Asian, 40% Caucasian, 16.4% Hispanic and 7% identified as other. English was not the first language for 38.2% of the participants. The final analyses included 51 participants as 4 were unable to complete the study in the allotted time.

#### *4.2. Learning Measures*

Participants' reading comprehension was measured using the Gates–MacGinitie reading comprehension test (GMRT, grades 10–12) [96] at pre- and post-test. Forms S and T were counterbalanced across participants such that those who were given form S at pretest were given form T at post-test or the reverse, in the training and delayed treatment control.

All participants also completed pretest and post-test comprehension assessments. The pretest text was titled *Red Blood Cells,* and the post-test text was titled *Cell Repair.* The pretests and post-tests include textbase questions (i.e., those that can be answered directly from the text) and bridging inference questions (i.e., those that require students to make a bridging inference between two sentences in the text).

#### *4.3. Perceptions Measures*

Participants completed a survey following their interaction with iSTART and the StairStepper practice game. Participants rated their experience with iSTART and StairStepper, including their enjoyment of the game and its features. Participants were also asked to rate their performance using the system. Items were rated on a 5-point Likert scale from (1) strongly disagree to (5) strongly agree.

#### *4.4. Procedure*

Participants self-selected into study A (delayed-treatment control) or study B (training condition) through the SONA research participant sign-up system. Scheduling of study (A) and study (B) was counterbalanced across weeks to prevent selection bias. Participants in Study A and Study B completed the same tasks for this experiment. Participants in the training condition experienced the intervention between the pretests and post-tests (see Table 3). However, participants in the delayed-treatment control experienced the intervention, iSTART self-explanation training and playing the StairStepper game during Session 3. This design allowed us to compare conditions while not depriving the control group of instruction and practice using StairStepper.

**Table 3.** Delayed treatment and training group session activities.


#### **5. Conclusions**

Results from standardized reading assessments suggest that many students struggle to develop proficiency in literacy skills that are critical to educational and career success. Unfortunately, these test results, in conjunction with limited time and resources, often lead instructors to focus on preparing students for the high-stakes assessments rather than helping them to develop more generalizable reading comprehension skills [3,5]. The StairStepper game in iSTART was designed to address these potentially competing objectives by offering an automated, game-based practice environment that supports students' learning of reading comprehension strategies while also preparing them for high-stakes assessments. This study sought to answer three research questions; (1) How do students respond to the StairStepper game-based practice?; (2) How will participants progress through the StairStepper game, based on text adaptivity and scaffolded feedback?; (3) How to do iSTART training and StairStepper practice influence participants' performance on a comprehension test and standardized assessment?.

This study suggests that students enjoyed the game interface, attempted to perform well, and found the gameplay to be motivating. Specifically, they believed that it had gamelike features and indicated that they would use this type of game to practice other skills. Regarding research question two, system data analysis results suggested that students progressed through text difficulty levels successfully and benefited from the scaffolding and feedback process. Finally, the results suggested that the scores of students who played the StairStepper game remained stable, whereas the students who did not receive training and play the StairStepper game demonstrated a decrease in their reading comprehension scores. Collectively, these results suggest that iSTART training and StairStepper gamebased practice were beneficial for students reading comprehension strategy use. This initial student suggests promise for implementing StairStepper into the classroom and into test prep and as a positive step toward helping students to excel in high-stakes testing and beyond.

**Author Contributions:** Conceptualization, K.S.M. and D.S.M.; methodology, K.S.M. and D.S.M.; formal analysis, T.A. and K.S.M.; investigation, K.S.M.; resources, D.S.M.; data curation, K.S.M.; writing—original draft preparation, T.A.; writing—review and editing, T.A., K.S.M., and D.S.M.; visualization, T.A.; supervision, D.S.M.; funding acquisition, D.S.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was supported by The Office of Naval Research through Grant number N00014-20-1-2623. The opinions expressed are those of the authors and do not represent views of the Office of Naval Research.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the American Psychological Association and approved by the Institutional Review Board of Arizona State University (00011488, 10 February 2020).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Data may be accessed by emailing the first author at tarner@asu.edu.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


## *Review* **Serious Games and the COVID-19 Pandemic in Dental Education: An Integrative Review of the Literature**

**Kawin Sipiyaruk 1, \* , Stylianos Hatzipanagos 2 , Patricia A. Reynolds <sup>3</sup> and Jennifer E. Gallagher 3**

<sup>1</sup> Faculty of Dentistry, Mahidol University, Bangkok 10400, Thailand


**Abstract:** The COVID-19 pandemic has forced faculties including dental schools into a 'new normal', where the adoption of remote or distance learning is required to minimise the risk of infection. Synchronous learning historically was favoured due to the perceived advantage of 'real time' interactions between instructors and learners; these interactions are not always possible in asynchronous settings. However, serious games can overcome this limitation of asynchronous learning. This integrative review explores the literature on serious games in dental education, to construct a conceptual framework of their strengths in this pandemic. Following consideration of inclusion and exclusion criteria, 15 articles on 11 serious games designed for dental education were included in this review. Our investigation points to an increase in the use of serious games since 2018. The findings of the review support the use of serious games in dental education during the recent crisis. Key strengths include positive educational outcomes, enhanced engagement and motivation, interactive asynchronous distance learning, a safe learning environment, and the advantage of stealth assessment. Consequently, the 'new normal' in education appears to support a very promising future for serious games, particularly in dental education. A conceptual framework is proposed to inform further research across all education settings and timeframes.

**Keywords:** asynchronous learning; COVID-19; dental education; distance learning; game analytics; game-based learning; integrative review; remote learning; serious games

#### **1. Introduction**

The COVID-19 pandemic has been rapidly spreading around the world due to the SARS-CoV-2 virus. This outbreak has impacted on varied areas, including participation in the educational field at all levels. Students are not allowed to conduct learning activities on campus as they need to minimise the risk of COVID-19 infection. Technology-enhanced learning (TEL), especially in a remote or distanced setting, enables instructors and students to control time, location, and pace, which are weaknesses of traditional education [1,2]. Consequently, TEL can be helpful in this pandemic where distance learning is required.

'Remote' or 'distance learning' can be conducted in either synchronous or asynchronous formats [3]. The term distance learning will be used within this paper. Both synchronous and asynchronous formats have advantages and disadvantages. During the pandemic, synchronous learning employed videoconferencing and webinars to replace face-to-face teaching. It allowed instructors and students to have interactivity in real time [4]. Asynchronous learning has readily been implemented for a period to improve flexibility. It allows students to learn at any time, but there could be a problem with the absence of real-time interactivity between instructors and learners [4]. However, serious games are advanced technological tools that can be implemented to enhance interactivity in asynchronous learning.

**Citation:** Sipiyaruk, K.; Hatzipanagos, S.; Reynolds, P.A.; Gallagher, J.E. Serious Games and the COVID-19 Pandemic in Dental Education: An Integrative Review of the Literature. *Computers* **2021**, *10*, 42. https://doi.org/10.3390/ computers10040042

Academic Editors: Carlos Vaz de Carvalho and Antonio Coelho

Received: 28 February 2021 Accepted: 25 March 2021 Published: 1 April 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Serious (rather than for entertainment purposes) games are those primarily designed for education and training [5]. They allow students to improve their competences using feedback provided by the game system until they complete a game task [6,7]. Students can be engaged while learning through the game components [8,9]. Safe learning environments can also be created within serious games for students in various fields including healthcare [10–13]. A rapid review of serious games in healthcare education found that they can offer a similar educational outcome to traditional strategies, but the learning approach used in games seems to be more engaging [11]. These arguments support the use of serious games in disciplines such as healthcare education.

In the current global context, where learning in face-to-face settings or in real situations is restricted, serious games should be considered as tools to enhance interactivity in healthcare education including dental education. In our recent review in 2018 we reported that serious games had been implemented in various fields in healthcare education, but very few had been developed to be used in dentistry [11]. As there had not been a review of the use of serious games in dental education since our rapid review, it was considered necessary to conduct this integrative review to explore developments in this fast-changing field and to evaluate their impact when used in extraordinary circumstances such as the COVID-19 pandemic. This would give us the opportunity to investigate whether there has been an increase in use of serious games in dental education since 2018.

#### **2. Theoretical Background of the Review**

Serious games have the advantage of combining game-based learning and TEL. The learning process within an educational game comprises the instructional content and the game characteristics, and these two components trigger a game cycle, where students are motivated to learn [14], the game cycle being an iterative learning process that engages user judgement, user behaviour and system feedback to lead to achievement of learning outcomes [14]. This model can be implemented to explain the concept of serious games [10]. The learning process in serious games can be further explained by the important role that failure plays [6]. Failure within the game allows users to improve their competence to complete a game task. Furthermore, entertaining components are required for serious games to engage users in the game cycle [5,14], otherwise users may stop playing before they can achieve the expected learning outcomes.

Performance assessment is another consideration when using a serious game. In the game engine, interaction between users and a game system (user-generated data) can be captured without interrupting the learning process, i.e., so called stealth assessment [7]. These serious game analytics will reveal how the competence of learners can be improved with formative feedback until game completion. Enhanced with TEL, serious games can provide immediate feedback, enabling students to recognise mistakes and reconsider strategies to complete the game [6]. The immediate feedback within a serious game can support users to learn from their experiences.

Since the outbreak of the COVID-19, social distancing has been recommended to minimise the infection risk, and therefore onsite learning in dental schools has been restricted. During this period, there has been more focus on remote online synchronous learning as a substitute for face-to-face settings, as instant feedback can be provided through real-time interactivity [15,16], with an argument that immediate response may not be possible in asynchronous learning [17,18]. However, immediate feedback is considered an important feature of serious games [19,20]. In addition, with TEL support, serious games may be used anytime and anywhere [2], and therefore they have the potential to create interactive learning environments for asynchronous settings in dental education. Consequently, serious games might overcome the limitations of other asynchronous learning approaches and help students to gain knowledge and skills with engagement and motivation during the COVID-19 pandemic. This integrative review aimed to analyse the literature concerning serious games in dental education, in order to construct a conceptual framework of their strengths in response to the COVID-19 pandemic.

#### **3. Methods**

An integrative review of the literature was selected as the most appropriate investigative tool to generate new concepts within the chosen context of serious games in dental education during the COVID-19 pandemic. The synthesis of literature addressing emerging topics is suitable for this type of review, with a view to constructing a new framework as an initial conceptualisation [21]. An integrative approach starts from (1) conceptually structuring the organisation of the review, (2) designing how to conduct it, and (3) writing up the outcome of the review through both critical analysis and synthesis of the literature [21]. The methodological search process was piloted and adjusted repetitively before performing the final search [22]. This rigorous method aimed to ensure the thoroughness of the review, in order to answer the following questions:


#### *3.1. Search Strategy*

To assure that as much available evidence was identified as possible, the literature search was conducted across seven databases, covering areas of education, technology, and healthcare, including the Educational Resource Information Centre (ERIC), Web of Science, Scopus, Embase, Medline, ProQuest Dissertations & Theses Global, and Cochrane Central Register of Controlled Trials. In addition, Google Scholar and the reference lists of identified articles were explored to search for relevant papers. Grey literature was also screened to enable serious games used in dental education to be identified wherever possible. Search terms and Boolean combinations were implemented to identify relevant literature, which included 'Serious game', 'Computer-based game', 'Digital game', 'Video game' and 'Online game', together with 'Dental education', 'Dental student' and 'Dentistry'. The last search was conducted on 31 January 2021.

#### *3.2. Inclusion and Exclusion Criteria*

All types of empirical study of the use of serious games in dental education published between 2000 and 2021 were included in this review; however, they were excluded if they were not relevant to computer-based serious games and if they were not designed for educating and training dental learners. They were not included if they were not available in English or in full-text.

#### *3.3. Literature Identified from the Search*

The initial search across seven databases identified 120 articles. In addition, three further studies were identified through Google Scholar and the reference lists of identified articles. After removal of 18 duplicates, the titles and abstracts of 102 papers were reviewed. Eighty-two articles did not meet the inclusion criteria because they were not empirical studies and/or not relevant to serious games for dental students or professionals. Twenty full-text articles were accessed, of which a further five were excluded: one was not available in English; one was not relevant to dental education; and three were studies regarding non-serious games. Consequently, after consideration of inclusion and exclusion criteria, a total of 15 articles were included. This process is presented in Figure 1.

**Figure 1.** The articles selection process for the integrative review.

#### **4. Results**

#### *4.1. Characteristics of Included Articles and Serious Games*

‐ The fifteen articles included in this review comprised seven journal articles [23–29], three conference papers [30–32], four book chapters [33–36], and one master's thesis [37], including 11 serious games in dental education. Three serious games (reported in four articles) were designed for pre-clinical dentistry with a focus on tooth morphology in 2014 and 2017 [30,37], histology in 2019 [28], and skull anatomy in 2020 [36]. Eleven articles were relevant to clinical dentistry, including dentin bonding for operative dentistry in 2011 [23], alginate mixing in 2013 [24], diagnosis and treatment planning in virtual dental patients in 2013 [33], dental public health in 2013, 2016 and 2017 [25,26,34], biosafety in 2018 and 2019 [27,35], dental anesthesia in 2019 [31], diagnosis in virtual endodontic patients in 2019 [32], and clinical skill assessment in 2020 [29]. These serious games and their details are presented in Table 1.

‐

‐ ‐

‐ Following synthesis of the relevant literature the following themes emerged: educational outcomes, engagement and motivation, asynchronous distance learning, safe learning environment, and assessment issues. These five themes were considered as the key attributes of serious game use during the COVID-19 pandemic in a remote or distance learning context. Table 2 presents frequencies of the themes as reported in the included articles.


#### **Table 1.**Table presenting the 11 serious games shortlisted for the review.


**Table 2.** A table presenting frequencies of themes as reported in the included articles.

#### *4.2. Educational Outcomes of Serious Games*

Educational outcomes were reported in seven out of eleven serious games [23,24,28– 30,32,34]. All of them were found to have positive impact on knowledge improvement with only three of them (discussing dentine bonding, tooth morphology, and dental public health) were evaluated using pre- and post-tests to assess competence of students before and after the use of serious games [23,30,34]. One article compared the effectiveness of serious games to traditional approaches and found no statistical difference [24].

Six serious games were measured using user perceptions in either quantitative or qualitative format and were perceived positively as effective learning tools [23,24,28–30,32]. For instance, most students reported that the 'OSCEGame' (Objective Structured Clinical Examinations) could increase time management skills and reduce anxiety, which could prepare them well for further examinations [29]. Students also felt more confident after completing the games [23,28]. Therefore, serious games seemed to have a positive educational impact in dental education.

#### *4.3. Engagement and Motivation*

Engagement and motivation seem to be another key theme emerging when evaluating serious games. Seven serious games were surveyed to gather user perceptions towards entertaining components [23,24,28,30,32,34,36], which were positively perceived by dental students as engaging learning tools. The application of game rules within serious games can make learning activities more engaging. For instance, a challenge can motivate and engage users in completing a game task; however, there should be a balance between challenges and player skills. It might have been difficult for students to be engaged with learning content if they needed to repetitively perform a game task [36].

Entertainment features of video games could also be embedded for enhancement of learning engagement and motivation included having a colourful interface and using interactive music [24,26]. Technologies can also make serious games more engaging. For example, autostereoscopy and natural user interfaces can be implemented to enhance sensation and interactivity, enabling dental students to interact with three-dimensional objects using their gestures [30]. A high-quality graphic could also be used to enhance visualisation for engagement in a new generation of learners (Figure 2), in addition to pedagogical impact [36]. The enhancement of engagement and motivation can be considered as a key strength of serious games over traditional learning approaches.

**Figure 2.** A screenshot from the skull anatomy game presenting an engaging graphic, reproduced with permission from Dall, R.

#### *4.4. Serious Games as Interactive Asynchronous Distance Learning Environments*

‐ ‐ ‐ Seven of the eleven identified serious games in dental education enabled learning activities to be conducted online [23,25–29,32–36]. With these online serious games, asynchronous distance learning can be considered as an important strength in this COVID-19 pandemic. Distance learning allows dental students to learn to minimise the risk of COVID-19 infection. Together with asynchronous learning, students can learn in convenient time at a suitable pace [38]. Although the OSCEGame had set a limited time for students to complete each station, they could repetitively play the game until they felt confident about the examination [29]. Therefore, with serious games, each student can spend time differently in each section of the game and overall, based on their readiness.

‐

‐ ‐

‐

‐ ‐

‐ ‐ ‐ ‐ ‐ There were three serious games requiring onsite settings to ensure learners had access to relevant materials [24,30,31,37]. Skills-O-Mat required a spoon attached to an accelerometer to capture how well students could mix alginate [24]. A serious game for dental anaesthesia implemented a haptic device for training students in conducting an anaesthetic procedure (Figure 3) [31]. Motion sensors seem to be important technologies for serious games in training psychomotor skills in dental education. According to a serious game for tooth morphology [30,37], although its learning outcomes were not psychomotor skills, auto-stereoscopy and natural user interfaces were implemented to enhance the interactivity of the game.

Instant feedback and immediate response appeared to be available in all included serious games in either formative or summative format [23,24,26–33,36]. The formative feedback allowed students to learn from their mistake [23,26], enabling them to improve their knowledge and skills. It appeared that informative feedback could be made more suitable, rather than being offered only in a numeric format [26]. The summative feedback would report how well students performed in a game task [24,32,36]; however, it could provide information on errors as a further improvement.

**Figure 3.** A user conducting a dental anaesthetic procedure using a haptic device [31], reproduced with permission from Nunes, F.L.S.

#### *4.5. A Safe Learning Environment within Serious Games*

‐ ‐ ‐ Serious games can simulate a learning environment where students can experience dental practice safely. Dental students could be exposed to simulated patients in serious games, instead of 'real' clinical settings, to initially develop competences in oral diagnosis and treatment planning [32,33], as well as in local anaesthesia of the maxillofacial region [31]. This could minimise the risk of COVID-19 infection, whilst developing their skills in preparation for further training in clinical settings when possible. ‐

‐ ‐ ‐ ‐ In terms of community-based dentistry, serious games can simulate a learning situation where students can operate in a safe environment. GRAPHIC, (Games Research Applied to Public Health with Innovative Collaboration), a serious game for dental public health education, allowed students to gain disciplinary practice experience in a virtual town [26,34]. Within the game, students were firstly allowed to explore information on the virtual town provided by the system (Figure 4); they were then required to select the best five health promotion programmes, considering information about the town and research evidence, in order to improve the oral health of the population. This opportunity allows dental students to conduct community-based practice without being exposed to risk in a real community, and thus the risk of COVID-19 infection can be minimised. ‐ ‐ ‐ ‐ ‐ ‐

**Figure 4.** A screenshot of GRAPHIC (Games Research Applied to Public **Figure 4.** A screenshot of GRAPHIC (Games Research Applied to Public Health with Innovative Collaboration), where information on a virtual town is provided [34], reproduced with permission from Springer Nature.

#### *4.6. Stealth Assessment in Serious Games*

Stealth assessment, i.e., an approach to performance-based assessments that embeds assessments within digital games in order to measure how students are progressing toward targeted goals [7], can be considered as another strength when applied to serious games. The GRAPHIC system could record how students interacted with the game, and therefore dental instructors could assess logs of their performance and behaviours when performing the game task from the activity log data [26]. Designing stealth assessment in GRAPHIC also allowed students who did not progress to be identified, and therefore they could get additional support from academic staff for the achievement of their learning outcomes.

#### **5. Discussion**

#### *5.1. Trends in Serious Game Use in Dental Education*

This integrative review found an increasing use of serious games in dental education. There have been seven serious game papers published since 2018 [27–29,31,32,35,36], compared with eight between 2011 and 2017 [23–26,30,33,34,37]. This trend was similar to the use of serious games in general areas of education [39], including other healthcare education areas [11]. A rise in development of serious games may result from the increase of user demand, given that digitally-savvy generations are increasingly participants in all levels of education. In addition, game development software, together with increasingly advanced technologies, have become more affordable in recent years.

#### *5.2. Potential in the Use of Serious Games in the COVID-19 Pandemic*

Based on the articles reviewed, it appears that serious games should be supported for use as effective learning tools in dental education during the COVID-19 crisis, where distance learning is required to minimise the risk of COVID-19 infection. Given the time that it takes to complete research and move through publication, it may be that staff took the opportunity during the slowdown to ensure their findings were reported at that time, in support of their potential use. This development could be explained by the key strengths of serious games, which are (1) positive educational outcomes, (2) engagement and motivation, (3) asynchronous distance learning, (4) provision of a safe learning environment, and (5) assessment. These themes are discussed in this section to identify how they can support the use of serious games during the pandemic. The value of the development of a conceptual framework is that it allows further understanding of the key strengths of serious games, whether their effectiveness is evaluated during a pandemic or not.

#### 5.2.1. Positive Educational Impact

Serious games can be considered as effective learning tools in terms of educational outcomes, as seen from the results of this review. Serious games have a positive impact on knowledge improvement amongst dental students, as there has been an increase of scores evaluated by pre- and post-assessments [23,30,34]. This outcome is broadly similar to serious games for other healthcare education areas [11,40]. In addition, most articles included in this review requested dental students to rate their perceptions of serious games [23,24,28–30,36]; the games were perceived by students as 'helpful' TEL tools in improving competence and preparing them for further studies.

In terms of learning design, serious games adapt a game concept to a learning process. According to the game cycle, introduced by Garris et al. [14], there are three components: 'user judgements', 'user behaviour', and 'system feedback'. In other words, when users preform an action in a game, the system should provide feedback for them to adapt their strategies to complete the game task, the so-called 'role of failure' [6]. Within serious games, a 'failure' is not a true 'failure', as it will enable learners to improve their knowledge and skills until they can complete a game task. Experiential learning can also be achieved while using serious games, allowing students to gain knowledge and skills through direct experience within games [41,42].

When comparing the effectiveness of serious games in terms of knowledge improvement, there seems to be no clear evidence in supporting them over other learning approaches. Only one article in this review compared the gaming approach with a passive format but found no statistical difference in terms of knowledge improvement [24]. The systematic review of serious games in healthcare education also reported similar findings, where the effectiveness of serious games over other learning approaches could not be sufficiently evident [40]. Consequently, it seems that serious games should be considered as a very helpful replacement of face-to-face learning formats during this pandemic, as they can provide positive educational outcomes at least as effectively as other learning approaches.

#### 5.2.2. Engagement and Motivation

Engagement and motivation appear to be key strengths of serious games over other educational technology tools. Based on the results of this review, serious games were perceived positively as engaging learning strategies [23,24,28,30,34,36]. Although serious games are designed mainly for educational purposes, entertainment components are still required to engage and motivate learners. According to the game cycle [14], users need to perform a game task repetitively, failing and receiving feedback until they can complete the game. Therefore, if a serious game is not sufficiently engaging and motivating, students may cease the game, before achieving any learning outcomes.

The implementation of gaming technologies such as immersive graphics, gesture or motion control, voice recognition, and auto-stereoscopy appears to enhance serious games by making them more engaging. In the review, visual and audio aspects were reported to make the serious games more engaging. Both sounds and graphics can be considered as the entertainment components of serious games [43], and appear to play a fundamental role in engaging users [44]. Advanced technologies may also be used to enhance the entertainment value of serious games. Based on the included articles, interactivity can be designed using auto-stereoscopy and natural user interfaces to make the game more engaging [30]. This aspect can be considered as important, as this new generation of students in dental schools are familiar with video games, from an early age, and therefore serious games designed with very basic technologies might not be engaging for them.

Game rules and challenges can have an impact on the engagement with and motivation provided by serious games. One included article pointed out that a problem with engagement could occur if too many attempts were required for the same task [36]. This is an issue explained by the flow theory, where appropriate balance between competencies and challenges enhances flow of activities [45], which can be applied to game design [46–48]. In other words, if a game task is too simple, it could be boring. On the other hand, if it is too challenging, users may feel frustrated and stop playing the game. However, a serious game may be designed at different levels to allow learners to select a challenge that is suitable to their level of competence and knowledge.

#### 5.2.3. Interactive Asynchronous Distance Learning

Of the eleven identified serious dental games, seven had already been using an online format [23,25–29,32–36]. Using an asynchronous format, these games allowed students to conduct their learning at a convenient time and suitable pace. Students with high knowledge and skills may progress through the game sooner than ones who require further improvement of competencies, in a personalised learning set up. Four of the serious games required students to conduct learning activities onsite [24,30,31,37], as they required specific equipment to capture the motions of students. However, they had the potential to be used in a distance learning setting, as these motion sensors appear to be affordable everyday equipment, such as smartphones, smartwatches, and game consoles.

Interactive asynchronous learning appears to take place in serious games, which can be considered as a unique strength for their use during the COVID-19 pandemic. As face-to-face sessions are constrained, online synchronous learning has gained more attention, and there is a real-time interactivity between instructors and students, where

instant feedback can be provided [15,16]. On the other hand, immediate response and instant feedback may not be provided in asynchronous learning [17,18]. However, this integrative review has shown that immediate feedback (formative or summative) could be provided within all the serious games we included in the review [23,24,26–33,36], and therefore an interactive learning environment can be embedded in distance educational settings. These arguments support the use of serious games as interactive asynchronous distance learning environments in the COVID-19 crisis.

#### 5.2.4. Safe Learning Environment

As outlined above, serious games can simulate a learning situation, enabling dental students to gain experiences in a safe learning environment. This review identified serious games used for experiencing clinical practice in supporting cognitive [32,33] and psychomotor skills [31]. Not only is there no harm to patients, but any mistakes in the game may increase the awareness of each student in clinical practice. This strength of serious games has also been found in other healthcare areas including medical and nursing education [11]. A serious game for dental public health education allowed dental students to be exposed in a virtual town, where they could gain experience of community-based practice in a safe environment [26,34]. Such simulated environments ensure harm reduction when learning dental treatments as well as oral health prevention and promotion, and also remove the actual infection risk of COVID-19.

#### 5.2.5. Stealth Assessment

Stealth assessment was discussed in one of the identified articles in the context of using log data analytics in a serious game to indirectly observe how students interact with the game [26]. Serious game analytics combining gaming and learning analytics seem to be an important feature for the improvement of a game [49]. Serious game analytics can capture user-generated data (by creating an activity log of interaction between users and a game system), which are valuable for game developers or educators to identify areas for improvement as well as to assess performance of learners. Stealth assessment can also determine how students are progressing toward targeted goals [7]. It was included in this review as it represents an important learning design element in capturing the performance and improvement of learners when progressing towards the expected learning outcomes of serious games.

With stealth assessment, the flow of serious games can be maintained, so students can be engaged in the learning activities without self-consciousness and time pressures [50]. Therefore, the actual behaviour of students can be evaluated through serious game analytics when capturing their activities in completing a game task. Stealth assessment is not beneficial specifically to the COVID-19 context, but it represents another unique trait of serious games.

#### *5.3. A Conceptual Framework of Serious Games' Strengths in the COVID-19 Pandemic*

Our review indicates that serious games have a positive educational impact in dental education, whereby learners can learn from their failure. Engagement and motivation can be considered as important as the learning activities themselves, since students need to be engaged with serious gaming activities to achieve learning outcomes. Immediate and interactive feedback within a serious game can also enhance an asynchronous distance learning environment, which is considered necessary in this COVID-19 pandemic. Log systems enable stealth assessment, where gaming activities of students can be recorded, allowing instructors to assess performance without interrupting the learning process. In addition, serious games can simulate learning in situations where face to face participation is not possible, for instance, allowing students to interact with virtual patients or communities to improve competencies in a safe environment. These key strengths of serious games work together in supporting dental students to achieve learning outcomes during the pandemic or in other situations when there are restrictions in engaging with face-to-face learning.

Figure 5 presents the key elements of the conceptual framework. It defines the relevant variables of our review and maps out how they relate to each other. In the Figure, strengths in boxes with a thicker border are the ones reported in most studies; positive educational impact, enhanced engagement and motivation, and interactive synchronous learning environment were considered in seven serious games, safe learning environment was discussed in four, whilst stealth assessment was reported in one serious game (Table 2). The strengths include traits that are specifically designed for educational purposes; however, entertainment features are still required to engage and motivate users to repetitively perform a game task [5,14]. The arrows represent properties of the strengths; solid arrows represent essential properties that support the learning process and dash arrows represent desirable aspects that can enhance learning design.

‐ ‐

‐ ‐

‐

‐

‐ ‐

‐ **Figure 5.** A conceptual framework of key strengths of serious games in the COVID-19 pandemic.

#### *5.4. Limitations of this Review*

‐ ‐ There appears to be an increase in the use of serious games in dental education reported over the past two years since the last review we completed [11]; however, no research on serious dental games during the COVID-19 pandemic was identified. Consequently, it did not seem possible to evaluate the effectiveness of serious games for use in the pandemic crisis in dental education. In addition, although several serious dental games included in this review were designed as being available in an online format, none of the developers provided access to the games. Therefore, their descriptions were based only on the information provided in the included articles.

#### *5.5. Implications for Future Research*

‐ ‐ This review has indicated the advantages of using serious games in dental education in extraordinary circumstances such as the COVID-19 pandemic. Further studies with robust methods, such as randomised control trials, are required to evaluate the effectiveness of serious games, compared with other learning approaches. In addition, future research should seek knowledge regarding the implementation of serious games in dental education both in normal and extraordinary circumstances such as in a pandemic crisis or a natural disaster. The conceptual framework presented in Figure 5 provides the basis of a useful tool to inform such research across all educational domains as innovations in TEL accelerate during and post-pandemic.

#### **6. Conclusions**

This integrative review revealed an increasing use of serious games since 2018. Our findings support the use of serious games in dental education during the COVID-19 pandemic and beyond, when and where the adoption of distance learning and teaching is necessary to minimise the infection risk. The conceptual framework derived from this integrative review combines the key supportive features of dental serious games i.e., (1) positive educational outcomes, (2) student engagement and enhanced learner motivation, the provision of (3) an interactive asynchronous distance learning, (4) safe learning environment, and (5) stealth assessment. The new normal for dental education forced by the COVID-19 crisis, consequently, appears to provide new opportunities for the use of serious games in dental education. However, future research should seek to employ robust methods to evaluate the effectiveness of serious games, in order to support learning strategies and their implementation in dental education.

**Author Contributions:** Conceptualization, K.S., P.A.R., S.H., and J.E.G.; methodology, K.S., P.A.R., S.H., and J.E.G.; validation, K.S., P.A.R., S.H., and J.E.G.; formal analysis, K.S.; investigation, K.S.; resources, K.S., P.A.R., S.H., and J.E.G.; Data Curation, K.S.; writing—original draft preparation, K.S.; writing—review & editing, P.A.R., S.H., and J.E.G.; visualization, K.S., P.A.R., S.H., and J.E.G.; supervision, P.A.R., S.H., and J.E.G. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data summarised in Table 1 of this review were analysed from 15 articles listed in the reference section [23–37].

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Computers* Editorial Office E-mail: computers@mdpi.com www.mdpi.com/journal/computers

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18

www.mdpi.com ISBN 978-3-0365-3809-9