**Current Trends in Game-Based Learning**

Editors

**Margarida M. Marques L ´ucia Pombo**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editors* Margarida M. Marques Department of Education and Psychology University of Aveiro Aveiro Portugal

Lucia Pombo ´ Department of Education and Psychology University of Aveiro Aveiro Portugal

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *Education Sciences* (ISSN 2227-7102) (available at: www.mdpi.com/journal/education/special issues/current trends game based learning supported mobile devices).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-2316-3 (Hbk) ISBN 978-3-0365-2315-6 (PDF)**

© 2021 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**


## **About the Editors**

#### **Margarida M. Marques**

Researcher at the CIDTFF - Research Centre on Didactics and Technology in the Education of Trainers, in University of Aveiro (Portugal). She has expertise in technology-supported game-based learning in science education and her research interests include scientific literacy and teacher professional development. As a member of the EduPARK project, she won the 2018 ECIU Team Award for Innovation in Teaching and Learning of the European Consortium of Innovative Universities (ECIU). Additionally, she has been teaching curricular units of High Degrees and Master Courses and supervising PhD and post-doc students in the Department of Education and Psychology. She holds a PhD in Didactics and Training, branch of Didactics and Curricular Development.

#### **L ´ucia Pombo**

Auxiliary Researcher and Vice-Coordinator of the CIDTFF - Research Centre on Didactics and Technology in the Education of Trainers, in University of Aveiro (Portugal). She holds two PhDs: in Biology and in Education. Her expertise is on Technology Enhanced learning, Science Education, mobile Learning, gamification and quality in Education. She is the Coordinator of the EduPARK project funded by FEDER-FCT. The EduPARK seeks to use outdoor learning strategies, by means of an interactive mobile Augmented Reality app that supports geocaching activities in formal, non-formal and informal contexts, in Aveiro city park, in Portugal. She was awarded by the prestigious European Consortium of Innovative Universities (ECIU) Team Award for Innovation in Teaching and Learning, in 2018. She has been lecturing curricular units, in the Department of Education and Psychology, such as Natural Sciences and Educational Intervention Projects, she supervises PhD and MA students in ICT in Education and she has been also involved in teachers training.

## *Editorial* **Current Trends in Game-Based Learning—Introduction to a Special Collection of Research**

**Margarida M. Marques \* and Lúcia Pombo \***

CIDTFF—Research Centre on Didactics and Technology in the Education of Trainers, Department of Education and Psychology, University of Aveiro, 3810-193 Aveiro, Portugal

**\*** Correspondence: marg.marq@ua.pt (M.M.M.); Lpombo@ua.pt (L.P.)

The potential of digital games to promote learning is a growing field of research. Researchers have been extensively analyzing the impact of games on learner motivation and engagement in educational settings [1] and even reporting the contribution of this approach to a variety of learning outcomes, such as concept understanding [2,3] and soft skills development [4].

A myriad of technological options can be used to support digital game-based learning. One popular technology in this context is the mobile device, considering its high penetration rate in our societies, even among young people. These can be combined with other technologies, such as Augmented Reality (AR) or Virtual Reality (VR), to increase students' motivation and engagement in learning processes [5,6].

Because of this, there is an emergent need to know and promote good practices in the development and implementation of game-based learning approaches in educational settings. This was the motto for the proposal of the Education Sciences (ISSN: 2227-7102) Special Issue "Current Trends in Game-Based Learning". This book is a reprint of this Special Issue, collecting a set of five papers that illustrate the contribution of innovative approaches to education, specifically the ones exploring the motivational factors associated with playing games and the technology that may support them.

Considering the above, the first work in this book, by Lúcia Pombo and Margarida M. Marques, presents a study where the "Educational Value Scale of Mobile AR Games" was used in an illustrative case: The EduPARK app. The study revealed that games sustained by mobile devices and integrating AR can have high educational value, particularly with students aged 10 to 15 years old.

Second is a piece from the same authors focusing on the need for teacher training in these innovative approaches. Hence, the authors conducted a case study of the impact of a teacher-training initiative on trainees' professional development. It revealed improvements in teachers' knowledge and experience with game-based learning, mobile learning, and AR, as well as their ability to identify both benefits and barriers to these approaches.

The third work, by Dionísia Laranjeiro, discusses the process of designing, developing, and evaluating a set of four thematic educational apps composed of a set of games suitable for preschoolers in autonomous or guided activities. This work was developed under the "Aprender XXI" (Learn XXI) project by a multidisciplinary team of educational researchers, technology developers, and end-users, in this case, children and kindergarten educators.

The fourth chapter, by Rita Tavares, Rui Marques Vieira, and Luís Pedro, focuses on the promotion of primary students' scientific competences and self-regulated learning through their interaction with a mobile app. More specifically, it presents the conception process of the interaction design with a strong theoretical base, as it considers a learning approach proposal that combines the Universal Design for Learning principles, the Inquiry-Based Science Education and the BSCS 5E Instructional Model.

Finally, the work of Friday Joseph Agbo, Ismaila Temitayo Sanusi, Solomon Sunday Oyelere, and Jarkko Suhonen presents a literature review on the use of VR in computer sci-

**Citation:** Marques, M.M.; Pombo, L. Current Trends in Game-Based Learning—Introduction to a Special Collection of Research. *Educ. Sci.* **2021**, *11*, 622. https://doi.org/ 10.3390/educsci11100622

Received: 22 September 2021 Accepted: 30 September 2021 Published: 11 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

ence education. Among other relevant findings, this study reveals that game-based learning and gamification have been leveraged for computer science education integrating VR.

#### **References**


## *Article* **The Potential Educational Value of Mobile Augmented Reality Games: The Case of EduPARK App**

#### **Lúcia Pombo \* and Margarida M. Marques \***

CIDTFF—Research Centre on Didactics and Technology in the Education of Trainers, Department of Education and Psychology, University of Aveiro, 3810-193 Aveiro, Portugal

**\*** Correspondence: lpombo@ua.pt (L.P.); marg.marq@ua.pt (M.M.M.)

Received: 22 September 2020; Accepted: 14 October 2020; Published: 16 October 2020

**Abstract:** New teaching methodologies are nowadays integrating mobile devices, augmented reality (AR), and game-based learning in educational contexts. The combination of these three elements is considered highly innovative, and it allows learning to move beyond traditional classroom environments to nature spaces that students can physically explore. The literature does not present many studies of this approach's educational value. The purpose of the study is to present an illustrative case of a mobile AR game in order to analyse its educational value based on the users' opinion, both teachers and students, and on logs of game results. Through a mixed method approach, the educational value scale was applied to 924 users after playing the EduPARK app in a Green City Park. Results revealed high educational value scores, especially among teachers and students of 2nd and 3rd Cycles of Basic Education (83.0 for both). Hence, this particular software seems to be more suitable for 10–15 years-old students who highlighted motivational features, such as treasure hunting, points gathering, the use of mobile devices in nature settings, and AR features to learn. This study empirically revealed that mobile AR games have educational value, so these specific game features might be useful for those who are interested in creating or using games supported by apps for educational purposes.

**Keywords:** educational value; mobile learning; game-based learning; augmented reality; mixed methods

#### **1. Introduction**

Game strategies are used in Education with the aim of improving the learning process by making use of the motivating effects of game elements and techniques. Both strategies of game-based learning and gamification intend to motivate students for learning. Engagement and motivation are key factors that influence the students' performance during a learning process [1]. However, game-based learning is not the same as gamification, as the first makes use of a game for learning, and the latter integrates a set of technical concepts (such as points, badges, and leaderboards) in the learning process. Games have been used for a long time in teaching and learning; however, they have not always been properly investigated.

The increasing use of mobile phones makes it possible to explore digital educational games in outside environments, and when combined with augmented reality (AR) content, the educative effect may be exponential and result in best practices [2]. With the use of AR, the real-world environment can be augmented by providing users with accurate digital overlays. AR is a promising technology that has the potential to encourage learners to explore learning materials from a totally new perspective. Additionally, technological advancements along with the proliferation of wireless mobile devices, such as smartphones and tablets, allow for widening the scope of educational AR applications [3].

The integration of each of the above-mentioned elements (games, mobile devices, and AR) in educational contexts is considered innovative [3–5]. Moreover, their combination has revealed diverse benefits for learning, both at the cognitive and emotional levels [6–8], and supports learning to move beyond traditional classroom environments to nature spaces that students can physically explore [9].

Regarding off the shelf games, Lin, Huang, and Lin [10] analyse the game design of "Pokémon GO" in contexts where parents play together with their children, and found that some of the most valued features are its user-friendly interface, going outdoors, visiting places of interest, and combining games with exercise.

In what concerns games developed for educational purposes, Preka and Rangoussi [4] combine quick response (QR) codes with AR to create a collaborative game for music learning in Early Childhood Education in indoors and outdoors activities. Evaluation results indicate that the AR–QR technology is a powerful tool that triggers and sustains children's interest during the learning process and can enhance their cognitive and collaborative skills, as well as their social interaction.

Another study [11] presents a technical framework, Mobile Augmented-Reality Games for Instructional Support (MAGIS), for the development of this type of game. The authors illustrated the usefulness of the framework for implementing outdoor location-based educational games through the analysis of the game "Igpaw: Intramuros" for History learning. The authors highlighted that players' enjoyment of the game tended to be adversely affected by weather conditions, long walking distances between markers, and devices' battery life.

The EduPARK app is an example of a successful mobile AR game for learning [12]. It was developed to foster collaboration [13], and authentic [14] and situated learning [15] outside the classroom, under the umbrella of constructivist learning [16], as the learner or app user assumes an active role by constructing new knowledge within the articulation between the learning experience and previous knowledge. The EduPARK app aims to be explored in situ, an Urban Green Park, which is a context that can be used to promote new modes of learning in science education, since the ability to understand ecosystems is enhanced by experiences in real environments [17]. The app gives access to excellent cross subjects' educational materials, both in Portuguese and in English. It comprises a very useful tool for Portuguese teachers and students to explore scientific knowledge by accessing contextualised and appealing information on biological and historical references that augment the experience of exploring a local Urban Green Park. This type of innovative educational resource is not common in Portuguese speaking countries, hence, adding to the relevance of the EduPARK app.

Users' subjective perceptions about educational software can be evaluated with different tools, for example, System Usability Scale (SUS) and Educational Value Scale (EVS). The SUS is a robust, effective, and inexpensive tool, developed by Brooke in 1986, to quickly measure users' subjective perceptions of computer systems usability [18]. With only 10 questions on a 5-point scale (ranging from "strongly agree" to "strongly disagree"), it is one of the most used tools for measuring perceptions of usability, with a scale varying from 0 to 100 [19]. Sauro [20] reviewed 500 studies and found out that a SUS score of 68 could be considered average.

The tool used in this study, the EVS, was developed based on the SUS to quickly and easily collect users' subjective rating of the educational value of an app for outdoor green settings, considering the following dimensions: (a) Learning value; (b) intrinsic motivation; (c) engagement; (d) authentic learning; (e) lifelong learning; and (f) conservation and sustainability habits. Taking this into account, two items from each dimension were included in the scale, as described in [9,21]. As this is a new tool, there is a need of further studies to establish standards to support values interpretation.

Previous studies <sup>9</sup> showed that the EduPARK game achieved an average Educational Value Scale (EVS) of 83.8 and an average System Usability Scale (SUS) of 80.2, according to 244 students attending the 2nd or 3rd Cycles of Basic Education. This demonstrates its high educational value and usability for students of these school levels, indicating that the app can be used, as reference, by the international community, namely those who are interested in designing educational apps based on previous good experiences. As the EduPARK app is intended to be used by several school levels (as explained in the next section), there is a need to aggregate data in order to yield a wider picture of its educational value and its adequacy for different target publics. Hence, the purpose of this study is to present the

EduPARK app, an illustrative case of a mobile AR game, in order to analyse its educational value based on the users' opinion and on logs of game results. Aggregated data includes 924 questionnaires, filled in by teachers and students of non-higher education contexts.

The analysis sustains a reflection on the enhancement of the educational value of mobile AR games by presenting the specificities of the EduPARK game, including its educational resources, such as 2D and 3D models that mix real and virtual worlds, combining familiar technology with outdoor learning strategies. Finally, some guidelines arise that might be useful to inspire other educational game producers by providing theoretical and practical frameworks that can be useful in other natural environments to open horizons and opportunities for Education.

#### **2. The EduPARK Game**

As the main purpose of this paper is to analyse the educational value of a mobile AR game accessed through the EduPARK app, there is a need to present the educational principles that guided its development and the app's features that may enhance learning.

The EduPARK app is the main product of the project with the same name. It was developed with the aim of supporting social constructivism approaches to teaching [16] in a game-based approach for student engagement and motivation. These are well known factors influencing learning [22]. Hence, the app was created to support the users' construction of meaning through experiences in an Urban Green Park, being meaning influenced by the interaction of the learners' prior knowledge with the new experiences, as well as by their interactions with others. When designing the users' interaction with the app, the followed principles were: (i) To stimulate the app users to become active participants in their own learning, in a student-centred learning process with hands-on activities; (ii) to foster collaboration among app users from the same work group, through debate of ideas before answering questions, instead of making use of only competitive approaches; (iii) to embed learning in a real context, a park, which is rich in biology, history, and mathematics learning opportunities, thus providing authentic and situated learning experiences; (iv) to offer multiple modes of representation, namely through AR contents with video, audio, text, and 3D models; (v) to allow users to progress at their own pace, not establishing limited time to answering the challenges posed through the app; and (vi) to provide feedback with a scientific explanation after answering the proposed challenges.

Besides the social constructivism approach, game features, such as the treasure hunt format with a friendly mascot giving hints and feedback, accumulating points by correctly answering the challenges, and the leaderboard to show the best performance, can promote motivation by making boring content more enjoyable [22,23]. All these features were integrated in the EduPARK teaching approach. Moreover, allying these game features with the use of mobile devices (still widely forbidden in formal education contexts) and AR technologies is another factor increasing motivation and engagement with learning [3].

As mentioned before, the real outdoor context where learning is promoted by the app is a park. These areas have high ecological and environmental value, so they should be preserved. The use of this educational mobile AR game in the selected park promotes positive attitudes towards nature conservation and sustainability in the community [22], thus adding to the app's educational value.

For educational relevance of the app and game approach, it was important to carefully analyse the National Curriculum to identify multidisciplinary issues (e.g., integrating Biology and History) to integrate in the educational guides (games), so that students may correlate the experiences promoted by the app with the aimed curriculum learning. The fact that the educational guides were designed to be explored in the park provides an example of a truly authentic context for situated learning, where the location is essential for learning [23].

The EduPARK app can be installed from the project site (edupark.web.ua.pt/app). It requires the update of quizzes after downloading; dismissing the use of mobile web in the park. The app was developed for Android using Unity 5 and Vuforia SDK, integrating AR and quiz games, in the logic of a treasure hunt [24]. Therefore, it was designed to provide a learning experience that requires the exploration of a Green Park, the Infante D. Pedro Park (Aveiro, Portugal). exploration of a Green Park, the Infante D. Pedro Park (Aveiro, Portugal). The app is intuitive and can be used autonomously, either individually or in a group, at any time

of a treasure hunt [24]. Therefore, it was designed to provide a learning experience that requires the

*Educ. Sci.* **2020**, *10*, x FOR PEER REVIEW 4 of 20

The EduPARK app can be installed from the project site (edupark.web.ua.pt/app). It requires

The app is intuitive and can be used autonomously, either individually or in a group, at any time using the game mode or explore freely mode. The last one allows accessing AR content, without the requirement of following a predetermined path. In the game mode, users are welcomed by the EduPARK mascot, a female monkey who explains the rules to the players. The game objective is to gather points by correctly answering quiz questions. The user, or group of users, who gets the higher score is the winner. So, the game is more interesting when several groups play at the same time in a friendly competition climate [12,25]. using the game mode or explore freely mode. The last one allows accessing AR content, without the requirement of following a predetermined path. In the game mode, users are welcomed by the EduPARK mascot, a female monkey who explains the rules to the players. The game objective is to gather points by correctly answering quiz questions. The user, or group of users, who gets the higher score is the winner. So, the game is more interesting when several groups play at the same time in a friendly competition climate [12,25]. For motivational and engagement purposes, the app allows the creation of several profiles that

For motivational and engagement purposes, the app allows the creation of several profiles that record the progress of the explored games. For each profile, it is possible to know all completed games, percentage of correct answers, number of visited markers, and number of found treasures [12]. record the progress of the explored games. For each profile, it is possible to know all completed games, percentage of correct answers, number of visited markers, and number of found treasures [12].

For the first-time experience of playing, the user is guided by a tutorial that introduces the app's features (Figure 1). The game includes: (i) Instructions for users to find locations in the park, in order to follow a predetermined path; (ii) questions whose answer requires observing the surroundings or analysing multimedia resources, sometimes in AR format; (iii) feedback providing an explanation about the correct answer; (iv) the number of accumulated points; (v) challenges to find virtual treasures (caches), enclosing extra credits—bananas—the number of assigned bananas decreases with the time needed to find the treasure; (vi) the accumulated bananas can be exchanged for clues to help players answer later questions or are converted into points at the end of the game [12]. For the first-time experience of playing, the user is guided by a tutorial that introduces the app's features (Figure 1). The game includes: (i) Instructions for users to find locations in the park, in order to follow a predetermined path; (ii) questions whose answer requires observing the surroundings or analysing multimedia resources, sometimes in AR format; (iii) feedback providing an explanation about the correct answer; (iv) the number of accumulated points; (v) challenges to find virtual treasures (caches), enclosing extra credits—bananas—the number of assigned bananas decreases with the time needed to find the treasure; (vi) the accumulated bananas can be exchanged for clues to help players answer later questions or are converted into points at the end of the game [12].

**Figure 1.** Several displays of the EduPARK app showing some of its main features. **Figure 1.** Several displays of the EduPARK app showing some of its main features.

The logic of searching for virtual treasures/caches is based on Geocaching principles, integrated into the game to increase user involvement and motivation. An AR icon is always available so that, at any time, users can point the camera on their mobile device to specific images, called AR markers, to access the AR content. The compass can be used at any time during the game, to support users' orientation through the park. It is also possible to access an interactive map to view the four game areas and various locations to visit during the game. These locations are usually associated with botanical species of the park, marked with physical plaques with AR markers giving access to AR content about the species. Other locations for players to visit are historical interest points with The logic of searching for virtual treasures/caches is based on Geocaching principles, integrated into the game to increase user involvement and motivation. An AR icon is always available so that, at any time, users can point the camera on their mobile device to specific images, called AR markers, to access the AR content. The compass can be used at any time during the game, to support users' orientation through the park. It is also possible to access an interactive map to view the four game areas and various locations to visit during the game. These locations are usually associated with botanical species of the park, marked with physical plaques with AR markers giving access to AR content about the species. Other locations for players to visit are historical interest points with "natural" AR markers, for example pre-existing tiles or information boards to access additional AR information [12,24].

"natural" AR markers, for example pre-existing tiles or information boards to access additional AR information [12,24]. For different target publics, there are different educational guides that encourage users to follow a path to promote learning relevant for the curriculum of Sciences, Mathematics, and History, among other subjects. The project created three guides aimed at students and teachers of different school levels: (i) 1st Cycle of Basic Education (CBE) comprising school years from 1 to 4 (attended by children with

6 to 9 years-old), (ii) 2nd and 3rd CBE, comprising school years from 5 to 9 (attended by children with 10 to 14 years-old), and (iii) Secondary, from 10 to 12 school years (attended by students 15 to 17 years old), and higher education. A fourth guide was also created for the public who visits the park, including questions of general culture, for example, about the typical aesthetic style of Aveiro [25]. with 10 to 14 years-old), and iii) Secondary, from 10 to 12 school years (attended by students 15 to 17 years old), and higher education. A fourth guide was also created for the public who visits the park, including questions of general culture, for example, about the typical aesthetic style of Aveiro [25]. Each guide consists of four stages that correspond to a specific area of the park, in which the

with 6 to 9 years-old), ii) 2nd and 3rd CBE, comprising school years from 5 to 9 (attended by children

*Educ. Sci.* **2020**, *10*, x FOR PEER REVIEW 5 of 20

For different target publics, there are different educational guides that encourage users to follow a path to promote learning relevant for the curriculum of Sciences, Mathematics, and History, among

Each guide consists of four stages that correspond to a specific area of the park, in which the user is challenged to search for points of interest, collect information, answer multiple choice questions, receiving immediate feedback whether or not they answer the question correctly. The questions may have associated multimedia resources, such as audio, photography, illustration, video, or 3D objects in AR [12]. user is challenged to search for points of interest, collect information, answer multiple choice questions, receiving immediate feedback whether or not they answer the question correctly. The questions may have associated multimedia resources, such as audio, photography, illustration, video, or 3D objects in AR [12]. At the end of each stage, the user has five minutes to find virtual treasures in a "treasure hunt"

At the end of each stage, the user has five minutes to find virtual treasures in a "treasure hunt" inspired by Geocaching principles. After this period, if the treasure is not found, the game proceeds normally initiating a new stage. Points are accumulated throughout the game, whenever the users answer the questions correctly and whenever they find an AR marker. At the end of the game, the users have access to the number of correct answers and the total number of accumulated points [24]. inspired by Geocaching principles. After this period, if the treasure is not found, the game proceeds normally initiating a new stage. Points are accumulated throughout the game, whenever the users answer the questions correctly and whenever they find an AR marker. At the end of the game, the users have access to the number of correct answers and the total number of accumulated points [24].

AR combines the real world with a virtual world, which can be three-dimensional and interactive in real time. Usually, the camera of a mobile device is used to detect a previously defined marker (image) that activates associated virtual content. The physical AR plaques, which support the AR markers, have a digital laser engraving on laminated vinyl with ultraviolet protection on galvanized plaques. These are fixed to the ground by means of external piles, in order to constitute permanent elements in the park. AR combines the real world with a virtual world, which can be three-dimensional and interactive in real time. Usually, the camera of a mobile device is used to detect a previously defined marker (image) that activates associated virtual content. The physical AR plaques, which support the AR markers, have a digital laser engraving on laminated vinyl with ultraviolet protection on galvanized plaques. These are fixed to the ground by means of external piles, in order to constitute permanent elements in the park.

The markers about the park's plants present a menu (Figure 2) that includes information about the plant, the leaf, the flower, the fruit, its origin, ecology, and curiosities. The users can choose what they want to have access to systematized information with a photo or image. The words specific to Botany, which may not be known to ordinary people, are underlined in blue, and their meaning can be known by clicking on them, like a glossary. The user can digitally interact with the 3D models of the plant leaves, being able to rotate them and observe the top and bottom page of the leaf, which is often useful to identify species. This feature is particularly advantageous when the specimen is leafless, in the case of deciduous species. All this information, associated with AR markers, is available in Portuguese and in English, so that foreign tourists can also use the EduPARK app to know more about the Park's species [12,24]. For greater user convenience, it is possible to freeze the AR information that appears on the mobile device's screen. This way, the user can turn away from the marker without losing the associated information. The markers about the park's plants present a menu (Figure 2) that includes information about the plant, the leaf, the flower, the fruit, its origin, ecology, and curiosities. The users can choose what they want to have access to systematized information with a photo or image. The words specific to Botany, which may not be known to ordinary people, are underlined in blue, and their meaning can be known by clicking on them, like a glossary. The user can digitally interact with the 3D models of the plant leaves, being able to rotate them and observe the top and bottom page of the leaf, which is often useful to identify species. This feature is particularly advantageous when the specimen is leafless, in the case of deciduous species. All this information, associated with AR markers, is available in Portuguese and in English, so that foreign tourists can also use the EduPARK app to know more about the Park's species [12,24]. For greater user convenience, it is possible to freeze the AR information that appears on the mobile device's screen. This way, the user can turn away from the marker without losing the associated information.

**Figure 2.** Augmented Reality (AR) markers triggering an example of an interactive menu with AR plant information. **Figure 2.** Augmented Reality (AR) markers triggering an example of an interactive menu with AR plant information.

Additionally, the app recognizes "natural" markers, pre-existing in the Park, such as tiles and information signs for monuments. These were frequently used to provide 3D contents produced by the EduPARK project, which aim to provide information that complements the reality observable in those locations. Next, a brief description of the produced contents is presented. *Educ. Sci.* **2020**, *10*, x FOR PEER REVIEW 6 of 20 Additionally, the app recognizes "natural" markers, pre-existing in the Park, such as tiles and information signs for monuments. These were frequently used to provide 3D contents produced by the EduPARK project, which aim to provide information that complements the reality observable in *Educ. Sci.* **2020**, *10*, x FOR PEER REVIEW 6 of 20 Additionally, the app recognizes "natural" markers, pre-existing in the Park, such as tiles and information signs for monuments. These were frequently used to provide 3D contents produced by the EduPARK project, which aim to provide information that complements the reality observable in Additionally, the app recognizes "natural" markers, pre-existing in the Park, such as tiles and information signs for monuments. These were frequently used to provide 3D contents produced by the EduPARK project, which aim to provide information that complements the reality observable in those locations. Next, a brief description of the produced contents is presented.

*Educ. Sci.* **2020**, *10*, x FOR PEER REVIEW 6 of 20

• Moliceiro (typical boat of the city) (Figure 3): On the ancient moliceiro's tile, the AR functionality overlays a real photograph of a current moliceiro to highlight the change of use from transportation of aquatic flora to fertilize farmlands to tourist transportation. those locations. Next, a brief description of the produced contents is presented. • Moliceiro (typical boat of the city) (Figure 3): On the ancient moliceiro's tile, the AR functionality overlays a real photograph of a current moliceiro to highlight the change of use from those locations. Next, a brief description of the produced contents is presented. • Moliceiro (typical boat of the city) (Figure 3): On the ancient moliceiro's tile, the AR functionality overlays a real photograph of a current moliceiro to highlight the change of use from • Moliceiro (typical boat of the city) (Figure 3): On the ancient moliceiro's tile, the AR functionality overlays a real photograph of a current moliceiro to highlight the change of use from transportation of aquatic flora to fertilize farmlands to tourist transportation.

transportation of aquatic flora to fertilize farmlands to tourist transportation.

transportation of aquatic flora to fertilize farmlands to tourist transportation.

**Figure 3.** The typical Aveiro boat AR image. **Figure 3.** The typical Aveiro boat AR image. **Figure 3.** The typical Aveiro boat AR image. **Figure 3.** The typical Aveiro boat AR image.

• Santo António (saint adored by the Portuguese population) (Figure 4): On this tile, the AR functionality overlays several buttons on parts with religious significance, which become interactive and give access to a brief description. • Santo António (saint adored by the Portuguese population) (Figure 4): On this tile, the AR functionality overlays several buttons on parts with religious significance, which become interactive and give access to a brief description. • Santo António (saint adored by the Portuguese population) (Figure 4): On this tile, the AR functionality overlays several buttons on parts with religious significance, which become interactive and give access to a brief description. • Santo António (saint adored by the Portuguese population) (Figure 4): On this tile, the AR functionality overlays several buttons on parts with religious significance, which become interactive and give access to a brief description.

**Figure 4.** The saint tile AR information and interactivity. **Figure 4.** The saint tile AR information and interactivity. **Figure 4.** The saint tile AR information and interactivity. **Figure 4.** The saint tile AR information and interactivity.

• Symmetric tile (Figure 5): On this tile, the AR functionality overlays an animated threedimensional tile that demonstrates its axes of symmetry, through folding. • Symmetric tile (Figure 5): On this tile, the AR functionality overlays an animated threedimensional tile that demonstrates its axes of symmetry, through folding. • Symmetric tile (Figure 5): On this tile, the AR functionality overlays an animated three-dimensional tile that demonstrates its axes of symmetry, through folding. • Symmetric tile (Figure 5): On this tile, the AR functionality overlays an animated threedimensional tile that demonstrates its axes of symmetry, through folding.

**Figure 5.** The symmetric tile AR model. **Figure 5.** The symmetric tile AR model.

**Figure 5.** The symmetric tile AR model. • Torreão (ancient water deposit) (Figure 6): On the building identification plaque, the AR functionality overlays a three-dimensional reconstruction of the building, animated by its **Figure 5.** The symmetric tile AR model. • Torreão (ancient water deposit) (Figure 6): On the building identification plaque, the AR functionality overlays a three-dimensional reconstruction of the building, animated by its • Torreão (ancient water deposit) (Figure 6): On the building identification plaque, the AR functionality overlays a three-dimensional reconstruction of the building, animated by its decomposition into the three main geometric solids that compose it. • Torreão (ancient water deposit) (Figure 6): On the building identification plaque, the AR functionality overlays a three-dimensional reconstruction of the building, animated by its decomposition into the three main geometric solids that compose it.

decomposition into the three main geometric solids that compose it.

decomposition into the three main geometric solids that compose it.

**Figure 6.** The water deposit AR model and its decomposition into geometric solids. **Figure 6.** The water deposit AR model and its decomposition into geometric solids. **Figure 6.** The water deposit AR model and its decomposition into geometric solids.

**Figure 6.** The water deposit AR model and its decomposition into geometric solids.

• Ducks' House (wooden construction for the ducks in the parks lake) (Figure 7): On an identification plaque, the AR functionality overlays a three-dimensional reconstruction of the ducks' house. The object allows its rotation on all axes. • Ducks' House (wooden construction for the ducks in the parks lake) (Figure 7): On an identification plaque, the AR functionality overlays a three-dimensional reconstruction of the ducks' house. The object allows its rotation on all axes. • Ducks' House (wooden construction for the ducks in the parks lake) (Figure 7): On an identification plaque, the AR functionality overlays a three-dimensional reconstruction of the ducks' house. The object allows its rotation on all axes. • Ducks' House (wooden construction for the ducks in the parks lake) (Figure 7): On an identification plaque, the AR functionality overlays a three-dimensional reconstruction of the ducks' house. The object allows its rotation on all axes.

**Figure 7.** The ducks' house AR model, in lateral and top perspectives. **Figure 7.** The ducks' house AR model, in lateral and top perspectives. **Figure 7.** The ducks' house AR model, in lateral and top perspectives. **Figure 7.** The ducks' house AR model, in lateral and top perspectives.

• Monument to Dr Jaime de Magalhães Lima (local personality) (Figure 8): On the monument, the AR functionality overlays the three-dimensional reconstruction, which has interactivity to allow the exploration of the geometric solids that compose it. This model is triggered by pointing the mobile device camera directly at the monument, not requiring a physical AR plaque. This was the marker that constituted the greatest challenge from the technological point of view, since the overlap of the 3D model on the physical monument requires a high precision in terms of dimension and positioning. • Monument to Dr Jaime de Magalhães Lima (local personality) (Figure 8): On the monument, the AR functionality overlays the three-dimensional reconstruction, which has interactivity to allow the exploration of the geometric solids that compose it. This model is triggered by pointing the mobile device camera directly at the monument, not requiring a physical AR plaque. This was the marker that constituted the greatest challenge from the technological point of view, since the overlap of the 3D model on the physical monument requires a high precision in terms of dimension and positioning. • Monument to Dr Jaime de Magalhães Lima (local personality) (Figure 8): On the monument, the AR functionality overlays the three-dimensional reconstruction, which has interactivity to allow the exploration of the geometric solids that compose it. This model is triggered by pointing the mobile device camera directly at the monument, not requiring a physical AR plaque. This was the marker that constituted the greatest challenge from the technological point of view, since the overlap of the 3D model on the physical monument requires a high precision in terms of dimension and positioning. • Monument to Dr Jaime de Magalhães Lima (local personality) (Figure 8): On the monument, the AR functionality overlays the three-dimensional reconstruction, which has interactivity to allow the exploration of the geometric solids that compose it. This model is triggered by pointing the mobile device camera directly at the monument, not requiring a physical AR plaque. This was the marker that constituted the greatest challenge from the technological point of view, since the overlap of the 3D model on the physical monument requires a high precision in terms of dimension and positioning.

**Figure 8.** The monument AR model and its interactivity. **Figure 8.** The monument AR model and its interactivity. **Figure 8.** The monument AR model and its interactivity. **Figure 8.** The monument AR model and its interactivity.

• EduPARK Mascot (the inspiration for this mascot was the fact that a female monkey lived in the park for several years, so it is commonly known as the Monkey Park) (Figure 9): On an

• EduPARK Mascot (the inspiration for this mascot was the fact that a female monkey lived in the park for several years, so it is commonly known as the Monkey Park) (Figure 9): On an

• EduPARK Mascot (the inspiration for this mascot was the fact that a female monkey lived in the park for several years, so it is commonly known as the Monkey Park) (Figure 9): On an

• EduPARK Mascot (the inspiration for this mascot was the fact that a female monkey lived in the park for several years, so it is commonly known as the Monkey Park) (Figure 9): On an identification plaque, the AR functionality overlays an animated three-dimensional model of the mascot and her living cage, allowing the visualization of the monkey that remains an iconic symbol of the park. *Educ. Sci.* **2020**, *10*, x FOR PEER REVIEW 8 of 20 identification plaque, the AR functionality overlays an animated three-dimensional model of the mascot and her living cage, allowing the visualization of the monkey that remains an iconic symbol of the park.

**Figure 9.** The EduPARK mascot AR animated model. **Figure 9.** The EduPARK mascot AR animated model.

In summary, the EduPARK app innovation relies on the articulation of the following components: (i) The use of new and easy to explore technologies, mobile supported AR; (ii) Geocaching-based learning in outdoor environments; and (iii) cross subjects educational materials (guides specific for different educational levels) [25]. In summary, the EduPARK app innovation relies on the articulation of the following components: (i) The use of new and easy to explore technologies, mobile supported AR; (ii) Geocaching-based learning in outdoor environments; and (iii) cross subjects educational materials (guides specific for different educational levels) [25].

#### **3. Materials and Methods 3. Materials and Methods**

Mixed method research approaches are known for combining qualitative and quantitative elements to achieve a level of understanding and corroboration in breadth and depth, not possible through either approach on its own [26]. So, a mixed method approach is used to analyse the effect of using the EduPARK app into the following dimensions of educational value: (a) Learning value; (b) intrinsic motivation; (c) engagement; (d) authentic learning; (e) lifelong learning; and (f) conservation and sustainability habits by triangulation of teachers and students' opinions, after playing the EduPARK game, with game results. Mixed method research approaches are known for combining qualitative and quantitative elements to achieve a level of understanding and corroboration in breadth and depth, not possible through either approach on its own [26]. So, a mixed method approach is used to analyse the effect of using the EduPARK app into the following dimensions of educational value: (a) Learning value; (b) intrinsic motivation; (c) engagement; (d) authentic learning; (e) lifelong learning; and (f) conservation and sustainability habits by triangulation of teachers and students' opinions, after playing the EduPARK game, with game results.

This section comprises four sections: (i) A context introduction regarding the EduPARK activity and study participants; (ii) data collection methods; (iii) data analysis strategies; and finally, (iv) sample description. This section comprises four sections: (i) A context introduction regarding the EduPARK activity and study participants; (ii) data collection methods; (iii) data analysis strategies; and finally, (iv) sample description.

#### *3.1. The EduPARK Activity and Study Participants*

for participants to use during the activity, whenever needed.

*3.1. The EduPARK Activity and Study Participants*  To collect data, the research team organizes activities involving schools and other educational entities (such as after school study centres) for a period of about one year, from March 2018 to April 2019, in Infante D. Pedro Park (Aveiro, Portugal). The EduPARK activity, organised under the project, To collect data, the research team organizes activities involving schools and other educational entities (such as after school study centres) for a period of about one year, from March 2018 to April 2019, in Infante D. Pedro Park (Aveiro, Portugal). The EduPARK activity, organised under the project, comprises:


depending on the age of the players). For the game playing, the EduPARK project provided mobile devices (smartphones or tablets) For the game playing, the EduPARK project provided mobile devices (smartphones or tablets) for participants to use during the activity, whenever needed.

the groups started playing the educational guide for their academic level, in a lagged departure

A total of 44 activities for 1007 students, from the 1st Cycle of Basic Education (CBE) to Secondary Teaching, and 122 accompanying adults, usually teachers, were organised both in formal and non-formal educational contexts. After student distribution by groups and resources checking, the groups started playing the educational guide for their academic level, in a lagged departure organization. Five activities (out of 44) were mixed in what concerns students' school level, and the games played in these sessions were either the one for the 1st CBE or the one for the 2nd/3rd CBE. Table 1 shows, per school level, the number of activities (total of 42), students (total of 1007), and teachers and other adults (total of 122) who accompanied the students in the activities between March 2018 and April 2019. Considering both students and teachers, the total of participants was 1129.

**Table 1.** The relation of number of activities, participant students, and accompanying teachers in the EduPARK activities, per school level. CBE: Cycle of Basic Education.


#### *3.2. Data Collection*

Data collection included a paper questionnaire and automatic app logging mechanisms, supporting triangulation of users' subjective points of view with their performance during the game (for example, number of right answers), which are objective results. At the end of the game playing activity, each participant (both students and teachers or other accompanying adults) was invited to complete an evaluation questionnaire. The questionnaire was similar for all the school levels and types of users (students and teachers); however, it included adaptations to the age and educational context of the respondents. Teachers who accompanied classes in the activity more than once filled in only one questionnaire, in the first activity, so the number of teacher questionnaires is lower than the number of effective participant teachers. This option supports the respondents in avoiding biases caused by increasing familiarity, as, according to Sauro [27], having prior experience with the system increases scores. For example, when analysing websites usability with Brooke's System Usability Scale [28], Sauro found that users who had previously experienced the website, tended to generate higher SUS scores (11% higher) than first-time users.

The set of questionnaires analysed in this study were used in previous studies of the EduPARK project [9,29]. The tool comprises four sections, with mostly closed-ended questions in a Likert scale, where 1 corresponds to 'strongly disagree' and 5 to 'strongly agree', and a minority of open-ended questions that complement the quantitative data and provide a level of insight not captured by the closed questions. One section collected basic demographic data, such as age and gender, their familiarity with mobile devices, and their opinion on mobile learning advantages and disadvantages. Another section is about the interest regarding the activity of playing the EduPARK game in the park; although this is not the focus of this work. Another section refers to the Educational Value Scale (EVS) (presented in [9] and reliability demonstrated in [21]), and the last one consists of the System Usability Scale (SUS) [18,28,30] with the minor adjustment of replacing the word "system" with "app", which does not appear to have an effect on the resulting scores [19]. The same authors highlighted SUS high reliability and pointed it as the most sensitive poststudy questionnaire, designed to assess perceptions of usability.

The questionnaire for students of the 1st CBE did not have questions on mobile learning advantages and disadvantages nor an open answer question to justify their opinion regarding the interest of the activity. Teachers' questionnaires had additional questions regarding the interest of the activity for their own practice and for their colleagues.

Only fully answered questionnaires were included in the study. All teachers' questionnaires were considered, however, some students' questionnaires were discarded, as some questions were not answered.

The app includes automatic mechanisms of game log generation. These mechanisms allowed the collection of anonymous information from finished games during the observation period (between March 2018 and April 2019). The information includes: (a) Final score (points gathered through correct answers and points gained through collecting bananas); (b) game time; (c) number of questions answered correctly and incorrectly; and (d) number of hunted Geocaching treasures.

All data collection, processing, and storage procedures respected research ethics principles. Data were collected anonymously and did not include any personal information or set of information allowing the identification of specific participants.

#### *3.3. Data Analysis*

The focus of this study is the educational value of mobile AR games, analysed through an illustrative example, the EduPARK app. So, the questionnaire data analysis focuses the section comprised by the EVS, EduPARK activity interest, including qualitative data. Game logs were also analysed to triangulate the users' opinions.

Data were analysed through scores computing, descriptive statistics, and content analysis of open response questions.

The computing of EVS scores is similar to the SUS computing process described by Brooke [18,28]. However, as described by Sauro and Lewis [19], to get a 12 items scale to range from 0 to 100, the sum of the 12 EVS items contribution is multiplied by 2.0834 (the result of the division of 100 by 48).

As questionnaires' open questions aimed solely to collect the users' reactions to the EduPARK activity, content analysis was performed to present illustrative citations of participants' answers.

Document analysis of the app game logs was performed. A table with the overall results, regarding game scores attained, number of correct and incorrect answers, game time, etc., is presented in the next section.

Finally, the questionnaire and app log data were triangulated to provide a more comprehensive knowledge of users' opinion regarding the EduPARK app educational value. This analysis is presented in the next section.

#### *3.4. Sample Description*

The survey allowed gathering information about the users' profile. Table 2 presents the characterization of each type of user in terms of number of returned questionnaires (valid and fully filled in), age, gender, school year, Android mobile device ownership, and its use to learn or to promote learning. The response rate was high in all types of users, more specifically, it was 85.1% for 1st CBE students, 97.0% for 2nd and 3rd CBE students, 87.8% for Secondary Teaching students, and 57.4% for accompanying teachers. It is worthwhile to note that several teachers accompanied groups of students in more than one activity; hence, they were counted more than once in the participants section. However, these teachers were asked to answer the questionnaire only in their first participation, capturing their immediate perceptions. As highlighted in the Materials and Methods section, the literature reports an increase of SUS scores with the increasing familiarity with the system under evaluation [27].

Students' ages vary within the expected values for their respective schooling levels in Portugal: 6–11 and mean value of 8.1 for 1st CBE, 10–15 and mean of 11.0 for 2nd and 3rd CBE, and 15–20 and mean of 16.6 for secondary teaching. Teachers' ages vary between those that are expected for graduated workforces, between 23 and 64 years-old, with a mean of 42.1.

The proportion of female and male students is balanced (47.4%, 53.1%, and 55.4% of females); however, the teaching class is composed mainly by female teachers (77.1%) according to the general scenario in Portuguese schools, where the female teachers percentage was 77.9 in 2019 [31].


**Table 2.** A characterization of each type of participant on the EduPARK activities, from March 2018 to April 2019.

In the 1st CBE, most participating students frequented year 2 (43.7%), followed by year 3 and year 4 students (both 18.0%), and fewer students in year 1 (6.9%), which are expectable results, as the EduPARK app requires reading skills. In the 2nd and 3rd CBE, most students frequented year 5 (76.6%), followed by year 8 students (15.9%), with a small participation of students from other school-years (7.6% for years 6, 7, and 9). This participation disparity can be associated with the National Directives for Natural Sciences Curriculum, as environment-related learning content is the focus of school-years 5 and 8. In the secondary teaching context, the participation in the EduPARK activity involves one class of each school-year. Comparing with the other school levels, this smaller level of participation may indicate a lower educative value of the app for secondary teaching. An alternative explicative hypothesis can be teachers' fears of adopting new teaching methodologies with classes of students that will be submitted to mandatory national exams. Regarding participating teachers, all had higher education qualifications, which is a prerequisite in the Portuguese Education System. A considerable portion (32.8% = 21.4% + 11.4%) continued studies further (Post-graduation, Master's degree, and Doctorate).

The majority of students and teachers owned Android mobile devices, such as smartphones or tablets. The lower Android device penetration rate is in the 1st CBE group (70.9%), and the higher rate is in the teachers' group (92.9%).

Finally, most participants mentioned using mobile devices to learn or to promote learning. Most students and teachers claimed they used mobile devices for learning purposes either sometimes (47.9%, 68.0%, 50.8%, and 45.7%) or frequently (26.7%, 20.1%, 44.6%, and 40.0%). The smallest answer proportion was registered in the "Never uses" option for all types of participants (25.4%, 12.0%, 4.6%, and 14.3%). According to the results, most of these students and their teachers are already quite familiar with mobile technologies and usually employ them for learning. The results seem to support the literature, regarding the proliferation of mobile devices [29], especially in what concerns the young population.

When questioned about advantages of the use of mobile devices to learn, students and accompanying teachers mostly indicated a positive perspective regarding mobile learning, and Figure 10 shows the percentage of agreement with each advantage sentence.

All sentences related with positive aspects of using mobile devices to learn achieved a frequency of at least 244 students of basic education from a total of 384 (63.5%), 36 Secondary students from a total of 65 (55.4%), and 46 accompanying teachers from a total of 70 (65.7%). Only 3.6% of basic education students did not recognize any advantage in mobile learning. Among the most acknowledged advantages are 'you can learn in a fun way' (81.8% for Basic Education students and 88.6% for accompanying teachers), and 'it is easy to find the information I want' (84.6%), for Secondary students. Moreover, 9.9% Basic Education students added new advantages, such as: 'Learn quickly', 'It's much more interesting', 'It's not boring', 'We can work as a team', and 'We can learn on other locations besides school'. On the

other hand, accompanying teachers (5.7%) reported 'The use of images', 'It is fast', and 'It promotes discovery learning'. *Educ. Sci.* **2020**, *10*, x FOR PEER REVIEW 12 of 20

**Figure 10.** The opinions of participant students (2nd/3rd Cycles of Basic Education and secondary teaching), and accompanying teachers, in percentage, about advantages in using mobile devices to **Figure 10.** The opinions of participant students (2nd/3rd Cycles of Basic Education and secondary teaching), and accompanying teachers, in percentage, about advantages in using mobile devices to learn.

learn. All sentences related with positive aspects of using mobile devices to learn achieved a frequency of at least 244 students of basic education from a total of 384 (63.5%), 36 Secondary students from a total of 65 (55.4%), and 46 accompanying teachers from a total of 70 (65.7%). Only 3.6% of basic education students did not recognize any advantage in mobile learning. Among the most acknowledged advantages are 'you can learn in a fun way' (81.8% for Basic Education students and 88.6% for accompanying teachers), and 'it is easy to find the information I want' (84.6%), for Secondary students. Moreover, 9.9% Basic Education students added new advantages, such as: 'Learn quickly', 'It's much more interesting', 'It's not boring', 'We can work as a team', and 'We can Regarding the difficulties of mobile learning, Figure 11 shows that 141 Basic Education students (36.7%), 29 Secondary students (44.6%), and 16 accompanying teachers (22.9%), recognized not having any difficulties in the use of mobile devices to learn. The most stated difficulties are the need for an internet connection (mentioned by 62.9% of accompanying teachers and 59.9% of basic education students), and increased battery consumption (mentioned 55.4% of secondary students). The EduPARK project approach contributes to reduce these constrains, as: i) The game supporting app was conceived for offline use, not requiring internet connection, so this is not an issue; ii) in activities promoted by the EduPARK team, the project provides full charged mobile devices, thus, not interfering with the mobile devices' battery of participants. *Educ. Sci.* **2020**, *10*, x FOR PEER REVIEW 13 of 20

**Figure 11.** The opinions of participant students (from 2nd and 3rd Cycles of Basic Education and Secondary Teaching), and accompanying teachers, in percentage, about difficulties in using mobile devices to learn. **Figure 11.** The opinions of participant students (from 2nd and 3rd Cycles of Basic Education and Secondary Teaching), and accompanying teachers, in percentage, about difficulties in using mobile devices to learn.

uploaded to the EduPARK web platform were submitted to document analysis.

*4.1. Users' Perceptions: Educational Value and Interest* 

achieved similar results in previous studies [9,29,32].

This section reports this study's main results and their discussion in light of the consulted literature. The data were collected in the EduPARK activities conducted between March 2018 and April 2019. First, the authors analysed the EduPARK app EVS for the entire aggregated dataset and for each of the considered target public: Students of 1st Cycle of Basic Education (CBE), 2nd and 3rd CBE, and secondary teaching, and their accompanying teachers and other adults (other school staff, parents, etc.). Data from open questions of the questionnaire about the interest of the EduPARK activity are also analysed through content analysis. Finally, app game logs of finished games

Figure 12 presents the participants' general perceptions on the app educational value. The EVS items are worded in positive and negative sentences alternatively. Overall, participants' perceptions are positive, as most respondents (strongly) agree with the scale positive formulated items and (strongly) disagree with the negative formulated items. For example, 629 (68.1%) participants strongly agree with the sentence "This app helps you/students learn more about topics we study/I teach at school". A similar amount, 639 (69.2%) participants, strongly disagree with the sentence "This app does not help to realize that it is important to protect nature." These results indicate that participants considered that the EduPARK app comprises all the dimensions of the educational value analysed in this study: Learning value, intrinsic motivation, engagement, authentic learning, lifelong learning, and conservation and sustainability habits. Subsets of students and teachers from this dataset

**4. Results and Discussion** 

Finally, 5.5% Basic Education students added new difficulties, such as 'This is only for androids', 'It is not possible to explore the game in the classroom', 'running out of battery', and 'preference to study in books'. The first mentioned difficulty will be considered in further similar apps, intending to produce games for both Android and iOS operating systems.

#### **4. Results and Discussion**

This section reports this study's main results and their discussion in light of the consulted literature. The data were collected in the EduPARK activities conducted between March 2018 and April 2019. First, the authors analysed the EduPARK app EVS for the entire aggregated dataset and for each of the considered target public: Students of 1st Cycle of Basic Education (CBE), 2nd and 3rd CBE, and secondary teaching, and their accompanying teachers and other adults (other school staff, parents, etc.). Data from open questions of the questionnaire about the interest of the EduPARK activity are also analysed through content analysis. Finally, app game logs of finished games uploaded to the EduPARK web platform were submitted to document analysis.

#### *4.1. Users' Perceptions: Educational Value and Interest*

Figure 12 presents the participants' general perceptions on the app educational value. The EVS items are worded in positive and negative sentences alternatively. Overall, participants' perceptions are positive, as most respondents (strongly) agree with the scale positive formulated items and (strongly) disagree with the negative formulated items. For example, 629 (68.1%) participants strongly agree with the sentence "This app helps you/students learn more about topics we study/I teach at school". A similar amount, 639 (69.2%) participants, strongly disagree with the sentence "This app does not help to realize that it is important to protect nature." These results indicate that participants considered that the EduPARK app comprises all the dimensions of the educational value analysed in this study: Learning value, intrinsic motivation, engagement, authentic learning, lifelong learning, and conservation and sustainability habits. Subsets of students and teachers from this dataset achieved similar results in previous studies [9,29,32]. *Educ. Sci.* **2020**, *10*, x FOR PEER REVIEW 14 of 20

EVS data are submitted to exploratory data analysis. Table 3 presents the computed EVS scores arithmetic means, medians, modes (measures of central tendency), standard deviation (measure of variability), as well as minimum and maximum values, for each type of user and for the entire dataset. This table reveals high arithmetic means and medians of EVS for all participants in this study: All above 80 for students in basic education and accompanying teachers, and around 70 for students in EVS data are submitted to exploratory data analysis. Table 3 presents the computed EVS scores arithmetic means, medians, modes (measures of central tendency), standard deviation (measure of variability), as well as minimum and maximum values, for each type of user and for the entire dataset. This table reveals high arithmetic means and medians of EVS for all participants in this study: All above 80 for students in basic education and accompanying teachers, and around 70 for students in secondary

**Table 3.** EVS scores descriptive statistics for each type of user of the EduPARK app, in the activities

**Mean** 83.0 83.0 69.1 83.0 82.0 **Median** 85.4 85.4 70.8 86.5 85.4 **Mode** 91.7 91.7 72.9 89.6 91.7

Further analysis included the multiple boxplot display shown in Figure 13. The boxplots in this figure present both summary statistics (minimum value, lower hinge, median, arithmetic mean, upper hinge, and maximum value) and raw data. According to Theus and Urbanek [33], boxplots show robust measures of location and spread of datasets, and they are known for providing visual aids to compare two or more datasets. In each boxplot, the vertical lines represent ordered data (from the minimum to the maximum values); the boxes contain the middle range data, from the 1st quartile (25th percentile) to the 3rd quartile (75th percentile); the middle lines in the boxes indicate the median value for each dataset; and the cross below the median indicates the arithmetic mean. Basic education datasets present potential outliers, which are the dots located 1.5 times below the size of the box. Therefore, the arithmetic mean of these two datasets must be treated with scepticism, as this central

**Standard deviation** 13.9 13.6 13.4 11.9 14.0

**Minimum-Maximum** 16.7–100.0 20.8–100.0 39.6–100.0 54.2–100.0 16.7–100.0

**1st CBE 2nd/3rd CBE Secondary Teaching Accompanying Teachers All Users** 

considered less relevant for secondary teaching students.

organized by the project, from March 2018 to April 2019.

**Type of User** 

**Descriptive Statistics** 

**EVS** 

secondary teaching. Additionally, the median is 85.4 for two EduPARK's target publics and for all

teaching. Additionally, the median is 85.4 for two EduPARK's target publics and for all users. It is worth to note that all groups of participants included several users that attributed the highest value possible for EVS (100). This preliminary analysis indicates that the EduPARK game educational value is considered high by the questionnaire respondents; however, it seems to be considered less relevant for secondary teaching students.


**Table 3.** EVS scores descriptive statistics for each type of user of the EduPARK app, in the activities organized by the project, from March 2018 to April 2019.

Further analysis included the multiple boxplot display shown in Figure 13. The boxplots in this figure present both summary statistics (minimum value, lower hinge, median, arithmetic mean, upper hinge, and maximum value) and raw data. According to Theus and Urbanek [33], boxplots show robust measures of location and spread of datasets, and they are known for providing visual aids to compare two or more datasets. In each boxplot, the vertical lines represent ordered data (from the minimum to the maximum values); the boxes contain the middle range data, from the 1st quartile (25th percentile) to the 3rd quartile (75th percentile); the middle lines in the boxes indicate the median value for each dataset; and the cross below the median indicates the arithmetic mean. Basic education datasets present potential outliers, which are the dots located 1.5 times below the size of the box. Therefore, the arithmetic mean of these two datasets must be treated with scepticism, as this central tendency measure is affected by outliers. In this case, the median is a suitable central tendency measure [33].

It is possible that the presence of potential outliers in the younger students' datasets is due to the high level of excitement related to the timing and location where the questionnaires were applied (just after playing the game, in the park). This was reported before <sup>9</sup> , and may have hindered students' concentration during the questionnaire filling. Moreover, possible reading difficulties of young students may also have biased the results. These hypotheses are supported by a previous study [21], where a relatively low Cronbach's coefficient α (0.653) was found for this dataset. According to Hair et al. [34], this value indicates a not yet acceptable reliability (0.7), however, other authors consider the value 0.6 as the lower bound of reliability acceptance, particularly in the early stage of research [35,36], which is the case of the EVS. To reduce the impact of reading difficulties, children were supported by the adults who accompanied each group, to assure they understood the questions and answer options, whenever needed.

As mentioned above, the medians of the two datasets of basic education students are 85.4, the one of the accompanying teachers is very similar, 86.5; and the median of secondary teaching students is 70.8, which is the lowest. Moreover, as secondary teaching students' median lies entirely outside the interquartile ranges of the remaining target groups, this dataset is likely to be different from the other three.

The interquartile ranges (box lengths) are small and similar for all the datasets: 16.7 for 1st CBE, 14.6 for 2nd/3rd CBE, 19.8 for secondary teaching, and 16.7 for accompanying teachers. Hence, all the datasets seem to concentrate near their median values.

Regarding distribution of the datasets, the box and whiskers considered together reveal the range of each dataset. Secondary Teaching students are the participants with the most scattered data, as its range is 60.4. Therefore, the consensus regarding the Educational Value of the EduPARK app among Secondary Teaching students was lower than among the other types of participants.

**Figure 13.** EVS scores boxplots for each target public of EduPARK.

As the interquartile ranges of basic education students and accompanying teachers' datasets roughly overlap each other, these datasets seem similar. On the other hand, the interquartile range for Secondary Teaching students' dataset is located below all the other interquartile ranges, also indicating there is a difference between them. These results support the initial analysis (presented in Table 3), as the EVS median values for basic education students and accompanying teachers are higher than the EVS median value for secondary teaching students. These results are supported by previous studies, where the mean EVS value was 83.8 for 244 students in 2nd and 3rd CBE [29] and 88.2 for a cohort of 45 teachers in a workshop [32]. This indicates that the EduPARK app has a high educational value. Moreover, in this study, teachers also assigned high EVS scores, revealing the EduPARK app is a mobile learning resource with a high educational value for practitioners.

To understand further the case of the educational value of the EduPARK app for Secondary Teaching, the cohort of teachers who accompanied Secondary Teaching students was analysed separately to collect indicators of possible differences in teachers' perspectives of the educational value of the app, according to the school level (basic education vs secondary teaching) they accompanied. These teachers constitute a small portion of this type of participant (7 in a total of 70), and assigned the following EVS scores: 81.3; 81.3; 89.6; 89.6; 93.8; 97.9; and 97.9. This yields a mean of 83.1 and median of 87.5, which is in line with the overall teacher dataset. Hence, despite this small number of teachers, this result supports the idea that, although the EduPARK app educational value for secondary teaching may not be immediately identified by teachers, the practitioners who participated with secondary teaching students revealed a very positive opinion regarding the app's educational value.

#### *4.2. Game Logs: Educational Value*

App game logs of finished games uploaded to the EduPARK web platform are presented in Table 4 to triangulate with data gathered from the questionnaire for more comprehensive knowledge regarding the EduPARK app educational value. Each log corresponds to the performance of a group of students (usually 3 or 4) who played the game collaboratively, in teams. These values (final score, game time, right and wrong answers, and found treasures) are generated automatically by the mobile devices and uploaded to the project web platform, after game over. The data are anonymous, and accessed only by the project team.


**Table 4.** The app game logs of finished games uploaded to the EduPARK web platform from March 2018 to April 2019.

Table 4 presents the app game logs organized per target group. The final score is calculated by the sum of points gathered through correct answers and points gained through collecting bananas in the hunted treasures. The achievement of higher scores indicates higher educational value, as teams need to observe their environment, to select information provided by the AR contents accessed through the app, to analyse alternative solutions for the challenges, to negotiate meaning among group members, and to provide the answer considered correct by the collective.

From the table, the final score average is higher in the 2nd/3rd CBE group of students, as well as the minimum and maximum final score, compared to the other type of students. This is in accordance to the EVS scores for this group, reinforcing the high educational value of the app for this school level.

Game time, presented in the format hour:minute, corresponds to the time consumed from starting the game to its end (when the final scores are displayed in the last screen). The average game time is decreasing with the age of students, as older students are quicker to read and to answer questions (reaching less minimum and maximum time to finish the game), when compared to the younger ones. The game time does not indicate better performance, because students could finish the game very quickly, for example, in just over half an hour, without answering correctly the questions and not properly exploring all the educational resources available in the game, for example, AR contents, 3D models, images, or additional information. Moreover, the variance in game time can also indicate that students progress in the activity at their own pace, providing enough time for students to interact with each other and their prior knowledge with the new experiences, and thus, supporting learning. The fact that all groups were able to finish the game also indicates that the educational guides supported by the app are adequate to their respective school levels.

The average of questions answered correctly is higher in the 2nd/3rd CBE group of students (24.4) comparing to the 1st CBE (21.6) and secondary teaching group of students (18.5). Accordingly, the minimum and the maximum number of right answers is also higher for this group of students. The opposite situation occurs with the wrong answers, showing that some basic education groups answered correctly all the questions (zero wrong answers), contrasting to 4 minimum wrong answers for secondary teaching students. Accordingly, the older students reached higher values of maximum number of wrong answers (23), indicating that this type of student revealed more interest in finishing the game quickly than having a good learning performance. It is worthwhile to note that the AR contents, which are mostly provided before the proposed challenges, support the students in finding the solutions. It is a matter of students deciding to explore them properly, in order to select and analyse the contextualized information, to have a good performance in the game. The differences in the means

of right and wrong answers, once more, indicates that the app is more suitable for the 2nd/3rd CBE group of students.

The averages of hunted Geocaching treasures are quite similar in the three types of students (varying from 3.6 to 3.8), which means that almost all group of students found the four virtual treasures available in the game. The Geocaching treasure hunt is a feature introduced in the game for motivational purposes, so the fact that the groups attempted to find the treasures is a strong indicator that they are engaged with the activity. It is worthwhile to note that finding the treasures is not essential to proceed with the game, as the groups could keep playing without finding any treasure.

Finally, the above presented game logs analysis triangulated with EVS scores (presented in the previous sub-section) reinforces the finding of the suitability of the EduPARK game as an educational resource, particularly for 2nd/3rd CBE students. Moreover, participants' perceptions are in consonance with students' performance in the game, as they achieved, overall, a good game performance.

#### **5. Conclusions**

This work addresses an identified need of research reported before [9], as the previous analysis of the Educational Value of the EduPARK app, with a smaller sample of 1st CBE students, is now expanded to include a higher number of respondents and other target publics.

The EduPARK game achieved an average EVS of 82.0, with higher values for the subsets of data referent to basic education students and for teachers, who experienced this mobile AR game in loco, that is, in a Green City Park. This applies to a sample of users who claimed: i) To have their own Android mobile devices (the lower penetration rate was registered in the younger students, with 70.9%); ii) to use them to learn or promote learning, at least sometimes (particularly among secondary teaching students, with 95.4%); and iii) to have a quite positive perspective regarding mobile learning, considering it is more advantageous than disadvantageous. The high educational value of this mobile AR game is supported by the data collected through the app logging mechanisms, as the groups of students achieved a good performance, overall. Educational resources that combine this set of innovative features, as being mobile, designed for outdoor use (namely in urban parks), with contextualized AR contents, and supporting game-based activities, may promote learning, both at a cognitive level and at an affective one, increasing motivation for learning.

The game revealed a particularly high educational value for basic education students, as secondary teaching students assigned it lower EVS scores. Moreover, the small number of activities with secondary teaching students (3 out of 42) seems to support the lower adequateness of this methodology for this school level. However, both basic education and secondary teaching teachers, who accompanied students in the activity, revealed a very positive opinion regarding the ability of the app to promote learning. This result may indicate that secondary teaching teachers are more reluctant in trying out new approaches with students of a school level that comprises national exams. This issue needs further study, as the number of teachers accompanying this school level was low (7 in a total of 70).

Secondary teachers' reluctance in adopting this mobile AR game may be due to dominant mentalities associating mobile devices, games, and parks to distraction, play, and leisure [12]. Nevertheless, early adopters among teachers seem quite optimist towards this approach and may promote changes in their colleagues' mentalities on how their students can learn. Still, more studies involving a higher number of students and teachers of this school level are needed.

Limitations of this study are related to the young age of most participants (405 students in 1st CBE and 384 students in 2nd and 3rd CBE). As discussed before, factors such as level of excitement and reading difficulties may have influenced the results, for this age group. Compensation methods may include adapted questionnaires in terms of vocabulary and supporting children in the interpretation of the items, which were the main strategies in the EduPARK activities. In future studies, the use of face emojis instead of the numbered Likert may be a powerful alternative for young students [19].

Another issue to consider is groups' constitution, which may also have influenced the results, as each student's participation in the game is lower in bigger groups. This factor may have an impact

on how the activity is perceived by the app players. However, this variation in groups' constitution could not be addressed, particularly with activities with a lot of participants, as it was related with the human resources available to accompany each group of children in each session.

Finally, as the EVS is a new data collection tool, more studies involving the rating of other educational mobile AR games are needed, to both improve and consolidate this data collection tool, and also to establish benchmarks and scale norms. Future research must address this issue, by implementing the EVS with more users and by analysing the educational value of other mobile AR games.

In terms of implications for research, this paper contributes to the mobile game-based AR learning literature, as it is an empirical study with evidence on the educational value of the integration of these elements in teaching practices. For practitioners, this work also bears the report of an example of excellent cross-subjects' educational materials—the learning game—that comprises a very useful tool for teachers and students to explore scientific knowledge by accessing appealing information on cross subjects references (such as biological and historical) of a local Urban Green Park.

**Author Contributions:** Conceptualization, L.P. and M.M.M.; Data curation, L.P. and M.M.M.; Formal analysis, M.M.M.; Investigation, L.P. and M.M.M.; Methodology, L.P. and M.M.M.; Project administration, L.P.; Writing—original draft, L.P. and M.M.M.; Writing—review & editing, L.P. and M.M.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by FEDER-Fundo Europeu de Desenvolvimento Regional funds through the COMPETE 2020-Operational Programme for Competitiveness and Internationalisation (POCI), and by Portuguese funds through FC-Fundação para a Ciência e a Tecnologia, grant number POCI-01-0145-FEDER-016542, and the work of the second author is funded by national funds (OE), through FCT–Fundação para a Ciência e a Tecnologia, I.P., in the scope of the framework contract foreseen in the numbers 4, 5 and 6 of the article 23, of the Decree-Law 57/2016, of August 29, changed by Law 57/2017, of July 19.

**Acknowledgments:** The authors thank the reviewers for their insightful comments.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


=0ahUKEwjt8YeUouTpAhWwBWMBHVpZAcwQ6AEITDAD#v=onepage&q=boxplot&f=false (accessed on 3 June 2020).


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **The Impact of Teacher Training Using Mobile Augmented Reality Games on Their Professional Development**

**Margarida M. Marques \* and Lucia Pombo \***

CIDTFF—Research Centre on Didactics and Technology in the Education of Trainers, Department of Education and Psychology, University of Aveiro, 3810-193 Aveiro, Portugal

**\*** Correspondence: marg.marq@ua.pt (M.M.M.); Lpombo@ua.pt (L.P.)

**Abstract:** Ongoing technology progress sustains innovative teaching approaches. Mobile devices, augmented reality (AR), and games are a few of the new resources that teachers have at their disposal to promote student learning. However, their effective integration into practices requires training, so there is a need to analyze the impact of training initiatives on teacher professional development. A case study is being conducted on the development process of mobile AR games for Science, Technology, Engineering and Mathematics (STEM) learning by 14 Portuguese in-service teachers in a 50 h workshop. This contribution refers to the analysis of this training's impact on teacher professional development through a questionnaire filled in at the beginning and end of the workshop. This study registered a higher impact on teachers' understanding of AR educative use, the less-known approach, compared to mobile and game-based learning. Moreover, teachers became more experienced with these approaches as learners, and reported having explored them with their students during the workshop period. Teacher ability to identify benefits and barriers in these approaches increased with the workshop, particularly the learning that could be promoted with mobile AR games. The presented set of barriers to implementation is relevant to future teacher professional-development initiatives.

**Keywords:** continuous teacher training; mobile learning; educative augmented reality; game-based learning; training impact; teacher professional development

#### **1. Introduction**

Mobile technologies' versatility and relatively low cost may be two factors concurring to their pervasiveness in modern industrialized societies. Hence, the availability of a resource that allows performing a variety of tasks, namely some related to information seeking and treatment, makes their integration in teaching and learning practices a logical step to take, having given rise to the term "mobile learning".

Mobile learning can be defined in different ways. Perspectives may fall into one of the following categories: (1) technocentric, referring to learning processes that occur through the use of handheld technologies, such as mobile phones; (2) mobility of the technology, highlighting the portability allowed by the small size of the supporting technologies; (3) relationship to e-learning, with mobile learning being considered an extension of elearning; (4) augmenting formal education, as a way of extending face-to-face education; (5) learner-centered, referring to learning when the learner is not at a fixed location, thus focusing on the learner's mobility [1–3]. Despite the perspective adopted, the use of mobile devices for educational purposes has been a growing field of research with a history of positive empirical results [4], and their use in game-based learning approaches has also been documented as effective [5].

Game-based learning usually refers to the use of gameplay with the purpose of promoting determined learning objectives [6]. With the motivation factor being one of the most cited arguments in the literature [7–10], other positive results include student

**Citation:** Marques, M.M.; Pombo, L. The Impact of Teacher Training Using Mobile Augmented Reality Games on Their Professional Development. *Educ. Sci.* **2021**, *11*, 404. https:// doi.org/10.3390/educsci11080404

Academic Editors: Greg Kessler and James Albright

Received: 31 May 2021 Accepted: 30 July 2021 Published: 5 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

satisfaction with the learning experience [11], increased student achievement across all schooling levels [12]; and facilitation of students' 21st-century skill development [13]. To achieve these positive results, some caution is required, namely incorporating learning theories into the game design [13], taking into account learners' personal factors, such as learning achievement or learning styles [14], and challenging players in a way that matches students' abilities [15]. Additionally, the competition created by games may increase students' engagement in challenging learning situations and improve their overall sense of enjoyment. When game-winning conditions require working with other players, collaborative dynamics can also be promoted [16].

Despite being considered effective learning tools [8], in real formal education scenarios, accepting digital games in the classroom is one of the initial barriers to overcome, which may be related to factors such as the availability (and reliability) of the supporting technology, prerequisite knowledge required of the teacher, and even financial and licensing issues [17]. For example, a study conducted by Russo, Bragg, and Russo [18] highlighted that although Australian mathematic primary teachers mentioned frequently using games in their practices, digital games were virtually not among their favorite types of games, which may be interpreted as a reluctancy to incorporate digital technologies to support game-based learning.

Future developments in mobile and game-based learning involve evaluating and analyzing game usage data, providing powerful tools on how to create better learning experiences, and developing game-based learning, supported by significant data about users' perception and their performance while playing [19].

These educational approaches, when combined with emerging augmented reality (AR) technologies, can enhance learning experiences, as they can enrich and contextualize learning information offered to learners [20]. AR supported by mobile games can move learning to outdoor settings, fostering authentic learning, and also personal and collaborative learning with a lifelong learning perspective [21]. Other authors [7,22] stress the unique affordances of AR, as an "immersive" interface that enables participants to interact with digital information embedded within the physical environment, supporting situated learning. Outdoor collaborative learning activities using AR become an approach scarcely found in the educational context, although with high potential in education [7,22]. Moreover, the incorporation of AR into educational practices for effective learning, instead of for merely beautiful scenography, requires teacher training in teaching methodologies with AR technologies [23], as teachers are often reluctant to use or integrate them into their science curriculum [24,25].

There is a scarcity of educational resources, such as educational mobile games, that integrate curriculum contents. Considering this scarcity, the EduPARK project: http: //edupark.web.ua.pt/?lang=en (accessed on 3 August 2021) developed and evaluated an app [26] that can be used autonomously, and at any time, using the "game" mode or the "explore freely" mode. It promotes authentic learning so that visitors can enjoy a healthy walk while learning. The game includes several learning guides for different target groups: teachers and students from basic to higher education, and also for park visitors and the public, from a lifelong learning perspective. The tourist guide is also offered in English. The guides integrate multidisciplinary issues under the Portuguese National Education Curriculum and propose interdisciplinary questions articulated to educational challenges along the park using the logic of a treasure hunt. The game enables visitors to explore and access information about the plant species living in the city park, historical references, different multimedia contents, and a park map, allowing interaction. The goal is to accumulate points by answering the questions correctly, visualizing AR marker contents that help to answer questions, and finding virtual caches/treasures (3D images) [27].

The literature has noted that teachers' adoption of mobile technologies may be influenced by factors such as their digital literacy, ICT anxiety, teaching self-efficacy, and perceived ease of use and usefulness [28]. Moreover, teachers need to develop the ability to implement technology in their practices [28]. Hence, the EduPARK project has organized

several short-term workshops for teachers so they can feel confident in using them in their practices, widening the use of mobile and game-based learning. However, the need to involve teachers in the creation of games and of educational resources to integrate into games encouraged the authors of this paper to propose, as trainers, a long-term accredited course with impact on teachers' career progression. The course was directed at Science, Technology, Engineering and Mathematics (STEM) teachers, and allowed them time to explore and experiment with tools for game-based learning, prompting teachers to develop learning content, as advised by De Freitas [29]. Hence, teachers were not only users, but also creators of educational games integrating AR contents.

The scarcity of teacher training on mobile game-based learning with AR makes it relevant to analyze their potential for teachers' practice changes, starting with the perceptions of the involved stakeholders [30]. In this paper, the authors take one step forward by analyzing not just teachers' perceptions, but also the impact of this workshop on the teacher trainees' professional development with respect to the main concepts/teaching approaches it addresses: mobile learning, AR use in education, and game-based-learning. This research comprises training opportunities that revolve around mobile game-based learning with AR, which are innovative, and therefore it is important to analyze their potential in teachers' practice changes.

The next sections briefly present and discuss the adopted materials and methods, and the results concerning the impact of this workshop on the teacher trainees' professional development around the three main concepts of this study: (i) mobile learning, (ii) AR use in education, and (iii) game-based-learning. Finally, the Conclusions section summarizes the main findings, some limitations, and lines of future work.

#### **2. Materials and Methods**

This research was conducted under a case study [31]. Case studies are acknowledged in the literature as effective methodologies to investigate and understand complex issues in real-world settings that do not aim to extrapolate probabilities through statistical generalization [31,32]. This research approach is adequate when researchers' want to understand a real-world case and assume that such an understanding is likely to involve important contextual conditions' [31].

The case in this study was the development process of educational resources by 14 in-service teachers during a 50 h workshop (25 h face-to-face and 25 h autonomous work) developed in the Center Region of Portugal between October 2020 and January 2021. The workshop aimed to promote the collaborative development of open digital educational resources that foster STEM learning based on a game approach and supported by mobile devices.

The research question that guided the work reported in this contribution was: What is the impact of this workshop on the teacher trainees' professional development with respect to: (i) mobile learning, (ii) AR use in education, and (iii) game-based-learning, in what concerns: (a) basic knowledge, (b) teacher experience as a learner, (c) reported use in teaching practice, and (d) opinions about benefits and barriers?

To answer the research question, an adaptative online questionnaire was applied at the beginning of the workshop and right after its end, so the results could be compared to analyze the workshop's impact on the teacher trainees' professional development.

The initial questionnaire comprised a sum of closed- and open-ended questions and was organized into four sections:


The final questionnaire was quite similar to the initial one, but without the demographic data section, as the population was already characterized. Hence, it comprised three sections:


For questionnaire content validity, both versions were analyzed by two educational researchers with different experiences. One was a teacher who was undertaking her doctoral studies. The other was an experienced researcher and methodology professor at a public university, with a Ph.D. and postdoctoral training. After introducing changes suggested by the educational researchers, they considered the questionnaire clear, understandable, and suitable for the target population.

In the present study, only data regarding the research question was analyzed (Appendix A presents Section 3 questions of the initial—Table A2—and final questionnaires—Table A2). The analysis included descriptive statistics for answers to closed-ended questions and content analysis for answers to open-ended questions. When relevant, initial and final responses were compared.

For the content analysis related to teachers' basic knowledge on the considered topics, each answer was classified as correct, partially correct, or incorrect, in accordance with the researchers' own perspectives on the topics. These were in line with the workshop activities, as the researchers were also the teacher trainers.

For the content analysis of the benefits and barriers associated with each teaching approach, the analysis categories developed in the previous study [30] were revised. For example, two categories ("It is easy or quick to find information" and "Supports higher interaction") were included in a broader category ("Supports better learning"). Moreover, in each teacher answer, the number of different types of benefits or barriers was identified.

#### *Study Participants*

The study participants were characterized according to their: (i) Demographic data; (ii) Motivations and expectations concerning the workshop; and (iii) Conditions for technology use in schools. These data are available in the previous study of Marques and Pombo [30]. To comply with the General Data Protection Regulations, the questionnaire included a closed-ended question respecting for informed consent to participate in this study. Out of 16 teachers attending the workshop, 14 agreed and signed the informed consent, so the following data are related only to those 14 teachers.

Concerning demographic data, in terms of gender, 12 were females and 2 were males. Ten teachers had a high degree, mandatory by Portuguese law, one had a post-graduation course, and three had a master's degree. All teachers were experienced: (i) two had 11 to 20 years of experience; (ii) eight had between 21 and 30 years, and (iii) four had more than 31 years of experience.

Three teachers lectured Mathematics in the 3rd cycle of basic education (CBE, corresponding to school years 7 to 9) or in secondary teaching (ST, years 10 to 12), six teachers lectured Physics and Chemistry in the 3rd CBE or SE, one teacher lectured Nature Sciences in the 3rd CBE, and six teachers lectured Mathematics or Nature Sciences in the 2nd CBE (years 5 and 6). Their classes varied from 16 to 20 students (4 teachers), 21 to 25 students (4), and 26 to 30 students (4).

With respect to motivations, in the closed-ended question regarding reasons for attending the workshop, the participant teachers selected: "Updating or acquiring knowledge" and "Possibility to have access to new resources" with 12 mentions each; and "Combination of the workshop topics", "Possibility of changing teaching practice" and "Professional valorization" with 10 mentions each. Concerning their expectations regarding the workshop, teachers mentioned they expected to learn more about the workshop approaches (9 mentions) to improve their teacher practice (7 mentions) and, consequently, to have an impact on students' motivation to learn (6), achievement (3), and behavior (1). Three teachers also mentioned they were curious about the workshop topics.

Concerning conditions for technology use, results indicated that participating teachers and their students had conditions from the technological point of view, although students had lower access to technology than teachers. All teachers reported to have data shown in the classroom, and seven had an interactive board. Students' access to these resources was lower—only two and five, respectively.

All teachers mentioned having computer access for their teaching practices, either desktop, portable, or tablet, and the same applied to the students of 12 of the 14 respondents. One teacher, who in a previous question reported tablet and laptop access only for teachers, highlighted in the open-ended question that it is unreliable technology: "The school has some laptops and some tablets, not always in the best conditions and some rooms have a desktop computer. All classrooms have a projector, but teachers need to use their own laptops." (Q5).

Twelve teachers reported having either a feature phone (2) or a smartphone (10), and one stated having both. Schools provided students' access to mobile phones in only five cases in this cohort. In addition, 12 teachers mentioned having an email account, whereas nine respondents revealed their students had access to email accounts.

Although the results revealed that teachers and students mostly had access to technology to use in the school, 11 classified the technology as reasonable, one as bad and another as good. It is worth noting that one teacher mentioned their school did not provide students access to any type of the considered technology, providing only internet access.

All respondents reported internet access for themselves and their students, although only classifying it as reasonable (13), and feeling some constraints in its use, such as slowness and insecurity, among others. Only one teacher classified their school internet connection as good.

As to the school policy on mobile devices use in classrooms, seven teachers mentioned it was allowed, five said it was forbidden, and two acknowledged not knowing. One step further, six respondents mentioned that the school provided guidance to students on the proper use of mobile devices, five reported this did not happen in their school, and three did not know.

No relevant technology-conditions changes were reported by the teachers at the end of the workshop.

#### **3. Results and Discussion**

This section presents and discusses the results obtained through the initial and final questionnaires, with the aim of analyzing the impact of the workshop on the teacher trainees' professional development. Therefore, it is organized according to the dimensions of the research question—basic knowledge, teacher experience as a learner, reported use in teaching practice, and opinions about benefits and barriers—for all the training topics (mobile learning, AR use in education, and game-based learning).

#### *3.1. Basic Knowledge 3.1. Basic Knowledge*

(mobile learning, AR use in education, and game-based learning).

*Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 6 of 22

Figure 1 presents the overall results regarding teachers' answers to the prompt to explain the concepts of mobile learning, educational use of AR, and game-based learning. Detailed analysis is presented, with the support of tables dedicated to each key concept. Figure 1 presents the overall results regarding teachers' answers to the prompt to explain the concepts of mobile learning, educational use of AR, and game-based learning. Detailed analysis is presented, with the support of tables dedicated to each key concept.

This section presents and discusses the results obtained through the initial and final questionnaires, with the aim of analyzing the impact of the workshop on the teacher trainees' professional development. Therefore, it is organized according to the dimensions of the research question—basic knowledge, teacher experience as a learner, reported use in

**Figure 1.** Frequency of teachers' types of answers to the questions on basic knowledge about mobile learning, educational use of AR, and game-based learning. **Figure 1.** Frequency of teachers' types of answers to the questions on basic knowledge about mobile learning, educational use of AR, and game-based learning.

Table 1 presents the types of answers given by teachers, their frequency, and examples of citations that illustrate each type of answer given by teachers when asked about the meaning of mobile learning, at the beginning and end of the workshop. It reveals that the number of incorrect answers decreased (from 3 to 1), the number of partially correct answers increased (from 2 to 4), and the correct answers remained the same (8); hence, small positive differences between the initial and the final phases of the workshop were Table 1 presents the types of answers given by teachers, their frequency, and examples of citations that illustrate each type of answer given by teachers when asked about the meaning of mobile learning, at the beginning and end of the workshop. It reveals that the number of incorrect answers decreased (from 3 to 1), the number of partially correct answers increased (from 2 to 4), and the correct answers remained the same (8); hence, small positive differences between the initial and the final phases of the workshop were registered.

registered. **Table 1.** Comparison of teachers' answers on their understanding of the mobile-learning concept, at the beginning and end of the workshop.


1 f: Frequency or number of teachers' answers included in each type of answer; <sup>2</sup> Q: questionnaire ID.

It is noteworthy to highlight that, even at the end of the workshop, all answers considered correct were simple and technocentric definitions of mobile learning, not considering, e.g., the mobility of learner [1–3], despite these issues being discussed in face-to-face sessions.

At the end of the workshop, when questioned about examples of mobile learning projects or initiatives, 10 teachers mentioned at least one educational mobile app example, which was an increase compared to the eight teachers in the first questionnaire [30]. Most teachers (8) mentioned the EduPARK app, which was used during the training. The Khan Academy was mentioned twice, the same as at the beginning of the workshop, although this platform is not necessarily used in mobile contexts.

The results presented above indicate a small impact of the workshop on teachers' ability to explain the concept of mobile learning. The strategies explored in the workshop included discussion in face-to-face sessions, analyzing texts about mobile learning to produce a reflexive text, and using mobile devices to learn in a training context. However, their impact on teachers' learning fell behind what was initially expected. Moreover, practical tasks, such as naming examples of mobile learning, seemed to be easier for teachers when compared to theoretical tasks, such as defining the concept. This was reinforced by a teacher's answer to the questionnaire prompt "Indicate at least one [workshop] activity that was too difficult and explain why": "The initial reflection was too theoretical. In my opinion, teachers mainly need to have a practical component in training. I think that searching a certain topic adds little to the improvement of teaching practice" (Q6). This teacher revealed that they did not value the literature search's potential contribution to teaching practice.

Table 2 reveals the types of answers given by teachers when asked about the meaning of educative use of AR, at the beginning and end of the workshop. It is possible to identify a decrease in the number of teachers acknowledging not knowing or not answering, from 4 to 1, and of teachers answering incorrectly, from 7 to 4. The same table also shows an increase in the number of teachers answering partially correct (usually only defining AR, and not mentioning its potential for education), from 3 to 7, and in the correct answers, from 0 to 2. These results revealed the trainees' difficulty in explaining the educative use of AR, although progress was made by teachers concerning this topic.


**Table 2.** Comparison of teachers' answers on their understanding of the use of AR for educational purposes, at the beginning and end of the workshop.

1 f: Frequency or number of teachers' answers included in each type of answer; <sup>2</sup> Q: questionnaire ID.

> Again, at the end of the training, when questioned about examples of AR use in education, 9 teachers presented at least one educational AR example; all mentioned the one explored in the workshop. Compared with the results from the beginning of the workshop, only 2 teachers mentioned adequate examples, and 12 did not present any example at

all [30]—there was an evident impact at this level. From these results, it is reasonable to claim that the workshop had an impact on teachers' knowledge of educative use of AR, particularly in what concerns the ability to provide concrete examples.

Finally, Table 3 presents the types of answers given by teachers when asked about their understanding of game-based learning, at the beginning and end of the workshop. The table reveals small differences between the initial and the final phases of the workshop, regarding "Did not know/answer", from 2 to 1, and "Answered correct, but in a simple way", from 9 to 11.

**Table 3.** Comparison of teachers' answers on their understanding of game-based learning, at the beginning and end of the workshop.


1 f: Frequency or number of teachers' answers included in each type of answer; <sup>2</sup> Q: questionnaire ID.

> At the end of the training, when questioned about examples of game-based learning, 12 teachers presented at least one illustrative example, and 7 mentioned the one explored in the workshop. At the beginning of the workshop, only 5 teachers were able to present valid examples of game-based learning [30], so the cohort of teachers revealed progress at this level.

> In summary, the presented results indicated that the workshop had a higher impact on the topic about which teachers knew less: the educative use of augmented reality. A smaller impact was registered concerning mobile learning and game-based learning. The impact was more evident, in all topics, when teachers were asked to mention examples.

#### *3.2. Teacher Experience as A Learner*

Figure 2 summarizes teachers' experience, as learners, with mobile learning, AR educational use, and game-based learning. At the beginning of the workshop, teachers reported the following previous experience: (a) All (14) had experienced mobile learning, even though only 6 presented valid examples; (b) none had learned with AR; and (c) 8 had experienced game-based learning [30].

When asked if they had used mobile devices, AR, and/or games to learn during the workshop, 13 teachers mentioned they experienced the three approaches; 1 teacher reported having used only mobile devices and games to learn, but not AR. When comparing these results with teachers' previous experience, it is possible to claim that all became more experienced with these approaches, particularly concerning AR. For 13 respondents, the workshop allowed them to experience this technology to learn for the first time. Only 1 teacher mentioned not using AR to learn during the workshop. This result may be interpreted in two different ways. Either this individual did not use AR technology, despite being given the opportunity to do so, or this teacher used it, but considered that this experience did not provide learning. As the questionnaires were anonymous, it was not possible to explore further this issue.

**Figure 2.** Comparison of teachers' experience as learners with mobile devices, AR, and game(s), before and during the workshop. **Figure 2.** Comparison of teachers' experience as learners with mobile devices, AR, and game(s), before and during the workshop.

valid examples of game-based learning [30], so the cohort of teachers revealed progress at

In summary, the presented results indicated that the workshop had a higher impact on the topic about which teachers knew less: the educative use of augmented reality. A smaller impact was registered concerning mobile learning and game-based learning. The impact was more evident, in all topics, when teachers were asked to mention examples.

Figure 2 summarizes teachers' experience, as learners, with mobile learning, AR educational use, and game-based learning. At the beginning of the workshop, teachers reported the following previous experience: (a) All (14) had experienced mobile learning, even though only 6 presented valid examples; (b) none had learned with AR; and (c) 8

#### When asked if they had used mobile devices, AR, and/or games to learn during the *3.3. Reported Use in Teaching Practice*

*3.2. Teacher Experience as A Learner* 

had experienced game-based learning [30].

this level.

workshop, 13 teachers mentioned they experienced the three approaches; 1 teacher reported having used only mobile devices and games to learn, but not AR. When comparing these results with teachers' previous experience, it is possible to claim that all became more experienced with these approaches, particularly concerning AR. For 13 respondents, the workshop allowed them to experience this technology to learn for the first time. Only 1 teacher mentioned not using AR to learn during the workshop. This result may be interpreted in two different ways. Either this individual did not use AR technology, despite being given the opportunity to do so, or this teacher used it, but considered that this ex-Figure 3 summarizes teachers' exploration of the workshop approaches in their practices. At the beginning of the workshop, 11 teachers reported previous experience in exploring mobile devices with their students to promote learning, although only 6 presented valid examples in response to the following question. These results seem to indicate that teachers are beginning to integrate mobile learning in their practices [30], as advocated by several Horizon Reports [33–35]. The fact that teachers began to promote mobile learning was in line with a previous study [24] and with the Portuguese State of Education Report [36]. *Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 10 of 22

perience did not provide learning. As the questionnaires were anonymous, it was not pos-

**Figure 3.** Comparison of teachers' use of mobile devices, AR, and game(s) in their practices, before and during the workshop. **Figure 3.** Comparison of teachers' use of mobile devices, AR, and game(s) in their practices, before and during the workshop.

Teachers who reported never having used mobile devices in their teaching practices before the workshop mentioned they could change that due to two main factors: (a) using mobile devices may increase students' motivation to learn; and (b) mobile devices' availability to most students [30]. Teachers who reported never having used mobile devices in their teaching practices before the workshop mentioned they could change that due to two main factors: (a) using mobile devices may increase students' motivation to learn; and (b) mobile devices' availability to most students [30].

No teacher had previous experience in using AR to promote their students' learning [30]. The Horizon initiative [37] has placed AR technology in the time-to-adopt group for K-12 educational contexts, highlighting its potential to provide powerful, contextual, and in situ visual and interactive learning experiences. However, our empirical results indicated that about 8 years later, teachers are still not exploring this technology with high educational value. Factors that could change this situation, according to teachers, are: (a) teacher knowledge on how to use AR to promote learning, developed in professional development initiatives; (b) knowing that the use of AR technology may increase students' No teacher had previous experience in using AR to promote their students' learning [30]. The Horizon initiative [37] has placed AR technology in the time-to-adopt group for K-12 educational contexts, highlighting its potential to provide powerful, contextual, and in situ visual and interactive learning experiences. However, our empirical results indicated that about 8 years later, teachers are still not exploring this technology with high educational value. Factors that could change this situation, according to teachers, are: (a) teacher knowledge on how to use AR to promote learning, developed in professional development initiatives; (b) knowing that the use of AR technology may increase students'

motivation to learn; (c) AR may facilitate teaching and learning processes; and (d) teacher

Seven teachers reported previous experience in promoting game-based learning [30]. From these, 6 presented valid examples, usually mentioning a quiz format. The game approach is one with high expression in international reports. In the Horizon series, it is presented as an effective and versatile approach with gains in student engagement, creativity, and authentic learning [38,39]. However, this seems to still be an approach with limited expression in teacher practices. This result contrasts with Russo, Bragg, and Russo's study [18], in which the majority of teachers mentioned using educational games at least once a week, revealing this approach can be popular among Australian practition-

According to this study's teacher cohort, factors that could promote higher exploration of games in formal education are: (a) using games, which may increase students' motivation to learn; (b) teacher knowledge of games that can be explored to promote learning; and (c) access to resources, either the games or their supporting technologies [30].

Regarding the use of mobile devices, AR, and games in teaching practices, during about three months, which corresponded to the workshop period, 12 teachers mentioned they promoted mobile and/or game-based learning, and 9 teachers mentioned using AR. As to the reasons for not exploring these approaches, teachers mentioned: (a) lack of opportunities or time ("It was not timely" (Q4)); (b) lack of resources ("Did not have conditions in the classroom" (Q16)); (c) lack of teacher readiness ("Because I still don't feel comfortable using it" (Q9)); (d) lack of students' skills ("Because students don't know how to use it for this purpose" (Q16)); and (e) COVID-related barriers ("Considering the pan-

Moreover, teachers had the opportunity to present additional (optional) comments on the workshop topics, and their answers revealed teachers generally sustained positive

demic context that we live in" (Q15)).

ers.

motivation to learn; (c) AR may facilitate teaching and learning processes; and (d) teacher willingness to change practice [30].

Seven teachers reported previous experience in promoting game-based learning [30]. From these, 6 presented valid examples, usually mentioning a quiz format. The game approach is one with high expression in international reports. In the Horizon series, it is presented as an effective and versatile approach with gains in student engagement, creativity, and authentic learning [38,39]. However, this seems to still be an approach with limited expression in teacher practices. This result contrasts with Russo, Bragg, and Russo's study [18], in which the majority of teachers mentioned using educational games at least once a week, revealing this approach can be popular among Australian practitioners.

According to this study's teacher cohort, factors that could promote higher exploration of games in formal education are: (a) using games, which may increase students' motivation to learn; (b) teacher knowledge of games that can be explored to promote learning; and (c) access to resources, either the games or their supporting technologies [30].

Regarding the use of mobile devices, AR, and games in teaching practices, during about three months, which corresponded to the workshop period, 12 teachers mentioned they promoted mobile and/or game-based learning, and 9 teachers mentioned using AR. As to the reasons for not exploring these approaches, teachers mentioned: (a) lack of opportunities or time ("It was not timely" (Q4)); (b) lack of resources ("Did not have conditions in the classroom" (Q16)); (c) lack of teacher readiness ("Because I still don't feel comfortable using it" (Q9)); (d) lack of students' skills ("Because students don't know how to use it for this purpose" (Q16)); and (e) COVID-related barriers ("Considering the pandemic context that we live in" (Q15)).

Moreover, teachers had the opportunity to present additional (optional) comments on the workshop topics, and their answers revealed teachers generally sustained positive perceptions, as revealed by the following citations: "It [mobile leaning] is an asset for the knowledge acquisition in a more fun and motivating way" (Q2); "AR is a tool that should be explored in lessons' (Q5); and 'It [the game] is an excellent tool for motivation" (Q1). On the other hand, some teachers seemed to envision difficulties: "I would like to be able to use them [mobile devices] in the classroom" (Q16); and "It [AR] is still not very disseminated in the teaching community" (Q14).

Facing these results, it is reasonable to consider that the workshop had an impact on teachers' practices, according to their self-reports. However, it is worthwhile to note that even after long-term training, a small group of teachers (2 to 5, varying with the considered approach) remained reluctant to try out the approaches for which they were training. These reluctancy and lack of readiness to explore mobile, AR, and game-based learning were not evident during the face-to-face training sessions, so it seems reasonable to assume that teachers tended to contribute less to the large-group discussion when they did not agree or did not feel comfortable with the approaches under analysis. In the future, teacher trainers must take this result into account in order to have a deeper impact on the practices of teachers who are reluctant regarding the exploration of mobile learning, AR, and game-based learning.

#### *3.4. Opinions about Benefits and Barriers*

Figures 4 and 5 and Tables 4–6 summarize teachers' answers to the open-ended question "What potential/advantages do you identify in using [mobile devices/AR/game(s)] to promote learning?".

perceptions, as revealed by the following citations: "It [mobile leaning] is an asset for the knowledge acquisition in a more fun and motivating way" (Q2); "AR is a tool that should be explored in lessons' (Q5); and 'It [the game] is an excellent tool for motivation" (Q1). On the other hand, some teachers seemed to envision difficulties: "I would like to be able to use them [mobile devices] in the classroom" (Q16); and "It [AR] is still not very dissem-

Facing these results, it is reasonable to consider that the workshop had an impact on teachers' practices, according to their self-reports. However, it is worthwhile to note that even after long-term training, a small group of teachers (2 to 5, varying with the considered approach) remained reluctant to try out the approaches for which they were training. These reluctancy and lack of readiness to explore mobile, AR, and game-based learning were not evident during the face-to-face training sessions, so it seems reasonable to assume that teachers tended to contribute less to the large-group discussion when they did not agree or did not feel comfortable with the approaches under analysis. In the future, teacher trainers must take this result into account in order to have a deeper impact on the practices of teachers who are reluctant regarding the exploration of mobile learning, AR,

Figures 4 and 5 and Tables 4–6 summarize teachers' answers to the open-ended question "What potential/advantages do you identify in using [mobile devices/AR/game(s)] to

Figure 4 reveals that teachers were able to mention more types of benefits and barriers at the end of the workshop, when comparing to the beginning. For example, teachers identified 24 types of benefits of mobile learning at the beginning, and 31 benefits at the end. Similarly, teachers identified 26 types of barriers for mobile learning at the beginning and 37 barriers at the end. Thus, the workshop seems to have had an impact on teachers' ability to acknowledge both benefits and barriers to the educative use of mobile devices,

inated in the teaching community" (Q14).

and game-based learning.

promote learning?".

AR, and games.

*3.4. Opinions about Benefits and Barriers* 

**Figure 5.** Comparison of the frequency of each type of benefit identified by teachers on mobile learning, educative use of AR, and game-based learning, at the beginning and end of the workshop. **Figure 5.** Comparison of the frequency of each type of benefit identified by teachers on mobile learning, educative use of AR, and game-based learning, at the beginning and end of the workshop.

In a more detailed analysis, Table 5 shows that initially, the most pointed types of benefits were: (a) "Motivates students to learn" (total of 26 teacher mentions); (b) "Supports better learning" (total of 12); and (c) "Allows learning in a funny way" (total of 8), with this last one having more expression regarding the game-based learning approach. At the end of the workshop, motivation (total of 25 mentions) was surpassed by better


**Table 4.** Numbers of types of benefits and barriers on the educative use of mobile devices, AR, and game(s), according to teachers at the beginning and end of the workshop.

<sup>1</sup> The first column indicates the number of types of benefits (numerals presented in bold) and the number of types of barriers (numerals also in bold) given in each questionnaire. f: Frequency of teachers' whose answers included each number of types of benefits or barriers. T: Total. A: Average.

**Table 5.** Benefits of exploration of mobile devices, AR, and game(s) in teaching practices, according to teachers at the beginning and end of the workshop.


1 f: Frequency of teachers' answers included in each type of answer. T: Total (values for all the benefits at the beginning and end of the workshop are resented in the grey cells).


**Table 6.** Barriers to exploration of mobile devices, AR, and game(s) in teaching practices, according to teachers at the beginning and end of the workshop.

1 f: Frequency of teachers' answers included in each type of answer. T: Total (values for all the benefits at the beginning and end of the workshop are resented in the grey cells).

> Figure 4 reveals that teachers were able to mention more types of benefits and barriers at the end of the workshop, when comparing to the beginning. For example, teachers identified 24 types of benefits of mobile learning at the beginning, and 31 benefits at the end. Similarly, teachers identified 26 types of barriers for mobile learning at the beginning and 37 barriers at the end. Thus, the workshop seems to have had an impact on teachers' ability to acknowledge both benefits and barriers to the educative use of mobile devices, AR, and games.

> Taking into account that the more benefits teachers identified, the more positive their perspectives could be considered, and that the more barriers teachers identified, the more negative their perspectives could be considered, Figure 4 seems to indicate that the workshop had a more intense impact regarding the barriers, which may indicate a moderate negative view. This was a surprising result after training on these educational approaches, but it is possible to hypothesize that knowing the approaches better made teachers more conscious of potential barriers to their implementation.

> Presenting results in more detail, Table 4 shows that at the beginning of the workshop, teachers pointed toward 0 to 3 different types of benefits and barriers for each workshop approach. They reported a total of 60 benefits (24 for mobile learning, 16 for AR, and 20 for games) and 61 barriers (26, 16, and 19, respectively), which seemed to reveal teachers' initial neutral perspective on the educative use of mobile devices, AR, and games [30].

> After the workshop, teachers pointed toward 0 to 4 types of benefits and 7 types of barriers for each workshop approach. They identified a total of 76 (31 for mobile learning, 24 for AR, and 21 for games) types of benefits and 95 (37, 27, and 31 respectively) types of barriers.

It is notable how, at both data collection moments, teachers seemed to associate the benefits and the barriers of mobile learning with the ones of AR, and also with those of game-based learning. For example, even at the end of the workshop, teachers mentioned similar benefits to all the approaches, as illustrated by the following citations from Q7: "[Mobile devices] promote engagement and motivation in students."; "[AR] fosters motivation, commitment, and enthusiasm for learning."; and "Extrinsic motivation is enhanced by the strategies of the game." This result may be related to the fact that all the approaches were focused on the same training course and exemplified with the same educational resource, a mobile app that supports both gaming and AR. However, even in the initial questionnaire, teachers associated these three approaches in similar ways. This was illustrated by the benefits of mobile learning, AR, and game-based learning reported in Q4: "Learning in a playful way" (the same sentence included in all the answers). Therefore, it seems that the association between the approaches occurred before the training.

Figure 5 summarizes the frequency for each type of benefit teachers mentioned in their answers at the beginning and at the e2nd of the workshop. For the two most frequent benefits, "Supports better learning" and "Motivates students to learn", there was an increase of frequency after the workshop.

In a more detailed analysis, Table 5 shows that initially, the most pointed types of benefits were: (a) "Motivates students to learn" (total of 26 teacher mentions); (b) "Supports better learning" (total of 12); and (c) "Allows learning in a funny way" (total of 8), with this last one having more expression regarding the game-based learning approach. At the end of the workshop, motivation (total of 25 mentions) was surpassed by better learning (total of 32), these being the most relevant benefits for the majority of teachers. These results were in line with a previous study [10], related to a short-term teacher training on the same topics, but focused on mobile learning, where student motivation, ease in finding information (which in this study was included in the better learning category), and technology availability stood out. The results indicated that teacher training supported more teachers in identifying learning promoted by these approaches, although with less intensity for games (an increase from 3 to 8 mentions in the present study) and higher intensity for AR (an increase from 4 to 13 mentions). Therefore, the workshop seems to have had a greater impact on teachers' ability to acknowledge how a technology, unknown to most of the cohort before the training (the AR), can support learning. Moreover, teachers' answers were longer and more elaborated at the end of the workshop, which may be interpreted as an impact of the workshop on teachers' understanding of the learning that can be potentiated by the approaches. This is illustrated by the following citations: "[AR] improves knowledge" (Q9 at the beginning); and "[AR] shows in three dimensions (3D) some topics covered in the classroom, so that the student can better understand the «reality». Greater interactivity" (Q9 at the end of the workshop).

Fun learning is a theme that was present in all workshop topics; however, teachers' focus on this feature decreased (from a total of 8 to a total of 4 mentions). This result seems to point out that training in these approaches contributes to the transformation of teachers' perceptions of mobile devices, AR, and games being used just for fun to perceptions that these approaches can effectively support deeper learning. The articulation of these elements may sustain mentality changes regarding learning, which is a claim that has been made previously [10].

Other types of benefits pointed out by teachers included aspects related to the availability of technological hardware (usually, associated with mobile devices) and software, its ease of use, and the fact that by using these approaches, teachers and students diversify teaching and learning experiences, particularly in formal education contexts. Some benefits frequently selected by teachers in the previous study [10] were not mentioned by this study's cohort, specifically "The information is up-to-date", "Does not waste paper", or "Facilitates teachers work, namely in assessment".

*Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 15 of 22

"Facilitates teachers work, namely in assessment".

made previously [10].

Figure 6 summarizes the frequency of each type of barrier teachers mentioned in their answers at the beginning and end of the workshop. Overall, most barriers registered higher frequencies at the end of the workshop. Figure 6 summarizes the frequency of each type of barrier teachers mentioned in their answers at the beginning and end of the workshop. Overall, most barriers registered higher frequencies at the end of the workshop.

these approaches can effectively support deeper learning. The articulation of these elements may sustain mentality changes regarding learning, which is a claim that has been

Other types of benefits pointed out by teachers included aspects related to the availability of technological hardware (usually, associated with mobile devices) and software, its ease of use, and the fact that by using these approaches, teachers and students diversify teaching and learning experiences, particularly in formal education contexts. Some benefits frequently selected by teachers in the previous study [10] were not mentioned by this study's cohort, specifically "The information is up-to-date", "Does not waste paper", or

**Figure 6.** Comparison of the frequency of each type of barrier, identified by teachers on mobile learning, educative use of AR, and game-based learning, at the beginning and end of the work-**Figure 6.** Comparison of the frequency of each type of barrier, identified by teachers on mobile learning, educative use of AR, and game-based learning, at the beginning and end of the workshop.

shop.

In a more detailed analysis, Table 6 shows that the most frequent types of barriers were: (a) "The hardware is not available or is too diverse" (from 18 teacher mentions to 15); (b) "Lack of (quality) internet connection" (from 10 mentions to 12); and (c) "Risk of student distraction" (from 10 mentions to 15). All these barriers were found in a previous In a more detailed analysis, Table 6 shows that the most frequent types of barriers were: (a) "The hardware is not available or is too diverse" (from 18 teacher mentions to 15); (b) "Lack of (quality) internet connection" (from 10 mentions to 12); and (c) "Risk of student distraction" (from 10 mentions to 15). All these barriers were found in a previous study [10], although with a much smaller expression: (a) 1 teacher (in a total of 26) mentioned the lack of access to mobile devices for some students; (b) 5 teachers selected the need for an internet connection or its lack of quality; and (c) 9 teachers selected student access to distractions. In fact, in the previous study, the most expressed barriers were the risk of developing mobile-device dependence, increased battery consumption, and school prohibition of mobile device use in classes. All these barriers emerged in the present study as well, but with different intensities. Nevertheless, these issues need to be considered by teachers and teacher trainers in order to effectively promote mobile learning, AR, and game-based approaches in teacher practices.

Other types of barriers pointed out by teachers at both data-collection moments included aspects related to lack of time (from 6 to 9 mentions), lack of teacher or student digital competence (from 5 to 7), risk of poorer learning (from 4 to 6), and lack of suitable digital resources (from 1 to 6). None of these barriers were identified in the previous

study [10]. However, other themes in common with the previous study, which emerged with small expression in this study, were: (a) School prohibition of use in classes (only for mobile devices); (b) Lack of teacher didactic competence in these approaches; and (c) Risk of student addiction (only for mobile devices in the first study and only for games in the present study). So, once again the same barrier themes were identified in both studies, although with different intensities.

Finally, it is noteworthy to highlight that teachers pointed out some barriers that were not exclusive to these approaches. For example, regarding lack of time, a teacher mentioned at the end of the workshop: "The time spent on classroom with gaming activities may lead to not being able to teach all the [curricular] content" (Q4). The same may be said regarding most teaching approaches that teachers do not know yet, which require a higher investment in terms of planning and implementation during lessons.

#### **4. Conclusions**

This work addresses the need to analyze the impact of continuous teacher training initiatives concerning new technology supporting teaching approaches on professional development. It WAS conducted under a case study [31] on the development process of mobile AR games for STEM learning by 14 in-service teachers during a 50 h workshop in Portugal. Hence, the present paper presented the analysis of the impact of this workshop on teacher trainees' professional development through a questionnaire filled in at the beginning and end of the workshop.

Regarding teachers' understanding of mobile learning, AR, and game-based learning, this study registered a higher impact on AR educative use, which was the less-known approach for teachers, compared to mobile and game-based learning. Teachers revealed difficulties in explaining concepts' definitions even at the end of the workshop; however, they demonstrated increased understanding and increased ability to provide concrete examples of each teaching approach.

In what concerns teachers' experience in educational contexts, teachers became more experienced with mobile learning, AR, and game-based learning as learners themselves, and reported having explored them with their students during the three-month workshop period. Hence, this study's results support the claim that the analyzed workshop promoted teacher practice changes, although only through self-reports. Future investigations may include teacher practice observations to ascertain the accuracy of teachers' claims.

Finally, teachers' ability to identify benefits and barriers through the workshop teaching approaches increased with the training, although with more intensity with respect to barriers. Nevertheless, the most mentioned benefits pointed out by teachers were related to the improved student learning and motivation that may be promoted by mobile AR games. On the other hand, among the barriers to the implementation of these approaches in teaching practices that stood out was the unavailability of proper hardware to support them, or even the hardware diversity that emerges from the bring-your-own-device option, the risk of student distraction, and lack of a quality internet connection. These barriers gain higher relevance if the 2019 State of Education Report [36] is considered, as it mentions the wear and tear of the Portuguese schools' computer park, and the internet connection fragility in the majority of schools. Therefore, the presented set of barriers to implementation is relevant both for in-service teachers and for teacher trainees preparing future professional development initiatives.

With only 14 participant teachers, this study did not aim to provide results generalizable to the entire Portuguese teacher population. However, this teacher cohort very closely reflected the Portuguese teacher profile [23,24] in terms of gender and experience. Hence, this study could be a good indicator of the teacher population status on these matters.

In sum, this study presented empirical evidence that long-term teacher training concerning the educative exploration of new technologies, which includes the creation of educational resources, may contribute to the transformation of teachers' perceptions. These seemed to evolve from a perspective that mobile devices, AR, and game-based approaches are considered just for the fun they provide, to perceptions that these approaches can effectively support deeper learning, and hence changing mentalities on how people can learn [40]. Consequently, a recommendation to educational researchers and teacher trainers emerged: to build upon these workshop methodologies in order to have an impact on teacher professional development in what concerns the integration of innovative teaching technologies in their practice.

**Author Contributions:** Conceptualization, M.M.M. and L.P.; data curation and formal analysis, M.M.M.; investigation, M.M.M. and L.P.; methodology, M.M.M. and L.P.; project administration, M.M.M. and L.P.; writing—original draft, M.M.M. and L.P.; writing—review and editing, M.M.M. and L.P. Both authors have read and agreed to the published version of the manuscript.

**Funding:** The work of the first author was funded by national funds (OE), through the University of Aveiro, in the scope of the framework contract foreseen in the numbers 4, 5, and 6 of Article 23 of the Decree-Law 57/29 August 2016, changed by Law 57/19 July 2017.

**Institutional Review Board Statement:** Ethical review and approval were waived for this study, due to being a study involving a small number of healthy adults, participating under informed consent, and with no sensitive data collection.

**Informed Consent Statement:** Written informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy issues.

**Acknowledgments:** The authors thank the participating teachers for accepting taking part of this study. The authors also thank the reviewers for their insightful comments.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

#### **Appendix A**


**Table A1.** Questions of the Section 3 of the Initial Questionnaire.


#### **Table A1.** *Cont*.

### **Question ID Type Questions in English (in Portuguese in the Original Questionnaire)** G6Q08 open-ended What potential/advantages do you identify in using game(s) to promote learning? (Please clearly present at least three strengths/advantages that may affect teachers.) G6Q09 open-ended What barriers/constraints do you recognize in using game(s) to promote learning? (Please clearly state at least three barriers/constraints that can affect teachers.) G6Q10 open-ended Do you have any comments regarding game-based learning?

#### **Table A1.** *Cont*.

#### **Table A2.** Questions of the Section 3 of the Final Questionnaire.



#### **Table A2.** *Cont*.

#### **References**


## *Article* **Development of Game-Based M-Learning Apps for Preschoolers**

**Dionísia Laranjeiro**

**Abstract:** Recent studies indicate tablets as the preferred devices of preschool children, due to portability, autonomy of use and variety of apps. There is also extensive evidence of the contributions of digital technologies in different areas of learning at these ages. The Aprender XXI project aimed to develop game-based learning apps, with content recommended in the Curriculum Guidelines for Pre-School Education (CGPE). The project used Design-Based Research (DBR) methodology, which combines scientific research and technological development. It was divided into three phases: preliminary study (literature review, search for existing apps, study of preschool curriculum), development (specifications, scriptwriting, design and programing) and evaluation (tests with users and conclusions). The preliminary study identified the needs to define robust apps. The evaluation with children and educator validated the development and defined improvements in the apps. As a result, we obtained four thematic apps—environment, health, citizenship and professions, composed of a set of games, suitable for autonomous use for children or for educational activities guided by educators in kindergarten. In addition, a website collects children's play data, which is represented with flowers in a virtual world, to illustrate their participation/collaboration for a better future.

#### **1. Introduction**

Today's children belong to a generation that is familiar with technologies, such as computers, tablets and the internet. Prensky [1] called this generation "digital natives", affirming that digital language is part of their lives and it can even change their thinking patterns. It is important to understand some characteristics and skills of these children to provide interesting learning, such as recognizing that they are used to receiving large amounts of information and instant gratification, they prefer access to non-linear information, they like to network and perform various tasks at the same time. This argument is criticized by Thomas [2], claiming that the concept of digital natives presupposes a homogeneous generation, which learns differently from previous generations. Hattie and Yates [3] reinforce that human capabilities are not that malleable. There is no scientific evidence that living in an age dominated by new technologies causes changes in the brain organization or in cognitive abilities. The European Commission assumed a balanced perspective, declaring that children up to eight years old can easily acquire digital skills but their abilities are limited to the level of cognitive development. Additionally, some children show more interest and skills than others. Their lives are not dominated by technologies, as they carry out many other activities. Sometimes children use technologies to complement their offline interests, for example, to search online and photograph [4]. The use of ICT in kindergarten is not consensual and there is still some resistance to integration in this educational context, caused by several reasons: lack of conditions and outdated devices in kindergartens, lack of adequate training for kindergarten teachers, beliefs of some parents and educators that the use of technologies at these ages is not beneficial [5]. These concerns are supported by researchers and child advocacy groups who argue that technologies can

**Citation:** Laranjeiro, D. Development of Game-Based M-Learning Apps for Preschoolers. *Educ. Sci.* **2021**, *11*, 229. https://doi.org/10.3390/ educsci11050229

Academic Editor: James Albright

Received: 16 April 2021 Accepted: 8 May 2021 Published: 12 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

impede physical, emotional, cognitive and social development in early childhood. Some risks include obesity, eyestrain, sleep disorders and social isolation [6]. Larry Cuban [7] claims that research on the use of technology in the learning of preschoolers is limited and ambiguous, and it cannot be confirmed whether preschoolers should use computers, for how long, for what tasks and under what conditions. Richard Mayer [8] agrees that there are strong claims about the potential of technology in transforming education that has not been scientifically tested. However, these high expectations have led to massive implementations of technological interventions in schools that have had failed results. He believes that the problem lies in the approach to learning with technology, which is focused on technology while it should be focused on the learner. In a technology-centered approach the goal is to give access to the latest technology, hoping that teachers and students will adapt and use it. The learner-centered approach focuses on the learner, the way learning happens and how the mind works. In this approach, it is technology that is adapted to the needs of students and teachers. The joint position statement of NAEYC and the Fred Rodgers Center argues that if the integration of technology in kindergarten is built upon solid developmental foundations, and educators are aware of the challenges and opportunities, the quality of the educational program is improved by using technologies for the benefit of all children. Some recommendations are that educators have digital literacy to make informed choices that can maximize learning opportunities; they should monitor the time and use of devices by children; technology must be used to support learning and expand access to new content. It should not replace creative games, outdoor experiences, and interactions with children and adults [9]. The American Academy of Pediatrics also set recommendations to help families with young children promote healthy media use. At home, children aged two to five should use the media together with their parents to help them understand the content. Use should be limited to one hour a day, with a pre-selection of quality content. From the age of six, parents must define time limits, locations and types of media that can be used, maintaining balance with physical activity, reading, social interaction and other activities essential to health. Children should not use screens one hour before going to bed, to avoid sleep disturbances [10]. Despite all the concerns and risks identified, extensive literature documents the contributions of digital technologies in the learning and skills development of pre-school children [11–13]. In language development, digital technologies can impact the fluency and complexity of speech, the development of verbal communication and written language, vocabulary, syntax and word recognition [14]. As examples, the use of word processing software can enhance the understanding of written code, and the creation of podcasts can enrich orality, expression and communication [15]. In math, there are specific applications that stimulate mathematical concepts, such as counting, classification or logical thinking. Drawing programs develop geometric and spatial knowledge. Programing encourages creative thinking and problem solving [14]. Internet allows contact with other natural, social and cultural realities. A good example is the use of Google Earth® for an interactive and precise exploration of the world, to understand space and local geography in relation to global geography [15]. Using various technologies, pre-school children acquire digital literacy, which includes the ability to search online, use the mouse and touchscreen and create multimodal texts [12]. For the development of artistic skills, there are interactive tools and environments, which can be used for exploring music, drawing, painting, animation, creative writing, poetry and storytelling. These are usually open-ended applications, quite versatile and adaptable to different contexts, and it is up to the educator to plan their use in learning activities. With regard to develop preschoolers' cognitive skills, carrying out learning activities based on apps and digital games improves abstract thinking, reflection, analysis and evaluation of information, problem solving, spatial representation, attention and memorization [13]. Social tools foster learning and teamwork, by providing communication and collaboration features, which can be used in group projects, in the classroom or remotely. This type of activities promotes interaction, team spirit and develops language and critical thinking. Exposure to other perspectives and divergence of opinions stimulates dialogue and joint

analysis of possibilities, which results in higher quality decisions. All this involvement increases the interest in the content and leads to better learning performances [16]. Despite the rapid evolution of technologies, there are some propositions in the literature with more than twenty years similar to the current ideas, regarding children of preschool age: they write on the keyboard and use computers and software with confidence; they face difficulties as challenges and almost always manage to overcome them; computers help children to learn, so the educator has to know how to help them, assuming a less instructional and more guiding role; computers encourage social interaction and communication, as children prefer to work with friends and talk while they work; cooperative work on the computer generates enthusiasm and interest in the activity; children with special educational needs can benefit from digital technologies, in physical, emotional and social terms [17,18].

Recent studies point out that access to mobile devices is universal for children up to eight years old and indicate tablets as their favorite devices. Using comparative data from 2011 and 2017, a report on the use of screen media by North American children ages zero to eight points out that the major change has occurred in the type of devices rather than in screen time, which remained approximately 2 h a day. The use of mobile devices has increased substantially (from 5 to 48 min a day) while the use of television, DVD players, computers, and video game consoles has decreased. The study indicates that 95% of children aged zero to eight have a smartphone at home and 42% have their own tablet. 67% of parents believe that the use of digital media helps their children in learning [19]. In Portugal, a study on the use of technology in early childhood also indicates the tablet as their preferred device. In terms of consumption, 50% of children play digital games and 50% of children aged three to five who access the internet, use their own tablet [20]. The mobile market has brought a wide range of educational apps that integrate different ways of understanding, knowing and expressing, valuing the multiple forms of intelligence linguistic, mathematical, creative and artistic. A learner with access to a selected set of educational apps can perform activities in languages, arts, sciences, among others [21]. Children in pre-school education can benefit from the characteristics of tablets, due to the appropriate size, portability and long battery life, which gives them autonomy in use and allows them to explore apps in various locations. The possibility of multitouch interaction also favors joint use with friends, educators and parents. There is a wide variety of apps with multimodal content, available for different educational purposes and appropriate for these ages [22]. Educational apps can be classified into three levels: instructional apps, based on exercise-reward, aimed at the acquisition of specific content and skills; manipulable apps that manage ideas and content, allowing multiple responses to a set of variables; constructive apps, with an open structure that serves to create or communicate, allowing learners to build a learning object from a set of available components. In terms of contribution to cognitive development, there is a growth in the learner's involvement and motivation from instructional to constructive apps [23]. Most educational apps are not directly related to specific curricular themes, but have versatility and flexibility to be used in different contexts.

In the development of digital pedagogical applications, it is necessary to take into account three components that influence learning: the child (to whom it is addressed), the adult (who guides learning) and technology. Older children, with previous experience, longer exposure and access time at home, can benefit more from technology in learning [12]. Adults have a mediating role, whether they are parents or kindergarten educators. They can plan and guide activities with technology, prepare questions, encourage interaction and experimentation. They must monitor the screen time and use of technological devices, support and intervene when necessary, promoting gradual autonomy [24]. With regard to technology, design mechanisms, teaching-learning approaches and content are conditioning factors for learning [12]. In terms of design, pedagogical applications for children must use graphics and actions that provide context; they must use simple and clear instructions, based on images; they must have an intuitive interface and interactivity for independent use, but also a challenging approach with multiple opportunities for

success, to maintain interest [24]. They must have clear and understandable feedback mechanisms for the child, that make connections between actions and results, guiding the child's performance. The content must be appropriate and meaningful to the child, allowing to explore different domains (cognitive, physical, emotional) and offering opportunities to complement learning with offline activities, such as drawings or playing games outside [25]. Digital games are a very appealing type of multimedia content for children, which can be used in the teaching-learning process. The learning of younger children is intuitive and action-oriented. Children experience and discover reality through error and success, observing the consequences of their actions, influenced by the context, with the support of adults [26]. While playing games, children have an interested and cooperative attitude. They are active in the search for information and oriented to achieve results [27]. Games present content in a fun way and give opportunities to practice, with immediate feedback on the results. They allow children to experience and solve challenges, encourage collaboration, teach how to respect rules, how to work individually and as a team. Games provide variety and flexibility of learning, for example, there may be individual games, games for small or large groups, closed or open games [28].

This paper presents the project Aprender XXI, an enterprise R&D project, involving a research team, a technological team and the participation of users from a kindergarten. This project aimed to develop mobile apps to promote the learning of pre-school children, using game-based learning strategies and addressing content areas, according to the CGPE, from the Portuguese Ministry of Education [29]. Technological development was driven by scientific research, to generate rigorous knowledge about the development process and the results achieved. The project lasted 24 months and adopted the DBR methodology, as it was considered an appropriate methodology for the context and the defined objectives.

#### **2. Materials and Methods**

The term DBR encompasses a group of research methodologies based on design and development, which have common characteristics [30], such as—design experiments [31], development research [32], educational design research [33].

This methodology allows exploring the potential of technologies in education to solve a real problem, bringing practical and scientific contributions [32]. It may include the development of technological products, materials and instructional activities that use technologies. DBR includes problem analysis, design and conception of an educational prototype, evaluation and review activities [34]. The user must be involved in the design and development process, namely in the use and evaluation of the prototype [35]. Scientific knowledge influences development, which is then tested in the field, bringing empirical data to improve the product and validate knowledge [32]. In conclusion, DBR is based, on the one hand, on rigorous and reflective research to build knowledge and design principles that can guide developments. On the other hand, it seeks to solve concrete problems, involving users and acting on the context [36].

For all these reasons, DBR methodology was chosen for the development of this project. The Aprender XXI apps were designed to explore the potential of mobile game-based learning in kindergarten. A kindergarten classroom was involved, with the participation of an educator and 22 children in the development process, which means that the prototypes of the apps were used and evaluated in the context (kindergarten), to be corrected and improved in a final version. The results achieved are the final versions of four game-based learning apps for preschool children.

The model proposed by Plomp [33] to operationalize the DBR methodology was adapted to the project, divided into three phases: Phase 1—preliminary study, which included the literature review, search for solutions that already exist in the market and acquisition of knowledge regarding learning contents and skills to be acquired at preschool age; Phase 2—development, started with the writing of technical specifications of the apps, the choice of learning content to be included in the apps, the adaptation and script of digital content, design and programing of technological products; Phase 3—evaluation, which

included tests with users in kindergarten and product improvements, ending with the final versions of the apps.

#### *2.1. Preliminary Study*

The first phase was the preliminary study necessary for the development of gamebased learning apps for children aged three to six years old. A literature review on children's learning with technology and games served to contextualize and substantiate the theoretical framework and to become aware of research already performed, such as recent cases of use of apps in learning practices in kindergarten. The platforms Scopus and Web of Science were selected for search and constitution of the documental corpus. In order to contemplate national projects, searches were also carried out in national journals and repositories of the Portuguese Universities. Further, complementary search was done on the aggregator system Google Scholar. Research equations were combinations and variations of the terms: apps/tablets, kindergarten/preschool and children/toddlers. The research was carried out in Portuguese and English languages. As criteria for inclusion, researchers considered articles of scientific journals, research reports, conference proceedings and doctoral theses with theoretical discussions and practical experiences of using apps for learning in kindergarten. Documents centered on other levels of education, or technologies that fell outside the scope (e.g., websites, CD-ROM) were excluded. Since it is not the objective of this article to present the entire literature review, some studies were selected to be presented in the "results of the preliminary study" section, due to their relevance to the project.

Another preliminary study carried out was the search and selection of educational apps suitable for these ages, to learn more about the "state of the art" and understand the existing offer that can be used in an educational context, identifying the features they offer, innovations, trends and good practices, as well as weaknesses that can be suppressed. To this end, a search was made for apps available in online stores (App Store® and Google Play®, 2017 versions). These stores have limitations in terms of organization, search and filtering, making it difficult to select apps, in an immense offer with daily growth. Thus, it was decided to select the most prominent apps in stores, by the number of comments or downloads. At the same time, a web search was done, combining some terms (apps, children/kids, kindergarten/preschoolers, learning, games) to find lists of apps and reviews from experts that reflect on the most used and well positioned apps for these purposes. This research allowed to find app sites, with information about their educational approach and content. In the Stores, it was possible to view demonstration videos and download the apps to test. Some apps were excluded because they were outside the scope of the project. As a result, a solid set of apps was obtained, to be subsequently evaluated. The evaluation considered dimensions relevant to the educational level, establishing the categories of analysis based on the CGPE. In relation to game dynamics, the presence of rules, obstacles and quantification of actions was verified. In terms of skills development, the possibility of creating, building, collaborating or cooperating was considered. Regarding the content, it was analyzed whether the focus was scientific, simulation of reality, or storytelling.

Since this project focuses on children aged three to six, who may attend kindergarten, it was necessary to understand what the approaches were to formal learning for children of this age group, as apps must respond to educators' pedagogical practices in order to be efficient. To this end, CGPE and educational guidelines from other European countries were studied, in order to define appropriate content at national level, but with the potential for internationalization.

#### *2.2. Development*

Based on the results of the preliminary study, the development phase began. Technological development consisted of a set of activities that were divided into smaller tasks with defined objectives, deadlines, human resources and dependencies. It started with the definition of specifications—detailed descriptions of what needs to be developed to

respond to the identified needs. The specifications delimit the scope of the project and present guidelines, common to the different functions and team members: content creation, design and programing [37]. Four types of specifications were defined:


After defining the specifications, content planning started, with the choice of themes and subthemes for each app, based on the CGPE. Screenwriting described what appears on each screen, in terms of content, graphics, interaction, feedback and sound. After the specifications and content were defined, the design began, with graphic studies, creation of characters, scenarios, objects and animations. The content was programed in four apps for mobile devices (Android® 4.0.3 or later and iPads® OS 8.0 or later) and an integrating website. The development also included internal tests, before the evaluation with users. In the internal tests, errors were corrected and interaction details were changed to improve the user experience.

#### *2.3. Evaluation*

The evaluation phase aimed to test the prototypes with users, to correct or validate the direction of technological development. It was carried out in a kindergarten classroom, with an educator and 22 children, aged three and six years old, as the use in the context and the interaction between users was essential for the evaluation [38]. Data collection was based on the observation of the use of prototypes by the children and the educator. At the end of the observation sessions, an interview was conducted with the educator. The specific objectives of this evaluation were:


Some kindergartens were identified in order to integrate a group of children in the project. It should be a heterogeneous group, to test the prototypes with different ages. The educator should have a predisposition to use technologies and a willingness to integrate the project in pedagogical activities. A kindergarten classroom was intentionally selected because it complied with the prerequisites. The invitation was formally made to the institution and the project was presented to the educator, who transmitted the information to the parents and obtained informed consent. The pilot project was approved and took place at the kindergarten, during the first period of the 2018–19 school year, involving the kindergarten teacher and her group with 22 children between the ages of three and six years old.

The data collection technique chosen was participant observation, as it is useful for studying small groups and events that last a short time. It is an opportunity to record information as it occurs in a context, to study non-verbal behaviors and people who have difficulty verbalizing their ideas, such as pre-school children [39]. Following the considerations of the authors, Cohen, Manion and Morrison [40], the observation was planned, defining when, where, how and what to observe. Four observation sessions were held, one for each app. The researcher interacted with the users (children and educator) in their natural environment, giving initial explanations about the apps. Then, the children used tablets to explore the apps, freely and/or with the guidance of the educator.

It was necessary to take into account some limitations of this data collection technique. The presence of an observer in the context can impact the behavior of those being observed or lead to disturbances of what is intended to be observed, which may cause changes in the outcomes of a study [41]. In a research with children, it is necessary to consider the way they look at adults, which can lead to inhibition or the search for attention and approval. It is difficult for children to accept an adult as an equal, although they can tolerate their presence in the group [42]. In this investigation, the children involved were used to receiving adults in the classroom to do specific activities, such as talking about professions, carrying out science experiments, cooking. In this sense, the researcher chose to follow the procedures they were used to, in order to reduce the strangeness element. She was introduced as a person who came to do an activity with tablets. During the sessions, she explained the games, helped the children punctually, but intervened as little as possible to make the use as natural as possible—individually, between peers or with the educator. This posture allowed the researcher to take notes during the observation, recording a large number of comments immediately, to reduce the possibility of bias in the data, which is another limitation of the observation technique [41]. An observation grid was created to fill with data collected in the sessions. The grid had a table for each app, with games identified by a name and separated by lines. In each line, there were fields to fill in, regarding the use of a child: (1) liked it or not; (2) understood or not; (3) comments during the game; (4) notes from the researcher, such as interaction with other children or with the educator. The data collected about each child were the name, age and gender, which were then stored separately and coded from C1 to C22.

Each session lasted the morning activity period. The observation was semi-structured, because it started from a set of topics that were intended to be observed, in order to obtain a rich description of the events that occurred in the context [40]. In addition, an interview was conducted as an instrument to collect more detailed and in-depth information from the kindergarten educator, to understand the possible use of apps in an educational context, and also to collect opinions and suggestions. To help conduct the interview, a guide was created with questions previously formulated to answer the objectives of the evaluation. It used open questions related to experience and opinion, to understand thoughts, beliefs and attitudes about certain topics [40]. The interview was conducted in the kindergarten. Data were analyzed to obtain the educator's perception and to assess the need to make changes in technological development.

#### **3. Results**

The results of each phase are presented separately and sequentially, as each phase influenced the next. The discussion and conclusions combine the results of the three phases.

#### *3.1. Results of the Preliminary Study*

The literature review focused on projects that contributed with results on the role of tablets in pre-school education and the feasibility of integration in this context. The complete study can be read in the preliminary study report [43]. For this article, some projects were selected in which kindergarten children used apps for learning, covering different areas of knowledge and skills. One study, on drawing activities with tablets, reports that children enjoyed the experience, showed persistence and quality in the drawings, developed familiarity with the tablet and used them progressively more time and more autonomously, over six weeks [44]. In another research, the tablet was used for literacy activities, with apps for creating narratives and acquiring vocabulary. Children understood the apps, played an active role in the games, shared the tablet, showed a good mood and communicated. The activity promoted language learning and increased speech skills [45]. The use of math apps in a Greek kindergarten had better results than teaching traditional mathematics [46]. The use of apps for multimedia production, in groups, generated curiosity, communication and collaborative work, stimulated creativity, hand-eye coordination and fine motor skills [47]. A dollhouse decoration game app allowed children to reproduce traditional play, work on spatial organization skills and learn the relationship between objects. At the same time, it stimulated cooperative work and creativity. A photo editing app allowed the creation of graphic compositions. These experiences showed the potential to influence the cognitive development, autonomy and creativity of children with mobile apps [48].

In the survey of educational apps for pre-school education, an extensive number of apps was found. The full study is available in the preliminary study report. A selection was made of the apps that best meet the project's goals, as they were suitable for the ages of three to six years old, and covered content areas proposed in the CGPE. As a summary, Table 1 presents some apps (or collections of apps), with categories relevant to the educational level, according to CGPE [49].


**Table 1.** List of selected apps. Adapted from Zagalo & Laranjeiro (2018).

In general, all apps have sets of games and correspond to several categories. The most prevalent categories are the presence of open rules, which allow children to act freely; the possibility of building/construction within the game, which stimulates creativity; and the simulation of reality, with activities that are not compulsory, but closely linked to their daily lives. This emphasizes the agency of players who are in a phase of discovering themselves and others. A less interesting point is the approach to scientific content in some apps, with greater rigidity and a more instructional approach, using game logics as overcoming obstacles to score, stating the need to quantify activities. Another category rarely found was collaboration/cooperation, which may be associated with parents' fears about their children's access to the online environment, but that narrows the potential of mobile apps. It was also concluded that, in the national context, there were few apps aligned with the CGPE. The learning apps in Portuguese were scarce, more oriented to the consumer market and, tendentially, instructional. Most of the educational apps found were in English, which may be an exclusion factor for use in Portuguese classrooms, particularly in pre-school education, when children have little contact with foreign languages.

To understand the approach to formal learning in kindergarten, the CGPE and educational guidelines from other European countries were consulted, with the intention of developing apps according to the content, and pedagogical practices of kindergarten teachers. European countries provide official guides for pre-school education, in documents that refer to learning objectives at an emotional, social and intellectual level. They give indications about pedagogical approaches, learning activities and assessment methods and define skills to be developed, such as language and communication. There are differences in the learning goals defined by country, but most emphasize reading, numeracy and

logical thinking, as well as the importance of the arts and knowledge of the world for children aged three to six. Pre-school institutions are free to decide the curriculum, the organization of activities and methodologies they want to adopt. A balance between individual and group work is recommended, and also between activities initiated by educators and children. About half of the countries believe that free play should be encouraged as an essential way of learning [50]. In Portugal, the CGPE follow an approach that is supported by the three main areas of human development: socio-affective, motor and cognitive. The document proposes that children develop through exploration and action on the world around them, with activities planned by the educator that should enhance the discovery of relationships with themselves, with others and with objects, and encourage reflection, understanding, transformation and complexification of knowledge. It distinguishes three content areas—personal and social training; expression and communication; knowledge of the world. Personal and social training integrates citizenship, multiculturality, sense of identity, education for values, such as tolerance, sharing and justice, seeking to make children conscious and supportive, with the ability to solve problems. The area of expression and communication refers to the acquisition of different forms of language, for the child to make their representations, interact with others, express thoughts and feelings. It is divided into the scientific areas of physical education, artistic education, oral and written language, mathematics. In the last area, knowledge of the world, the child's natural curiosity is stimulated, fostering new situations for discovering and exploring the world. Three components are used: scientific methodology, sciences and technologies. The CGPE recognize the child as the central element of learning and consider playing as the natural and spontaneous way of learning. It advises a holistic approach to content areas, in a process of articulated construction of knowledge [29].

#### *3.2. Results of the Development Phase*

After the preliminary study was completed, development began. Four apps were defined. Each app refers to a main theme, with five games that address sub-themes, accordingly with the content areas and knowledge domains defined in the CGPE.

#### 3.2.1. Nature Kids

CGPE.

Science—

Science—

autonomy

Science—

Science—

Science—

physical/natural world

Democratic coexistence

physical/natural world

physical/natural world

physical/natural world

physical/natural world

3.2.2. Healthy Kids

Independence and

Knowledge of the world

Personal and social training

Expression and

Knowledge of the world

Personal and social training

the world

the world

Knowledge of the world

Expression and

Plantations Knowledge of

Meteorology Knowledge of

Forest

Feed animals

Recycling

Nature Kids is an environmental education app, which explores concepts and challenges related to sustainability, biodiversity, recycling, forest and animal life preservation (Figure 1). It has five games: (1) Forest—find baby animals in the forest (squirrels, rabbits, deer. . . ) and hand them over to their parents; (2) Feed the Farm Animals—feed the cows, chickens, pigs and other animals on a farm, according to their requests, that are presented with illustrations of food in speech bubbles; (3) Plantations—the whole process of sowing or planting until harvesting fruits and vegetables—putting seeds, water, fertilizer, watching them grow, harvesting; (4) Recycling—separate waste for the correct containers—paper, glass, packaging; (5) Meteorology—dress up characters, according to the weather conditions. *Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 10 of 24

The main domains of the CGPE addressed in this app are the knowledge of the phys-

Characteristics of animals, identify similarities and differences

Characteristics of animals, identify similarities and differences

Making decisions, taking into account the well-being of others;

Characteristics of plants, nature preservation, respect for the environment; phenomena and transformations in the natural

Respect for the environment; properties of objects, identify

Climate influence in daily life; similarities and differences of materials (clothes); phenomena/transformations in the natural

distinguish food and its importance for health

similarities and differences of materials

Healthy Kids is the health education app, that addresses the subthemes of healthy eating, hygiene, oral health, physical activity (Figure 2). It has five games: (1) Creating Healthy Snacks—using available food, children can create sandwiches and drinks; (2) Supermarket—shopping for healthy food at the supermarket, selecting options from the shelves. When passing the food in the cashier, it will check if the choices are healthy; (3) Dentist—going to the dentist, clean the teeth, treat caries and put a brace; (4) Physical Activity—choose an activity to train outdoors, such as cycling, rollerblading, skateboarding; (5) Beach—going to the beach, put on sunscreen, put on a cap, stay in the shade, play

spect for the environment, properties of objects, climate and influence in daily life). Table 2 summarizes the learning that can be promoted by the Nature Kids app, according to the

and citizenship Respect for the other, solidarity, interventional attitude

communication Math Identify numbers, quantities, count; locate objects and group by

**Figure 1.** Print screens of games from the app Nature Kids: (**a**) Recycling; (**b**) Plantation; (**c**) Feed the Farm Animals. in the sand, see the flags. **Figure 1.** Print screens of games from the app Nature Kids: (**a**) Recycling; (**b**) Plantation; (**c**) Feed the Farm Animals.

**Table 2.** Nature Kids games, content and learning. **Game Content Area Knowledge Domain Learning, Attitudes, Know-How That Can Be Explored** 

of animals

characteristics

of animals

world

world

communication Math Identify and locate objects; group by characteristics

The main domains of the CGPE addressed in this app are the knowledge of the physical and natural world (characteristics of animals and plants, nature preservation and respect for the environment, properties of objects, climate and influence in daily life). Table 2 summarizes the learning that can be promoted by the Nature Kids app, according to the CGPE.


**Table 2.** Nature Kids games, content and learning.

#### 3.2.2. Healthy Kids

Healthy Kids is the health education app, that addresses the subthemes of healthy eating, hygiene, oral health, physical activity (Figure 2). It has five games: (1) Creating Healthy Snacks—using available food, children can create sandwiches and drinks; (2) Supermarket—shopping for healthy food at the supermarket, selecting options from the shelves. When passing the food in the cashier, it will check if the choices are healthy; (3) Dentist—going to the dentist, clean the teeth, treat caries and put a brace; (4) Physical Activity—choose an activity to train outdoors, such as cycling, rollerblading, skateboarding; (5) Beach—going to the beach, put on sunscreen, put on a cap, stay in the shade, play in the sand, see the flags.

**Figure 2.** Print screens of games from the app Healthy Kids: (**a**) Beach; (**b**) Supermarket; (**c**) Physical Activity. **Figure 2.** Print screens of games from the app Healthy Kids: (**a**) Beach; (**b**) Supermarket; (**c**) Physical Activity.

The main learning contents relate to health care and safety of the body, well-being and responsible choices, which are included in the areas of world knowledge and personal and social training of the CGPE. Table 3 summarizes the learning that can be promoted by the Healthy Kids app, according to the CGPE. The main learning contents relate to health care and safety of the body, well-being and responsible choices, which are included in the areas of world knowledge and personal and social training of the CGPE. Table 3 summarizes the learning that can be promoted by the Healthy Kids app, according to the CGPE.



Areas—distribute children in the areas of the kindergarten classroom. In each area, there 3.2.3. Citizen Kids

can only be a certain number of children (e.g., dollhouse can have 4 characters, library— 2, paintings—3). The player has to distribute the characters in the areas, taking into account their wishes that are shown in thought bubbles. When there are too many characters Citizen Kids in the citizenship app, that addresses the subthemes of civility, multiculturalism and preservation of cultural heritage (Figure 3). It has five games: (1) Play Areas—distribute children in the areas of the kindergarten classroom. In each area, there

wanting to go to an area, they have to wait for their turn. They stay in the queue, until the

culturalism and preservation of cultural heritage (Figure 3). It has five games: (1) Play

Play Areas

Playground

Behaviour Board

Personal and

Personal and social training

Expression and

Personal and social training

can only be a certain number of children (e.g., dollhouse can have 4 characters, library—2, paintings—3). The player has to distribute the characters in the areas, taking into account their wishes that are shown in thought bubbles. When there are too many characters wanting to go to an area, they have to wait for their turn. They stay in the queue, until the player removes one that has been there for a long time. (2) Playground—distribute characters in the playground, similarly to the previous one, but outside. The characters must be distributed among different spaces: slide, sandbox, hopscotch, soccer. (3) Behavior Board—a board divided into two sides. On the left side there is the correct symbol and on the right side there is the wrong symbol. In the game, children explore a room where they discover notes with correct attitudes (lift the finger to speak) and wrong attitudes (push, throw tantrums) and they can build a board, choosing the correct and wrong behaviors, with the notes they prefer. (4) Clothes of the World—dress dolls with clothes that represent different countries and customs in the world. Players can choose a country on the map and see the typical costume. Then they can open the closet to choose clothes and dress the doll according to the costume or otherwise, as they wish. When they open the closet the clothes change, to give them several options; (5) Preservation of Heritage—on a walk down the street, the character goes through places that he can help preserve, for example, picking up papers from the floor, washing a statue with a hose, fix a broken church-stained glass, tearing wall posters. *Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 12 of 24 player removes one that has been there for a long time. (2) Playground—distribute characters in the playground, similarly to the previous one, but outside. The characters must be distributed among different spaces: slide, sandbox, hopscotch, soccer. (3) Behavior Board—a board divided into two sides. On the left side there is the correct symbol and on the right side there is the wrong symbol. In the game, children explore a room where they discover notes with correct attitudes (lift the finger to speak) and wrong attitudes (push, throw tantrums) and they can build a board, choosing the correct and wrong behaviors, with the notes they prefer. (4) Clothes of the World—dress dolls with clothes that represent different countries and customs in the world. Players can choose a country on the map and see the typical costume. Then they can open the closet to choose clothes and dress the doll according to the costume or otherwise, as they wish. When they open the closet the clothes change, to give them several options; (5) Preservation of Heritage—on a walk down the street, the character goes through places that he can help preserve, for example, picking up papers from the floor, washing a statue with a hose, fix a broken church-stained glass, tearing wall posters.

Citizen Kids app addresses the CGPE, in the field of personal and social training, areas of democratic coexistence and citizenship, exploring contents about civility and rules of living in society, multiculturalism and diversity, solidarity and preservation of the natural and cultural heritage. Table 4 summarizes the learning that can be promoted by the Citizen Kids app, according to the CGPE. Citizen Kids app addresses the CGPE, in the field of personal and social training, areas of democratic coexistence and citizenship, exploring contents about civility and rules of living in society, multiculturalism and diversity, solidarity and preservation of the natural and cultural heritage. Table 4 summarizes the learning that can be promoted by the Citizen Kids app, according to the CGPE.

#### **Table 4.** Citizen Kids games, content and learning. 3.2.4. Busy Kids

Democratic

Democratic

coexistence/citizenship; Independence/autonomy

Identity/self-esteem;

coexistence/citizenship

**Game Content Area Knowledge Domain Learning, Attitudes, Know-How That Can Be Explored**  social training Democratic coexistence/citizenship; Independence/autonomy Respect for the other, solidarity, critical and interventional attitude towards the world around them, making decisions taking into account the well-being of the other Expression and communication Math Identify numbers, identify, count and locate objects; group objects by characteristics Busy Kids is the profession awareness app that offers games about different jobs and routines (Figure 4). It has five games. (1) Postman—the character has to sort mail, deliver mail to addresses in a street, following a map and directional arrows to advance in the scenario. (2) Fashion Designer—the player chooses fabric patterns, cuts fabrics according to shapes, decorates clothes with props; (3) Builder—mix cement, lay bricks, build walls according to different geometric shapes; (4) Office Worker—uses the computer, answers the phone, uses paper, pens, stamps and other office supplies. (5) Mechanic—changes tires, paints cars, replaces headlights, among others.

group objects by characteristics

Respect for the other, solidarity, critical and

interventional attitude towards the world around them, making decisions taking into account the well-being of

Awareness of belonging to a group and respect for rules and others members; ability to make choices and assume

the other

responsibilities

communication Math Identify numbers, Identify, count and locate objects;


#### **Table 4.** Citizen Kids games, content and learning.

Busy Kids app exposes children to contents in the content area of knowledge of the world, suggested in the CGPE, such as knowing the community, life situations and cultural practices, but also, addresses the expression and communication area. Table 5 summarizes the learning that can be promoted by the Citizen Kids app, according to the CGPE.

Clothes of the

Preservation of Heritage

World

Expressions and

Knowledge of the world

Personal and social training

Knowledge of the world

Personal and social training

tires, paints cars, replaces headlights, among others.

Communication Visual Arts Ability to recognize elements of visual communication

cultural ties

properties

the need for preservation.

Busy Kids is the profession awareness app that offers games about different jobs and routines (Figure 4). It has five games. (1) Postman—the character has to sort mail, deliver mail to addresses in a street, following a map and directional arrows to advance in the scenario. (2) Fashion Designer—the player chooses fabric patterns, cuts fabrics according to shapes, decorates clothes with props; (3) Builder—mix cement, lay bricks, build walls according to different geometric shapes; (4) Office Worker—uses the computer, answers the phone, uses paper, pens, stamps and other office supplies. (5) Mechanic—changes

Identify physical, social and cultural aspects of the community and identify differences and similarities with other communities; know and respect cultural diversity

Know and value manifestations of cultural heritage; respect diversity; accept their social and cultural identity, in relation to others; recognize and value social and

Know and value manifestations of cultural heritage; recognize and value social and cultural ties, recognizing

Recognize central elements of the community, highlighting physical, social and cultural aspects; identify cultural practices; identify objects and their

Science—physical/natural

Identity/self-esteem;

coexistence/citizenship

Science—physical/natural world; social world

Independence/autonomy;

coexistence/citizenship

world

Democratic

Democratic

3.2.4. Busy Kids

**Figure 4.** Print screens of games from the app Busy Kids: (**a**) Builder; (**b**) Office Worker; (**c**) Postman. **Figure 4.** Print screens of games from the app Busy Kids: (**a**) Builder; (**b**) Office Worker; (**c**) Postman.


Busy Kids app exposes children to contents in the content area of knowledge of the world, suggested in the CGPE, such as knowing the community, life situations and cul-**Table 5.** Busy Kids games, content and learning.

3.2.5. Common Approach, Multidisciplinary Content and Integrating Website

The apps are based on image (illustration, animation) and sound (music and sound effects), which are suitable ways to communicate with children, regardless of their native language and literacy level. Written text and audio narration were excluded. Scenarios are

filled with interactive and animated elements, to incite children to freely explore different spots. Objects with small interactivity stimulate curiosity and discovery, as well as an understanding of the world around them (e.g., touching a perched bird, the bird flies; Touching a switch, the light turns on or off; Touch a tap, the water flows). These small interactions allow the child to appropriate concepts and make sense of the world. In certain games, there are concrete tasks. Other games have open rules, so children build freely and progress according to their will and speed.

In addition to the knowledge and skills immediately associated with each game, there are other learnings that are more subtly integrated to explore multidisciplinary, such as mathematics. In the Forest game, the child has to count and find the baby animals that parents had lost. In the Construction game, the player respects geometric shapes to create a wall. In the Postman game, children interpret maps to orient the character and move around. Other knowledge domains that are suggested in the CGPE can be explored with the Apps. One it the introduction to scientific methodology, which proposes to carry out activities that allow the child to question, hypothesize, predict responses, experiment, collect, organize and analyze information and draw conclusions. This type of activity can be promoted with the support of the educator, in the exploration of various games. Examples: which foods are healthy? What happens if the boy does not put on sunscreen? What do animals eat? How will the seed end up in fruit?

Additionally, the use of technologies is a learning area for CGPE—to recognize and use technological equipment, with care and safety. This is promoted by using tablets to play the apps, individually and autonomously, or in peer or group activities, with guidance from the educator.

These apps do not promote competition among children, but cooperation, since they all want to contribute to a better world, which is translated into a system of flowers in a virtual world. When children play a game, a flower appears in a virtual world inside the app, symbolizing their individual participation. But they can also send their contribution to the project's website, where all the flowers of all the children are represented. It symbolizes their total participation in the global world (Figure 5).

This aggregator site gathers data on the use of apps and shows the contribution of all users to a better world. Children can see the graphic representation of a world in which they participate. Thus, children begin to learn that their actions have a personal impact and a global impact, and they feel that they have contributed to a better future, which is extremely important in environmental, citizenship, public health, economy and employment issues, the main themes of the apps. On this website, parents and educators find educational guides for each app, with the objectives of the games, how to play, conversation topics before, during and after playing the games, and suggestions for educational activities to do with the children.

#### *3.3. Results of the Evaluation Phase*

The results of the observation (children using the apps) and the interview (educator's perception) served to implement corrections and improvements in the final versions of the apps, in order to make the products more efficient for educational activities in kindergarten and to achieve a greater interest and involvement of the children. A pilot study report details all the improvements made in each game, in the four apps [51]. For this article, the results of each observation session are summarized, stating the main improvements made in the final versions of the apps.

bolizes their total participation in the global world (Figure 5).

ance from the educator.

Additionally, the use of technologies is a learning area for CGPE—to recognize and use technological equipment, with care and safety. This is promoted by using tablets to play the apps, individually and autonomously, or in peer or group activities, with guid-

These apps do not promote competition among children, but cooperation, since they all want to contribute to a better world, which is translated into a system of flowers in a virtual world. When children play a game, a flower appears in a virtual world inside the app, symbolizing their individual participation. But they can also send their contribution to the project's website, where all the flowers of all the children are represented. It sym-

**Figure 5.** Children's participation in the Aprender XXI World. **Figure 5.** Children's participation in the Aprender XXI World.

#### This aggregator site gathers data on the use of apps and shows the contribution of all 3.3.1. Observation Session of the Nature Kids App

users to a better world. Children can see the graphic representation of a world in which they participate. Thus, children begin to learn that their actions have a personal impact and a global impact, and they feel that they have contributed to a better future, which is extremely important in environmental, citizenship, public health, economy and employment issues, the main themes of the apps. On this website, parents and educators find educational guides for each app, with the objectives of the games, how to play, conversation topics before, during and after playing the games, and suggestions for educational activities to do with the children. *3.3. Results of the Evaluation Phase*  The results of the observation (children using the apps) and the interview (educator's perception) served to implement corrections and improvements in the final versions of the apps, in order to make the products more efficient for educational activities in kindergarten and to achieve a greater interest and involvement of the children. A pilot study report details all the improvements made in each game, in the four apps [51]. For this article, the results of each observation session are summarized, stating the main improvements made in the final versions of the apps. 3.3.1. Observation Session of the Nature Kids App In the Forest game (delivering baby animals to their parents), some children (C2, C7, C11, C18) found it difficult to catch the baby animals, as they were very small (squirrel and rabbit). The animals were enlarged in the final version. C13 found two baby animals In the Forest game (delivering baby animals to their parents), some children (C2, C7, C11, C18) found it difficult to catch the baby animals, as they were very small (squirrel and rabbit). The animals were enlarged in the final version. C13 found two baby animals and tried to pick them at the same time, which was not possible. It was also fixed. The game always started with the same animals and the children wanted to see them in another order. It was necessary to make it random. In the Meteorology game (Figure 6) (dressing characters according to the weather) the gameplay had to be explained to the first children who played (C4, C6, C8). They did not understand how to change the climate and how to see different clothes. This confirmed the need to implement graphic instructions and feedback to guide the action, for autonomous use by children. C20 found an error, which has been fixed. A character dressed according to the weather did not look happy. The game Feeding the Farm Animals was not well understood. Children did not understand that the animals were making specific requests, using speech bubble (meat, fish, water...) and that they had to satisfy the needs requested at that time. They played the game without obeying the rules, feeding the animals as they wanted. C11 gave the dog meat, then she gave him water because he could be thirsty. The dog had made no request at all. After the researcher explained the gameplay, children found it difficult to satisfy the requests at the right time. It was decided to remove this rule from the game. The objective changed to feed the animals, without having to obey time or specific requests, but according to the animal's diet (carnivore, herbivore, granivore). Graphic feedback was implemented, showing whether the animal liked it or not. The educator suggested that the animals could respond positively with a heart. The Plantations game also had to be changed. The game had several actions—sowing/planting, watering, fertilizing, harvesting. Since the children planted, the plantations started to grow and were ready to be harvested, regardless of whether the children carried out the actions. Children took some time to do these actions. C2 planted strawberries and harvested, without taking any intermediate action. The game has been corrected to make some steps mandatory for plants to grow. The Recycling game generated collaboration and communication. C5 and C6 played together, passing the tablet to each other. Several children pointed out the right container and commented where each type of garbage should be placed.

container and commented where each type of garbage should be placed.

and tried to pick them at the same time, which was not possible. It was also fixed. The game always started with the same animals and the children wanted to see them in another order. It was necessary to make it random. In the Meteorology game (Figure 6) (dressing characters according to the weather) the gameplay had to be explained to the first children who played (C4, C6, C8). They did not understand how to change the climate and how to see different clothes. This confirmed the need to implement graphic instructions and feedback to guide the action, for autonomous use by children. C20 found an error, which has been fixed. A character dressed according to the weather did not look happy. The game Feeding the Farm Animals was not well understood. Children did not understand that the animals were making specific requests, using speech bubble (meat, fish, water...) and that they had to satisfy the needs requested at that time. They played the game without obeying the rules, feeding the animals as they wanted. C11 gave the dog meat, then she gave him water because he could be thirsty. The dog had made no request at all. After the researcher explained the gameplay, children found it difficult to satisfy the requests at the right time. It was decided to remove this rule from the game. The objective changed to feed the animals, without having to obey time or specific requests, but according to the animal's diet (carnivore, herbivore, granivore). Graphic feedback was implemented, showing whether the animal liked it or not. The educator suggested that the animals could respond positively with a heart. The Plantations game also had to be changed. The game had several actions—sowing/planting, watering, fertilizing, harvesting. Since the children planted, the plantations started to grow and were ready to be harvested, regardless of whether the children carried out the actions. Children took some time to do these actions. C2 planted strawberries and harvested, without taking any intermediate action. The game has been corrected to make some steps mandatory for plants to grow. The Recycling game generated collaboration and communication. C5 and C6 played together, passing the tablet to each other. Several children pointed out the right

**Figure 6.** Child playing the Meteorology game, educator guiding, several classmates watching. **Figure 6.** Child playing the Meteorology game, educator guiding, several classmates watching.

#### 3.3.2. Observation Session of the Healthy Kids App

3.3.2. Observation Session of the Healthy Kids App In the Supermarket game, C22 was the first to play and immediately understood the game. He chose assorted food on the shelves and went to the cashier to pay. The cash register screen showed a sad face when the chosen foods were not healthy. C22 repeated the game, buying only healthy foods. The rest of the children commented, to help him choose the right food. C1 found a bug, which was fixed in the final version. He put the character of the game walking in the opposite direction. The character left the screen and In the Supermarket game, C22 was the first to play and immediately understood the game. He chose assorted food on the shelves and went to the cashier to pay. The cash register screen showed a sad face when the chosen foods were not healthy. C22 repeated the game, buying only healthy foods. The rest of the children commented, to help him choose the right food. C1 found a bug, which was fixed in the final version. He put the character of the game walking in the opposite direction. The character left the screen and did not appear again. In the Dentist game, C10 was the first to play. He did not realize that he had to clean all teeth in the exposed mouth. He received an explanation from the researcher and followed the rest of the steps to the end, without any doubt. The educator felt that there was little distinction between step 2 (removing the caries and opening a cavity in the tooth) to step 3 (putting mass on the tooth). The cavity and the mass looked similar. In the final version, changes in shape and color were made in both steps. The instruments were difficult to use on the lower teeth, even for five-year-old children. It has been corrected by increasing the touch area on the screen. In Creating Healthy Snacks game, C12 found it very fun to create a giant sandwich and a drink with all the elements available. Then, the educator told her to create a healthy sandwich. Other children gave suggestions and commented. The game was well accepted, with no doubts or need for changes. The Beach game was well understood. C8 found an error. He was unable to give the character water to drink. He dragged the bottle to his mouth, but nothing happened, and the character continued to show thirst (Figure 7). It was necessary to increase the touch area of the mouth. The Physical Activity game did not attract so much interest or any interaction with each other. Children entered the garage, chose a prop (skateboard, bicycle, rollerblades) for the character, used it outdoors, in the garden, and left the game.

did not appear again. In the Dentist game, C10 was the first to play. He did not realize that he had to clean all teeth in the exposed mouth. He received an explanation from the researcher and followed the rest of the steps to the end, without any doubt. The educator felt that there was little distinction between step 2 (removing the caries and opening a cavity in the tooth) to step 3 (putting mass on the tooth). The cavity and the mass looked similar. In the final version, changes in shape and color were made in both steps. The instruments were difficult to use on the lower teeth, even for five-year-old children. It has been corrected by increasing the touch area on the screen. In Creating Healthy Snacks game, C12 found it very fun to create a giant sandwich and a drink with all the elements available. Then, the educator told her to create a healthy sandwich. Other children gave suggestions and commented. The game was well accepted, with no doubts or need for changes. The Beach game was well understood. C8 found an error. He was unable to give the character water to drink. He dragged the bottle to his mouth, but nothing happened, and the character continued to show thirst (Figure 7). It was necessary to increase the touch area of the mouth. The Physical Activity game did not attract so much interest or any interaction with each other. Children entered the garage, chose a prop (skateboard, bicycle, rollerblades) for the character, used it outdoors, in the garden, and left the game.

**Figure 7.** Detail of child playing the Beach game, trying to give water to the character. **Figure 7.** Detail of child playing the Beach game, trying to give water to the character.

#### 3.3.3. Observation Session of the Citizen Kids App

3.3.3. Observation Session of the Citizen Kids App The games Play Areas and Playground (Figure 8) were highly requested, being played by eight and nine children, respectively. In both games, children wanted to move the characters to different areas and see if they were happy. The other children watched and made comments that indicated they understood the games. Some comments: C3 (while watching)—"you are always moving them around!"; C4 (while playing)—"because they want to"; C12 (while watching)—"Otherwise, they will be sad!"; C3—"But they have to wait". In certain interactions, children dragged a character to an area, but it did not stay there. This was fixed in the final version, in both games. The game Board of Behaviors was not well understood, even by the five-year-old children. It was necessary to explain everything, step by step. They did not realize that the board had one area for the right behaviors and another for the wrong behaviors. In the final version, it was necessary to change the graphics to better delineate the different areas. C8 played the game under the guidance of the researcher and did not like it. C14 also needed guidance, but he filled the board to the end. It was necessary to improve the game, for autonomous use, inserting graphical explanations and feedbacks. The game Preservation of heritage was understood and played six times by children aged three, four and five. The game has a path with tasks, The games Play Areas and Playground (Figure 8) were highly requested, being played by eight and nine children, respectively. In both games, children wanted to move the characters to different areas and see if they were happy. The other children watched and made comments that indicated they understood the games. Some comments: C3 (while watching)—"you are always moving them around!"; C4 (while playing)—"because they want to"; C12 (while watching)—"Otherwise, they will be sad!"; C3—"But they have to wait". In certain interactions, children dragged a character to an area, but it did not stay there. This was fixed in the final version, in both games. The game Board of Behaviors was not well understood, even by the five-year-old children. It was necessary to explain everything, step by step. They did not realize that the board had one area for the right behaviors and another for the wrong behaviors. In the final version, it was necessary to change the graphics to better delineate the different areas. C8 played the game under the guidance of the researcher and did not like it. C14 also needed guidance, but he filled the board to the end. It was necessary to improve the game, for autonomous use, inserting graphical explanations and feedbacks. The game Preservation of heritage was understood and played six times by children aged three, four and five. The game has a path with tasks, which were carried out by children without the need for explanation. After finishing the game, C19 made the course again without stopping, trying to be the fastest. The others commented, interested. It was possible to notice that they already knew the tasks (e.g., C12—"Next, you have to clean the posters on the wall"). In the game Clothes of the World, C13 and C15 had difficulty dragging shoes and glasses. The need to increase the size of these elements was noted and corrected in the final version. They also did not realize that they could choose another country by touching the map, nor change the clothes inside the closet, by closing and opening its doors. It was necessary to add visual clues.

closet, by closing and opening its doors. It was necessary to add visual clues.

which were carried out by children without the need for explanation. After finishing the game, C19 made the course again without stopping, trying to be the fastest. The others commented, interested. It was possible to notice that they already knew the tasks (e.g., C12—"Next, you have to clean the posters on the wall"). In the game Clothes of the World, C13 and C15 had difficulty dragging shoes and glasses. The need to increase the size of these elements was noted and corrected in the final version. They also did not realize that they could choose another country by touching the map, nor change the clothes inside the

**Figure 8.** Child playing autonomously the game Playground. Friends watching and commenting. **Figure 8.** Child playing autonomously the game Playground. Friends watching and commenting.

3.3.4. Observation Session of the Busy Kids App 3.3.4. Observation Session of the Busy Kids App

The most chosen game of all apps was the Builder game, played by 15 children (Figure 9). Some played together, in collaboration. Three-year-old C2 was the first to play. As he had doubts, the researcher explained the game and all the children listened. However, some difficulties have arisen. The interaction for making cement consisted of touching the materials and touching the machine's switch to turn it on. Seven children tried to drag the materials into the machine and nothing happened. In the final version, the interaction was changed, from touching to dragging, to be more intuitive. They also did not realize that they had to press the switch to start the cement machine. The size of the switch and the luminous graphic feedback were increased. In the second part of the game, they had to build a wall, but five children did not take the trowel to put the cement. It has been corrected by highlighting the trowel in the instrument menu. After placing the cement and brick, four children took the hammer and broke the brick, not understanding the function of the hammer. It was necessary to separate the construction instruments from the destruction instruments in the menu. As the walls could have different shapes, the educator considered that this was a good game to learn mathematics. The game Fashion Designer was not spontaneously chosen by any child. C18 volunteered at the end, to experiment. The researcher explained the first step—choosing a fabric. From there, he played alone, without difficulty, understanding all the possibilities. In the Office Worker game, C15 was the first to play. She thought the goal was to write on the computer that appeared in the game. It was explained to her that she could explore other office supplies (phone, pad, pens, among others). The game's keyboard was simplified, it only contained vowels, space key and emojis. The educator suggested placing a virtual keyboard with all the letters, so that the children could practice writing. She also considered that the phone was useful for The most chosen game of all apps was the Builder game, played by 15 children (Figure 9). Some played together, in collaboration. Three-year-old C2 was the first to play. As he had doubts, the researcher explained the game and all the children listened. However, some difficulties have arisen. The interaction for making cement consisted of touching the materials and touching the machine's switch to turn it on. Seven children tried to drag the materials into the machine and nothing happened. In the final version, the interaction was changed, from touching to dragging, to be more intuitive. They also did not realize that they had to press the switch to start the cement machine. The size of the switch and the luminous graphic feedback were increased. In the second part of the game, they had to build a wall, but five children did not take the trowel to put the cement. It has been corrected by highlighting the trowel in the instrument menu. After placing the cement and brick, four children took the hammer and broke the brick, not understanding the function of the hammer. It was necessary to separate the construction instruments from the destruction instruments in the menu. As the walls could have different shapes, the educator considered that this was a good game to learn mathematics. The game Fashion Designer was not spontaneously chosen by any child. C18 volunteered at the end, to experiment. The researcher explained the first step—choosing a fabric. From there, he played alone, without difficulty, understanding all the possibilities. In the Office Worker game, C15 was the first to play. She thought the goal was to write on the computer that appeared in the game. It was explained to her that she could explore other office supplies (phone, pad, pens, among others). The game's keyboard was simplified, it only contained vowels, space key and emojis. The educator suggested placing a virtual keyboard with all the letters, so that the children could practice writing. She also considered that the phone was useful for children to train their parents' contact numbers. The change to the keyboard was not implemented, due to lack of space on the game screen. The Postman game also had to be explained. The first part, separating the types of letters and packages, was apprehended immediately. The second part, delivery mail following a map and arrows, was more difficult to understand and play. However, the gameplay remained, because it trains important skills, such as orientation, interpretation of maps and the sense of laterality. The Mechanic game was easily understood, although there were doubts about the tools for fixing dents, filling tires

*Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 19 of 24

and painting. It was necessary to change the illustrations of these items. Three-year-old C16 spent four minutes decorating the car, worried about the aesthetic. items. Three-year-old C16 spent four minutes decorating the car, worried about the aesthetic.

immediately. The second part, delivery mail following a map and arrows, was more difficult to understand and play. However, the gameplay remained, because it trains important skills, such as orientation, interpretation of maps and the sense of laterality. The Mechanic game was easily understood, although there were doubts about the tools for fixing dents, filling tires and painting. It was necessary to change the illustrations of these

**Figure 9.** Children collaborating, playing together the Builder game, educator helping. **Figure 9.** Children collaborating, playing together the Builder game, educator helping.

3.3.5. Interview with the Kindergarten Educator

3.3.5. Interview with the Kindergarten Educator The interview with the educator aimed to obtain information to characterize the context where the tests took place and to evaluate the apps, as to relevance (content validity), consistency (construct validity) and expected practicality, that is, to understand if the ed-The interview with the educator aimed to obtain information to characterize the context where the tests took place and to evaluate the apps, as to relevance (content validity), consistency (construct validity) and expected practicality, that is, to understand if the educator would use the product and identify obstacles to its use [35].

ucator would use the product and identify obstacles to its use [35]. Regarding the context, the educator already carried out technological activities with the children, such as watching videos on YouTube®, searching on Google®, printing images. In the kindergarten classroom, they had a computer, where children could write letters or draw pictures. The educator also promoted digital communication. Children used Messenger® in kindergarten to send emojis to their parents. The educator used Skype® to make video calls with the children, when they were sick at home. They did not have a tablet in the kindergarten room, but the educator recognized that it was the most appropriate and preferred device. Regarding the expectation of use, the educator did not know many learning apps and thought that the Aprender XXI apps were in accordance with the interests of the children and the thematic areas they usually work on. The educator liked all the apps and gave concrete suggestions in some games. One suggestion was for games where the buttons are integrated into the scenario. A greater distinction should be made between buttons that have interaction and scenery props that are only illustrative. It can be a visual cue on the buttons (brightness, animation) or, after some time without action, a help could appear, such as an arrow indicating a path. The educator also had suggestions about the structure and content of the educational guides for parents and educators. It could be a guide per game. Each guide should have an introduction about the game, goals and gameplay, so that the educator understands it first and then can explain the game to the children, guide the use and plan activities. Regarding the advantages, the Regarding the context, the educator already carried out technological activities with the children, such as watching videos on YouTube®, searching on Google®, printing images. In the kindergarten classroom, they had a computer, where children could write letters or draw pictures. The educator also promoted digital communication. Children used Messenger® in kindergarten to send emojis to their parents. The educator used Skype® to make video calls with the children, when they were sick at home. They did not have a tablet in the kindergarten room, but the educator recognized that it was the most appropriate and preferred device. Regarding the expectation of use, the educator did not know many learning apps and thought that the Aprender XXI apps were in accordance with the interests of the children and the thematic areas they usually work on. The educator liked all the apps and gave concrete suggestions in some games. One suggestion was for games where the buttons are integrated into the scenario. A greater distinction should be made between buttons that have interaction and scenery props that are only illustrative. It can be a visual cue on the buttons (brightness, animation) or, after some time without action, a help could appear, such as an arrow indicating a path. The educator also had suggestions about the structure and content of the educational guides for parents and educators. It could be a guide per game. Each guide should have an introduction about the game, goals and gameplay, so that the educator understands it first and then can explain the game to the children, guide the use and plan activities. Regarding the advantages, the educator said that this type of activity is very stimulating for children, pleases them and transmits knowledge in the way they like. It also promotes communication and collaboration in a joint activity. As for the constraints, she indicated lack of time, the size of the group and the lack of equipment. This is a very large group (22 children) to give personalized attention in this type of activity, which is something that all the children want to do at the same time. Referring to the devices, she said that due to the results of this activity,

educator said that this type of activity is very stimulating for children, pleases them and

group and the lack of equipment. This is a very large group (22 children) to give personalized attention in this type of activity, which is something that all the children want to she would ask the institution or even the parents, to purchase tablets for the classroom's educational activities.

After the evaluation with users, changes were made to the games and the final versions of the apps were completed. The project ended with the release of the four apps on the App Store® (iOS 8.0) and Google Play® (Android 4.0.3), the online publication of the website with the virtual world that flourishes with the participation of children in apps, and educational guides for parents and educators.

#### **4. Discussion**

Following the DBR methodology, the project Aprender XXI was divided into three phases: preliminary study, development and evaluation. The preliminary study served as a theoretical framework for contextualizing and supporting the project, and to identify the existing needs that helped define the objectives and content of the apps. The search for educational apps suitable for these ages allowed to know the "state of the art" and understand the existing offer in the market that can be used in an educational context, as well as to identify limitations and opportunities for development. To end this phase, Portuguese curriculum guidelines for pre-school education were studied, along with guidelines from other countries. The analysis of data collected in the preliminary study was essential in the development phase—definition of specifications, selection of contents, scriptwriting, design and programing of the games. The theoretical knowledge was integrated in a practical way in the apps, such as opting for open games (e.g., Office Worker) or games that allowed to manipulate several variables (e.g., Meteorology, Mechanic), because they have a greater cognitive and creative contribution, and promote group interaction and communication [23,25]. Games were planned to allow children to experiment different scenarios, to solve challenges (e.g., Builder, Healthy Snacks), to respect rules (e.g., Behavior Board, Playground, Play Areas), providing variety and flexibility of learning [28]. Design decisions considered the literature: use of illustrations and animations that provide context; have a simple interface and interactivity that is easy to learn and use; image-based feedback related to actions [24]. The search for apps in the market showed a lack of apps suitable for educational activities in kindergarten, according to CGPE and in Portuguese language. The content chosen was based on the areas suggested in pre-school education guidelines [29], adapted to games that could be used by educators in a multidisciplinary way, combining, for example, mathematics, geography, citizenship and environment [21].

The kindergarten sessions were important to test the products with the public (children and educator), to verify the need to change the course of development or to continue the implementation according to the plan. The results of the evaluation showed that the apps met the interests of the children, were pedagogically useful and provided a new educational tool for the educator. In this way, the evaluation validated the options that had been taken in the development phase. There were also evaluation results that added knowledge and led to improvements in the games. It became clear the importance of making interactive elements more visible, so lighting effects or animation were added to the buttons feedback, and some objects or outlines were increased. Visual instructions were introduced in all games to guide children in their autonomous use. Some games had specific corrections, as a result of observing their use by the children. Minor bugs were also detected and fixed. These corrections were made, before the release of the final versions of the apps in the Stores, that were linked to the website, with the flower system to visualize the cooperation and participation of children.

#### **5. Conclusions**

This project aimed to develop game-based learning apps for preschoolers, using the DBR methodology to support technological development with scientific research in the area of technologies in education. The information collected in the three phases (preliminary study, development and evaluation), the involvement of the target audience (educator and children) and the triangulation with theoretical studies and existing apps guided the technological development, resulting in robust products and contributions to the theory. Summarizing some principles of this study:


Some limitations to the study are related to characteristics of the DBR methodology and to the development context, a micro company where the researcher is also a project manager. It was necessary to involve and combine several people with different profiles and rhythms—researchers, technological team and target audience [52]. Scientific research requires time to collect and analyze data at each stage. The involvement of the educator and children is conditioned by timetables, school calendars and pedagogical plans, so it was necessary to adjust the research to these restrictions to achieve their participation. The company's team had reduced availability, because it had several projects with tasks and deadlines running at the same time. This resulted in a limitation of the project, which was the impossibility of involving the target audience in all phases, particularly in the preliminary study, to survey the needs. It would be interesting in a later version, to include questionnaires or focus groups involving the target audience from the beginning. Another limitation is the difficulty in generalizing the results, since, in DBR, the sample of participants is not representative of reality, but rather small and intentional [32]. Thus, the products are tested with small groups of users and released on the market. Subsequently, with continued use and new data, they evolve to improved versions. Thus, the design principles presented in this article follow the recommendations of several authors [32,33,53], to be contextualized, so that they can be interpreted, adapted and contribute to knowledge. It would be interesting to repeat the evaluation in other kindergarten classrooms to increase the degree of generalization.

In terms of impact for end users, the results of the evaluation indicate that the products have the potential to promote the learning of pre-school children, since the games have contents covered in kindergarten, the children showed interest in playing the games and the educator identified possible educational activities to do with apps. A more in-depth case study on the use of apps in kindergarten would be interesting to measure learning outcomes and suggest an implementation plan in the classroom, with a certain number of sessions, duration and learning objectives.

As a final note, the researcher considers that the project Aprender XXI is an example of good practice, for a research project developed in a business context in the area of education technologies. The project represented a business effort to reinforce internal R&D skills to create knowledge and greater competitiveness. It stimulated technological experimentation and scientific dissemination led by an SME. The results (four apps in two stores) were able to generate value for the company and integrate an initial internationalization strategy.

**Funding:** This research was funded by the European Union through the European Regional Development Fund, the 2020 Center Program, under the project "Aprender XXI: Game-based m-learning for children in kindergarten", grant number CENTRO-01-0247-FEDER-009828).

**Institutional Review Board Statement:** Ethical approval was not sought for the present study because no more than minimal risk was identified. This study meets the necessary ethical requirements and does not include activities or results that pose safety problems for the participants. Data were properly anonymized and informed consent was obtained at the time of original data collection. Data storage meets current Data Protection regulations.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study, with the assent of children and permission of their parents.

**Data Availability Statement:** Data are contained in supplementary material, available in open access on Zenodo [43,51]. The following supplementary materials are available online: website (http://aprenderxxi.criamagin.com) (accessed on 5 April 2021); App Nature Kids (https://itunes. apple.com/us/app/nature-kids/id1459394172 (accessed on 5 April 2021) e https://play.google. com/store/apps/details?id=com.criamagin.naturakids (accessed on 5 April 2021)); App Healthy Kids (https://itunes.apple.com/us/app/healthy-kids/id1459395328 (accessed on 5 April 2021) e https://play.google.com/store/apps/details?id=com.criamagin.healthykids (accessed on 5 April 2021)), App Citizen Kids (https://itunes.apple.com/us/app/citizen-kids/id1459383445 (accessed on 5 April 2021) e https://play.google.com/store/apps/details?id=com.criamagin.citizenkids (accessed on 5 April 2021)); App Busy Kids (https://itunes.apple.com/us/app/busy-kids/id1459400995 (accessed on 5 April 2021) e https://play.google.com/store/apps/details?id=com.criamagin.busykids (accessed on 5 April 2021)).

**Acknowledgments:** The author thank the reviewers for their insightful comments.

**Conflicts of Interest:** The author declares no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


## *Article* **Mobile App for Science Education: Designing the Learning Approach**

**Rita Tavares 1,\* , Rui Marques Vieira <sup>1</sup> and Luís Pedro <sup>2</sup>**


**Abstract:** This paper reports research work related to a wider study, aimed at developing a mobile app for Science Education in primary-school. Several studies reveal that Science Education can be improved by using technology, namely educational software. However, to promote a structured use of technology, innovative learning approaches must be designed for educational software. This paper aims to answer how the interaction between students and a mobile app for Science Education can promote students' scientific competences development and self-regulated learning. To achieve this, a learning approach was designed, combining the Universal Design for Learning principles, Inquiry-Based Science Education and the BSCS 5E – teaching model for Science Education designed by the *Biological Sciences Curriculum Study*, which results in the acronym of the model. The 5E is related to each phase of the model: Engagement; Exploration; Explanation; Elaboration; Evaluation. The proposed was based on a grounded, participatory, and user-centred approach, crossing literature contributions with data collected among primary-school teachers through the application of a questionnaire (*n* = 118). Data collected allowed deductions about the expected adequacy of the learning approach, according to Nieveen's criteria for high quality educational interventions. This adequacy was revealed through the teachers' conceptions about the potential impact of the conceptualized mobile app (i) to provide a comprehensive and practical Science Education learning; and (ii) to enhance students' scientific competences development and self-regulated learning. The paper aims to contribute to the design of an innovative learning approach in Science Education and to share it with other researchers since it can be expanded to other educational software.

**Keywords:** mobile application; Science Education; learning approach; scientific competences; Universal Design for Learning; Inquiry-Based Science Education; BSCS 5E; Educational Data Mining

#### **1. Introduction**

The importance of Science Education is increasing within a more democratic, informed and enabled society, faced with the great challenges that the technological world brings to new generations [1–7]. For this reason, Science Education is advocated from the early years, especially in primary schools. Some of the main reasons pointed out for this are its potential (i) to help students to develop (new) ideas and to "make sense of the world", from what they hear and see in their daily lives; (ii) to promote opportunities for the clarification of students' (pre-)concepts and to confront these with scientific evidence; (iii) to avoid students' belated conceptual change, by testing their ideas through scientific experiences; and (iv) to promote and enhance positive attitudes towards Science, extending and/or amplifying the number of students who will pursue scientific careers, and, among others, decreasing the problem of the growing lack of girls' interest in Science [2,3,6,8,9]. In this regard, almost all European countries propose as main objectives (a) to improve students' understanding regarding the application of Science in real life; and (b) to strengthen Science

**Citation:** Tavares, R.; Marques Vieira, R.; Pedro, L. Mobile App for Science Education: Designing the Learning Approach. *Educ. Sci.* **2021**, *11*, 79. https://doi.org/10.3390/ educsci11020079

Academic Editor: Diego Vergara Received: 3 December 2020 Accepted: 1 February 2021 Published: 18 February 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Education in schools in order to increase recruitment in areas such as Mathematics and Technology [4].

For the last 30 years, several authors and organizations have highlighted the importance of a deeper understanding of the potential and/or effectiveness of the different Science Education learning approaches, for instance, in students' scientific competences development [10,11]. Furthermore, in the last decade, many European countries have been promoting several actions and efforts to integrate Science Education in a more contextualized way, e.g., by supporting the development and maintenance of international networks and databases to share and provide free (digital) educational resources and practices [10]. Despite these initiatives, Science Education is far from assuming the same importance as such disciplines as mother tongue and Mathematics in primary school [4].

In European countries Science Education is taught as one general integrated subject in primary-school, based on the broad acceptance that in real life knowledge and practice are not split. This approach highlights the integration and iteration between knowledge and practice, between theory and action, purporting a meaningful and contextualized knowledge construction practice. This comprehensive approach relates scientific concepts to other disciplinary contents/subjects and can help the students to develop logical reasoning, critical thinking, and an integrated and extended knowledge of reality [11]. This can also potentially enhance students' interest, stimulating new ideas, questions, and the understanding of (new) complementary concepts based on personal experience and/or real situations.

Several studies reveal that Science Education can be improved by using technology, namely personal computers, smartphones, tablets and different types of educational software [12–19]. The increasing usage of technological devices is an international trend, underlined in several reports of the development and usage of digital educational resources such as mobile applications (mobile apps) [17,20–22]. In the last OECD report related to innovation in education, the importance of these devices in Science Education is underlined, namely to enhance the development of students' content and procedural knowledge [23].

In line with the above mentioned, this paper reports research work related to a wider study aimed at developing a mobile app for Science Education in primary-school. One comprehensive research question has been designed: *Which type of mobile app can promote primary-school students' scientific competences development and self-regulated learning?* To answer this, eight additional questions have oriented the wider study. The present paper is focused on one of those eight additional questions, related to the interaction between the students and the mobile app and how that can promote students' scientific competences development and self-regulated learning: *How can students interact with the mobile app and how can the mobile app respond in real time to students' interactions, simulating the teaching and learning process and promoting the students' scientific competences development and self-regulated learning?* To answer this question, a learning approach was designed, combining the Universal Design for Learning (UDL) principles, Inquiry-Based Science Education (IBSE) and the BSCS 5E (5E) – teaching model for Science Education designed by the Biological Sciences Curriculum Study, which results in the acronym of the model. The 5E is related to each phase of the model: Engagement; Exploration; Explanation; Elaboration; Evaluation. This approach was designed and validated by crossing literature contributions with data collected among primary-school teachers through the application of a questionnaire *(n* = 118). Among other aspects not reported in this paper, the questionnaire application allowed deductions about the expected adequacy of the proposed learning approach.

Our study aimed (i) to contribute to the design of innovative learning approaches [3,5,23] by combining the UDL principles, the IBSE and the 5E; and (ii) to contribute to (research in) Science Education by proposing a learning approach that can be expanded and/or applied to other digital educational solutions besides mobile apps, aiming at (a) to facilitate approaches to scientific concepts/topics/phenomena; and (b) to promote students' scientific competences development: scientific knowledge, skills and attitudes, with particular focus on self-regulated learning.

Because our research was developed in Portugal, based on the most recent published data, our study also aimed to contribute to enhance and promote innovative practices in Portuguese primary schools, regarding the following reported aspects [23]:


In the following sections, the authors present how the study was implemented (Section 2) and how it can contribute to the above aspects, namely by proposing an innovative literature-based learning approach (Section 3.1); and presenting the expected adequacy of the integration of the proposed digital educational resources in the mobile app (Section 3.2); the potential impact of the conceptualized mobile app (Section 3.3); and the expected adequacyof the proposed learning approach in promoting scientific competences development (Section 3.4).

#### **2. Materials and Methods**

Since the wider study aimed at the development of a mobile app according to the future end-users' needs and expectations, a participatory and user-centred design approach was adopted. At the same time, the mobile app development was research-based.

For this mixed approach, a research plan was conceptualized involving (i) scientific knowledge deepening; (ii) the collaboration between researchers, experts and future endusers; (iii) mixed methods; and (iv) interactive, cyclic and flexible phases of analysis, design, development, implementation, evaluation and revision.

This research plan resulted in a participatory framework proposal nested within the larger framework of Educational Design Research [24]. *Educational Design Research* revealed to be the most adequate methodological approach to the study rationale since it [25,26]:


In line with the above, the study fits in with this methodological approach by:


The present paper reports on the Preliminary research focused on the analysis and design of the proposed mobile app (see Figure 1), presenting and discussing a literaturebased learning approach proposal and data collected among primary-school teachers through the application of a questionnaire (*n* = 118).

**Figure 1.** Conceptual scheme of the wider study.

Data collected with the questionnaire application was used in other moments of the wider study (e.g., to define the mobile app's target audience, i.e., 4th grade primary-school; to define the topic to be approached in the mobile app, i.e., the Human Body). In the study, data collected via the questionnaire allowed deductions about the expected adequacy of the proposed learning approach and the potential impact of the conceptualized mobile app.

Since the mobile app's learning approach is quite comprehensive, for its design four correlated moments within the Preliminary research were considered:


preliminary proposal of the conceptual EDM framework for Science Education – *Relational structure: questions and events that result in the conceptual EDM framework for Science Education* [28].

All the study phases were implemented in Portugal, the country where the authors developed their research. For this reason, the questionnaire and the original data are only available in Portuguese [27,29].

#### *2.1. Participants*

Since teachers represent one of the most legitimate and reliable sources of information regarding the teaching and learning process, to ground and validate the proposed learning approach the authors opted to involve teachers in the Preliminary research reported in this paper.

Since the aim was to survey primary-school teachers' conceptions about their knowledge and their educational practices in Science Education using digital educational resources, the study sample was a convenience sample [29,30]. This option was to assure that most of the teachers answering the questionnaire used technological resources to teach Science Education. The study's population profile was the following: *Primary-school teachers that use and/or frequently interact with digital educational resources to teach Science Education.*

To select our sample, the authors first surveyed teachers with the mentioned profile. For this, Portuguese open access repositories dedicated to Science Education on the web that included registered primary school teachers were searched. From the search, the authors were able to find an open access repository in Portugal with these requirements: *House of Sciences* – originally *Casa das Ciências* [10].

This repository delivers digital educational resources in subjects such as Introduction to Science, Biology, Physics, Geology, Mathematics and Chemistry, allowing teachers to upload, share, access and/or download the following digital educational resources: animations, simulations, videos, interactive presentations, games, interactive whiteboard resources, documents, and activities exploration guides. According to the House of Sciences stakeholders, at the time of the survey the repository had 1046 primary-school teachers registered. From those, 118 primary-school teachers answered the questionnaire.

#### *2.2. Data Collection*

The paper reports on the Preliminary research, focused on analysis of the mobile app's requirements and design, presenting and discussing a proposal for a literaturebased learning approach and data collected among primary-school teachers through the application of a questionnaire (*n* = 118).

The questionnaire was a multidimensional and *a priori* structured instrument, crossing four instruments validated and implemented in national and international settings. The authors adapted and adopted ten items and conceptualized another six. For this, the following stages were considered: (1) design of the pilot version of the questionnaire; (2) validation of the pilot version (3) implementation of the pilot version; (4) analysis of the data gathered; (5) design of the final version of the instrument; and (6) implementation of the final version.

Regarding the adapted item, the following instruments were used: Avaliação do Impacte Programa de Formação de Professores do 1.º Ciclo do Ensino Básico em Ensino Experimental das Ciências nas práticas docentes de ensino experimental (Evaluation of the Impact of the Primary-School Teachers Training Programme in Experimental Science Teaching on the Teachers' Practices) [31]; and TIMSS 2015 Grade 4 Teacher Questionnaire [32]. For the adopted items, the following instruments were considered: Self-Efficacy Teaching and Knowledge Instrument for Science Teachers [33]; and Survey of Preservice Teachers' Knowledge of Teaching and Technology [34]. A formal consent was requested of the instruments' authors, both to translate and to apply the adopted items.

According to the proposed and adopted methodology presented in Section 2 and Nieveen's criteria [35], in the Preliminary research teachers' conceptions provided as-

sumptions about the expected adequacy of the proposed learning approach regarding its (a) consistency; (b) expected practicality; and (c) expected effectiveness (see Table 1).


**Table 1.** Nieveen's criteria for high quality educational interventions [35] (p. 94).

The questionnaire's application was authorized and supported by House of Sciences stakeholders, by sending an e-mail containing the invitation to participate in the study and the hyperlink to the questionnaire to all the possible participants (*N* = 1046). In this way, it was assured that only primary-school teachers registered in the repository participated in the study, and allowed to control the number of e-mails read to reinforce the invitation if needed.

The instrument was implemented using the *University of Aveiro Questionnaires* platform [27] and it was available for 38 days. Besides the first e-mail containing the invitation sent on 27 February 2017, the House of Sciences stakeholders sent two more e-mails to reinforce the participation on March 13th 2017 and on April 3rd 2017.

The instrument was validated by five external experts and by the *Portuguese School Surveys Monitoring* team – originally *Monitorização de Inquéritos em Meio Escolar* (MIME). The preliminary pilot version of the instrument was sent to the external experts, who analysed it with a qualitative approach. Based on their appreciation, a convergence analysis was adopted [36]. Considering the consensus of comments, suggestions, points of view and ideas pointed out by the experts, the final pilot version of the instrument was generated and submitted for the MIME's validation and approval to be implemented in a school setting. Once approved, the questionnaire was implemented among a random sample of primary school teachers, according to the defined profile (*n* = 17).

The final version of the questionnaire comprised nine questions (Q). On the first page, the study and some clarifications about the instrument were presented, such as the normal duration of time to fill it in (around eight minutes). The second page was related to participants' informed consent and agreement to participate in the study. The third page comprised two questions, one Likert scale with eight items related to the teachers' knowledge and educational practices in Science Education (Q1); and one dichotomous "Yes/No" question asking the participants if they used digital educational resources to teach Sciences (Q2). By answering "Yes" the participants proceeded with the remaining questions (pages four and five), and by answering "No" the participants move to Q7 (page five).

The fourth page had four questions, all closed-ended one-choice or multiple-choice questions: Q3 related to the frequency of usage of digital educational resources; Q4 related to which school grades were most privileged to use them; Q5 related to the most used digital educational resources; and Q6 related to the usage of digital educational resources in Science Education (e.g., explore concepts/topics using games).

Finally, the fifth page comprised three questions. The first two were closed-ended multiple-choice questions: Q7 related to the Experimental Science Education topics most commonly explored by teachers, and Q8 related to the two topics most easily explored using digital educational resources. The last question (Q9) was an open-answer question related to the potential of the conceptualized mobile app.

Because the questionnaire was implemented in Portugal, the adopted items of the Q1 scale [36,37] were translated into Portuguese. To assure the correct version of the translation, the items were submitted to a process of translation and back translation, ensured by two external experts. To assure the internal consistency and reliability of the questionnaire, as mentioned, a pilot version of the instrument was applied to 17 primary-school teachers, allowing its validation. For both the pilot and the final version of the instrument, the adequacy of the sample, the internal consistency, and the reliability was verified [37–45] (see Table 2).


**Table 2.** Measures of the internal consistency and reliability of the questionnaire.

\* *p* < 0.01.

In the present paper, to deduce the expected adequacy of the proposed learning approach, revealed through the teachers' conceptions about the potential impact of the conceptualized mobile app (i) to provide comprehensive and practical Science Education learning; and (ii) to enhance students' scientific competences development and self-regulated learning, from the nine questions available in the questionnaire, data from the teachers' answers to Q5, Q6, and Q9 were considered:


By asking the participants if they use digital educational resources to teach Science (Q2) the sample was split into two independent groups: group one–*Primary-school teachers that use digital educational resources to teach Science,* and group two–*Primary-school teachers that do not use digital educational resources to teach Science.* Q2 data analysis, verified that only 20.3% of the teachers did not use digital educational resources. This result confirmed the adequacy of the convenience sample, and collected significant data from which to infer the expected adequacy of the proposed learning approach.

From 230 answers to the questionnaire, 78 were incomplete and 34 were not properly saved in the *University of Aveiro Questionnaires* platform due a system failure. For these reasons, 118 complete and valid answers were considered, which will be analysed in the following sections for the three questions.

#### *2.3. Data Analysis*

For Q5 and Q6–closed-ended multiple-choice questions–descriptive statistical analysis was applied, using the software SPSS Statistics 24® (IBM Corporation, Armonk, NY, USA). Descriptive statistical analysis was conducted to deduce (i) the expected adequacy of the integration of the proposed digital educational resources in the mobile app; (ii) the potential impact of the conceptualized mobile app; and (iii) the expected adequacy of the proposed learning approach to promote scientific competences development.

For Q9–open-answer question–content analysis was conducted using the software webQDA® (Universidade de Aveiro, CIDTFF–Research Centre on Didactics and Technology in the Education of Trainers, Micro I/O, and Ludomedia). Content analysis was conducted to deduce the potential impact of the conceptualized mobile app and its potential future usage, namely regarding the expected adequacy of the proposed learning approach. For this, a deductive system of categories was designed to analyse the teachers' answers. The system of categories was designed according to the "User Experience Honeycomb" [46], reflecting the theoretical frameworks adopted (UDL, IBSE and 5E) and the proposed learning approach (see Table A1–Appendix A).

The system of categories was framed in the content analysis software by creating a project with the defined categories organized in a coding scheme, and with the data sources: teachers' answers to Q9 organized by identifiers (ID) [47]. The IDs were analysed according to the categories. To assure the adequacy of the system of categories and the coded IDs, clone versions of the project were submitted for experts' validation. For this, two external experts coded 10% of the coded ID selected randomly. Then, the coded ID were crossed and the reliability (*r*) of the system of categories was calculated according to the following [48]: *r* = *Ta* (*Ta*+*Td*) , where *TA* represents the total of agreements and *Td* the total of disagreements between the researchers' coded ID and the experts. By applying this validation approach, the system of categories' reliability was verified (*r* = 0.73). Although the value is low [44,45,49–51], according to DeVellis [51] in some Social Sciences studies, namely those with a small sample such as the present one, reliability values from 0.6 can be considered as acceptable. For this reason, *r* = 0.73 was considered acceptable to proceed with the system of categories application.

To deduce the expected adequacy of the proposed learning approach, aspects related to the *User Experience* –*Valuable* subcategory and the aspects related to the *Scientific Competences* category were analysed, finding for both a total of 387 references. The original references were translated into English, bearing in mind textual coherence. Although some of the translated sentences suffered small adjustments, semantic, idiomatic, cultural, and conceptual equivalence were preserved.

#### **3. Results**

#### *3.1. Literature-Based Learning Approach Proposal*

The proposed learning approach designed for the mobile app combines the Universal Design for Learning (UDL) principles, the Inquiry-Based Science Education (IBSE) and the BSCS 5E (5E). UDL is an educational framework focused on the development of learning environments, designed to meet individual learning differences and to promote and facilitate the learning process [52,53]. Therefore, UDL proposes that the curriculum must be designed to promote equal learning opportunities, since information access is not enough for students' knowledge development [54]. Knowledge construction will depend on several aspects, such as learning goals, teaching and learning approaches, educational resources adopted and learning assessment methodology [53]. In this regard, UDL sets three main principles presented as follow according to the study scope [53–55]:


Regarding the theoretical frameworks adopted towards the teaching and learning process simulated by the mobile app, in the last decade several authors have implemented the IBSE approach according to the 5E inquiry curriculum model [1,56,57]. Both approaches propose five highly related phases, adopted in the study according to two complementary points of view:

	- Orientation phase: to stimulate students' curiosity about a certain scientific concept/topic;
	- Conceptualization phase: to confront students' (pre-)concepts and/or inquire about them, and to promote the generation of new ideas/assumptions;
	- Investigation phase: to lead the students to plan and apply investigation processes (e.g., collect, analyse and interpret data to test the assumptions);
	- Conclusion phase: to lead the students to draw conclusions by comparing/confronting their (pre-)concepts with new evidence;
	- Discussion phase: to confront students' ideas and/or results, promoting a reflection and learning process (self-)evaluation - this phase is adopted as transversal to the previous phases).
	- Engage phase: to stimulate students' interest and promote their personal and active learning involvement;
	- Explore phase: to lead the students to build their own understanding about concepts/topics, by confronting and experimenting with scientific phenomena;
	- Explain phase: to promote the students' opportunity to communicate their knowledge/findings and to establish a theoretical framework;
	- Elaborate phase: to lead the students to apply their (new) knowledge, deepening scientific concepts/topics and/or proceeding towards new learning paths;
	- Evaluate phase: to help students to develop self-awareness about their learning path and about their knowledge construction (this phase is adopted as transversal to the previous phases).

Besides the intrinsic relationship between both approaches found in the literature and the proposed complementary points of view endorsed for each one, their simultaneous adoption aims to address the importance of comprehensive and practical activities in the development of scientific competences [2,58–60]. Thus, the proposed mobile app first allows the students to contact (new) scientific concepts/topics and/or confront their previous knowledge, hoping that such confrontation will allow them to become actively engaged in reflective, exploratory and (self-)evaluative activities. For this, the mobile app integrates five types of digital educational resources, organized according to learning sequences (the mobile app games levels) and three learning management components.

Both the learning sequences and the learning management components are supported by an Educational Data Mining (EDM) framework integrated in the mobile app, according to the adopted methods and techniques. So that the mobile app can provide formative feedback, recommendations and real-time help and tailored to the students' needs, the app will need to provide (in terms of computer programming) a structure that allows the system (the mobile app) to read every single interaction between students and the mobile app (and *vice versa*). For this, as presented in Figure 1, in the Preliminary research the authors have also defined which questions should be "asked" of the system and which events should be read and analysed through the EDM methods and techniques. The set of questions and events resulted in the so-called *Relational structure: questions and events that result in the conceptual EDM framework for Science Education* [28]. The *Relational structure* is based on the learning approach proposed, relating it to the possibility of the mobile app (i) to assess in real-time students' performance levels; (ii) to identify in real-time difficulties experienced by the students; and (iii) to guide students in real-time students along the most adequate learning path [61].

The EDM framework and its influence on the availability of digital educational resources and learning management components aims to simultaneously promote the students' development of scientific competences–scientific knowledge, skills and attitudes– and self-regulation with regard to attitudes [1,62–67] (see Figure 2). Thus, the mobile app includes several game levels related to different scientific concepts/topics. Each level supports a learning sequence organized according to the following digital educational resources: animation, game, simulation, quiz, and information areas. The integration of these resources reflects the relationship between the UDL principles and the theoretical frameworks adopted [68–77] (see Table 3).

**Figure 2.** Proposed learning approach scheme.

**Table 3.** Relationship between the five types of digital educational resources and the three learning management components integrated in the mobile app with the UDL principles, the IBSE and the 5E approaches, and the student's scientific competences.



**Table 3.** *Cont.*


**Table 3.** *Cont.*

The integration of animations aims at the representation of scientific information, enhancing the introduction and presentation of scientific concepts/topics. By integrating games, the authors intend to provide means of engagement, expression, and interaction with the information, enhancing the exploration of scientific concepts/topics. The simulations intend to be a means of expression and interaction with the information, enhancing the application of scientific knowledge and skills. By integrating the quizzes, the authors aim to provide a means of knowledge expression, enhancing the deepening of scientific knowledge and skills. Finally, by integrating information areas the authors intend to provide means of scientific information representation, enhancing the deepening of scientific knowledge and skills, and helping the students to proceed in their learning path.

As referenced, besides the integration of digital educational resources, the mobile app includes three components related to the learning management process: available help, formative feedback, and recommendations. As mentioned, these components are related to the authors' proposal of an EDM framework for Science Education integrated into the mobile app, aiming simultaneously at students' development of scientific competences, and at students' self-regulation: reflection, (self-)evaluation and self-awareness [28]. The focus on self-regulation aims that the students are able to (i) to identify personal interests and learning needs; (ii) set learning objectives and pathways according to personal interests and needs; and (iii) search for personal skills consolidation and deepening learning opportunities.

Regarding the IBSE and the 5E approaches, the mobile app game levels (the learning sequences) and the learning management components allow the students to go through the five phases of the adopted approaches by exploring scientific concepts/topics of introduction, exploration, application and deepening activities. Each level is related to a specific scientific concept/topic which means that the level related to "A" comprises one animation, one game, one simulation, one quiz, and information areas related to "A"; the level related to "B" comprises one animation, one game, one simulation, one quiz, and information areas related to "B"; and so on.

According to the proposed learning approach scheme (see Figure 1) and the previous examples, for instance, the students can begin to explore game "A", continue to animation "A", browse an information area "A", answer quiz "A" and explore the simulation "A2; or the students can run each learning sequence: animation "A" → game "A" → simulation "A" → quiz "A" → information areas "A". The mobile app is set to propose this last sequence of exploration, so the student can go through the theoretical learning structure proposed. However, to allow the students to set learning objectives and pathways according to personal interests and needs, and to promote the students' development of self-awareness about their learning path and their knowledge construction (5E Evaluate phase), the mobile app allows students to explore each learning sequence both linearly and non-linearly. To ensure that all the phases are completed, students can only advance to the next level (learning sequence) when they complete the previous one.

Thus, either exploring the mobile app linearly and/or non-linearly, students will have the opportunity to develop scientific knowledge (factual, conceptual and procedural), scientific skills and attitudes by exploring the five types of digital educational resource. In addition and simultaneously, the three learning management components will ideally help to promote the development of students' scientific competences and self-regulated learning, since the mobile app (i) gives the students the opportunity to choose what digital educational resources are more suitable for their learning path, and so to personalize it; and (ii) supports students' digital educational resources exploration by giving them real time formative feedback and recommendations, and identifying when they need help (e.g., to propose that the students access available help to solve a problem).

In the following section, data analysis from primary-school teachers' answers to the questionnaire (*n* = 118) detail the expected adequacy of the proposed learning approach.

#### *3.2. Expected Adequacy of the Integration of the Proposed Digital Educational Resources in the Mobile App*

For Q5 and Q6 descriptive statistical analysis was applied, calculating percentages according to the total sample.

As presented above, the proposed learning approach provides learning sequences–the mobile app game levels. Each level has a set of five correlated digital educational resources: an animation, a game, a simulation, a quiz and information areas. To deduce the expected adequacy of the integration of these types of digital educational resources in the mobile app, the teachers' answers to Q5 were analysed: *From the following options, please select the digital educational resources you frequently use in Science lessons using computers (including tablets).* From the listed resources, the most frequently used by teachers in their Science lessons were animations (68.6%) and games (55.9%) (see Figure 3).

**Figure 3.** Primary-school teachers' conceptions about the digital educational resources most frequently used in their Science lessons using computers (including tablets) (*n* = 118).

To deduce the expected adequacy of the usage of the listed digital educational resources to introduce, explore, apply, and/or deepen scientific concepts/topics, data gathered from Q6 was analysed: *From the following options, please tell us how you use digital educational resources in Science lessons?* The analysis demonstrated that teachers used mainly

animations to introduce scientific concepts/topics (57.6%); simulations to explore scientific concepts/topics (45.8%); games to apply scientific concepts/topics (52.5%); and simulations to deepen scientific concepts/topics (39%) (see Figure 4.)

**Figure 4.** Primary-school teachers' conceptions about how they use digital educational resources in Science lessons (*n* = 118).

Finally, the proposed integration of digital educational resources in the mobile app, aiming at scientific concepts/topics' introduction, exploration, application and deepening according to UDL principles (see Table 3), was matched with the collected data (see Table 4).

**Table 4.** Proposed integration of the digital educational resources vs. Teachers' conceptions about the expected adequacy of the usage of the listed digital educational resources (*n* = 118).


#### *3.3. Potential Impact of the Conceptualized Mobile App*

Regarding the *User Experience* –*Valuable* subcategory, data analysis demonstrated that most of the teachers privileged a mobile app providing digital educational resources correlated with the possibility of implementing *Orientation* and *Engage* phases, and to promote scientific information *Representation* (66 references registered). The teachers' answers revealed the following aspects related to the potential of the mobile app to:


For the aspects related to the potential of the mobile app to implement *Conceptualization* and *Explore* phases, and to promote *Engagement, Expression,* and interaction with scientific concepts/topics, 29 references were registered in teachers' answers. The references emphasized aspects related to the potential of the mobile app to promote:


Analysing data for aspects related to the potential of the mobile app to implement *Investigation* and *Explain* phases, and to promote multiple ways of *Expression* of scientific competences and interaction with scientific concepts/topics, 39 references were registered in teachers' answers. The references emphasized aspects related to the potential of the mobile app to help the students and/or the teachers:


For the aspects related to the potential of the mobile app to implement *Conclusion* and *Elaborate* phases, and to promote multiple ways of *Expression* of scientific competences, 20 references were registered in teachers' answers. The references emphasized aspects related to the potential of the mobile app to help the students:


Analysing data for aspects related to the potential of the mobile app to implement *Discussion* and *Evaluate* phases, and to promote multiple ways of *Engagement* with scientific concepts/topics, six references were registered in teachers' answers. The references emphasized aspects related to the potential of the mobile app to help the students:


#### *3.4. Expected Adequacy of the Proposed Learning Approach to Promote Scientific Competences Development*

Regarding the *Scientific Competences* category, data analysis demonstrated that most of the teachers consider that the use of a mobile app integrating correlated digital educational resources could promote *students' Scientific Knowledge* development (47 references

registered). The references emphasized aspects related to the potential of the mobile app to promote:


For aspects related to the potential of the mobile app to promote students' *Scientific Skills* development, 16 references were registered in teachers' answers. The references emphasized aspects related to the potential of the mobile app to promote:


Analysing data for the aspects related to the potential of the mobile app to promote students' *Scientific Attitudes* development, 40 references were registered in teachers' answers. The references emphasized aspects related to the potential of the mobile app to enhance students':


#### **4. Discussion**

Data analysis allowed inference about the expected adequacy of the proposed learning approach and its potential to promote a comprehensive and practical approach in Science Education, by allowing exploration of correlated digital educational resources, namely animations, games, simulations, quizzes and information areas. Furthermore, the teachers' answers also allowed inferences about the expected adequacy of the proposed learning approach to promote the student's scientific knowledge, skills and attitudes development.

First, the study demonstrated that most of the teachers adopting technology in their Science lessons used mainly animations (68.6%) and games (55.9%) (see Figure 2). The fact that only 38.1% of the teachers selected simulations from the options available in Q5 could be an indicator of the low availability of this typology in the Portuguese language (develop and/or adapted), as well as the low availability of simulations for primary school grades at the time of data gathering [78,79]. Since quizzes are a game typology, in Q5 and Q6 this typology was considered as a game option [72,80–83]. Regarding information areas, since they are not very common, they were not presented in the questionnaire. In this regard, it is important to refer to the fact that, within the Preliminary research, between 1 September 2015 and 18 September 2017 a survey was performed a survey related to the state of the art of mobile apps for Science Education, among others, (i) for primary-school students; (ii) developed by Portuguese stakeholders; (iii) available in Portuguese language; (iv) including (correlated) digital educational resources such as animations, games, simulations, quizzes, and information areas. The survey found nine mobile apps with those features, four from those from the same collection/stakeholder providing information areas. Another piece of useful evidence is the fact that, despite all the digital educational resources that *House of Sciences* provides, information areas are not available. This means that the study sample was not familiar with this type of digital educational resource. Instead, it was familiar with animations, simulations, videos,

interactive presentations, games, interactive whiteboard resources, documents and activity exploration guides.

Regarding the use of digital educational resources in Science lessons, the study also demonstrated that most of the teachers used animations and games to introduce and apply scientific concepts/topics, respectively (see Figure 3). When matching teacher conceptions with the leaning approach proposal, although most of the teachers used simulations to explore and to deepen concepts/topics in their Science lessons, the verified frequency of usage was below 50% of the total sample (see Table 4). Thus, games typology was considered as a means to explore scientific concepts/topics, as proposed in the designed learning approach, and quizzes and information areas to deepen scientific concepts/topics. It is also important to mention that, at the time of data gathering, most of the available simulations in the Portuguese language and for primary school grades were similar to games [78,79]. In this regard, by choosing to maintain the proposed learning approach, the mobile app, simultaneously, allows diversification of the typology of digital educational resources available, and allows the students to use the most appropriate resources for each one of the possibilities proposed: to introduce, explore, apply, and deepen scientific concepts/topics.

Whether using technology in their Science lessons or not, data analysis demonstrated that most of the teachers expressed that they privileged a mobile app integrating correlated digital educational resources (i) to promote the students' orientation and engagement in the learning process, and (ii) to (re)present scientific information (see Section 3.3). Data analysis also demonstrated that most of the teachers privileged the use of a mobile app like the proposed one to promote students' scientific knowledge and attitudes development (see Section 3.4). Since references to all the defined categories and subcategories were found, data analysis predicted the expected adequacy of the proposed learning approach regarding the following: (i) integration of correlated digital educational resources in the mobile app; (ii) adoption of the UDL principles; and (iii) the simultaneous adoption of the IBSE and 5E theoretical frameworks from two related points of view: the teachers' role in the learning process simulated by the mobile app (IBSE) [28,84] and the students' learning process when interacting with the mobile app (5E) [30,62]; providing a comprehensive and practical Science Education approach and facilitating students' scientific competences development and self-regulated learning. In this regard, the proposed learning approach revealed itself to be adequate in providing a comprehensive and practical Science Education learning tool, and enhancing students' scientific competences development and self-regulated learning [2,58–60].

Finally, by crossing the theorized aspects with data collected, the expected adequacy of the proposed learning approach according to teachers' conceptions could be deduced. Therefore, the participatory and user-centred design approach allowed to confront and ground the literature-based rationale of the proposed learning approach. Data analysis and its impact detailed in the study aimed to share the foundations of the proposal with other researchers and enhance the importance of the design and (preliminary) validation of technological educational solutions among future end-users. Furthermore, the proposed learning approach and the adopted method to design and demonstrate its potential could represent a contribution to the development of innovative learning approaches in (Science) Education, aiming at students' engagement and helping them to deepen, understand and develop (new) competences.

The study presented minor limitations. Besides the limitation mentioned in Section 2.2 related to the fact that from the 230 answers to the questionnaire, 78 were incomplete and 34 were not properly saved in the *University of Aveiro Questionnaires* platform due a system failure, the authors had to wait 23 days for formal consent to adapt to Portuguese language items of one of the instruments. Three e-mails had to be sent to reinforce the request. This constraint resulted in a one-month delay in the following processes: (i) translation of the adopted items; (ii) backtranslation of the adopted items; (iii) questionnaire pilot version design; (iv) questionnaire pilot version validation; and (v) questionnaire pilot version implementation in the *University of Aveiro Questionnaires* platform.

The authors support the idea that the learning approach proposal can contribute to (research in) Science Education by proposing the combination of the UDL principles, the IBSE and the 5E. From data analysis, the authors also support the idea that the proposed approach can (a) facilitate scientific concepts/topics/phenomena approaches; (b) represent an opportunity for students to explore scientific contents/topics in an organised, comprehensive, and practical approach; (c) promote students' scientific competences development (scientific knowledge, skills and attitudes); and (d) facilitate students' self-regulated learning. Since the proposed learning approach is quite comprehensive, it could be expanded to other educational software. With this paper, the authors also aim to contribute to Education Science by providing a validated questionnaire aimed at surveying primary-school teachers' conceptions about their knowledge and educational practices in Science Education using digital educational resources.

With wider study, among other aspects, the authors aim at to contribute to Education Science by (a) providing a participatory framework proposal for guiding researchers through an educational mobile app development [24], representing an opportunity for researchers in Education and Multimedia (in Education) to develop educational software based not only on the state-of-the-art, literature in the area and their own rationales, but also on the users' perceptions, ideas and needs, and on experts' validation; and (b) deepening knowledge in the area of EDM and proposing a relational structure [28] and a conceptual EDM framework developed for the mobile app, both extendable to other Science Education software, representing a new perspective on technology enhanced learning and on how to extract valuable educational data to guide students' scientific competences development and self-regulation of learning, as well as to help teachers and researchers to understand and support students in those processes.

Future work is related to the mobile app development, testing and implementation in a school setting, to investigate the adequacy of the learning approach, namely the actual practicality and the actual effectiveness. By validating practicality, the authors intend to understand if the mobile app is usable in the settings for which it has been designed and developed. Finally, by validating its actual effectiveness, the authors intend to investigate mobile app usage and relevance according to the desired outcomes: students' scientific development and self-regulated learning.

**Author Contributions:** Conceptualization, R.T.; Methodology, R.T.; Formal analysis, R.T.; Investigation, R.T.; Data curation, R.T.; Writing—original draft preparation, R.T.; Writing—review and editing, R.T., R.M.V. and L.P. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by *Fundação para a Ciência e a Tecnologia* – FCT I.P. – Portugal, grant number PD/00173/2014 within the *PhD Program Technology Enhanced Learning and Societal Challenges*; and SFRH/BD/107808/2015 under the *Human Capital Operational Program*, supported by the *European Social Fund* and national funds of the *Ministry of Science, Technology and Higher Education*.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study. All the participants have been informed about the aim of this study and of the use of data collected.

**Data Availability Statement:** The anonymized data is available open access on *figshare*® [29]. This study meets the necessary ethical requirements and does not include activities or results that pose safety problems for the participants. All data were treated confidentially, and the participants were treated anonymously in the study.

**Acknowledgments:** The authors thank the reviewers for their insightful comments.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **Appendix A**

**Table A1.** Deductive System of Categories: potential and future usage of the mobile app regarding the expected adequacy of the proposed learning approach.



#### **Table A1.** *Cont.*

#### **References**


## *Article* **Application of Virtual Reality in Computer Science Education: A Systemic Review Based on Bibliometric and Content Analysis Methods**

**Friday Joseph Agbo 1,\* , Ismaila Temitayo Sanusi <sup>1</sup> , Solomon Sunday Oyelere 2,\* and Jarkko Suhonen <sup>1</sup>**


**Abstract:** This study investigated the role of virtual reality (VR) in computer science (CS) education over the last 10 years by conducting a bibliometric and content analysis of articles related to the use of VR in CS education. A total of 971 articles published in peer-reviewed journals and conferences were collected from Web of Science and Scopus databases to conduct the bibliometric analysis. Furthermore, content analysis was conducted on 39 articles that met the inclusion criteria. This study demonstrates that VR research for CS education was faring well around 2011 but witnessed low production output between the years 2013 and 2016. However, scholars have increased their contribution in this field recently, starting from the year 2017. This study also revealed prolific scholars contributing to the field. It provides insightful information regarding research hotspots in VR that have emerged recently, which can be further explored to enhance CS education. In addition, the quantitative method remains the most preferred research method, while the questionnaire was the most used data collection technique. Moreover, descriptive analysis was primarily used in studies on VR in CS education. The study concludes that even though scholars are leveraging VR to advance CS education, more effort needs to be made by stakeholders across countries and institutions. In addition, a more rigorous methodological approach needs to be employed in future studies to provide more evidence-based research output. Our future study would investigate the pedagogy, content, and context of studies on VR in CS education.

**Keywords:** computer science education; virtual reality; VR; content analysis; bibliometric analysis; immersion; 3D simulation; presence; game-based learning

#### **1. Background of the Study**

Virtual reality (VR) has recently become a popular technology in different contexts such as entertainment, military, and education [1]. VR combines technologies to provide an immersive presence through highly interactive objects in a virtual environment but stimulates users' sensory awareness to perceive being in an almost natural environment. The use of VR in education to support training, teaching, and learning through 3D simulation and visualization of learning content in a virtual presence has grown recently [2]. This increasing VR application growth in the educational field is evident, as revealed by the literature, including a recent VR study in computer science education [3]. VR technology provides an opportunity to develop a state-of-the-art smart learning environment with a high level of interaction, engagement, and motivation for an enhanced learning experience [1–8]. This study refers to computer science (CS) education as the art and science involved in learning and teaching computer science, including computing, algorithmic and computational thinking [9]. For example, the science behind curriculum design, pedagogical approach, and instructional tools and techniques educators adopt to support computer science teaching and learning.

**Citation:** Agbo, F.J.; Sanusi, I.T.; Oyelere, S.S.; Suhonen, J. Application of Virtual Reality in Computer Science Education: A Systemic Review Based on Bibliometric and Content Analysis Methods. *Educ. Sci.* **2021**, *11*, 142. https://doi.org/ 10.3390/educsci11030142

Academic Editor: João Piedade

Received: 22 February 2021 Accepted: 16 March 2021 Published: 23 March 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

This study investigated the role of VR in CS education by conducting a comprehensive content and bibliometric analysis of relevant articles published between 2011 and 2020 in journals and conferences. Bibliometric and content analysis of articles focused on VR in CS education would provide a deeper understanding of the evolution of research conducted in this field and how VR applications have advanced CS education over the years [4,10,11]. From the standpoint of bibliometric mapping analysis, this study investigates the publication growth of studies on VR in CS education within the last 10 years, reveals the most active authors and affiliations contributing to the development of VR in CS education, and anticipates the future direction on the basis of the co-occurrence pattern analysis of current studies. In addition, this study explicates the role of VR in CS education from the perspective of methodological approaches used in studies related to VR in CS education [7], the kind of data collected for such studies, the sample size, and the types of data analysis conducted.

Research on VR in education has claimed several benefits, such as positively affecting users' attitude [12,13], presenting an effective and efficient learning and training environment [14,15], and increasing students' motivation to learn within a virtual environment [14–17]. Furthermore, many systematic review studies related to VR in education have been published in recent years. However, there have been only a limited number of such studies focused on computer science education. For example, Pirker et al. [3] conducted a systematic literature review of VR in CS education, focusing on the technology used to deploy VR applications for CS education, the learning objectives, and challenges recorded in studies related to VR in CS education. Pirker and colleagues revealed that VR desktop applications using Oculus Rift and HTC Vive dominate the technology currently used to deploy VR in CS education. On the other hand, the majority of studies on VR in CS education focused on cognitive learning with topics such as fundamental components of algorithms and object-oriented programming [3].

Similarly, Oyelere et al. [1] studied VR games in CS education, focusing on developmental features such as the technology, pedagogy, and gaming elements used in such studies. In terms of technology, Oyelere et al. [1] finding was in congruence with that of Pirker et al. [3], where Oculus Rift, HTC Vive, and PC-based applications dominate the technology aspect. Both studies show that mobile-based VR applications for CS education are still growing, with less than 15% of deployment of VR applications on mobile devices.

We could find only a few studies regarding recent studies that focused on content and bibliometric analysis of articles related to VR in education. For example, Arici et al. [11] conducted content and bibliometric mapping analysis of augmented reality (AR) in science education. Lorenzo et al. [17] investigated VR articles' scientific production for inclusive learning of people with autism spectrum disorder (ASD). Sobral and Pestana [18] studied a bibliometric analysis of articles related to VR application to learn about dementia from 1998 until 2018 by focusing on articles' intellectual structure and emerging trends. Lai et al. [19] conducted a bibliometric analysis of VR research in engineering education published and indexed in the Scopus database that spans over 26 years. Thus, Lai et al. [19] provided valuable insights in terms of article production, trends, and co-occurrence network of VR studies within the field of engineering. Another bibliometric study related to VR in CS field-specific was recently conducted by Enebechi and Duffy [20]. This study [21] focused on bibliometric analysis of VR and artificial intelligence (AI) articles in mobile computing and applied ergonomics.

While all these related studies highlighted above are relevant and provided essential knowledge about the field, our current research would expand on the existing research rather than re-inventing the wheel. For example, while the work of Pirker et al. [3] mainly focused on the technology used to deploy VR application for CS education, the learning objectives, and challenges recorded in studies related to VR in CS education, our research would address the aspect of methodological approach used in studies on VR in CS education, kind of data collected for such studies, the sample size, and types of data analysis conducted. The majority of these related studies analyzed a small sample size, limiting the study, and

cannot justify the generalization of their findings. For example, Pirker et al. [3] analyzed 13 pieces of data, Lorenzo et al. [17] revealed 18 articles, Lai et al. [19] conducted bibliometric analysis on 274 articles, and Enebechi and Duffy [20] presented a content analysis of 8 papers. Our study took a different approach by analyzing more extensive data to discover more profound knowledge in the field. It is worth mentioning that our study drew motivation from [11] by focusing content analysis of variables such as materials and method trends, sample sizes, and method of an investigation conducted by articles on VR in CS education in the last 10 years. The authors hope that the approach used in this study would contribute to the existing knowledge in terms of unveiling how VR has supported CS education and what scientific achievement have been made in this field.

As a result of this comprehensive content and bibliometric analysis of studies on VR in CS education, we hoped that our findings would contribute to the existing knowledge by providing a deeper understanding of VR applications' role in honing CS education over the last decade. In addition, the authors believe that this study will unveil information regarding what scholars have made a scientific achievement in this field in terms of advancing teaching and learning of CS topics in the different contexts, which will serve as a boost for active researchers. In contrast, new scholars would derive motivation and valuable resources for future studies. To achieve objectives, this study set out to answer the following research questions:


#### **2. Methods**

The method explored in this study was centered on content and bibliometric mapping analysis. This study followed the recommended workflow for science mapping provided by Aria and Coccurullo [21] to conduct our bibliometric mapping analysis. In contrast, the approach shown by [11] was followed to present the content analysis, respectively.

#### **Article selection process**

The article selection process for this study includes 3 phases similar to the one presented by [4], namely, (i) literature search and data collection; (ii) data extraction, loading, and conversion; and (iii) data synthesis. A graphical representation of the data collection process is presented in Figure 1, showing detailed actions in each phase.

(i) Literature search and data collection

This study obtained data from 2 databases, i.e., the Web of Science (WoS) and the Scopus databases. These 2 databases have been acclaimed to contain comprehensive data of scientific outputs relevant to this study [14]. To conduct an extensive data collection needed for this study, we define the search keywords to include "virtual reality" "VR", "computer science", and "computing education". A number of common protocols for data collection were applied to both databases. They include the same search keywords used in combination with the binary operators such as "OR" and "AND" across the 2 databases, limited time span to the period from 2011 to 2020, and language selected as "English". Table 1 presents details of the search protocol, how they were applied in each database, and the result obtained.

WoS

Scopus

in combination with the binary operators such as "OR" and "AND" across the 2 databases, limited time span to the period from 2011 to 2020, and language selected as "English". Table 1 presents details of the search protocol, how they were applied in each database,

TOPIC: ("virtual reality" OR "VR") AND TOPIC: ("computer science"

TOPIC: ("virtual reality" OR "VR") AND TOPIC: ("computer science"). Refined by: DOCUMENT TYPES: (ARTICLE OR PROCEEDINGS PA-PER) AND PUBLICATION YEARS: (2020 OR 2014 OR 2019 OR 2013

Timespan: All years. Indexes: SCI-EXPANDED, SSCI, A&HCI, ESCI.

(TITLE-ABS-KEY ("virtual reality" OR "VR") AND TITLE-ABS-KEY ("computer science" OR "computing education")) AND PUBYEAR >

(TITLE-ABS-KEY ("virtual reality" OR "VR") AND TITLE-ABS-KEY ("computer science" OR "computing education")) AND PUBYEAR > 2010 AND PUBYEAR < 2021 AND (LIMIT-TO (DOCTYPE, "cp") OR LIMIT-TO (DOCTYPE, "ar")) AND (LIMIT-TO (LANGUAGE, "Eng-

After data from the independent databases were collected and downloaded in BibTex format, we conducted data extraction and conversion into a comma-separated values CSV file to merge the 2 datasets from WoS and Scopus. The process of merging the data is presented in Table 2, followed by executing command line instructions (CLI) shown in Figure 2. R-studio is an integrated development environment for R programming language (https://rstudio.com)) software was used to combine the data into a single CSV file before uploading it to biblioshiny (Biblioshiny is a web interface for bibliometrix r-package (https://www.bibliometrix.org/Biblioshiny.html)) for bibliometrix R-package [17].

OR 2018 OR 2012 OR 2017 OR 2011 OR 2016 OR 2015)

OR "computing education"). <sup>80</sup>

**Outcome** 

58

1058

962

Total

**Table 1.** Data search procedures and obtained amount of data.

**Database Description of the Protocol Combination of Search String Based on Database Algorithm Search** 

2010 AND PUBYEAR < 2021.

(ii) Data extraction, loading, and conversion

After merging both files, we removed 49 duplicated documents. <sup>971</sup>

lish")).

and the result obtained.

Applying the search keywords in quotation to the WoS TOPIC field

Additional conditions were applied by limiting the results to only articles and proceedings papers, with time span set to 2011–2020.

Applying the search keywords in quotation to Scopus title, abstract, and keywords field with binary operators and limiting the time span

Applying additional conditions by limiting to only articles and confer-

with binary operators.

to 2011–2020.

ence papers.

**Figure 1.** The procedure followed to obtain data used for bibliometric and content analysis. **Figure 1.** The procedure followed to obtain data used for bibliometric and content analysis.


#### (ii) Data extraction, loading, and conversion

After data from the independent databases were collected and downloaded in BibTex format, we conducted data extraction and conversion into a comma-separated values CSV file to merge the 2 datasets from WoS and Scopus. The process of merging the data is presented in Table 2, followed by executing command line instructions (CLI) shown in Figure 2. R-studio is an integrated development environment for R programming language (https://rstudio.com, accessed on 18 January 2018) software was used to combine the data into a single CSV file before uploading it to biblioshiny (Biblioshiny is a web interface for bibliometrix r-package (https://www.bibliometrix.org/Biblioshiny.html, accessed on 18 January 2018) for bibliometrix R-package [17].


**Table 2.** Data conversion and merging steps.

*Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 6 of 25

**Figure 2.** (**A**) shows the set of commands to be executed in R-Studio command line instructions (CLI) to implement the conversion and to merge of data downloaded from Web of Science (WoS) and Scopus databases; (**B**) shows the output of the executed commands; (**C**) depicts the console for the CLI where the line execution returns a value including line errors. **Figure 2.** (**A**) shows the set of commands to be executed in R-Studio command line instructions (CLI) to implement the conversion and to merge of data downloaded from Web of Science (WoS) and Scopus databases; (**B**) shows the output of the executed commands; (**C**) depicts the console for the CLI where the line execution returns a value including line errors.

> **3. Results**  *3.1. Findings from Bibliometric Mapping Analysis*  This section presents our findings from the bibliometric analysis on the basis of the data generated from WoS and Scopus databases. This bibliometric analysis intends to provide insight into how studies on the use of VR for CS education have grown in the last 10 After completing the steps in Table 2, we executed the line of commands (lines 6 to 15) in Figure 2 to complete the remaining process of data conversion and merging. This merging of the two converted points of data by running the command in line 11 of Figure 2A triggered the R- Function that identified 49 similar articles from WoS and Scopus databases. The identified similar articles were removed to avoid having duplicate data. Removing duplicate articles left the remaining data at 971, which was uploaded to biblioshiny for bibliometric mapping analysis. The search was conducted on 2 January 2021.

> years. In addition, the result reveals authors, institutions, and countries who have been

Furthermore, the result presents how studies on VR in CS education have had an impact in terms of their citations and authors co-occurrence pattern analysis. The section delineates the analysis of common keywords used in articles on VR for CS education, thereby

presenting the thematic area of the current research landscape and topic hotspots.

#### (iii) Data Synthesis

In Table 3, we present the synthesized data used for the bibliometric analysis. However, for the content analysis, 3 researchers screened the entire data by reading each paper's abstract to decide whether it was relevant or not. Further criteria for selecting relevant papers suitable for the content analysis included:

**Table 3.** Data synthesis indicating the primary information about the data and document type.



After applying the criteria, we arrived at 39 papers that met the content analysis requirements presented in Section 3.2.

#### **3. Results**

#### *3.1. Findings from Bibliometric Mapping Analysis*

This section presents our findings from the bibliometric analysis on the basis of the data generated from WoS and Scopus databases. This bibliometric analysis intends to provide insight into how studies on the use of VR for CS education have grown in the last 10 years. In addition, the result reveals authors, institutions, and countries who have been contributing to the field by actively publishing research related to VR in CS education. Furthermore, the result presents how studies on VR in CS education have had an impact in terms of their citations and authors co-occurrence pattern analysis. The section delineates the analysis of common keywords used in articles on VR for CS education, thereby presenting the thematic area of the current research landscape and topic hotspots.

3.1.1. Research Publication Growth of Articles on the Use of VR in Computer Science Education 3.1.1. Research Publication Growth of Articles on the Use of VR in Computer Science Education

Figure 3 shows the articles' distribution in terms of the publication year regarding the article production and development across 10 years. The overall publication trend of articles related to VR in CS education shows that 2011 witnessed the highest production year, reaching 148 articles, followed closely by 135 articles in 2018. Figure 3 shows the articles' distribution in terms of the publication year regarding the article production and development across 10 years. The overall publication trend of articles related to VR in CS education shows that 2011 witnessed the highest production year, reaching 148 articles, followed closely by 135 articles in 2018.

**Figure 3.** Annual scientific production of articles on virtual reality (VR) in computer science (CS) education. **Figure 3.** Annual scientific production of articles on virtual reality (VR) in computer science (CS) education.

The publication volume decreased from 2012 to 2015 and from 2019 to 2020. There was an increase in article production from 2016 to 2018 before the slight decline until 2020. This trend occurred probably because the selected articles were limited to only education, leaving out other domains, such as health, business, entertainment, and media. The publication volume decreased from 2012 to 2015 and from 2019 to 2020. There was an increase in article production from 2016 to 2018 before the slight decline until 2020. This trend occurred probably because the selected articles were limited to only education, leaving out other domains, such as health, business, entertainment, and media.

3.1.2. Most Active Authors, Institutions, and Countries Publishing Articles on the Use of 3.1.2. Most Active Authors, Institutions, and Countries Publishing Articles on the Use of VR in Computer Science Education

VR in Computer Science Education Regarding authors' production over time, we investigated the top 20 authors. Our findings showed that most of those top authors were already publishing articles on VR in CS education by 2011. However, about half of those authors were not active from 2019. As shown in Figure 4, many articles related to VR in CS education were published be-Regarding authors' production over time, we investigated the top 20 authors. Our findings showed that most of those top authors were already publishing articles on VR in CS education by 2011. However, about half of those authors were not active from 2019. As shown in Figure 4, many articles related to VR in CS education were published between 2011 and 2020.

tween 2011 and 2020. As we can see in Figure 4, the author Li Y. had the highest publication over time, having had several articles published yearly for 7 years from 2011 to 2020, except in 2013, 2014, and 2016. With the least productivity over time was the author is Dengel A., with As we can see in Figure 4, the author Li Y. had the highest publication over time, having had several articles published yearly for 7 years from 2011 to 2020, except in 2013, 2014, and 2016. With the least productivity over time was the author is Dengel A., with publications only in 2019 and 2020.

publications only in 2019 and 2020. We analyzed the top 20 authors' number citations across the production years (mindex) regarding their impact. M-index is calculated by dividing the total number of citations by the total number of years of production. In order words, this study measures the authors' impact by dividing the H-index by the total number of years of production. Note that the total years of production varied for different authors. Although the total number of years investigated in this study remained at 10, some authors did not start publishing from 2011; therefore, such an author's total number of years of production would count We analyzed the top 20 authors' number citations across the production years (mindex) regarding their impact. M-index is calculated by dividing the total number of citations by the total number of years of production. In order words, this study measures the authors' impact by dividing the H-index by the total number of years of production. Note that the total years of production varied for different authors. Although the total number of years investigated in this study remained at 10, some authors did not start publishing from 2011; therefore, such an author's total number of years of production would count from the year the author published his/her first paper. For example, Dengel A. started publishing articles on VR in CS education in 2019; hence, the total number of

years remained at two. Therefore, the m-index would be the total number of citations in 2019 and 2020, divided by 2. remained at two. Therefore, the m-index would be the total number of citations in 2019 and 2020, divided by 2. remained at two. Therefore, the m-index would be the total number of citations in 2019 and 2020, divided by 2.

**Figure 4.** Top 20 authors publishing articles on VR in CS education between 2011 and 2020: the size of each circle indicates the number of articles. The amount of boldness of the circles shows the number of citations in that year. **Figure 4.** Top 20 authors publishing articles on VR in CS education between 2011 and 2020: the size of each circle indicates the number of articles. The amount of boldness of the circles shows the number of citations in that year.**Figure 4.** Top 20 authors publishing articles on VR in CS education between 2011 and 2020: the size of each circle indicates the number of articles. The amount of boldness of the circles shows the number of citations in that year.

As shown in Figure 5, the authors' m-index was highest at 1.0 (to a single decimal). Therefore, the result indicates that Dengel A., with the highest m-index, remained the most impactful author at the end of 2020. This finding suggests that Dengel A. had had an unbroken research activity in the area of VR in CS education since the first publication and had received a significant number of citations. As shown in Figure 5, the authors' m-index was highest at 1.0 (to a single decimal). Therefore, the result indicates that Dengel A., with the highest m-index, remained the most impactful author at the end of 2020. This finding suggests that Dengel A. had had an unbroken research activity in the area of VR in CS education since the first publication and had received a significant number of citations. As shown in Figure 5, the authors' m-index was highest at 1.0 (to a single decimal). Therefore, the result indicates that Dengel A., with the highest m-index, remained the most impactful author at the end of 2020. This finding suggests that Dengel A. had had an unbroken research activity in the area of VR in CS education since the first publication and had received a significant number of citations.

**Figure 5.** Top-20 authors' impact analysis within 10 years. **Figure 5.** Top-20 authors' impact analysis within 10 years. **Figure 5.** Top-20 authors' impact analysis within 10 years.

Our analysis revealed some top universities regarding institutions (authors' affiliations) and countries fronting VR in CS education. As shown in Figure 6, some of these universities, to name a few, were the University of Southern California, USA; Aalborg University, Denmark; and University of Rennes, France. Our analysis revealed some top universities regarding institutions (authors' affiliations) and countries fronting VR in CS education. As shown in Figure 6, some of these universities, to name a few, were the University of Southern California, USA; Aalborg University, Denmark; and University of Rennes, France.

**Figure 6.** Three-field plot of active institutions and countries publishing articles related to VR in CS education between 2011 and 2020. **Figure 6.** Three-field plot of active institutions and countries publishing articles related to VR in CS education between 2011 and 2020.

> Figure 6 shows the USA as the most productive country in terms of publishing articles related to VR in CS education in countries. From the European continent, France, Denmark, Italy, the UK, Germany, and Spain made significant contributions. Only China Figure 6 shows the USA as the most productive country in terms of publishing articles related to VR in CS education in countries. From the European continent, France, Denmark, Italy, the UK, Germany, and Spain made significant contributions. Only China and Japan made contributions regarding VR in CS education from the Asian continent.

and Japan made contributions regarding VR in CS education from the Asian continent. 3.1.3. Keywords Co-Occurrence Patterns of Studies on the Use of VR in Computer Science Education

3.1.3. Keywords Co-Occurrence Patterns of Studies on the Use of VR in Computer Science Education A keywords co-occurrence pattern (KCP) focuses on understanding the knowledge A keywords co-occurrence pattern (KCP) focuses on understanding the knowledge components and knowledge structure of a scientific field by examining the links between keywords in the published articles within the same area [4].

components and knowledge structure of a scientific field by examining the links between keywords in the published articles within the same area [4]. Figure 7 focuses on keyword co-occurrence patterns of studies on the use of VR in computer science education. As observed in Figure 7, the root keyword in the field remains "virtual reality". Other keywords that are frequently used by articles on VR in CS education are shown in red color. For instance, we notice keywords such as gamification, simulation, higher education, mixed reality, serious games, and more. In addition, as expected, keywords that define the characteristics of virtual reality technology were seen to be strongly connected to the root keyword. For example, we observe a thick line connecting keywords such as immersion, interaction, and presence, to the root keyword "virtual Figure 7 focuses on keyword co-occurrence patterns of studies on the use of VR in computer science education. As observed in Figure 7, the root keyword in the field remains "virtual reality". Other keywords that are frequently used by articles on VR in CS education are shown in red color. For instance, we notice keywords such as gamification, simulation, higher education, mixed reality, serious games, and more. In addition, as expected, keywords that define the characteristics of virtual reality technology were seen to be strongly connected to the root keyword. For example, we observe a thick line connecting keywords such as immersion, interaction, and presence, to the root keyword "virtual reality". Moreover, virtualization, cloud computing, and virtual machine are keywords that show a strong connection. Other keywords that show a close relationship to virtual reality include augmented reality and computer science.

reality". Moreover, virtualization, cloud computing, and virtual machine are keywords that show a strong connection. Other keywords that show a close relationship to virtual

reality include augmented reality and computer science.

**Figure 7.** Co-occurrence patterns of authors' keywords in articles on VR in CS education between 2011 and 2020. **Figure 7.** Co-occurrence patterns of authors' keywords in articles on VR in CS education between 2011 and 2020.

Furthermore, Figure 8 presents a visualization of frequently used keywords in VR for CS education. It is clear from the size of the nodes that other related terms used for virtual reality, for example, "virtualization" and "virtual environment" were found to be highly connected to "computer science" and "education". In addition, some pedagogical concepts for teaching and learning, such as games, gamification, collaborative learning, and immersive learning, are visible in the network. Figure 8 also shows clustering of concepts where terms such as virtualization, virtual environment, computer science, and education form clusters depicted with different colors. Furthermore, Figure 8 presents a visualization of frequently used keywords in VR for CS education. It is clear from the size of the nodes that other related terms used for virtual reality, for example, "virtualization" and "virtual environment" were found to be highly connected to "computer science" and "education". In addition, some pedagogical concepts for teaching and learning, such as games, gamification, collaborative learning, and immersive learning, are visible in the network. Figure 8 also shows clustering of concepts where terms such as virtualization, virtual environment, computer science, and education form clusters depicted with different colors.

One way to examine how VR application has influenced CS education is to analyze trending topics over the period considered in this study. Figure 9 presents the trending topics or approaches scholars have explored to provide VR intervention for CS education. One way to examine how VR application has influenced CS education is to analyze trending topics over the period considered in this study. Figure 9 presents the trending topics or approaches scholars have explored to provide VR intervention for CS education.

This study analyzed the authors' keywords to determine what research hotspot in terms of topics and approaches have been explored by VR applications in CS education in the last decade. This analysis was conducted through the word cloud of authors' keywords, which gives a pointer to what has been the scholars' interest. This analysis also provides insight regarding the future outlook of VR interventions in CS education. Figure 9 delineates that virtualization, cloud computing, the virtual world, and virtual machine dominate VR studies in CS education between the years 2011 and 2015. In addition, slightly different changes were observed where keywords such as computer science education, serious games, and higher education emerged among the trending topics between This study analyzed the authors' keywords to determine what research hotspot in terms of topics and approaches have been explored by VR applications in CS education in the last decade. This analysis was conducted through the word cloud of authors' keywords, which gives a pointer to what has been the scholars' interest. This analysis also provides insight regarding the future outlook of VR interventions in CS education. Figure 9 delineates that virtualization, cloud computing, the virtual world, and virtual machine dominate VR studies in CS education between the years 2011 and 2015. In addition, slightly different changes were observed where keywords such as computer science education, serious games, and higher education emerged among the trending topics between 2015 and 2017.

2015 and 2017.

*Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 11 of 25

**Figure 8.** Visualized authors' keywords co-occurrence analysis of articles on VR in CS education: these are among the highest number of repetitive keywords within the field. **Figure 8.** Visualized authors' keywords co-occurrence analysis of articles on VR in CS education: these are among the highest number of repetitive keywords within the field. *Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 12 of 25

Furthermore, it was observed that between the years 2018 and 2020, new keywords

This section presents the content analysis findings to address some of the research questions (RQ4 to RQ7). Moreover, an overview of the data analyzed in this section is presented as an Appendix A. In the Appendix A, information regarding the study focus and outcome are highlighted to showcase how the selected articles have employed VR in

According to Figure 10, 47% of the articles used a quantitative design approach, 16% used a qualitative design, 3% used mixed design, and 12% utilized a design and development research approach. In comparison, others may include review/meta-analysis re-

3.2.1. Trends of the Research Methodology Employed in Articles on the Use of VR in

**Figure 9.** (**a**–**c**) Word cloud showing the trending topics of VR in CS education in terms of authors keywords. While (**a**) shows the trending topics between 2011 and 2015, (**b**) presents the topics between 2015 and 2017, and (**c**) depicts the trending topics on VR in CS education between 2018 and 2020. **Figure 9.** (**a**–**c**) Word cloud showing the trending topics of VR in CS education in terms of authors keywords. While (**a**) shows the trending topics between 2011 and 2015, (**b**) presents the topics between 2015 and 2017, and (**c**) depicts the trending topics on VR in CS education between 2018 and 2020.

CS education.

Computer Science Education

search accounts for 5%.

Furthermore, it was observed that between the years 2018 and 2020, new keywords such as augmented reality, immersion, presence, gamification, game-based learning, and human–computer interaction were added to the trending topics. Therefore, topics such as immersion, presence, human–computer interaction, gamification, and game-based learning dominate the list of research hotspots in recent times. This finding suggests that one of the most appreciated learning and teaching approaches used by studies on VR application in CS education is game-based learning.

#### *3.2. Findings from Content Analysis*

This section presents the content analysis findings to address some of the research questions (RQ4 to RQ7). Moreover, an overview of the data analyzed in this section is presented as an Appendix A. In the Appendix A, information regarding the study focus and outcome are highlighted to showcase how the selected articles have employed VR in CS education.

3.2.1. Trends of the Research Methodology Employed in Articles on the Use of VR in Computer Science Education

According to Figure 10, 47% of the articles used a quantitative design approach, 16% used a qualitative design, 3% used mixed design, and 12% utilized a design and development research approach. In comparison, others may include review/meta-analysis research accounts for 5%. *Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 13 of 25

**Figure 10.** Frequency of research methods in articles on VR in CS education between 2011 and 2020. **Figure 10.** Frequency of research methods in articles on VR in CS education between 2011 and 2020.

found to be the most used research method over the years.

Figure 11 revealed the research method trends related to VR in CS education in the

and in 2014, and witnessed an increase in 2020. While mixed methods are almost inexistent, qualitative and other methods showed no significant distribution variations over time. Review and meta-analysis began to be used in 2019 as the quantitative design was

Figure 11 revealed the research method trends related to VR in CS education in the past 10 years. The use of quantitative methods increased in 2018 and declined from 2019 to 2020. The next prominent method utilized is the design research method used in 2011 and in 2014, and witnessed an increase in 2020. While mixed methods are almost inexistent, qualitative and other methods showed no significant distribution variations over time. Review and meta-analysis began to be used in 2019 as the quantitative design was found to be the most used research method over the years. *Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 14 of 25

**Figure 11.** Trends of research methods in articles on VR in CS in the past 10 years. **Figure 11.** Trends of research methods in articles on VR in CS in the past 10 years.

3.2.2. The Most Preferred Data Collection Tools and Sampling Methods in Articles on the Use of VR in Computer Science Education 3.2.2. The Most Preferred Data Collection Tools and Sampling Methods in Articles on the Use of VR in Computer Science Education

Data collection tools and sampling methods in research conducted on VR in CS education show that the questionnaire (46%) remains the most used tool. However, quite a number of studies (23%) either did not conduct evaluation or did not specify what method of data collection was used. Data collection tools and sampling methods in research conducted on VR in CS education show that the questionnaire (46%) remains the most used tool. However, quite a number of studies (23%) either did not conduct evaluation or did not specify what method of data collection was used.

As shown in Figure 12, the use of interviews (13%) is still growing as fewer studies have been seen to use the method. As shown in Figure 12, the use of interviews (13%) is still growing as fewer studies have been seen to use the method.

3.2.3. Sample Populations and Sample Sizes in Articles on the Use of VR in Computer Science Education

According to Figure 13, the most commonly used sample size in articles published between 2011 and 2020 fell between 11–20 participants. Closely followed were 1–10 persons and 51–100 people. Although other studies utilized samples between 21–50 and 101–200 respondents, a few studies did not specify the sample size they used.

*Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 15 of 25

**Figure 12.** Data collection tools and sampling methods of articles on the use of VR in CS education. **Figure 12.** Data collection tools and sampling methods of articles on the use of VR in CS education.

**Figure 13.** Frequency of use of sample sizes in articles. **Figure 13.** Frequency of use of sample sizes in articles.

**Figure 14.** Most preferred data analysis method between 2011 and 2020.

Science Education

studies as "others/not specified".

cation.

3.2.4. Most Preferred Data Analysis Methods in Articles on the Use of VR in Computer

The findings show that most studies were performed using descriptive analysis regarding the most preferred data analysis conducted in studies focused on VR in CS edu-

Other preferred analysis methods, as shown in Figure 14, are meta-analysis and content analysis. Moreover, some studies adopted a theoretical approach while some other studies did not conduct any form of research, and therefore we categorized these types of

3.2.4. Most Preferred Data Analysis Methods in Articles on the Use of VR in Computer Science Education The findings show that most studies were performed using descriptive analysis regarding the most preferred data analysis conducted in studies focused on VR in CS edu-

The findings show that most studies were performed using descriptive analysis regarding the most preferred data analysis conducted in studies focused on VR in CS education. cation.

3.2.4. Most Preferred Data Analysis Methods in Articles on the Use of VR in Computer

Other preferred analysis methods, as shown in Figure 14, are meta-analysis and content analysis. Moreover, some studies adopted a theoretical approach while some other studies did not conduct any form of research, and therefore we categorized these types of studies as "others/not specified". Other preferred analysis methods, as shown in Figure 14, are meta-analysis and content analysis. Moreover, some studies adopted a theoretical approach while some other studies did not conduct any form of research, and therefore we categorized these types of studies as "others/not specified".

**Figure 13.** Frequency of use of sample sizes in articles.

*Educ. Sci.* **2021**, *11*, x FOR PEER REVIEW 16 of 25

**Figure 14. Figure 14.** Most preferred data analysis method between 2011 and 2020. Most preferred data analysis method between 2011 and 2020.

#### **4. Discussion**

Science Education

The bibliometric method's potential is seen by earlier research [4]. It was opined that bibliometric study advances complement meta-analysis and qualitative research for the scientific evaluation of literature. This study delved into VR's role in CS education to provide a deeper understanding of the evolution of research conducted in this field and anticipate the future direction on the basis of the analysis of the co-occurrence pattern of keywords used in studies conducted in the last 10 years. The study contributes to knowledge by presenting valuable findings that can boost the morale of prolific scholars who have been contributing to this field and researchers and practicing managers who may be starting to research into VR for CS education. This current study obtained its bibliometric and content analysis data from the Web of Science and Scopus databases.

The bibliometric analysis of articles related to the use of VR in CS education, together with the methodological research trends over the last 10 years, was revealed. Bibliometric analysis results showed that the year 2011 was the highest in article production (148 articles). This result was closely followed by the year 2018 with 135 articles. This finding implies that between 2012 and 2017, articles related to VR in CS education dwindled. Regarding the authors' production over time, Li Y. had the highest number of articles produced in the field, which is not surprising as the author consistently published in 2011–2012, 2015, and 2017–2020.

Moreover, we analyzed studies' impact by investigating the number of citations obtained by authors within 10 years. The analysis was focused on the m-index of each author. Considering the 10 years duration in this study, we calculated the m-index by dividing the total number of citations by the total number of years authors have been publishing. For example, Dengel A. emerged as the most impactful author because this author had produced one paper per year for only two years. This means that Dengel's impact analysis was computed on the basis of the output of these two years. However, it was surprising to discover that Li Y., who had the highest number of articles produced over the years, was not as impactful as Dengel A., who had a limited number of articles published within just two years. Earlier studies have examined intrinsic factors affecting the number of citations of articles [22,23]; however, some indicators are not directly related to the quality or content of articles' extrinsic factors [24]. The previous finding reveals that price index, number of references, keywords, and length of studies are essential explanatory factors [24]. It can be concluded that it is likely that Li's articles are easily accessible to researchers via open access medium. The relevancy of their topic or even the quality of their paper in terms of content and presentation may account for the citations and rapid impact.

Regarding the institutions and countries contributing to VR in CS education, the results further showed that the University of Southern California, USA; Aalborg University, Denmark; and the University of Rennes, France, remain the top universities in terms of publishing VR in CS education articles. On the other hand, the USA emerged as the most productive country. However, other countries from Europe (France, Denmark, Italy, the UK, and Germany) and Asia (China) are making a significant contribution towards advancing CS education using VR technology. The co-occurrence pattern of authors' keywords revealed that VR characteristics are leveraged for CS education. For example, immersion, presence, interaction, and gamification are being explored in advancing CS education [1,16,18]. Moreover, these keywords also form the research hotspots in VR, primarily to support learning. Therefore, this study anticipates that VR in CS education would continue to be researched within the scope of these keywords [14].

The content analysis results showed that quantitative studies (47%) dominate the studies in terms of research methodology. The reason for quantitative method preference may be due to the simplified way of presenting quantitative research, as well as less time and effort required to conduct and analyze quantitative data [25]. It might also be the case that the generalization and replicability that the quantitative approach provides accounts for its dominance in the studies. The percentage for the use of mixed methods studies was meager, reflecting that the use of mixed approach studies presents methodological difficulties and challenges [12]. It is safe to conclude that only a few studies consider the potential of mixed-method research, which adds rigor and validity to research through triangulation and convergence of multiple and different sources of information [26,27]. Moreover, few qualitative studies have been conducted in the last 10 years. This may have been due to the rigor and non-use of numbers, making it difficult to simplify findings and observations [25]. On the contrary, Johnson and Christensen [28] assert that reliance on collecting non-numerical primary data such as words and pictures makes qualitative research well-suited for providing factual and descriptive information.

Regarding the frequency of the sampling size utilized over the years, the most used sample sizes were 11–20. We were surprised to find out that most published articles on VR in CS education were evaluated with about 11 to 20 participants. Since the research method's preference was quantitative research, we expected that many studies would have used more participants to arrive at a generalized outcome. Although studies that used 51–100 sample sizes were also seen in the result, one could have thought that 20 participants may be too small for a quantitative study. According to Faber and Fonseca [29], very small samples undermine the internal and external validity, while huge samples tend to transform minor differences into statistically significant differences.

Our findings revealed that the questionnaire is the most used data collection tool, while descriptive analysis remains the preferred data analysis method. One way to reflect on this result is that the questionnaire seems more straightforward, quicker, and cost-effective to collect data from participants. Moreover, the preference for descriptive analysis may be used to simplify data efficiently [30]. The researcher may have adopted this data analysis method to reduce the time and effort required to format and present beneficial, easily

interpretable results to practitioners, policymakers, and other researchers to understand a phenomenon better.

#### **5. Conclusions**

This study provides a comprehensive view of scientific papers on VR in CS education published in peer-reviewed journals and conferences between 2011 and 2020. Two main approaches were explored to answer the research questions presented in this study. First, the bibliometric analysis answered the questions regarding the article production growth in the field within a decade, prolific scholars and their affiliations publishing to advance VR in CS education, and research hotspots in the field may guide scholar's future research focus. Second, content analysis of articles that met the inclusion criteria for this study was analyzed to provide a methodological overview of studies conducted on VR in CS education. Several findings were presented in this study. These findings show that VR research for CS education has fared well; however, some of the years (between 2013 and 2016) witnessed low article production. The study also revealed the prolific scholars and authors' impact analysis in this field and provided insightful information regarding research hotspots by analyzing the authors' keywords co-occurrence.

Regarding the scientific methodology and data sampling technique used by studies on VR in CS education, the most preferred is the quantitative method. At the same time, the questionnaire was the most used data collection technique. Moreover, descriptive analysis was mainly used to analyze data in studies on VR in CS education.

This study witnessed a limitation regarding the content analysis. It would be interesting to see the educational context where VR technology is being used and the learning contents deployed in the VR application for CS education. Nonetheless, this study contributes to knowledge in significant ways. The study revealed that pedagogical approaches such as game-based learning and gamification were explored for VR education in CS education. The findings from this study can provide insight into how VR technology research has progressed in a decade. Moreover, the result can be generalized since this study could obtain relevant data from two databases (WoS and Scopus) to conduct its analysis. The process for merging these data is another contribution as scholars interested in running a similar study would find this helpful study. Our future study would address the limitations by providing answers regarding the pedagogy, content, and context of studies on VR in CS education.

By implication, we conclude that findings from this study suggest that even though scholars are leveraging VR to advance teaching and learning in the field of CS, more effort needs to be made, especially from continents, countries, and institutions that were not reported among the top-20 list revealed in this study. In addition, a more rigorous methodological approach needs to be employed in a future study to provide more evident-based research output. For example, our study revealed only a few studies that used a mixedmethods approach, which has been more rigorous in terms of quality of scientific research.

**Author Contributions:** Conceptualization, F.J.A., I.T.S.; Data curation, F.J.A., I.T.S.; Formal analysis, F.J.A., I.T.S., S.S.O. and J.S.; Investigation, F.J.A., I.T.S. and S.S.O.; Methodology, F.J.A., I.T.S., S.S.O. and J.S.; Project administration, F.J.A.; Resources, F.J.A., I.T.S.; Software, F.J.A., I.T.S., S.S.O. and J.S.; Supervision, S.S.O. and J.S.; Validation, S.S.O. and J.S.; Writing—original draft, F.J.A. and I.T.S.; Writing—review & editing, F.J.A., I.T.S., S.S.O. and J.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data sharing is not applicable to this article.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**

**Table A1.** Published articles contained in the content analysis of VR for CS education (2011–2020).



**Table A1.** *Cont.*


#### **Table A1.** *Cont.*

#### **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Education Sciences* Editorial Office E-mail: education@mdpi.com www.mdpi.com/journal/education

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18

www.mdpi.com ISBN 978-3-0365-2315-6