**Improving the Teaching of Hypothesis Testing Using a Divide-and-Conquer Strategy and Content Exposure Control in a Gamified Environment**

### **David Delgado-Gómez 1,\*, Franks González-Landero 2, Carlos Montes-Botella 1, Aaron Sujar 1,3, Sofia Bayona 3,4 and Luca Martino <sup>5</sup>**


Received: 30 October 2020; Accepted: 16 December 2020; Published: 19 December 2020

**Abstract:** Hypothesis testing has been pointed out as one of the statistical topics in which students present more misconceptions. In this article, an approach based on the divide-and-conquer methodology is proposed to facilitate its learning. The proposed strategy is designed to sequentially explain and evaluate the different concepts involved in hypothesis testing, ensuring that a new concept is not presented until the previous one has been fully assimilated. The proposed approach, which contains several gamification elements (i.e., points or a leader-board), is implemented into an application via a modern game engine. The usefulness of the proposed approach was assessed in an experiment in which 89 first-year students enrolled in the Statistics course within the Industrial Engineering degree participated. Based on the results of a test aimed at evaluating the acquired knowledge, it was observed that students who used the developed application based on the proposed approach obtained statistically significant higher scores than those that attended a traditional class (*p*-value < 0.001), regardless of whether they used the learning tool before or after the traditional class. In addition, the responses provided by the students who participated in the study to a test of satisfaction showed their high satisfaction with the application and their interest in the promotion of these tools. However, despite the good results, they also considered that these learning tools should be considered as a complement to the master class rather than a replacement.

**Keywords:** education; learning environments; educational games; engineering students

#### **1. Introduction**

Statistics is a core subject in almost all university degrees. As an example, Garfield and Ahlgren pointed out that the University of Minnesota offered 160 statistics courses, taught by 13 different departments [1]. Learning statistics is important as it provides the capacity of conducting logical reasoning and critical thinking, enhancing interpretation and evaluation skills and facilitating dealing with highly abstract concepts [2].

However, many students find learning statistics difficult and unpleasant as it can be appreciated in several studies that reported high levels of statistical anxiety [3–6] or "statisticophobia" [7–9]. Dykeman observed that, compared to students enrolled in general education courses, students in

statistics courses had lower levels of self-efficacy and higher levels of anxiety [10]. Ferrandino indicates that the teaching of statistics can be difficult for many reasons that have persisted over time, across disciplines, and across borders [11]. Ben-Zvi and Garfield pointed out different causes as, for example, that some statistical ideas are complex and/or counter-intuitive, that students lack of the required mathematical knowledge, or that they struggle in dealing with the context of the problem [12]. These difficulties can also be noticed in the large number of research articles devoted to explain students' misconceptions related to several statistical concepts [13–16]. These misconceptions appear in almost all the statistical topics, including descriptive statistics [17,18], probability [1,19], or statistical inference [14]. Therefore, developing tools that facilitate the acquisition of statistical concepts is essential. This need increases with the proliferation of online courses on statistics [20–22]. As an example, Reagan points out that several colleges are offering online statistics courses to enable students to complete their academic curriculum [23]. Mills and Raju also highlight the need to learn about how to effectively implement statistical courses [24].

To facilitate the understanding of the different statistical concepts and to correct the misconceptions, several methodologies have been proposed. Two of the most popular approaches are active learning [25] and the flipped classroom [26]. Active learning was initially defined as "anything that involves students in doing things and thinking about the things they are doing" [27]. A more recent and meaningful definition is the one proposed by Felder and Brent [28]. They define active learning as "anything course-related that all students in a class session are called upon to do other than simply watching, listening and taking notes". The use of active learning techniques in statistical courses has raised students' confidence in understanding statistics [29], increased knowledge retention [30], and improved students' performance [31].

In the flipped classroom methodology, knowledge acquisition takes place out of the classroom so that the class time can be devoted to more interactive activities. These can be designed, for instance, to polish the knowledge or to increase students' motivation. The flipped classroom has shown several benefits when it has been used to teach statistics. Wilson's [26] implementation of a flipped classroom in an undergraduate statistics course had a positive impact in students' attitude and performance. Winquist and Carlson found that students who took an introductory statistics course using the flipped classroom approach outperformed the ones that took the same course in a traditional lecture-based approach [32]. They observed that the implementation of a flipped classroom in an undergraduate statistics course had a positive impact in students' attitude and performance. McGee, Stokes, and Nadolsky adapted the flipped classroom methodology to identify students' misconceptions so they can correct them during the master class [33]. The results in Reference [34], when studying mathematics achievement and cognitive engagement, indicate that students in a flipped class significantly outperformed those in a traditional class and those enrolled in the online independent study class. Moreover, flipped learning with gamification promoted students' cognitive engagement better than the other two approaches. Faghihi et al. conclude that students who used the gamified system scored above the median, and their performance was greater than with the alternative method [35], whereas the results reported by Jagušt, Boticki, and Sin suggest that gamification improved student math performance and engagement [36]. Sailer et al. affirm that gamification is not effective per se, but that different game design elements can trigger different motivational outcomes [37]. In particular, their results show that badges, leaderboards, and performance graphs positively affect competence need satisfaction, as well as perceived task meaningfulness.

This article focuses on hypothesis testing which, according to Brewer [38], is probably the most misunderstood and confused of all statistical topics. This assertion can also be appreciated in the review conducted by Castro-Sotos et al. about the different students' misconceptions of statistical inference [14]. Difficulties in distinguishing sample and population [39], confusion between the null and the alternative hypothesis [40], misunderstandings about the significance level [41], and misinterpretation of the *p*-value [42] are only some of the different misconceptions related to hypothesis testing.

Apart from the above-described general purpose methodologies that have been used to facilitate the learning of any statistical topic, there are some research works devoted to teach hypothesis testing. One of the first works is the one conducted by Loosen, who built a physical device to explain hypothesis testing [43]. He found that his students were enthusiastic about the apparatus, but he did not report any quantitative measure. Schneiter [44] built two applets for teaching hypothesis testing. However, just like Loosen, she did not report about their performance. A more recent work is the one conducted by Wang, Vaughn, and Liu [45]. These researchers showed that the use of interactive animations improves the understanding of hypothesis testing but not students' confidence. Other works have proposed the use of specific examples, such as a two-headed coin [46] or a fake deck that only contains red cards (instead of half deck being red cards and the other half being black cards), to demonstrate hypothesis testing [47].

In this article, a new approach is proposed to facilitate the understanding of hypothesis testing, based mainly on the divide-and-conquer strategy, on the content exposure control, and on the use of a gaming environment that increases student motivation.

#### **2. Material and Methods**

The proposed approach is supported by six pillars. These aim at providing a solution to the factors that we assume are responsible for the difficulties in learning hypothesis testing: individual learning differences, misconceptions, knowledge gaps, and lack of attitude and motivation. The six pillars are:


The following section describes how the proposed approach, based on the previous pillars, is implemented in an application. After that, in Section 2.2, the design of an experiment, aiming at evaluating the performance of the proposed approach, is described. The obtained results are presented in Section 3. Finally, the article concludes in Section 4 with a discussion of the obtained results.

#### *2.1. The Hypothesis Testing Learning Tool*

The learning tool has been developed using the game engine Unity3D [55] and the Next-Gen User Interface (NGUI) library [56]. Following, its main components are described, beginning with an explanation of the main screen and its elements: buttons that lead to video lectures and problems, and the game elements. After that, the video lectures, problems, and content exposure control mechanisms are explained in more detail.

#### 2.1.1. The Main Screen

Figure 1 shows the main screen of the application, in which its different elements are shown. In particular, it contains the following elements:

**Figure 1.** Main screen of the application.

• **Buttons**. The application has two types of buttons: circular and square. Each circular button leads to a video in which a key concept of hypothesis testing is explained. The concept explained in each video is indicated on the label placed under each button. The courseware includes five concepts that are: type of contrast (mean, variance, proportion), sample vs. population, the null hypothesis, the alternative hypothesis, and the *p*-value and its interpretation to obtain conclusions. The square buttons present the student with a series of questions related to the concept explained in the previous circular button. As it will be detailed, these questions are intended to ensure

that the student does not misunderstand any concept. Until the student demonstrates sufficient knowledge of these questions, the following videos and problems are blocked.


#### 2.1.2. Videolectures and Questions

As mentioned, the buttons lead to video lectures and assessment questions. Next, we describe their functioning.

#### Videolectures

Each video explains one of the five above discussed concepts and has an approximate average duration of 6 min. Students can watch each video as many times as desired so that if they are not able to solve the problems related to a concept, they can watch the associated video again. The only restriction is that students can only watch the videos that they have unlocked.

#### Problems and Questions

As mentioned, our approach is based on a problem-based learning strategy [52]. For the construction of the problems, the difficulty level has been designed to stay within the student's Zone of Proximal Development (ZPD) [57], keeping it understandable and reachable, but challenging enough.

Each of the five key concepts is evaluated with six questions. These questions are obtained from six practical problems that the student must solve. One of these problems is shown in Figure 2. It can be observed that, in this problem, the student must identify the type of contrast, recognize the data sample, formulate the null hypothesis, enunciate the alternative hypothesis, and, based on the *p*-value, determine if it is possible to reject the null hypothesis or not. It is important to note that students do not initially receive all the questions related to a problem at once. Instead, they only receive the question, for each problem, corresponding to the current concept, remaining within their ZPD. Questions related to more advanced concepts appear invisible until the student has reached that level. That is, if a student has currently watched the video that explains the concept of the data sample, he/she will only receive the second question for each of the six problems. Below, these questions are described in more detail.


the student must identify the problem values belonging to the sample and enter them in the correct text boxes, that is, in the boxes labeled with the corresponding notation to the sample statistics and the sample size. In addition to these text boxes, there are other text boxes related to different population parameters in which the student must not include any value.


**Figure 2.** One of the problems (translated to English) included in the courseware.

As previously mentioned, these questions evaluate students' knowledge about each concept and are responsible for determining if a student can advance to the next concept or not. When a student correctly answers the six questions, he/she receives two stars, and, when he answers four or five questions correctly out of the six, the student receives one star. In these two cases, the following concept is unlocked, and the student can move forward. When the student answers less than four questions correctly, he/she does not receive any star and must redo these questions. Following the scaffolding approach, each time the student answers a question erroneously, he/she receives formative feedback in the way of a message that helps him/her to identify his/her error, facilitating his progress. Therefore, it is important to emphasize that the questions do not only assess students' knowledge; the fact of receiving feedback and having to repeat the questions until achieving a minimum knowledge allows students to better understand the concept and to correct their misconceptions.

In addition, the student receives 10 points per question answered correctly on the first attempt. As there are 6 questions, it is possible to obtain a maximum of 60 points per level. The questions answered correctly on the second and subsequent attempts provide 5 points. The student must repeat the level if he/she was not able to correctly answer up to four questions. Students that correctly

answered four or five questions can repeat a level, until having correctly answered the six questions and obtaining the two stars.

#### *2.2. Evaluation of the Approach Application*

An experiment was conducted to evaluate the benefits of applying the proposed approach with respect to a traditional master class. Following, the details of the experiment are presented.

#### 2.2.1. Sample

A total of 89 students, who were studying the subject of Statistics belonging to the degree of Industrial Engineering, participated in the study. Students were informed that their participation was voluntary and was under no circumstances considered in their academic evaluation. In order to make the data collected anonymous, a random number was given to each student to access the application and sign the tests with it. None of the researchers involved in data collection and analysis were teaching the students who participated in the experiment. After the study, all students were granted access to the application and all the material.

#### 2.2.2. Assessment of Learning Methods

In order to assess the students' knowledge acquisition, two comparable problems were prepared. These two problems were intended to determine whether the student had been able to identify: (i) the type of contrast; (ii) the data pertaining to the sample together with the appropriate notation; (iii) the null hypothesis; (iv) the alternative hypothesis; and, finally, (v) the interpretation of the *p*-value in relation to the problem in question. Each of the problems was scored either with 1 (totally correct), 0.5 (partially correct), or 0 (incorrect). These two problems were:

Question (Q1): The average expenditure per customer in a store was 89 euros before the recession. Currently, taking a sample of 70 shopping carts, an average of 86 euros with a standard deviation of 9 euros is obtained. According to these data, and assuming a significance level of 0.05, could we affirm that you can see the effect of the current recession? Provide your answer by identifying the type of contrast, the sample data, the null and alternative hypothesis, assuming a *p*-value of 0.003.

Question (Q2): The ideal weight for 1.80 m tall men is 75 kg. Given a sample of 45 men that are 1.80 m tall in Spain, the average weight turns out to be 77 kg with a standard deviation of 8.5. According to these data, can we say that the Spanish are too fat? Provide your answer by identifying the type of contrast, the sample data, the null and alternative hypothesis, assuming a *p*-value of 0.06 and a significance level of 0.05.

#### 2.2.3. Experimental Set-Up and Study Groups

The 89 students who participated in the experiment were divided in 3 groups according to their interest in using the application and their time availability. Students who decided to use the application indicated their time availability to participate before and/or after the master class. With this information, we tried to balance the number of participants in each of the groups as much as possible. The first subgroup, which will be referred as G1 from hereafter, was composed of 10 students. The students belonging to this group used the developed application some days before the class during a session of one hour. These students did not have any prior knowledge about hypothesis testing when they participated in this activity. Days later, they attended to the master class in which hypothesis testing was explained. The second group, which will be referred as G2, was the control group. It was made up of 60 students who attended only to the master class. Finally, the group G3 contained the remaining 19 students who enrolled for attending to the master class and days later had access to the application for one hour. There were two students of the G1 who used the application, but they did not attend later to the master class. There were also 4 students from group G3 who attended to the master class, but they did not appear later to use the application.

Next, we describe the order in which each group answered the assessment questions. Group G1 received the Q1 question immediately after their session (prior to the lecture) and the Q2 question after the lecture. Groups G2 and G3 received the Q1 question after the lecture class (none of them had performed used the application). In this way, it was possible, on the one hand, to use Q1 to compare the application of the proposed approach and the master class. On the other hand, it also allowed comparing the performance of groups G2 and G3, which is important to ensure that there are no differences between students who had signed up for using the application from those who did not, as both groups had simply attended the master class by the time they answered the question Q1. Lastly, students belonging to G3 answered question Q2 after using the application. In this way, it was possible to analyze if it is preferable that students to conduct the session with the application before or after the lecture. The design of the experiment is shown in Figure 3. From now on, we will refer to the different sets of scores through the pair formed by the student group and the question presented.

**Figure 3.** Experimental design including the 3 study groups (G1, G2, and G3), the two assessment questions (Q1 and Q2), and the results of the statistical analysis.

Additionally, the students who participated in the extra session filled out a questionnaire to evaluate the acceptance of the application. This questionnaire contained eight five-point Likert items and two open-ended questions.

#### **3. Results**

In this section, we present the results obtained. Figure 3 shows the mean and the standard deviation of the scores obtained for each one of the participant groups in the different phases of the study. Before examining these data and the significance of the differences, it is important to present a result obtained when analyzing the distribution of the scores. This analysis revealed that the distribution of the scores obtained by the students who used the application followed a Gaussian distribution, whereas, when students had only attended to the lecture class, it followed an exponential distribution. Concisely, the goodness-of-fit test to normality Kolmogorov–Smirnov yielded *p*-values of 0.99, 0.38, and 0.51 for the answers obtained in the pairs-question group G1-Q1, G1-Q2, and G3-Q2, respectively. Similarly, a Chi-square goodness-of-fit test showed that the distribution scores in G2-Q1 and G3-Q1 followed an exponential distribution (*p*-values 0.14 and 0.18, respectively).

After analyzing the distribution of scores, several hypothesis tests were performed to understand the results obtained. First, we compared the students' scores in the scenario (G1, Q1) with those obtained in (G2 and G3, Q1). A *t*-test showed that the differences were significant (*p*-value < 0.001). This result indicates that the proposed approach seems to be more effective to teach hypothesis test concepts than the lecture class.

Similarly, a *t*-test also showed that the differences between (G2 and G3, Q1) and (G3, Q2) were also significant (*p*-value < 0.001). This result also indicates that, similarly to the previous result, the application of the proposed approach after the master class significantly improves the knowledge acquired by students during the lecture.

Another interesting comparison was the one between the scores obtained in (G1, Q1) and (G1, Q2). The corresponding *t*-test provided a *p*-value close to 0.05, which seems to indicate that the lecture class reinforces the knowledge obtained initially with our approach.

In the comparison of the scores (G1, Q2) and (G3, Q2), the *t*-test indicated that the results were not significant (*p*-value 0.14). However, although this difference is not significant, performing the application prior to the class seems to provide better results, which is consistent with the flipped classroom methodology approach.

Finally, the scores obtained in (G2, Q1) and (G3, Q1) were compared to determine if the participants who used the application after the master class and those who participated only in the master class were similar. Since the scores in these groups followed an exponential distribution, the non-parametric Mann–Whitney test was used to analyze the difference. This test showed a *p*-value of 0.22, indicating that there were no significant differences between these two groups.

Regarding the subjective acceptance questionnaire about the application of the proposed approach, Table 1 shows the obtained results. This table shows the eight items that were presented to the students together with the mean and the standard deviation of their responses. These numbers indicate that the students considered that the application helps to improve assimilation of concepts and that it should be promoted. This can be seen in the values obtained for items 1 and 2, in which the means were 4.30 and 4.80, respectively. The students also considered that the experience was positive (item 8). However, despite these numbers, the students consider this application as a complement to the course but not as a replacement to the teacher (items 5, 6, and 7). On the other hand, despite that explanatory messages were clarifying (item 3), this questionnaire also allowed discovering elements that need to be improved, such as its comfort of use (item 4). In particular, the participants complained about the fact that our video player did not have the option of reproducing only certain parts of the video.

**Table 1.** Results from questionnaire about the use of the application.


In addition to the eight items above, students answered two open-ended questions. The first question asked the student how he/she had felt when his/her name moved up on the leader-board. This question was answered by 14 students, 13 of whom (92.8%) provided answers related to increased motivation, while the remaining student replied that he did not paid attention to the leader-board. The second question asked students if they had felt the need to overtake other participants on the leader-board. Of the 16 students who answered this question, 13 (81.2%) answered affirmatively and justified their response on the basis of competitiveness. The remaining three students answered that their goal was either to learn or their personal improvement.

The students' subjective satisfaction, together with the objective values shown above assessing students' knowledge, show the benefits of applying the proposed approach.

#### **4. Discussion and Conclusions**

In this article, a new approach has been proposed for the teaching of hypothesis testing. It has taken into account different elements that have been proposed in the literature to facilitate statistical learning, such as the inclusion of video lectures that allow students to learn at their own pace and a problem-based learning approach to encourage active learning. Additionally, the proposed approach includes a mechanism for content exposure control. In this way, students cannot access to any new concept until they demonstrate having understood the previously presented concepts. This mechanism provides several benefits. First, it avoids that, if the student has understood a concept in a wrong way, this error spreads to the following concepts. Second, when students make an error, they receive immediate formative feedback about it and have the opportunity to correct errors as they occur, providing a good basis to understand the next concept. In addition, it increases the students' confidence making them capable of solving the different questions. Finally, the application that incorporates the proposed approach contains elements of gamification to increase student's motivation.

In the experiment, carried out to identify the advantages and disadvantages of the proposed approach with respect to a traditional class, it was observed that students assimilated the concepts much better when they benefited from the proposed approach. This was true regardless of whether our approach had been done before or after the traditional class, although results seem to indicate that the advantage is greater if it is performed earlier, which is consistent with the flipped-class learning approach.

The students who used the application based on our approach were able to complete almost perfectly the six questions that were formulated with respect to the five concepts involved in the resolution of a hypothesis test. Conversely, in the traditional class, they correctly answered 1.34 concepts on average, in the best of cases. A possible explanation of this remarkable difference can be attributed to the fact that, in a master class, the different concepts are explained sequentially. If students are not able to fully understand one of them, they will have serious difficulties in understanding the subsequent ones, limiting their participation to taking notes that they will use later to try to understand the subject. Regarding the gamification elements, more than 50% of the students expressed that these elements enhanced their competitiveness and made learning more entertaining.

If we compare our proposal with others in the literature, we observe that our students also experimented difficulties understanding the statistical concepts when following the master class, corroborating the conclusions of the review made by Castro-Sotos et al. [14]. In addition, as suggested in Boyle et al. [2] and Sailer et al. [37], we included game design elements, such as stars as game badges, a leader board, a scoring system to indicate performance, and even a flag carrier to show the progress of the most advanced student in the session. Regarding performance, though most studies on learning statistics propose improvements, few carry out a formal evaluation. Wang, Vaughn, and Li [45] did evaluate performance after using different animation interactivity techniques, and their results indicated that the increase of animation interactivity could enhance student achievement improvement on understanding and lower-level applying but not on student remembering and higher-level applying. In terms of confidence improvements, there were no significant differences between their four groups. In turn, our results, depicted in Figure 3, reflect high student satisfaction and a mean significant difference of about 2 points out of 5 (*p*-value < 0.001). As discussed above, our work stands out for dividing complexity into simpler concepts that students can only access in a controlled, progressive

manner, and for including elements of gamification, demonstrating a beneficial effect on understanding complex issues and obtaining positive results in terms of student satisfaction.

Some limitations of this research should be noted. First, the generalization of the conclusions of this study to other research contexts, subjects, and group sizes should be studied, due to our limited sample size. Despite this fact, our results showed significant improvement in performance. Another current limitation is that most learning platforms do not include facilities to implement our proposal to control content exposure, which means that it must be specifically implemented for each case. In this study, we created a small learning engine. We hope that, as more studies demonstrate the usefulness of this approach, tools, such as Moodle or EdX, could consider implementing such functionality.

These results show the need to rethink traditional lectures to include the benefits derived from methodologies that incorporate current technologies. The acquisition and understanding of difficult concepts by dividing them into steps or simpler concepts allows students to build the learning. The content exposure strategy not only controls the progress of students, but it is also a very interesting tool for the teacher to see what concepts are hard to master, as well as to evaluate whether it is worth dividing them even further or generating additional material. In addition, it offers the advantage of providing an overview of the overall progress of the group. Our work shows that, just by using the application that implements the methodology, the students' results are reasonably good. This could offer the possibility of redesigning the master class in a different way, to be more oriented to clarify concepts or to increase the cognitive level of learning.

We plan to apply the proposed approach for learning different subjects. In addition, not only gamification elements could be included, but also video games themselves. For example, we are currently developing a video game to teach quality control. In this video game, the student is responsible for controlling the filling of a bottling company. To do this, using an avatar, the student must collect various samples, measure them, and create the control charts. And later, he/she will have to detect whether or not the process is under control.

This work is only a first step, but its extensions are immediate. For example, since the application has been well accepted by the students and its usefulness has been demonstrated even without the intervention of the teacher, it could be offered as a self-learning tool aimed at pre-balancing basic statistical knowledge among students before starting other courses. It would also be interesting to dynamically create specific itineraries for each student, tailored according to the particular results obtained. Future versions can include the adaptive selection of exercises that are based on the detected errors, fostering deliberative practice [58], since this would enable students to pay more attention to their weaker areas. Furthermore, with the growth of online courses, especially with COVID-19, providing learner-centered tools that ensure good understanding becomes even more relevant. We want to finish this article with a phrase that we heard a student say to another when the proposed activity ended: "Now, finally, I understood it all".

**Author Contributions:** Conceptualization, D.D.-G. and S.B.; methodology, D.D.-G.; software, F.G.-L.; formal analysis, D.D.-G.; investigation, D.D.-G.; resources, D.D.-G. and C.M.-B.; data curation, D.D.-G.; writing—original draft preparation, D.D.-G. and S.B.; writing—review and editing, A.S., D.D.-G., L.M., and S.B. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Abbreviations**

The following abbreviations are used in this manuscript:

ZPD Zone of Proximal Development

#### **References**


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Application in Augmented Reality for Learning Mathematical Functions: A Study for the Development of Spatial Intelligence in Secondary Education Students**

**Francisco del Cerro Velázquez and Ginés Morales Méndez \***

Energy Engineering and Teaching Innovation Research Group in Technology, Faculty of Education ("C" Building), Campus de Espinardo, University of Murcia, 30100 Murcia, Spain; fcerro@um.es **\*** Correspondence: gines.morales@um.es; Tel.: +34-868-887-696

**Abstract:** Spatial intelligence is an essential skill for understanding and solving real-world problems. These visuospatial skills are fundamental in the learning of different Science, Technology, Engineering and Mathematics (STEM) subjects, such as Technical Drawing, Physics, Robotics, etc., in order to build mental models of objects or graphic representations from algebraic expressions, two-dimensional designs, or oral descriptions. It must be taken into account that spatial intelligence is not an innate skill but a dynamic skill, which can be enhanced by interacting with real and/or virtual objects. This ability can be enhanced by applying new technologies such as augmented reality, capable of illustrating mathematical procedures through images and graphics, which help students considerably to visualize, understand, and master concepts related to mathematical functions. The aim of this study is to find out whether the integration of the Geogebra AR (Augmented Reality) within a contextualized methodological environment affects the academic performance and spatial skills of fourth year compulsory secondary education mathematics students.

**Keywords:** augmented reality; spatial intelligence; STEM; mathematics; Geogebra AR; secondary education

### **1. Introduction**

The term function in mathematics is defined as any relationship between two or more variables that can be represented graphically. Function learning provides students in Compulsory Secondary Education (ESO) with their first contact with the identification, visualization, and interpretation of the relationship between two independent variables and is therefore a key point of transition within mathematical development figure [1]. The cognitive transition of graphically representing a constant, linear, affine, quadratic, exponential, absolute value, inverse proportionality, and logarithmic function from its algebraic expression is included in the curriculum of this educational stage and tends to be a challenge for most students.

This study is based on research integrating ICT in the classroom, where we can detect their benefits and drawbacks, design resources to help implement these technological tools, collect and analyze data, and reflect on the results. These action research elements provide a backdrop for teachers to recreate a digital and proactive environment in the classroom within a contextualized methodology that favors the teaching-learning processes of mathematics, with the aim of making students the protagonists in the construction of their knowledge.

Several studies claim that the inclusion of ICT in the teaching and learning of mathematics helps students to visualize how changes in one variable affect others immediately, thus improving their experience and interaction with learning compared to solving formulas so as to obtain the answer [2–6]. It is common for students to associate the representation of functions with a collection of isolated points rather than a single entity, making it difficult

**Citation:** del Cerro Velázquez, F.; Morales Méndez, G. Application in Augmented Reality for Learning Mathematical Functions: A Study for the Development of Spatial Intelligence in Secondary Education Students. *Mathematics* **2021**, *9*, 369. https://doi.org/10.3390/math9040369

Academic Editor: Francisco D. Fernández-Martín

Received: 13 December 2020 Accepted: 8 February 2021 Published: 13 February 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

to visualize and interpret graphically [7–9]. As a consequence, students often do not visualize and interpret correctly the representations of graphic functions as a solution in itself, therefore they do not manage to conceive the transition process from algebraic language to visual language and vice versa. Therefore, we pose the following questions: How could ICT based on augmented reality facilitate the process of representation, visualization, and analysis of algebraic functions? Is this cognitive-visual process linked to students' spatial intelligence?

#### *1.1. Spatial Intelligence*

According to Bishop's theory, an individual acquires the capacity for spatial visualization through three distinct stages of development [10]. In the first stage, children learn topological spatial visualization, where they can understand the relationship between different objects in space, i.e., the location of an object within a group of objects, the isolation of the object, etc. In the second stage of development, they acquire projective representation, where they can conceive how an object will look from different perspectives. Finally, the final stage of the development of spatial visualization is based on combining spatial projection skills with distance measurement.

On the other hand, spatial intelligence corresponds to one of the eight intelligences of the model proposed by Gardner [11] in the theory of Multiple Intelligences (MI). This type of intelligence implies having the capacity to perceive the visual world with accuracy, to mentally recreate objects or models, even in the absence of physical stimuli, and to carry out transformations or modifications of them.

In the study of the so-called knowledge areas of Science, Technology, Engineering, and Mathematics, better known by its popular homonym in English as STEM, this type of intelligence is fundamental for students to develop the ability to transfer numerical data and two-dimensional projections to three-dimensional objects with ease [12,13]. Within the contents of the subjects of Secondary Education, this skill has numerous applications, such as the conception and construction of spatial models, the analysis of geometric objects, the interpretation of diagrams, and the identification of functions among others.

The term spatial intelligence covers five fundamental skills: Spatial visualization, mental rotation, spatial perception, spatial relationship, and spatial orientation [14].

Spatial visualization [15] denotes the ability to perceive and mentally recreate twoand three-dimensional objects or models. Several authors [16,17] use the term spatial visualization to indicate the processes and abilities of individuals to perform tasks that require seeing or mentally imagining spatial geometric objects, as well as relating these objects and performing geometric operations or transformations with them.

Shepard and Metzler [18] define mental rotation as the cognitive ability to rotate ideal representations of dimensional and/or three-dimensional objects or models, and can be described as the movement of representations through the brain to help conceive each of its views or perspectives regarding a turn.

According to Gibson [19], spatial perception is defined as the ability to visually perceive and understand external spatial information, such as characteristics, properties, measurements, shapes, the position, and movement of an object in relation to an individual.

On the other hand, the spatial relationship determines how an object is located in space in relation to another reference object and this skill is the basis of cognitive development for walking and trapping objects in space [20].

Finally, we can refer to spatial orientation as a fundamental ability to move and locate oneself in space [21,22], being necessary for such common activities as writing straight, reading, differentiating between right and left, and, in general, locating objects and orienting them in space.

These five skills are malleable and can therefore be reinforced through the use of multi-sensory tools or applications that stimulate and improve these abilities [23]. However, the traditional method for teaching visual and spatial skills to students is based on analyzing and interpreting two-dimensional images, orthogonal views, and graphics on a

blackboard or paper. This method has obvious limitations, as it hinders the conceptualization and assimilation of contents due to the lack of interaction between students and the representations [24].

This study relates the development of spatial skills to the representation of two and three-dimensional functions in mathematics, and demonstrates that augmented reality technology contributes to the improvement of spatial skills and the understanding of highly visual content. This might be due to the observation and experimentation of the models from different angles and relative positions, respecting the individual learning pace of each student. Some studies [25,26] state that visual and spatial abilities can be improved by emerging technologies such as augmented reality. The integration of this technology in the classroom favors a constructivist approach to learning by allowing teachers to introduce tangible and proactive experiences in the classroom where students interact and manipulate with the learning object. As educators, we must show a positive attitude towards the integration of ICT in education, as it effectively changes the way students learn [27], however, a lot of work still needs to be done in order to achieve a systematic development of augmented reality for educational purposes.

#### *1.2. Augmented Reality as a Methologocial Resource in Teaching-Learning Processes*

Augmented Reality, AR henceforth, offers multiple benefits that support the teachinglearning process. The applications of AR allow the human-machine interaction to be more natural by enabling the preservation of the user's environment, providing a real frame of reference which the user can rely on to perform certain actions. This process can be achieved through the superimposition of virtual objects in a real environment. Students can experience the ability to combine their real environment with a virtual one designed, in this case, by themselves.

This technology allows any real environment to be enriched with digital information through the use of a camera and software that in recent years has focused its development on mobile devices which, due to their portability, contribute to off-site learning, where any scenario can be transformed for training purposes [26,28,29].

The reports of New Media Consortium [30–35] that identify and describe the trends, challenges, and technological advances in education, estimate that AR technology will be established in secondary and higher education classrooms in the short term as an information access tool that will generate new applications of technology in the learning process.

This indicator, together with the omnipresence of mobile devices, which have become the main tool for accessing information in different formats and in an immediate form, can be used as access portals to Open Educational Resources (OER) that adapt the pace of learning to the needs of each user; it combines an AR-mobile device binomial that equates access to learning opportunities and facilitates the provision of mobile, interactive, individualized, and adapted learning services [26].

The integration of AR technology into the field of education has enabled an evolution of the educational model. Initially, this technology was used only as a tool for immediate access to digital information, involving students in the theories of behaviorism and objectivism. Recently the applications of this technology are undergoing some changes, with students moving from being recipients to providers of knowledge and the teacher taking on the role of guide and tutor with the objective that students generate knowledge using this technology in an interactive way, where the main theories of this new model are: Cognitivism, constructivism, and constructionism [36].

The fact that the educational scene is one in which the acquisition of digital competences is particularly relevant must be noted [37], although the vast majority of technological tools and resources do not promote the same learning opportunities for all. The Sustainable Development Goal 4 aims to ensure inclusive, equitable, and quality education and promote continuous learning opportunities for all. Mobile devices are driving a revolution in education, allowing learners to access learning resources anywhere, anytime. Therefore, the role of mobile learning is relevant, as it has the ability to help break down eco-

nomic barriers, differences between rural and urban areas, as well as functional limitations. The omnipresence of mobile devices is changing the way people interact with information and their environment. In addition, the continuous improvement of the hardware of these devices and their reduction in cost, positions them as the first tool for accessing the most widespread information worldwide [26]. Consequently, in order to conduct this study, mobile devices were chosen as the learning platform, since all students had one or had access to them, thus guaranteeing access to training for all students.

Thanks to new technologies, we enter for the first time a place where we interact with real objects and at the same time with virtual ones, which allow us to remember previous learning and restructure our thinking, thus giving meaning to what we perceive from the surrounding world. As Vigotsky [38] stated, people develop ways of interpreting and strategies to relate to physical and cognitive space in such a way that this type of interaction can be established with tools and systems that provide various types of stimulation, thus it is certain that the use of AR will lead to substantial changes in the way knowledge is accessed, interpreted, and communicated, which must be considered in the field of education [39].

AR as an integrated technology in teaching acquires a dimension that emphasizes sensory transformation, so if it is integrated into the teaching-learning processes it could promote meaningful and contextualized learning acquired through multiple sensory experiences [40].

This technology can be used in education to represent 3D models of objects that, because of their size, cost, danger, distance and tangibility, are not within the real reach of students. Moreover, working in contexts with AR, there is a direct interaction with the environment or the object of study, making learning more meaningful.

With the representation of objects in 3D through AR technology we have the freedom of spatial exploration, so students can really perceive and understand space as it is. In addition to spatial perception, students can view models in space and modify parameters that alter their geometry. In this way, the spatial visualization is exercised and they can rotate or flip these representations to visualize each of their perspectives or views, thus promoting spatial rotation. At the same time, and while the user observes the parameters that correlate various objects recreated in space and places the designs in the plane, the skills of spatial relationship and orientation are also developed. With all this, we stimulate, work, and enhance all the fundamental fields of spatial skills established by Maier in 1994 [14] through a multi-sensorial tool, such as AR and mobile devices.

#### *1.3. Geogebra AR as a Tool to Support the Learning of Mathematical Functions*

In accordance with the constructivist theory, it is believed that technology can help students in teaching-learning processes. One of the first technological tools for learning functions is graphical calculators, which emerged as an instrument to enable students to solve systems of equations, represent graphs, and perform other tasks with variables [41]. Despite their benefits, these calculators have limitations when solving and representing certain expressions due to their small output interface. In addition, they must be implemented cautiously, as many students have difficulties when using symbols, which can be counterproductive and slow down the resolution of operations [42].

The most recent graphical interfaces offer direct manipulation mechanisms for the representation of mathematical functions, allowing users to interact intuitively and directly in the visualization they are editing, providing immediacy and simplicity when obtaining results, and helping their interpretation and learning. The term direct manipulation describes a style of interaction that stands out for the following characteristics: Continuous representation of objects and actions of interest; change from complex command syntax to manipulation of objects and actions; fast, incremental, and reversible actions that have an immediate effect on the selected object [43]. Therefore, direct manipulation is, by far, the most common type of interaction in mobile applications, and it is found to a greater extent in AR interfaces, since it provides us with an immediate handling of virtual objects in our real environment.

Numerous research studies claim that didactics through AR applications positively influence students' attitude and motivation towards learning [44–53], providing an active teaching environment where the capacity for enquiry and research is encouraged, while promoting the development of autonomous student work in their learning [26,54]. Likewise, several studies state that the correct integration of AR applications in the classroom improves students' learning results [55–59].

Despite the numerous research studies cited on AR resource didactics, few are concerned with the possible impact of AR technology on spatial intelligence [12,60] and, thus, there is an interest in conducting research so as to determine if there is a real contribution of AR to the acquisition of spatial skills.

In order to explore the development of spatial intelligence in relation to mathematical learning, our classroom experience revolves around the open source application, Geogebra AR, for mobile devices which helps students learn analysis, geometry, algebra, and calculus. This mathematical application is specifically designed for educational purposes. It allows the dynamic drawing of geometric constructions of all kinds, as well as the graphic representation, algebraic treatment, and calculation of functions in a simple and effective way, which permits us to use it as a support tool for the study, promoting mathematical selflearning. There is a large volume of research that has shown that Geogebra, in its version for personal computers, has been effective for the teaching-learning of mathematics [61–65], improving the understanding of abstract concepts and enabling their correlation through a meaningful and effective learning experience.

In its AR version, it allows us to generate 3D objects and mathematical functions, which we can place on an imaginary plane in our real environment (Figure 1a) and then experiment with them in a tangible way, being able to visualize and rotate them with total freedom, which helps to improve the understanding of the function itself through manipulative learning. The user interface of the Geogebra AR application is direct and intuitive. At the bottom of the screen, it includes a section where we can introduce the algebraic expressions of our naturally defined functions, as they appear in the textbooks or as they are written by the teacher on the blackboard, through a virtual keyboard incorporated in the mobile device, generating immediately the graphic representations of the introduced functions (Figure 1b).

**Figure 1.** Geogebra AR (Augmented Reality) interface: (**a**) Surface detection and (**b**) introduction and representation of functions.

Through the application menu, located in the upper left corner, we can search and open existing resources, save and share our work, as well as make changes to the program settings (hide or show axes, change the coordinate grid, distances between axes, hide or show descriptions or labels, etc.).

The application design promotes the learning and analysis of mathematical functions, not only generating them in AR, but also emphasizing the cognitive-visual process that

occurs when an object is built in space. In particular, introducing the algebraic expression of defined functions, representing them in space and interacting with them in AR, is a major cognitive step in the transition from algebraic expression, through 2D linear designs, to the 3D object representation that covers the five fundamental skills of Maier's spatial intelligence [14].

#### **2. Materials and Methods**

#### *2.1. Researh Design*

The research approach adapted for this study is based on a quasi-experimental design. Two pre-test/post-test models were applied to each of the two ordinary class groups, formed by students who do not have any type of special educational need, that participated in the study: One to assess the level of spatial ability and the other to determine the level of learning of mathematical functions. The experimental group underwent a contextualized methodology that integrated the binomial RA-mobile devices for the use of the Geogebra AR application in the study of mathematical functions, while in the control group, a traditional teaching-learning methodology was used. At the end of the experience, the experimental group completed a questionnaire in order to obtain the students' perceptions after using Geogebra AR.

#### *2.2. Researh Objectives*

The research question posed is whether there is a significant difference between students who use the application of Geogebra AR in a contextualized methodological environment and those who use traditional teaching-learning methods with regard to their spatial intelligence and the level of learning acquired. In order to assess the scope of these research objectives, the following hypotheses are established:


#### *2.3. Sample*

The total number of participants was 48 students, who were taking the subject Academic Mathematics in their 4th year of ESO, taught by one of the teachers who conducted this study. Out of the total number of participants, the 47.92% (f = 23) belonged to the experimental group and 52.08% (f = 25) belonged to the control group, presenting no significant curricular adaptations. The sample used in the research is non-probabilistic and, as a consequence, the results cannot be generalized with statistical precision [66].

#### *2.4. Data Collection Instrument*

The study uses three different instruments to collect information: A pre-test/post-test model to evaluate spatial intelligence, a second pre-test/post-test model, which is a written test to detect previous knowledge, and another one to evaluate the learning standards of the functions block within the curriculum of the subject Academic Mathematics in the 4th year of ESO. Finally, the students were given a questionnaire to detect the motivation levels of the experimental group.

There are several standardized tests to measure a person's ability in the first two stages of spatial development. For our study the Purdue Spatial Visualization Test: Rotations (PSVT:R) has been used because of its design to evaluate a person's ability in the second stage of spatial development [67]. Figure 2 presents a random question extracted from the PSVT:R test. This 12-item test has been used as an evaluation instrument at the beginning and end of the experience in the experimental and control group, with the aim of identifying the level of visualization and spatial rotation that the students started from, and to evaluate the impact on the spatial intelligence of the students through the experience in the classroom with the Geogebra AR application, as an aid for the analysis and study of mathematical functions.

**Figure 2.** Sample Purdue Spatial Visualization Test: Rotations (PSVT:R) test question (correct answer D).

Likewise, and in the perspective of evaluating the learning of mathematical functions within the block of contents of functions in the curriculum of Academic Mathematics in the 4th year of ESO in Spain established by the Royal Decree 1105/2014 [68], an individual written test of detection of an initial assessment of knowledge and another final assessment test made up of 8 items that includes the evaluable learning standards were used as data collection instruments, having been both instruments designed by the authors of the study.

After the final test, the experimental group carried out a 10-item Likert scale questionnaire with 6 answer options so as to identify the feasibility, motivation, and students' perception of the experience, thus evaluating the AR enriched learning environment. The questionnaire focused mainly on determining the following aspects:


Finally, the reliability of the evaluation instruments designed by the authors of the research (written test and Likert questionnaire) is established by means of Cronbach's internal consistency coefficient α [69], considered by several researchers to be one of the most appropriate statistical methods to obtain quality values [51,70,71]. Table 1 shows that the internal consistency reliability indexes are adjusted to a high level for each one of the scales that constitute the evaluation instruments elaborated.

**Table 1.** Internal consistency reliability coefficient for designed tests.


Once the data from the PSVT:R test and the individual written test were collected, they were analyzed using descriptive and inferential statistics. The descriptive statistics are composed of the mean obtained from the pre-test and post-test results, the standard deviation, the range, etc. On the other hand, for inferential statistics, a student *t*-test with a 5% confidence level is used along with a bilateral test to test the study hypothesis.

#### *2.5. Learning Experience*

In May 2019, the classroom experience was carried out with 4th year ESO Academic Mathematics students, distributed in 12 class sessions within a three-week period. The objective of this trial was to determine the scope and limitations of integrating the mobile device in the classroom with the Geogebra AR application (Figure 3), as a support for the

analysis and study of mathematical functions, in addition to checking its impact on the spatial intelligence of the students.

**Figure 3.** Students working in the classroom during the development of the experience.

The learning standards that are evaluated within the block of content of functions of the curriculum of the subject of Academic Mathematics in the 4th of ESO in Spain, established by Royal Decree 1105/2014, explicitly indicates that students must explain and graphically represent the relationship model between two magnitudes for cases of linear, quadratic, inverse proportionality, exponential, and logarithmic relationship, using technological means, if necessary. This makes it flexible enough to allow the introduction of other teaching methods such as approaches based on new technologies, in our case Geogebra AR, which facilitates the exploration, representation, and analysis of functions among other things. Therefore, by integrating Geogebra AR as a support to the teachinglearning of functions, students can explore and develop cognitive schemes that allow them not only to draw graphs of functions, but also to enhance proactive self-learning by achieving a progression in the development of analysis, application, reflection, and interpretation of knowledge.

#### *2.6. Generated Material*

To carry out the experience in the classroom, worksheets were generated, integrating the mobile device as a platform for access to classroom learning through the application Geogebra AR in order to solve the proposed activities. In relation to the above, it should be noted that the teachers do not necessarily have to follow the textbook, but they can create their own work material, in this case cards linked to objects in AR. In order to do this, teachers must have enough knowledge. In this sense, some authors design their own activity cards or OER work materials in what they call "production of augmented materials" which is generally systematic and sequential, adapting to the learning rhythm and needs of each user [12].

The collection of contents generated deals with aspects such as the representation, study, and analysis of functions such as: Constant, affine, linear, quadratic, absolute value, inverse proportionality, exponential, logarithmic, and trigonometric. These materials were used in paper format (Figure 4), so that the students could solve the activities in written form while superimposing in the work card the graphic representations in AR generated by Geogebra AR. A QR code was located at the bottom of each worksheet, giving access to downloading the application.

**Figure 4.** Worksheets with RA content, with QR code for access to the Geogebra AR application.

In this way, students interact directly with the object of study with total freedom of spatial exploration, rotating or flipping the representations to visualize the function in total detail and from any perspective. It is important to emphasize that the activities that are part of the collection of exercises are not far from a traditional teaching methodological framework of mathematical functions, which gives a great advantage when integrating technological AR tools as Geogebra AR.

Although students had never used interactive mathematical software in AR as a teaching tool before, Geogebra AR's smooth learning curve allowed us to design a classroom experience with a discovery-based learning format. Therefore, instead of dedicating teaching sessions to explain the operation, tools, or elements of the program interface, a routine was established in the classroom based on brief instructions and directed activities through proactive and tangible learning that made students gradually master the software according to the demands of each activity, their needs, and inquiry. As in the development of any other training unit, students were assigned tasks to perform outside school hours. The use of the binomial RA-mobile devices allows students to access information regardless of where they are, thus combining classroom work with online work, which results in an educational model closer to the needs of new generations known as b-learning [72]. This has a greater significance nowadays due to the change of paradigm that the educational system is facing in times of Covid-19, and due to the leading and essential role that technologies have taken, we are facing a scenario in which we must help strengthen self-learning and autonomy in students, as well as motivate them to help capture their interest and enhance their desire to investigate [73].

#### **3. Results**

During the execution of the experience it was observed in the experimental group that, firstly, the students quickly learned to generate graphic functions through the application as an alternative to the traditional system of representation. Secondly, students learned to visualize and analyze graphical solutions as an alternative to algebraic solutions. Thirdly, students moved from conceiving a graph as a collection of isolated points, to thinking of a graph as an entity, which caused them to begin doing comprehensive studies and analysis of function behavior. Fourthly, students understood the conceptualization of a function and understood the relationship of variables over them. Fifthly, it was detected that the students experimented freely and autonomously with the Geogebra AR application and contributed to the rest of the group with their perception of the operations carried out. It should be noted that these interpretations were typical of students from higher education levels.

Finally, one of the findings observed in the experimental group is that students related the different solutions between the systems of equations through their graphic representations. This shows us that students are able to visualize and identify a point or a line of intersection in a graph as a solution to a system of equations (Figure 5).

**Figure 5.** Students in the classroom working the graphical intersection through Geogebra AR.

*3.1. Analysis of the Variation in the Teaching-Learning of Mathematical Functions*

The descriptive statistical results of the initial knowledge assessment test within the function content block for both the experimental and the control group are shown in Table 2.


The experimental group with 23 participants obtained a mean score in the initial evaluation test of 5.7478, while the control group obtained a mean score of 6.0921. A *t*-test for independent samples was carried out to determine if there was a significant difference between the mean score of the two groups in the initial assessment test with a level of reliability of 5%. These results are shown in Table 3.

According to the results of Table 3, the Levene test has a value of 0.818, which is higher than 0.05, therefore assuming that the group variations are equal. The value of the test for bilaterality for the experimental and control group is 0.448 for both cases, which implies that the difference in measurements is not statistically significant at a probability of 0.05. The results show that there is no statistically significant difference (*p* > 0.05) between the mean value of the two groups based on the results of the initial evaluation test. This statistically indicates that students in both groups had similar performance levels at the beginning of the research. Therefore, any difference in performance observed later can be attributed to the use of the Geogebra AR application.


**Table 3.** T-test of results obtained in the initial evaluation test.

Table 4 compares the descriptive statistics of both groups according to the results obtained by the students in the final assessment test that collects the evaluable learning standards. The experimental group obtained a mean score in the final test of 7.3391, a standard deviation of 1.61125, and a mean error of 0.33597. On the other hand, the mean score of the control group was 6.0841, the standard deviation was 1.52334, and the mean error was 0.30467. The mean score obtained in the final evaluation test by the students of the experimental group is significantly higher than that of the control group.

**Table 4.** Descriptive statistics of the results obtained in the final assessment test.


The results obtained from the t-test for independent samples are shown in Table 5. The statistic of the Levene test is 0.034, which is less than 0.05 and therefore, it is not assumed that the group variations are equal with respect to the results obtained in the final knowledge evaluation test. The bilateral value is less than 0.05, which implies that the difference in means is statistically significant at a level of 0.05. These results indicate that the students of the experimental group achieved higher scores than the students of the control group. Therefore, according to the results of the t-test, we can reject the null hypothesis (there is no statistically significant difference in the performance scores of the students exposed to the Geogebra AR application and those who are not exposed to it) in favor of the alternative hypothesis (there is a statistically significant difference in the performance scores of the students exposed to the Geogebra AR application and those who are not exposed to it).

After analyzing the existence of a relationship between the groups, it is worth asking the intensity of their relationship, for which we use the mean of the effect size in ANOVA. The results of this test are shown in Table 6, where it can be observed that 20.4% of the variation in the teaching-learning of student functions can be attributed to the use of the Geogebra AR application.


**Table 5.** T-test of the results obtained in the final evaluation test.

**Table 6.** Measures of association between groups.


#### *3.2. Analysis of the Variation of Visualization and Spatial Rotation Skills*

In the same way, an analysis using descriptive statistics and the *t*-test of the pre-test and post-test PSVT:R was carried out in order to find out if there were any significant differences. By doing so, the impact of the Geogebra AR application is evaluated with the aim of improving the capacity of visualization and spatial rotation of the students.

The descriptive statistical analysis of the pre-test based on the PSVT:R model for the experimental and control groups is shown in Table 7. The participants in the experimental and control groups obtained a mean score of 4.9643 and 5.3332, respectively.

**Table 7.** PSVT:R pre-test descriptive statistical results.


A t-test for independent samples was conducted so as to determine if there was any significant difference between the mean score of the two groups of the pre-test based on the PSVT:R model with a 5% confidence level, the results are shown in Table 8. The Levene test had a value of 0.137 which, being higher than 0.05, means that the group variations are equal. The result of the bilaterality test was 0.425 for equal variances and 0.422 for different variances, so the difference in the means is not statistically significant with a probability of 0.05. Along with the results of the t-test carried out with the scores of the initial evaluation test, it was detected that the groups had a similar level of spatial intelligence at the beginning of the investigation. In this case, any difference detected later in terms of the improvement of the students' visualization and spatial rotation skills can be attributed to the integration of the Geogebra AR application in the classroom methodology.


**Table 8.** T-test results obtained in the PSVT:R pre-test.

Table 9 shows the results of the descriptive statistical analysis according to the scores obtained in the PSVT:R post-test for the two groups. The experimental group obtained a mean score in the post-test of 7.0652, a standard deviation of 1.60574, and a mean error of 0.33482. On the other hand, the mean of the control group was 5.6664, the standard deviation was 1.73463, and the mean error was 0.34693. It should be noted that the mean score obtained by the experimental group in the PSVT:R post-test was significantly higher than that of the control group.

**Table 9.** PSVT:R post-test descriptive statistical results.


The results obtained from the *t*-test for independent samples in relation to the scores obtained from the two groups in the PSVT:R post-test are shown in Table 10. The value of the Lenvene test is 0.029, which is lower than 0.05, so it is detected that the group variations are not equal. The bilateral test has a value of less than 0.05, implying that the difference in means is statistically significant at a probability of 0.05. For the results obtained in the *t*-test in relation to the scores obtained in the final written test, the students of the experimental group reached higher scores in the PSVT:R test than the students of the control group, therefore, according to the results of the *t*-test, the null hypothesis (there is no statistically significant difference in the level of spatial intelligence of the students exposed to the Geogebra AR application and those not exposed to it) was rejected in favor of the alternative hypothesis (there is a statistically significant difference in the level of spatial intelligence of the students exposed to the Geogebra AR application and those not exposed to it) with respect to the improvement of the students' visualization and spatial rotation skills.


**Table 10.** T-test results obtained in the PSVT:R post-test.

In addition, Table 11 shows the impact of the Geogebra AR application in the obtained scores, which shows that 21.3% of the improvement of the visualization and spatial rotation skills can be attributed to the integration of the Geogebra AR application in the classroom methodology.

**Table 11.** Measures of association between groups for PSVT:R.


#### *3.3. Descriptive Analysis of the Evaluation Questionnaire*

Finally, Table 12 presents the results obtained in relation to the data obtained from the Likert scale questionnaire in order to determine the motivation, feasibility, and perception of students in relation to the experience with AR technology. A total of 52.17% of students agreed to use AR resources for content learning and 65.21% believe that AR tools have helped improve their visualization and spatial rotation skills. It is noteworthy that virtually all students report having worked with great motivation and interest, and the vast majority of them confirm the ease of use of the application Geogebra AR.



#### **4. Discussion**

The results of the final evaluation test and the post-test demonstrate that there is a statistically significant difference in the level of achievement reached by students in the experimental group compared to those in the control group. From the findings of the study, it is evident that students who were exposed to a learning methodology with Geogebra AR (the experimental group) obtained better results both in the level of learning achieved in the formative unit functions and in their visualization and spatial rotation skills, compared to those students who were not exposed to learning supported by AR tools (the control group). Therefore, this finding suggests that the use of the Geogebra AR application as a support in the process of teaching and learning mathematical functions improved the academic performance and spatial intelligence of the students. This finding is related to the findings of Kaufmann and Schmalstieg [22] and del Cerro and Morales [12] about the effectiveness of AR tools in the teaching-learning processes in STEM knowledge areas and, especially,

in all subjects where spatial intelligence is fundamental for the development of learning. In addition, the results of this study coincide with those obtained by Hohanwarter [74], who through the use of graphic software improved student performance in the study of functions. Likewise, the findings of this study also coincide with previous research where the software Geogebra was used in its version for personal computers with the objective of improving learning results in the subject of mathematics [61,75,76].

The students in the experimental group were exposed to a not yet fully established educational technology, which most likely captured their attention and interest during the lessons in which it was incorporated into classroom methodology. The interactive and dynamic nature of Geogebra AR allowed students in the experimental group to represent, visualize, rotate, analyze, and compare graphs of mathematical functions with ease. This allowed students to better understand the concept of function, identifying a greater number of characteristics of the function in relation to its form than the control group. In addition, the integration of this technology managed to enhance the proactive learning of students, as well as awakening their inquiry and need to know more. The students of the experimental group had the possibility to verify and evaluate the correction and accuracy of the results of their exercises in an autonomous way through Geogebra AR. Similarly, the experimental group was able to draw and analyze several graphs at the same time without having to perform algebraic calculations, tables of values, or draw by hand each one of them through the application, Geogebra AR. This, in general, made them complete the proposed activities in class in a shorter time than the control group, a factor that may have contributed to the deepening of contents and the higher score in the final written test than the one obtained by the control group.

For all these reasons, it is recommended that teachers integrate tools such as Geogebra AR as support in the resolution of activities for teaching mathematical functions, since it has proven to be effective in improving learning by reducing the effort of students in the tedious task of drawing functions manually, allowing them to focus on other more relevant elements, such as exploring and analyzing them.

Before the study, we discovered that not many ESO students could manipulate and use mathematical software effectively due to their lack of training, but this was not the case with Geogebra AR, in which most students excelled in its intuitive and simple operation, obtaining great results.

The integration of tangible tools such as Geogebra AR in a classroom changes the role of teachers, relocating them as a permanent guide that gives students more freedom and autonomy, as well as encouraging critical and creative thinking, instead of just being a transmitter of knowledge.

As authors, we can assert that the process of integrating Geogebra AR into classroom methodology has been simple and satisfactory. However it must be taken into account that educators must be well trained in the use and integration of ICT, such as mobile devices [77] and AR, in the teaching-learning processes. In this sense, if they are applied in an adequate way and always within a contextualized methodology, it can be very useful in not only facilitating teaching-learning processes, but also making them more interactive, motivating, and interesting [26].

#### **5. Conclusions**

Our study explicitly sought to transform the teaching-learning processes of mathematics, with the purpose of promoting mathematical skills linked to spatial intelligence, instead of focusing only on learning specific mathematical content. The integration of Geogebra AR through a contextualized methodology in the teaching-learning process of mathematical functions meant a significant difference in the levels of academic achievement and spatial intelligence of the students exposed to it [12,26]. The results also showed that the students had a positive perspective on the use of the application which managed to capture their attention and increase their motivation from the beginning.

AR technology has come to transform the concept of what, until now, was not possible to implement in the subject of mathematics, allowing efficient and effective learning experiences in the classroom, which must be accompanied by appropriate resources and methods to deepen and stimulate the skills of students [29]. The study evaluates the academic and cognitive achievement of students through the scores obtained in each of the tests and addresses other factors, such as motivation, which have influenced students to obtain this performance. Therefore, we can say that the value of any technology integrated in the classroom depends largely on the level of student engagement.

Lastly, Geogebra AR has proven to be an effective tool in teaching mathematical functions and improving students' spatial intelligence. Therefore, we recommend that teachers integrate this software in the development of learning activities, which can also be adapted for the development of other concepts, with other curricula at different teaching levels. Therefore, its relevance in the field of mathematics covers a wide range of possible uses.

This study was developed around the subject Academic Mathematics of the 4th year of ESO in order to investigate the effect of integrating Geogebra AR in the teaching-learning of functions. Given the scope and potential of the models learned in an interconnected and ubiquitous environment not yet established, the conclusions drawn from this work should be taken with prudence [78]. Therefore, the generalization of the results of this study to other content and levels of mathematical education should be made with caution.

Our findings can be used as a starting point for future research. For example, studies can be carried out to analyze the impact of the Geogebra AR application through mobile devices as part of the learning of mathematics in different situations and contexts inside and outside the classroom (b-learning). This includes integrating our study to the current educational context to effectively stimulate self-learning, improve levels of attention, and motivate students through the paradigm shift caused by the Covid-19 pandemic.

Finally, we recommend that future studies perform qualitative meta-analyses to assess educators' perceptions towards the use and integration of emerging ICTs, such as AR, in the teaching of STEM areas.

**Author Contributions:** Conceptualization, F.d.C.V. and G.M.M.; methodology, F.d.C.V. and G.M.M.; software, G.M.M.; validation, F.d.C.V. and G.M.M.; formal analysis, F.d.C.V.; investigation, F.d.C.V. and G.M.M.; data curation, F.d.C.V. and G.M.M.; writing—original draft preparation, F.d.C.V. and G.M.M.; writing—review and editing, F.d.C.V. and G.M.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data are not publicly available due to privacy restrictions.

**Acknowledgments:** To the teachers, students, and institutions involved.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Classroom Methodologies for Teaching and Learning Ordinary Differential Equations: A Systemic Literature Review and Bibliometric Analysis**

**Esperanza Lozada 1, Carolina Guerrero-Ortiz 2, Aníbal Coronel 1,\* and Rigoberto Medina <sup>3</sup>**


**Abstract:** In this paper, we develop a review of the research focused on the teaching and learning of ordinary differential equations with the following three purposes: to get an overview of the existing literature of the topic, to contribute to the integration of the actual knowledge, and to define some possible challenges and perspectives for the further research in the topic. The methodology we followed is a combination of a systematic literature review and a bibliometric analysis. The contributions of the paper are given by the following: shed light on the latest research in this area, present a characterization of the actual research lines regarding the teaching and learning of ordinary differential equations, present some topics to be addressed in the next years and define a starting point for researchers who are interested in developing research in this field.

**Keywords:** teaching differential equations; teaching mathematics; mathematical modeling; solving problem

#### **1. Introduction**

The teaching and learning of ordinary differential equations has experienced a dramatic change in the last two decades [1–19]. The motivation for innovation in the traditional teaching obey different reasons, at least three of those are given below. First, from the second half of the 20th century until now, the ordinary differential equations have been recognized as useful tools for teaching and learning mathematical models arising in different areas of science like physics, biomathematics, engineering, and chemistry [20–26]. Second, in our current era, the development of information technologies has strongly influenced and modified the traditional ways of inquiring in science. In particular, the information technologies have increased the innovation and application of numerical methods which are essential to solve a wide class of differential equations and are also useful to understand some qualitative properties [10,11,19,27–30]. Third, as a consequence of the above, in the last years, more attention has been given to the transformation of teaching and learning mathematical concepts by incorporating didactic methodologies to encourage students to be actively engaged in their learning process [21,31–34]. Thus, in a brief sense, the changes in the teaching of differential equations has been mainly influenced by the incorporation of active learning didactic methodologies and technology enriched learning environments.

Traditionally the curricula in many careers, like engineering, physics, mathematics, or statistics, begin with three courses of calculus (differential, integral, and several variables) and they are followed by an ordinary differential equations course. From the last decade of the twentieth century, several efforts to change the calculus curriculum have been proposed and conducted by numerous authors worldwide [15,35,36]. Specifically, in the teaching of differential equations, the changes consider new contents, new pedagogical methods,

**Citation:** Lozada, E.; Guerrero-Ortiz, C.; Coronel, A.; Medina, R. Classroom Methodologies for Teaching and Learning Ordinary Differential Equations: A Systemic Literature Review and Bibliometric Analysis. *Mathematics* **2021**, *9*, 745. https:// doi.org/10.3390/math9070745

Academic Editors: José-María Romero-Rodríguez, Francisco D. Fernández-Martín, Gerardo Gómez-García and Magdalena Ramos Navas-Parejo

Received: 3 March 2021 Accepted: 24 March 2021 Published: 31 March 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

and the incorporation of the exploration of dynamical systems concepts with graphical (or qualitative) and numerical approximations by using technological resources. Nowadays, the study of curricular modifications that must be undertaken in order to adequately overcome the diverse deficiencies and difficulties in the teaching and learning of differential equations is a very active topic of research with different subjects and perspectives.

Despite the interest in the curricular innovation for the teaching and learning of ordinary differential equations, we have found that the research lines for the next years are still diffusely stated. Some advances for integration of the findings from the research can be seen in the articles [21,37]. In [21], the authors developed an extensive bibliographic survey of 16 works published between 2000 and 2011. Meanwhile in [37], the authors focused on the factors that influence the problem solving abilities for undergraduate students in differential equations. However, to the best of our knowledge, there is not a literature review with an open period of time and the specific didactic methodologies are missing in those works. In other words, a literature review to find, critically evaluate, and synthesize the relevant research topics related to the teaching and learning of ordinary differential equations remains open.

Consequently, in the light of the increasing development of research related to teaching methods and, in order to lead emerging trends and challenges of teaching and learning ordinary differential equations, it is evident the lack of a systematic study of the existing literature. To shed light on this gap, in this paper we present an analysis of the literature related to teaching and learning differential equations, based on a systematic review and a bibliometric investigation. We propose a systematic arrangement of the main existing literature. Thus, we follow the methodology of five steps introduced in [38]: (i) framing questions for a review, (ii) identifying relevant work, (iii) assessing the quality of studies, (iv) summarizing the evidence, and (v) interpreting the findings. For step (i), we considered three questions. In the case of (ii), we retain 120 articles that come from the following databases: Web of Science, Scopus, Qualis, Zbemath, and Scielo. In step (iii), we provide some statistical properties of the retained literature. In steps (iv) and (v), we expose explicitly the subjects of ordinary differential equations covered by the research, the teaching methodologies used in the classroom, and also we present the answers to the questions of step (i). Then, in step (v), we summarize our findings.

We survey a set of 120 articles from 1970 to 2020, where initially two standard classifications for teaching differential equations were identified. The first considers the traditional and contemporary teaching methodologies pointed, for example, by [1]. The second classification includes the separation in analytic, graphical, and numerical approaches advised by some authors [39,40]. However, these classifications are currently imprecise, since the actual state-of-the-art in the research is extensive and there are works which are out of those classes, for instance the teaching under the mathematical modeling as a cyclic process can gather the analytic, graphical, and numerical approaches as a particular phase of the cycle.

The main improvements for the research field which are established in this article are described below. From our analysis of the set of 120 articles we mainly obtained the following contributions:


Moreover, the review of the literature shows an increasing trend since the first research around 1990. The best ranked journal regarding to the h-Index in the area of Mathematical Education is "The journal for research in mathematics education" and the most prolific author is Chris Rasmussen with 13 articles in the collected list, and the article with the largest number of cites in Google scholar is [15] with a total of 196 citations. Some conclusions are established by bridging the different influential perspectives of the main works. We also highlight some possible challenges and perspectives for further research of the topic.

The paper is organized as follows. In Section 2, we describe the methodological approach used in this research. In Section 3, we formulate the questions that guide the review. In Section 4, we describe how the relevant work was identified. In Section 5, we develop a bibliometric analysis of the literature. In Sections 6 and 7, we summarize the review and present a discussion. Finally, in Section 8, we draw some conclusions with short comments about some possible challenges and perspectives for further research.

#### **2. Research Methodology**

In order to define the methodology supporting this research, we recall that there are at least three approaches related with the literature review: the bibliometric analysis, the systematic literature review, and narrative review [41,42]. The goal of a bibliometric analysis is to develop a quantitative research by applying statistical methods in order to evaluate several characteristics of specific bibliographic information like journals, research institutions, geographic location, and other characteristics [43]. The narrative literature review is developed to provide an overview of a large spectrum for some specific topic chosen by the author and is based on available literature on their particular interest, is descriptive, and written in a friendly readable format [44]. Meanwhile, the systematic literature review has two principal goals: to develop an extensive literature search with a very detailed process; and to give a critical evaluation of the selected literature. Moreover, the researchers who develop literature review recognize that the systematic reviews contain an explicit a priori strategy which is detailed and comprehensive, reducing the appraising when identify the relevant studies.

For the present study, our methodology is a combination of a systematic review and a bibliometric analysis. More precisely, firstly we develop a systematic review of the literature following the five steps introduced in [38]:

*Step 1.* Framing questions for a review.

*Step 2.* Identifying relevant work.

*Step 3.* Assessing the quality of studies.

*Step 4.* Summarizing the evidence.

*Step 5.* Interpreting the findings.

Particularly, in Step 2 we generate a list of references that was explored using a bibliometric analysis with particular well defined quantitative indicators in Step 3.

#### **3. Framing Questions for a Review (Step 1)**

We follow the discussion given by Benitti [45] to establish the following three research questions:

*Question 1:* What are the studies developed for teaching and learning of ordinary differential equations with a reported classroom experiences? What types of didactic methodologies have been used in those studies?

*Question 2:* What topics of ordinary differential equations have been explored in the previous studies?

*Question 3:* What are the results for the effectiveness of traditional and new didactic methodologies to teach and learning ordinary differential equations, as reported in previous studies?

#### **4. Identifying Relevant Work (Step 2)**

To answer our research questions, we drew on multiple resources to identify the topics of differential equations and the teaching methodology that were most mentioned in the papers. We proceed in several steps as is specified below (see Figure 1 for a summary):

Retained: 120 Articles focused on research in classroom

**Figure 1.** Schematic presentation of the process used for identify the relevant work. For specifications, consult the items (a) to (e) in Section 4.

	- Web of Science (https://clarivate.com/products/web-of-science accessed on 8 August 2020),
	- Scopus (https://www.elsevier.com/solutions/scopus accessed on 8 August 2020),
	- Qualis (http://qualis.capes.gov.br accessed on 8 August 2020)
	- Zbmath (https://zbmath.org accessed on 8 August 2020), and
	- Scielo (https://scielo.org accessed on 8 August 2020).

**Ordinary differential equations.** Differential equation; solution to differential equations; graphical interpretation; graphical solution; qualitative solutions; numerical solutions; analytic solutions; first order equations; higher order equations; Laplace transform; power series method; variable separable equation; reducible to variable separable equation; homogeneous equation; reducible to homogeneous equation; exact equation; reducible to exact equation; Bernoulli equation; linear equation; Ricatti equation; phase plane; isoclines; slope fields; equilibrium; stability of solutions; initial value problems; boundary value problems; scalar equations; systems of equations; linear; nonlinear.

**Didactic methodologies.** Teaching methodologies; students' understanding and difficulties; interpretation of solutions; registers of representations; mathematical modeling; mathematical models; problem-based learning; problem solving; error analysis; mathematics teaching practices; real world situation; computational resources; mathematical application; classroom discourse; didactic of differential equations; critical discourse analysis.

More specifically the strings are given in Appendix A. First we searched the list of selected words in all fields of the search engine of databases, i.e., in titles, article keywords, abstracts, author, topic, and full paper text.

The search on Web of Science was restricted to all journals indexed to "Science Citation Index Expanded (SCI-Expanded)", "Social Sciences Citation Index (SSCI)", "Arts & Humanities Citation Index (A&HCI)", and "Emerging Sources Citation Index (ESCI)". We get a total of 342,179 publications. Then, we refined the results using the "Document Types" option by "article" and the option "Web of Science Categories" by "Social Sciences Mathematical Methods or Education Educational Research or Education Scientific Disciplines" generating a list of 3366 articles.

In Scopus, when restricting the search to Document Type "article", a total of 23,967 publications were found. Then, we refined the option "subject area" by selecting "psychology or "social sciences", getting a list of 4276 articles.


a well detailed report of the experience. Thus, by a revision of all 405 papers, we deduce that there were a total of 262, 23 and 120 articles belonging to types classroom notes, curriculum, and research in classroom, respectively. In Figure 2, we present a classification by year and by decade from 1970 to 2020. An isolated case, which is not presented in Figure 2, is the classroom note [47] published in 1913.

**Table 1.** Number of journals for Mathematics Education on Qualis and zbMATH databases, see also Appendix B.


**Figure 2.** Number of articles from 1970 to 2020. Here the abbreviation notes, curr, and research are used for the types *notes*, *curriculum* and *research in classroom*, respectively. (**a**) Articles from 1970 to 1979. (**b**) Articles from 1980 to 1989. (**c**) Articles from 1990 to 1999. (**d**) Articles from 2000 to 2009. (**e**) Articles from 2010 to 2020. (**f**) Articles by decades from 1970 to 2020.

> On the other hand, we also have identified and counted the geographic location declared by the authors in the corresponding affiliation of each article, see Figure 3. We registered the affiliations of each coauthor and then we counted all coincidences of a given region location. The regions with the highest number of records are United States of America (USA), United Kingdom (UK), and Australia with 110, 86, and 29 records, respectively. The ranking is followed by Brazil, Denmark, Germany, India, Israel, Mexico, Spain, and Turkey, which have between 6 and 29 records, see Figure 3a for percentages. Moreover, the following 50 regions have at most 6 records (less than 2%):

Argentina, Azerbaijan, Bahrain, Brunei, Canada, Chile, China, Colombia, Costa Rica, Cuba, Czechia, Ethiopia, France, Ghana, Grece, Holland, Hungary, Iceland, Iran, Iraq, Italy, Kenya, Lebanon, Libya, Lithuania, Malaysia, Netherlands, New Zeland, Nigeria, Norway, Perú, Poland, Portugal, Romania, Russia, Saudia Arabia, Serbia, Singapore, Slovakia, Slovenia, South Africa, South Korea, Spalj, Sweden, Switzerland, Taiwan, Ukraine, United Arab Emirates, and Uruguay.

**Figure 3.** Percentages of the number of papers according to the geographic location declared by the authors in the corresponding affiliation of each article. We remark that the percentages are rounded off by its integer part, then apparently in (**a**) and (**d**) the total percentages are more than 100%. (**a**) Regions for authors with all types, (**b**) regions for authors with articles of *notes* type, (**c**) regions for authors with articles of *curriculum* type, (**d**) regions for authors with articles of *research in classroom* type.

In the case of *notes*, we find that UK (74 records), USA (61 records), and Australia (20 records) are the regions with the highest number of records. Brazil, Germany, India, Israel, Mexico, Spain, and Turkey, appear with more than 4 and less than 20 records. Moreover, we get that the following 41 regions have less than 4 records each, see Figure 3b:

Argentina, Azerbaijan, Bahrain, Brunei, Canada, China, Colombia Cuba, Denmark, Ethiopia, France, Ghana, Grece, Holland, Hungary, Iceland Iran, Italy, Kenya, Libya, Lithuania, Malaysia, Netherlands, New Zeland, Nigeria, Norway, Peru, Poland, Portugal, Russia, Saudia Arabia, Serbia, Singapore, Slovakia, South Africa, South Korea Spalj, Switzerland, United Arab Emirates, and Uruguay.

In the ranking for regions with publications related to *curriculum*, the first two places are for UK, USA, Brazil, and Denmark with a total of 6, 6, 2, and 2 records, respectively. Moreover, each of the following 9 regions: Australia, Canada, Chile, China, Hungary, Spain Taiwan, Turkey, and Ukraine, have associated 1 record, Figure 3c. Now, corresponding to articles of type *research in classroom*, the regions with highest number of registered affiliations are USA, Brazil, and Mexico with 43, 14, and 11 records, respectively. The ranking

of *research in classroom* type regions is followed by Australia, Chile, Costa Rica, Denmark, Germany, Israel, Lebanon, Netherlands, Spain, Turkey, and UK with the percentages given in Figure 3d. Moreover, the following 19 regions appear with less than 2 records: Argentina, Colombia, New Zeland, South Korea, Sweden, Ukraine, Canada, Cuba, Czechia, France, Iran, Iraq, Malaysia, Norway, Romania, Singapore, Slovakia, Slovenia, and Taiwan.

Hereinafter, unless stated otherwise, **the retained list** or **the retrieved list** refer to the 120 papers which will be analyzed and are explicitly given by the following references: [1–23,27–34,39,48–135]. The other 285 articles (*notes* and *curriculum*) will be presented and analyzed in a forthcoming work by the authors.

#### **5. Assessing the Quality of Studies (Step 3)**

In this section, in order to assess the quality of the 120 articles of *research in classroom* type retrieved and selected in Section 4, we develop a bibliometric study by considering several characteristics to capture the impact of articles, authors, and journals. Amongst the literature characteristics and indicators, which are frequently used in bibliometric analysis, we consider the number of citations, the ranking of authors, the ranking of journals, and the geographic location [43]. Thus, we identify the following characteristics in the analyzed documents:


2000s decade is one of the most prolific, since 8 of the top 10 articles are authored or coauthored by Chris Rasmussen.

Some additional bibliometric characteristics of the *research in classroom* articles, are the following: 92 articles are written in English, 18 in Spanish, and 10 in Portuguese, from which 3 articles [29,61,66] are applied for teaching and learning ordinary differential equations in high school students and the rest of articles (117) for undergraduate students.


**Table 2.** The top four journals according to the number of published articles.

**Table 3.** The top 11 journals according the H index reported by Scimago Journal & Country Rank, where particularly there are 7 journals in the subject area of Education. The quartil and subject area of Physical review special topics is not assigned yet.



#### **Table 4.** Authors with the highest number of articles in the retained list.

**Table 5.** Top 10 articles followed by the number of citations in Google scholar.


#### **6. Summarizing the Evidence (Step 4)**

To approach the answer to the questions presented in Section 3, we gathered and selected the relevant information from the retained list of publications (see last paragraph of Section 4). In Table 6, a synthesis with focus on didactic methodology and topics taught or evaluated is showed. More details related to the didactic methodologies (traditional methodology, mathematical modeling, etc.) will be presented in Section 7. The articles with empty topic are those where the topic covered was not specified. Moreover, related to the question of the reported effectiveness of the new didactic methodologies in comparison with the traditional methodology, we found that few articles address explicitly this topic. From the list in Table 6, the following articles: [33,34,55,57,64,66,76,79,84,89,92,123,126,135] provide an explicit treatment of effectiveness.

**Table 6.** Summary of didactic methodologies and the topics of ordinary differential equations declared on the list of retained list of papers (see last paragraph of Section 4). Here Ref. is used for abbreviation of the reference number in the list of references.


**Ref. Didactic Methodology Topics Taught or Evaluated** [49] Information and communication technology Scalar: Verhulst model, generalized logistic. [50] Active learning Scalar: first order, linear, Bernoulli [51] Active learning Scalar: first order, applications to exponential decay problems [52] Projects-based learning [53] Projects-based learning Scalar and systems: populations model, linear system [54] Mathematical modeling Scalar: first order, applications to mixing problems [55] Active learning [56] Geometric and qualitative solutions Scalar: first order [57] Traditional methodology Scalar: linear higher order [58] Mathematical modeling Systems: equilibrium [59] Traditional methodology Scalar: first order [60] Information and communication technology Scalar: applications to electronic circuits [61] Others Scalar: First order [62] Others [63] Mathematical modeling Scalar: Malthus model [64] Traditional methodology, Geometric and qualitative solutions Scalar: first order, second order, graphical solution, slopy fields [65] Mathematical modeling [66] Active learning Scalar: first order, second order, several applications (biomedical, scientific, and social-economic contexts) [67] Active learning Scalar: First order, Verhulst equation [68] Mathematical modeling Scalar and systems: linear, exponential, logistic, and ecology applications [69] Geometric and qualitative solutions Scalar: first order [70] Geometric and qualitative solutions Scalar: first order [71] Geometric and qualitative solutions, Active learning Scalar and systems: first order, autonomous differential equations, slope fields, Lotka–Volterra models [72] Geometric and qualitative solutions Scalar and systems: graphical solutions [73] Active learning, Information and communication technology Scalar: first order, applications to mixing problems [74] Information and communication technology Systems: first order, validation with real data [75] Information and communication technology Scalar: first order [76] Traditional methodology Scalar: first order, applications to kinetics [77] Others Scalar: Laplace transform [78] Mathematical modeling Systems: first order [79] Active learning Scalar: first order, second order, slope fields, several applications [80] Traditional methodology Scalar: first order [81] Active learning [82] Traditional methodology Scalar: first order, higher order [83] Others [84] Geometric and qualitative solutions, Active learning Scalar: first order, autonomous differential equations, slope fields [85] Active learning Scalar: first order, Newton's law of cooling [86] Active learning Systems: first order, linear, slope fields, Lotka–Volterra models [87] Active learning Scalar: first order, autonomous differential equations, slope fields [88] Mathematical modeling Systems: Lotka–Volterra model, phase plane, equilibrium solutions,

**Table 6.** *Cont.*

phase trajectories

**Table 6.** *Cont.*


**Ref. Didactic Methodology Topics Taught or Evaluated** [123] Traditional methodology, Information and communication technology Scalar: definition of differential equations, graphical solution, applications. [124] Active learning Scalar: first order, Malthus model [125] Active learning Scalar: first order [126] Traditional methodology Scalar: first order [127] Mathematical modeling Scalar: first order [128] Mathematical modeling Scalar: first order, Bernoulli's equation [129] Traditional methodology Several topics [130] Mathematical modeling Scalar: first order, population mathematical model [131] Traditional methodology Scalar: first order, freefall problems [132] Active learning Scalar: first order [133] Active learning Scalar: first order [134] Mathematical modeling Scalar and systems: first order, applications to exponential decay problems, Lotka–Volterra model [135] Projects-based learning Several topics of noise and vibrations concepts

**Table 6.** *Cont.*

#### **7. Interpreting the Findings (Step 5)**

After gathering, filtering, synthesizing, and analyzing the main contributions of each paper of the retained list, in this section we address the answers to the framing questions introduced in Section 3.

#### *7.1. Question 1: What Are the Studies Developed for Teaching and Learning of Ordinary Differential Equations with a Reported Classroom Experiences? What Types of Didactic Methodologies Have Been Used in Those Studies?*

To answer this question, we recall Section 4 where we identified 405 articles which were classified in *notes* (262), *curriculum* (23), and *research in classroom* (120), see Figure 1. In the case of *notes* and *curriculum* types of articles, there are no reported empirical applications of classroom experiences. Thus, there are 120 articles with classroom experiences, which are explicitly specified at the end of Section 4 and in the first column of Table 6. Now, regarding the didactic methodologies, we have identified seven groups:


Each classification is discussed below. There are many works that can be included in more than one classification, so we decided to include the paper in a group according to the aim declared by the authors.

#### 7.1.1. The Traditional Teaching and Learning Methodology

The traditional teaching is focused in solving ordinary differential equations by applying algebraic or analytic methods, where solving means that we can find an explicit or implicit expression for the unknown function [69]. Those methods are characterized by being algorithmic, procedural, symbolic, and particularly related with a specific type of differential equation. For instance, the traditional teaching of first-order ordinary differential

equations can be summarized in two steps: (i) the educator introduces the general form of the equation by writing the following two equivalent forms

$$\frac{dy}{d\mathbf{x}} = f(\mathbf{x}, y) \quad \text{or} \quad M(\mathbf{x}, y)d\mathbf{x} + N(\mathbf{x}, y)dy = 0\_r$$

where *<sup>f</sup>* , *<sup>M</sup>* and *<sup>N</sup>* are given functions from *<sup>D</sup>* <sup>⊂</sup> <sup>R</sup><sup>2</sup> to <sup>R</sup>, followed by the introduction of the classification as separable, homogeneous, exact, linear, Bernoulli and others, depending on the functions *f* , *M*, *N*, see Table 7; and (ii) the educator teaches the students their own algorithmic solution technique for each class of equation, where the algebraic manipulation and the integration of functions are essential techniques common to all classes. Two similar steps of teaching are also applied to higher-order ordinary differential equations and for first-order systems of differential equations. Thus, according to [123], the traditional approach to teaching differential equations consists of the use of a wide variety of algebraic or analytic methods for solving different type of problems.

**Table 7.** Typical classification of first-order ordinary differential equations.


The articles [1,2,39,57,59,64,76,80,82,123,126,129,131] address aspects related to the traditional approach to teaching ordinary differential equations. Such as, development of algebraic abilities, student's difficulties of learning, uses of different mathematical representations, among others. The articles [1,2] are in the boundary between traditional and new didactic methodologies of teaching and learning differential equations, since the author discusses the relationship between procedural and conceptual learning. In [57], the authors propose a didactic material to develop skills for solving non-homogeneous higher-order ordinary differential equations by the use of indeterminate coefficient and constant variation methods. In a broad sense, the didactic material proposed by the authors consist of a list of algebraic exercises to select the appropriate method and apply the corresponding algorithmic technique. In [59,82], the author's aim was to measure the undergraduate student's mathematical knowledge through several tests. Although, the authors do not give information about the pedagogical methodology used to teach ordinary differential equations, we observed that the questions in their tests evaluate the processes of finding solutions rather than evaluating the concepts. In the article [64], the authors discuss the prevalence of traditional teaching based on analytic methods and the slow incorporation of geometric methods, they argue that the incorporation of new teaching techniques require a new learning communication skills. A similar approach to [64] is presented in [80,129], where the authors establish a study to identify the difficulties of students to develop a conceptual understanding and to use symbolic representations, meanwhile, learning differential equations based on a procedural teaching. For their part, the authors of [39,123] introduced a widely documented discussion about the characteristics of traditional methods and describe the main disadvantages. In the papers [76,131], a new method to get an analytic solution of first order differential equations is proposed. In [126], the author investigates a mnemonic acronym designed for the pedagogy of first-order ordinary differential equations. The aim in this paper is to develop a critical analysis, and propose a pedagogical model with the potential to move mnemonics from being viewed as a particular tricks where learners repeat some information which they do not understand altogether; towards a deeper, more conscious experience where learners are fostered to think beyond the mnemonic.

On the other hand, several authors have developed a broad research and discussion related to the constrains of traditional learning of differential equations. Here we mention some of the main concerns reported in the literature: the students prefer to learn algebraic methods of solution because it gives them an exact answer, however, these methods present difficulties to converting symbolic information into graphical information and vice versa [72]; student learning with the traditional method is limited because it is focused on applying and mastering algebraic procedures [2]; the main difficulties of students are related with the unsuitable choice of the solution method or an incorrect integration [3]; and the students learning in traditional methodology present some difficulties to contextualize the concepts of ordinary differential equations because they are not able to interpret correctly the terminologies out of the algebraic meaning [2,119]. Consequently, the students develop misunderstandings and learning difficulties related to differential equations [15]. It is widely documented that traditional methods for teaching and learning of ordinary differential equations are not suitable for conceptual learning, and therefore other methodologies are required [1,16,69]. Aspects like the learning in different classroom environments, the design of instructional sequences of activities, and the prompting to rethink theoretical issues as graphical representations, mathematical modeling, and even social interactions, need a further theoretical and empirical investigation [15].

Even though the traditional method of teaching and learning ordinary differential equations has several disadvantages, specifically it is passive to develop concept learning, should not be discarded entirely, since the learning of differential equation concepts needs capability in calculus concepts and skills [136]. Moreover, any change in the teaching methodologies (lecture notes, worksheets, and demonstration materials) should be implemented carefully, considering that although the students may have knowledge on concepts and skills to work with functions, differentiation, integration, and graphical representation of the derivative function, they may be unable to utilize these resources in a differential equations course [3,96].

#### 7.1.2. Qualitative and Numerical Approach to Teaching Differential Equations

As noted in various sources, the traditional teaching of ordinary differential equations has been focused in the teaching of analytic methods, however is also know that those methods are restricted to solve only few types of of equations. In the last decades, we have witnessed the incorporation of graphical and numeric solutions methods to the teaching of differential equations. The practice of these qualitative methods is becoming more frequent in the classroom due to its potential to approach solutions of several types of ordinary differential equations [39,40]. However, in practice, there are some drawbacks. For instance, the order and the non-linearity of the equation which does not permit the universal application of those methods. In our list, 14 articles are focused on exploring the teaching of graphical solution, qualitative behavior and numerical solution of ordinary differential equations [3,4,14,22,39,56,64,69–72,84,97,109]. In the articles related to the teaching of qualitative analysis of ordinary differential equations, the focus is mainly in the learning of several concepts like graphical solution, direction fields, stability, and increasing or decreasing behavior of the solution, interpretation of situations based on the behavior of solutions. Meanwhile the articles on numerical solution are focused to introduce the concept of numerical solution and the construction of the numerical solution by application of the standard schemes like Euler and Runge–Kutta.

There are some works related to qualitative approaches that deserve special mention [137–140]. These works were pioneers in the exploration of new teaching and learning methods for the teaching and leaning differential equations, but they do not appear with our search criteria. The works [137–139] are out of the selected databases where we looked (see Section 4, item (a)) and the work [140] belongs to *notes* type of articles.

In recent years, the list of papers about the teaching of graphical and numerical solution of ordinary differential equations has been increased by the incorporation of technology. Those articles will be presented below on the Section 7.1.5.

7.1.3. Active Learning Methods

In the literature, there is not a unique definition of active learning, although this term is frequently used to refer the classroom practices that engage students in learning activities, such as reading, writing, discussion, or problem solving, that promote higher-order thinking [141]. The active learning methods are student-centered teaching methodologies which provide the students the opportunity to participate in mathematical investigation or problem-solving groups, where they construct and share knowledge in communities while maintaining an appropriate feedback on their work from experts and peers. Several research studies conducted in the last years have evidenced that active learning environments developed for students present better performance and retention than traditional and passive teaching.

In the last decades, a great number of instructional strategies have been proposed to foster the "active learning" approach. For instance, the inquiry-based learning, problembased learning, the collaborative learning, the flipped classroom, problem solving and modeling activities, thinking-based learning, competencies-based learning, etc. Particularly, in the case of the teaching ordinary differential equations, we found 36 works [3,6,7,15,16,23, 30,50,51,55,64,66,67,71,73,79,81,84–87,89,96,106–114,121,124,125,132], which are organized as follows:

(a) **Inquiry-based learning.** The "inquiry-based learning" is one kind of active learning methodology with several implementations in math classroom and its particular form of implementation is the "inquiry-based instruction" [71]. The methodology of inquiry-oriented instruction consists of four main steps: the generation of ways for reasoning of students, the analysis of student contributions, the development of a shared understanding, and the connection of finding in the development of research tasks to standard mathematical language and notation. Thus, the inquiryoriented instruction generates classroom environments where the students practice an authentic research mathematical activity meanwhile they discover mathematical concepts, answering to purposefully designed tasks.

The inquiry-based instruction for ordinary differential equations is researched in the following articles [15,16,71,79,81,87,89,106–110,124,125]. In [71], the author reports the findings about the students' work with concepts related to slope fields, horizontal and vertical translation of solutions, systems modeling species interaction, and graphical solution of scalar autonomous differential equations. The author concludes that several advantages are generated by the inquiry oriented environment. Particularly he pointed out the following results: the students showed a notable cognitive gain in understanding and thinking; through the intervention of the instructor guiding the discussion the students reinvented knowledge; and they expressed their satisfaction with the inquiry instruction environment. In [79], the authors focus on the teaching of slope direction fields and the conception of solutions. Through a quantitative analysis, they showed that the students were able to successfully identify direction fields when the ordinary differential equation was given in analytical form, matching the appropriate direction field and the solution curve. They also found that students improved their understanding of the concept of solution for an ordinary differential equation as a result of the inquiry oriented intervention. The authors claim that the training had a long-lasting impact. In [81], discourse analysis is used to study the students mathematical narratives when learning the basic concepts of ordinary differential equations in a inquiry-oriented classroom environment, particularly the student's positions and beliefs related to learning mathematics. The articles [15,16,87,89,106–110,125] are part of the line of research introduced by Chris Rasmussen and collaborators. These papers are mainly focused on studying the retention of mathematical knowledge, students reasoning with mathematical ideas, and conceptual understanding, in the context of learning differential equations. From these studies, the inquiry-oriented methodology stands out for its potential to facilitate the development of mathematical reasoning ability and fostering meaningful learning. With a different perspective, in the article [124], the authors discuss the knowledge and capacity of the instructor to manage whole-class discussions concluding that the teacher's knowledge is a valuable component to be considered in the curricular reforms or in the classroom reforms under the inquiry-oriented perspective.


The works related with the active methodologies of mathematical modeling, flipped classroom, and projects-based learning will be commented on in Sections 7.1.4–7.1.6, respectively.

#### 7.1.4. The Mathematical Modeling Based Methodology

The mathematical modeling has a long history and a wide spectrum of applications in modern science. However, modeling is not defined in a unified single sense and, in the context of mathematics education, it has been conceptualized in a variety of ways, for instance as a process, a skill, and as a theory for student learning [8]. Over the last decades, research in mathematical modeling has increased highlighting several approaches to the teaching of mathematics and developing of students' modeling abilities. Mathematical modeling has become part of the educational standards in many institutions worldwide, being included in the curriculum of different scholar levels and careers from pedagogy, science, technology, and engineering. The researchers in mathematical modeling have

emphasized different pedagogical goals as developing of modeling competencies through centered subject activities, orquestation of teaching and learning processes, developing of critical understanding of different situations, and students' motivation [143,144].

In the context of Mathematics Education, mathematical modeling has also been considered as a didactic methodology where we can find many approaches. Here we mention at least two of these: (i) research works motivated in curricular reasons and use some contextualized examples arising from validated mathematical models and, (ii) the papers that propose implementing mathematical modeling to involve the students in the treatment of real-world or life problems enhancing their career formation abilities [145]. Notice that in the case of (i) and (ii) the modeling can act as a vehicle for teaching mathematics or as content to be learned. This is, in the case (i), the modeling is a mean for attainment curricular contents and, in (ii), the modeling seeks first to nurture and enhance the ability of students to solve authentic real-world or life-like problems. In the case of (ii), the mathematical modeling process has been described as a cyclic process involving phases which are well discussed in [8,9,143,144,146]. A wide and documented discussion of meanings, approaches, priorities, challenges, and research perspectives associated with the mathematical modeling is presented in [145].

In the conceptualization of mathematical modeling cycle, there are several phases involving the process and sub-process of learning [146]. An example of the representation of the modeling process is presented in Figure 4 which was introduced by [147] and cited in [9]. The mathematical modeling is used to transit between two systems called the real world and the mathematical theories or representations. The process of mathematical modeling typically starts when the modeler has a question in the real world, which is referred as realworld situation on the diagram. Then, the modeler observes the situation mathematically by exploring the characteristics of the system which can be described by mathematical quantities and determine the relation between those quantities. After that, in the process known as mathematization or abstraction, the modeler considers some "conditions and assumptions" and replaces the real world by a mathematical entity (mathematical model) in terms of mathematical properties and parameters. The mathematical model is analyzed by applying the specific mathematical theory, deducing some mathematical conclusions which are transferred back to the real-world situation by examining if the conclusions of the mathematical model have a coherent answer to the original question. If the answer is ambiguous or has clear limitations, the modeler can repeat the cycle by considering new and more insightful observations and then improving the mathematical model.

Specifically, in the retained list, the articles [4,5,8,9,12,13,17,18,20,22,48,49,54,56,58, 63,65,68,74,78,88,91,94,98,100,101,115,116,118,119,127,128,130,134] are related to some approaches to the mathematical modeling for the teaching of ordinary differential equations. These works were developed between the years 2004 and 2019, with the exception of [78,130]. The inclusion of [78] in the list of mathematical model papers for teaching ordinary differential equations obey to the fact that the author introduced an example of a real-life problem which is analyzed by the application of ordinary differential equations. Meanwhile, in [130], the author addressed the teacher training and recommended to include tests questions to enhance students to experience higher thought levels. Particularly, he exemplified and analyzed a question related with mathematical models for describing population dynamics with ordinary differential equations. The rest of articles (i.e., the works from 2004 to 2019) have diverse and disperse approaches for mathematical modeling. However, we can distinguish some similar characteristics which allow the definition of the following four groups:

(a) **Development of skills for mathematical modeling.** We find some articles where the aim was to study the development of mathematical modeling abilities in order to solve real problem models by employing mathematical theory knowledge related to ordinary differential equations [8,17,20,54,63,65,68,88,91,100]. The papers [20,63] are focused on the teaching and learning of mathematical models, particularly in the construction and application of mathematical models through mathematical activities. In [20], the authors present two activities, one of them is based on mathematical models already known in the literature of ordinary differential equations and, the other one is based on the treatment of quantitative information for a new situation, concluding that different approaches to mathematical modeling lead to different actions of the students. In [8], the author introduces the methodological tool "Modeling Transition Diagrams" for capturing and representing the individual modeling process which uses this tool to examine the mathematical thinking while the students participate in modeling activities. The authors of article [65] are interested in the experience of implementing a mathematical modeling course, they report that the students adopt different approaches to learn mathematical models and conclude that after the experience, the students appreciate mathematical models, and suggest the usage of mathematical modeling to engage students into higher level learning approaches. The authors of [68,88] report the results of an innovative approach for teaching mathematical modeling with emphases in topics of environment, ecology, and epidemiology. Particularly, in [88] the students were involved in the solution of real-life problems adjusted to their region, by using the mathematical modeling tools were encouraged to pay attention to environmental issues like survival and sustainability. The paper [91] is focused on how to use ordinary differential equations as a pedagogical strategy to introduce students to the concepts of mathematical modeling. The author of [100] presents an application of mathematical modeling as a contextualized activity in several topics of an integral calculus with a small introduction to some topics of ordinary differential equations. In [17], the author studies the transposition of the mathematical modeling process used by the experts into the learning and teaching of mathematical modeling for undergraduate students.


usage of registers of representation for making relationships between the context and elements in ordinary differential equations [13], and the role of mathematical modeling to establish a relation between mathematics and other sciences [4,5,98].

(d) **Modeling activities using ordinary differential equations to teach other concepts.** Other articles are focused on the study of mathematical models based on ordinary differential equations for teaching concepts of other areas of mathematics or even other disciplines. More precisely, in [22] a study where the students were involved in the learning of concepts like drug administration by using simulations of the mathematical was developed. This experience was supported on modeling drug administration regimes for asthma through systems of coupled differential equations. In [115], the authors are focused in the teaching of concepts from cardiovascular physiology by using an analogous mathematical model to electronic circuits. In [116], some concepts of mechanics are introduced to the students through modeling fighter pilot ejection. In [118,119], the authors study how students understand units and rate of change when working with ordinary differential equations. In [30], some concepts of physical dynamic systems like the stability using mathematical models based on ordinary differential equation systems are studied; and in [128] the authors study some concepts of fluid dynamics using models based on the Bernoulli equation.

The articles [12,18,74,101] will be commented on Section 7.1.5; and [54,134] are presented on Section 7.1.3 and [49,56] on Section 7.1.2.

**Figure 4.** A diagram for the mathematical modeling cycle introduced in [147] and cited in [9]. The notations C&A and C&A are used for "conditions and assumptions" and "properties and parameters", respectively.

#### 7.1.5. Information and Communication Technology-Based Methodologies

The increase of technology has challenged researchers worldwide to explore the roles technology plays and how transforms the teaching and learning of mathematics [148]. Particularly, in the case of ordinary differential equations, the information and communication technology has also become one of the essential hallmarks of contemporary educational landscape and several studies have been developed in the last years [32]. The studies of advantages, effectiveness, and other properties of technology are dynamic and have been constantly improved in recent years. For instance, an advantage of a simulation software as a learning platform is that students can solve more problems and develop abilities to achieve higher-level learning in less time than before when using traditional platforms [27].

The pedagogical methodologies based on the information and communication technology are diverse, including some learning activities like the following ones: the implementa-

tion of algorithms by writing computer codes, the analysis of some statements problems to be translated into a computer program, use of an specific software to solve problems or to learn some concepts, split a complex problem in a more small problems which integration permits the solution, conjecture some properties, and simulate the solutions in order to support the development of the proofs. Now, in the case of ordinary differential equations, it is well-known the existence of at least three approaches to solve an equation: the analytic, the qualitative, and the numeric solutions. With support on the information and communication technology, it is possible to implement pedagogic methodologies that address these approaches to the solution of ordinary differential equations. More precisely, from the retained list of papers, the articles related with information and communication technology are: [3,4,10–13,18,19,27–29,32–34,49,60,73–75,92,94,95,99–103,115,120,122,123], which can be arranged in three groups:

(a) **Computer algebra system.** The concept of computer algebra system is widely used to refer a type of software package that is used in learning some concepts by the manipulation of some appropriate mathematical formulae, and it is used in those cases where the algebraic, graphic, or algorithmic manipulations are tedious tasks with a low level of learning [149]. There are several papers focused in the usage of technological tools to find the analytic, numeric, or graphical solution of differential equations or even to analyze the qualitative behavior. Specifically, the articles [3,4,10–13,19,28,29,32,49,73,74,94,95,99–101,103,120,123] are related to the computer algebra system approach. In [28], the use of the software "Scientific Notebook" is studied to obtain the analytic and graphical solution of ordinary differential equations. The authors of [49] are focused on researching the teaching of differential equations through mathematical modeling in a computer enriched environment. In [29], it is reported a study where the students were encouraged to develop simulations of freefall problem by using a spreadsheet based on mathematical models. The authors study if the activities contribute to the mathematical, physical, and technological knowledge of students. The paper [3] discusses the cognitive process developed by students when participating in a teaching module for ordinary differential equations, which is based on problem solving and the usage of the VoyageTM200 calculator. The authors of [4,11] are interested in analyzing the different representations developed by students when learned ordinary differential equations using a computer algebra system as mediator. Indeed, in [4] some results about the application of spreadsheets and the HPGSolver software for visualizing and interpreting the properties of a given phenomenon arising in population dynamics are reported, and [11] contributes to study the connections between symbolic and graphical representations. The authors of [10,94] use the software Modellus to teach some properties of a Lotka–Volterra type system by using numerical simulations. In the research developed in [12,13], it is reported how the students were able to use several digital tools such as Excel, Derive, Wolfram-Alpha, Geogebra, to explore ordinary differential equations and their solutions. Particularly in [12], the students used an Applet to visualize and interpret the behavior of solutions of ordinary differential equations, some students' difficulties were found in this work; and in [13] the students were encouraged to use different digital tools as mentioned before and a computer package "GeomED" particularly designed to visualize and analyze the direction fields. In the research reported in [73] the software called STELLA was used to simulate the physical cascade system. In [74], the authors are focused on teaching mathematical models building for some given physical situations and in the numerical validation using technology. In [95], the authors use Maple to assist students in understanding the construction of analytic solution into the classroom. The authors of [99] present the experience of a project for teaching mathematics at the Massachusetts Institute of Technology and particularly present the result of a developed software called "mathlets" which was used for teaching concepts of dynamical systems. The author of [32,100,101] presents an experience of teaching several topics of calculus and ordinary differential

equations using an integrated learning environment enriched with projects, mathematical modeling, and information and communication technology. In the article [103], some innovative ways to use free network computing laboratory called NCLab to the teaching of differential equations and applications are presented. In [120], the authors research how Maple helps the students in algebraic skills and construction of graphs, meanwhile the students learn some concepts related with the Laplace transform. The authors of [123] investigate the usage of Web-based simulations to learn ordinary differential equations. In [19], the authors studied the development of several mathematical thinking processes when the students learn ordinary differential equations using the software Maxima.


Particularly, in [33,34] the authors apply the flipped classroom to study the teaching of topics related to ordinary differential equations. In [33], the authors study the effectiveness of flipped classroom to develop skills related to the application of MAT-LAB/Simulink in the solution of ordinary differential equation mathematical models arising in a chemical course. Meanwhile, in [34], the authors combine the flipped classroom methodology with the cycle of mathematical model in order to study the introductory concepts of ordinary differential equations. In both works, supported on strong evidence, the authors conclude that the flipped classroom improves the active learning achievement of students.

Additionally, we observe that there are some papers in which digital tools are used without reporting particular results about the use of technology on their studies.

#### 7.1.6. Project-Based Learning

According to the philosophy, concepts and examples of research projects in calculus are provided in [150], we can describe a research project as a multistep take-home assignment which is developed individually or in groups with a concerted effort in long period of time, for instance one or two weeks. The statements of the projects are carefully designed and include some parts expecting to get stuck even in the best students, such that the learners seek for help from their instructors, from whom receive hints, additional exercises, and supplementary readings. Moreover, the projects can be designed for different learning goals. Some projects consider real world problems in order to help the students to discover the applications of mathematics and their utility to study the affine sciences like physics, biology, chemistry, or engineering. One of the key goals when working with projects is to guide the learners to construct formal proofs by exploration of particular examples. For major details on project-based learning in calculus, we refer to [150].

Concerning the application of project-based learning in differential equations, we refer to the following articles from our retained list: [30,31,52,53,90,101,135]. The authors of [31] use mathematical projects arising in biology in the context of modeling tumor growth by differential equations. In [52,53], the authors combine the ideas of mathematical modeling and project-based learning methodologies to design projects to teach some concepts of ordinary differential equations. The authors argue that the project itself contributes to the development of students' competency for project work in science even in the introductory university courses. The authors of [90] are focused into researching the perceptions of the students when writing projects in the context of a differential equations course and conclude that the methodology is appropriate to develop some skills beyond the usual academic content of concepts and procedures. The students participating in the project recognized that they improved their capacity of scientific communication with each other when analyzing and solving real-life problems. An increase in their critical thinking was also observed. In [101], similar to [52,53], is also integrated modeling and project-based methodologies in the context of classroom environment based on the information and communication technology. The authors of [30] give a preliminary report of a series of projects applied in a course of ordinary differential equations. In [135], the author uses the methodology of projects to teach some concepts such as noise, vibration, and harshness, which are part of an undergraduate course in the mechanical engineering program. Particularly, the author studies the mathematical knowledge of students related to differential equations and linear algebra and evaluates the effectiveness of the methodology.

#### 7.1.7. Other Methodologies

In the list of retained articles, we have that the works [21,61,62,77,83,104,105,117] are out of the groups presented before, although their topic of research is related to the teaching of ordinary differential equations and applications. However the didactic methodologies used are not explicitly presented or their goals are not precisely the teaching and learning ordinary differential equations in classroom experiences, for instance [21] is a review or [117] presents the results of a pilot research project.

#### *7.2. Question 2: What Topics of Ordinary Differential Equations Have Been Explored in the Previous Studies?*

From our retained list of 120 chosen articles, we can distinguish five groups for the topics covered in the teaching of differential equations:

**(a) Basic concepts of ordinary differential equation.** We refer to as basic concepts the definition of ordinary differential equation and their solutions. For instance, in [72], the author analyzed the answer of students to the question "What comes to your mind when you are asked to solve an ODE?" in two instants of a course, at the beginning and after the intervention. He found that firstly all students think about concepts related to the analytic solution and in the second two-thirds of students consider a change of their answers including some concepts related with the qualitative approach. A similar study was conducted in [69], where the answers of students to the following exam question were analyzed:

> *In your own words, define a differential equation. Explain what constitutes a solution to a differential equation. How can you represent geometrically a differential equation? Can the geometric representation of the differential equation help in sketching approximate solutions? In your opinion, how would you solve a differential equation?*" [69] (p. 654)

In the same study, the results of a semi-structured interview to the students who were asked six questions related with the definition of ordinary differential equation, the solution concept, the concept of geometric solution, and feeling of learning differential equations were also presented. In relation to the student construction of the concept solution a framework of four facets (context-entity-process-object) is introduced to analyze that type of constructions developed, see also [114]. The teaching of the concept of equilibrium solution in the case of scalar equations was investigated in [87]. More recently in [79], the authors research on the students conceptions about the solution of ordinary differential equations. Moreover, there are some works focused in the basic concepts related with graphical and numerical solution of an ordinary differential equation. In the case of graphical solution, researchers explore new ways for the students to interpret and give meaning to the information represented by a slope field. The initial value problem or Cauchy problem, autonomous differential equations, and the asymptotic behavior of solutions are also widely studied [12,71,84]. Regarding the numerical solution, the students have been introduced to learn the concepts of stability of the solution with respect to the initial condition and the coefficients of the equation by empirical examples [29].

Other concepts related with analytic solutions of first order (exact equations, linear, Bernoulli, etc.) and higher order (homogeneous, no homogeneous, coefficients variation, etc.) are treated in [9,19,57,64,79,82,89,95].

**(b) Biomathematical models.** There are several works that introduce some models arising in biomathematics which are based on differential equations. It is possible to find different types of population growth models, for example models from epidemics transmission. In those papers, the authors also pay attention to the introduction of qualitative analysis of solutions.

In the case of scalar models we have the articles [4,12,13,20,31,49,63,91,98,124], where the authors introduce the Malthus or Gompertz models and the Verhulst type models. Firstly, related with Malthus or Gompertz models, in [31] is presented research where the students are introduced in the study of population models according to:

$$\frac{dN}{dt} \quad = \quad rN,\tag{1}$$

$$N(0) \quad = \quad N\_{0\prime} \tag{2}$$

contextualized to the case of *N*(*t*) representing the density of carcinogenic cells of a tumor at the time *t*, with *N*<sup>0</sup> the measured initial density and *r* is a positive constant. A similar topic of ordinary differential equations is also developed by [63,91,98,124]; particularly in [98] the authors study a model for disinfection and modify the assumption on *r* by considering that *r* is a negative constant. Now, concerning with Verhulst type models, in [20] the authors use the mathematical modeling to teach the population models of the form

$$\frac{dN}{dt} \quad = \quad rN\left(1 - \frac{N}{K}\right) - p(N)\_\prime \tag{3}$$

$$N(0) \quad = \quad N\_{0\prime} \tag{4}$$

where *N*(*t*) is the number of individuals at time *t* living in a given bounded region; *r* and *K* are positive constants used for the increasing rate and the caring capacity, respectively; *p*(*N*) is the predation function; and *N*<sup>0</sup> is the initial population. The attention in [20] is reduced to predation function satisfying the properties *p*(*N*) → 0 when *N* → 0 and *p*(*N*) → *β* when *N* → ∞, with *β* a positive number, for instance considering *p*(*N*) = *BN*2/(*α*<sup>2</sup> + *N*2) with *α* a positive constant. We notice that when *p*(*N*) = 0 the model (3)–(4) is reduced to the Verhulst or logistic equation, which is also treated by [49]. A similar model is taught by [4,12,13] where *p*(*N*) = 3/2 and *p*(*N*) = 2, respectively.

On the other hand, in the case of systems of differential equations, we have the Lotka–Volterra model in competence of species and epidemiology, which are treated by [10,71,86,88,94,97,101,109,134]. In [10], the authors use mathematical modeling for describing the transmission of Malaria to the humans by the female mosquitoes of the genus Anopheles, given by the following system

$$\frac{dX}{dt}\_{\text{m-1}} = \left. \frac{dp}{N} Y(N - X) - \text{gX}\_{\text{\textquotedblleft}} \right| \tag{5}$$

$$\frac{dY}{dt} \quad = \ \frac{ac}{N}X(M - Y) - \nu Y\_\prime \tag{6}$$

$$X(0) \quad = \quad X\_{0\prime} \tag{7}$$

$$\begin{array}{rcl} \Upsilon(0) &=& \Upsilon\_{0\prime} \end{array} \tag{8}$$

where *X*(*t*) is the number of infected humans in time *t*; *Y*(*t*) is the number of (female) mosquitoes infected at time *t*; *N* is the total population of humans; *M* is the total population of mosquitoes; and *a*, *c*, *p*, *g* and *ν* are positive constants. The system (5)–(8) is a particular example of the wide class of the models well known as Lotka–Volterra like systems and is used to model competence of species, which are also treated by [71,86,88,94,97,101,109,134].

Other common topics covered by the articles in teaching biomathematical modeling are related to some advances in model design and mathematical analysis. In the case of mathematical modeling, the core of teaching is focused on the simplification of some biological phenomenon using mathematical concepts recognized by the group of students involved in the experience. Related with the mathematical analysis, the works draw attention to understanding the meaning of the equations in the biology context and to the characteristics of the behavior of the solutions. For instance, in [10] the students belong to a course in an undergraduate program in Biology. The students had a previous knowledge about the disease of malaria caused by a parasite of the genus Plasmodium from a female mosquitoes of the genus Anopheles and they also mastered some concepts of calculus. The research reports, that firstly the aim of the modeling design was to increase the relations that the students could build between calculus concepts and Biology elements. In addition, the most important simplifications associated to Biology were stated as follows: the period of incubation is discarded; the human natality and mortality are ignored; the progressive acquisition of immunity in humans is ignored; and infected mosquitoes will prevail infected until death. Then, precisely stating the variables and parameters and, considering

the behavior of populations interactions students formulated the model given by (5)–(8). The main two dependent variables at time *t* are the infected humans and the infected (female) mosquitoes populations given by *X*(*t*) and *Y*(*t*), respectively. Two parameters to be considered are total population of humans and mosquitoes given by *N* and *M*, respectively. To deduce the equation (5), describing the change over time of population for infected humans by interaction with mosquitoes, it is assumed that and infected mosquito bites a health human with a certain probably and the sick persons are recovered. The factors *N* − *X* and *ap*/*N* represent the health human and the number of bites given by a mosquito per unit of time *a*/*N* with a probability of health humans to be infected equal to *p*, respectively. Meanwhile, the recovered of infected humans is described by the term *gX* with *g* a parameter for the recovery rate. Similar arguments are used to deduce the Equation (6), mainly the term (*ac*/*N*)*X*(*M* − *Y*) is the change of infected mosquitoes when a non-infected mosquito bites into an infected human in a unit of time *a*/*N* with a probability to be infected equal to *c*, and the term *νY* is the infected mosquitoes that die at mortality rate *ν*. Second, concerning the mathematical analysis of (5)–(8), the authors observe that the system is non-linear and prevents the students from achieving analytical solutions and allows them access to the solutions using the software Modellus. The students worked with Modellus were guided by a set of activities that strengthen the concepts of calculus like functions, tangent line, derivative, and maxima and minima.

**(c) Scalar-based models**. We have some work using mathematical models based on scalar differential equations to teach some concepts of differential equations. For mathematical models based on first order scalar equations, we have four groups of articles. Firstly, we have the increasing (or decreasing) mathematical models based on an ordinary differential equation of the form

$$\frac{d\alpha}{dt} = ka, \quad \alpha(0) = \alpha\_{0\prime} \tag{9}$$

where *k* is a positive (or negative) constant, *t* is the time, and *α* is the measurement of some physical quantity such that the initial time is *α*0. In [51], the authors propose five activities in the context of problem solving and guided discovery methodologies, where particularly the four labeled activities are contextualized to radioactive decay modeled by (9) with *α* the quantity of radium in a body which is decreasing in time. The radioactive decay in the context of mathematical modeling is also considered by the authors of [39] where *α* is the number of radioactive atoms. A close problem is the model for uranium decay *p* (*t*) = −0.0003*p*(*t*) + 0.3 explored in [3], which is described as a variation of (9), with *p*(*t*) the amount of mercury in a given reservoir at any instant of time *t*. Related with the increasing behavior we have the works Malthus or Gompertz type described in the Biomathematical models, see the works for (1)–(2). Moreover, in [76] the authors use a difference equation of the form

$$\frac{[A]\_{t\_2} - [A]\_{t\_1}}{t\_2 - t\_1} = -k \left( [A]\_{t\_2} - [A]\_{t\_1} \right)^m, \quad k > 0, \ m > 0, \mu$$

arising in kinetic reactions and introduce the teach of convergence of discrete models to continuous models of the form (10) or to teach the relation of difference and differential equations. A second group of works are [3,8,17,29,39,48,51,85,100,116,117,131,132], where the authors use mathematical models based on first order differential equations. Here we distinguish four types of mathematical models. Firstly, we have the well known "freefall mathematical model", which is given by a differential equation of the type

$$m\frac{dv}{dt} = mg - bv^2, \quad v(0) = v\_0 \tag{10}$$

with *m* denoting the mass of a body, *g* is the acceleration due to gravity, *b* is a constant associated to air resistance, *v*<sup>0</sup> is the initial velocity of the body, *t* is the time, and the unknown *v* is the velocity of the body. In [29], the author uses numerical methods to simulate the solution of (10) in the case of vacuum (*b* = 0) and with air resistance (*b* > 0). Ref. [8] is focused on the research of mathematical thinking process when the students analyze and solve a freefall problem, and in [131] the authors are focused on the analytic solution of (10) by the variable separation method. Third, the model for describing "Newton's law of cooling" given by a differential equation of the form

$$\text{MC}\frac{d\theta}{dt} = -h(\theta - \theta\_d), \quad \theta(0) = \theta\_0. \tag{11}$$

where *h* is a positive constant called the convective cooling coefficient, *θ<sup>a</sup>* represents the environment temperature of cooling medium, *M* is the mass of the body, *C* is the specific heat, and *θ*(*t*) is the unknown temperature of the body in a time *t* with known initial condition *θ*0. The model of type (11) is treated in [39,85,91,100]. The fourth type of mathematical model is based on "Kirchoff and Ohm laws" given by

$$\frac{d\mathcal{U}\_{\mathfrak{c}}}{dt} + \frac{1}{RC}\mathcal{U}\_{\mathfrak{c}} = 0, \quad \mathcal{U}\_{\mathfrak{c}}(0) = E\_{\mathfrak{c}}$$

with *RC* as the constant for the resistance of the capacitor, the unknown *Uc* is the voltage in the capacitor, and *E* is the voltage of the capacitor at *t* = 0; this equation is studied in [17,18].

On the other hand, a second group of scalar models of second order are presented in [5,99], where the authors use mathematical models arising in electric circuits and vibration problems, respectively. Indeed, in [5] the authors consider the model

$$I^{\prime\prime}(t) + 2\lambda I^{\prime}(t) + \omega^2 I(t) = 0, \quad I(0) = 2, \quad I^{\prime}(0) = 0\_{\prime\prime}$$

where the *I* is the current intensity crossing the circuit and in [99] the authors use an interactive software for explore the equation

$$\mathbf{x}^{\prime\prime}(t) + b\mathbf{x}^{\prime}(t) + k\mathbf{x}(t) = k\cos(\omega t), \quad \mathbf{x}(0) = \mathbf{x}\_0, \quad \mathbf{x}^{\prime}(0) = \mathbf{x}\_1.$$

where *b*, *k*, *ω*, *x*<sup>0</sup> and *x*<sup>1</sup> are constants and *x* is the displacement of the mass from equilibrium in a spring-mass system. In the case of [5], the authors study physical concepts such as the inductance and resistance and in [99] the authors study some concepts of Mechanical Vibration Theory like amplitude and phase.

**(d) Systems based on mechanical theory**. The works [92,117] consider second order systems arising in Mechanical Vibration Theory. To be more precise, in [92] the authors consider a system modeling a two-mass two-spring vibration system of the following type

$$
\frac{d^2}{dt^2} \begin{pmatrix} y\_1 \\ y\_2 \end{pmatrix} = \begin{pmatrix} -(k\_1 + k\_2)/m\_1 & k\_2/m\_1 \\ k\_2/m\_2 & -k\_2/m\_2 \end{pmatrix} \begin{pmatrix} y\_1 \\ y\_2 \end{pmatrix},
$$

where *m*1, *m*<sup>2</sup> are the masses of two bodies connected by two springs with constants *k*<sup>1</sup> and *k*<sup>2</sup> and fixed at the top and *y*<sup>1</sup> and *y*<sup>2</sup> are the displacement from the equilibrium of the bodies. Moreover, in [92] several concepts like amplitude, modes of vibration, period, and frequency are taught.

**(e) Other concepts**. There are some works focused on the teaching and learning of other topics of differential equations like the Theorems of existence and uniqueness [1,2,111,112], Laplace transform [6,19,120], and bifurcation concept [31,135].

*7.3. Question 3: What Are the Results for the Effectiveness of Traditional and New Didactic Methodologies to Teach and Learning Ordinary Differential Equations, as Reported in Previous Studies?*

The effectiveness of a new methodology is usually an implicit motivation. However, in a practical research, the aim of a specific paper is usually defined explicitly in terms of other topics which are considered relevant to study in order to improve the teaching and learning process. Then, given that the effectiveness is implicitly transversal to all articles proposing innovative didactic methodologies for ordinary differential equations, here the works where effectiveness was explicitly mentioned were included [33,34,55,57,64,66,76, 79,84,89,92,123,126,135].

Concerning the evaluation of the effectiveness, we distinguish four groups of articles: (i) works where only the effectiveness of the new didactic methodology was evaluated [33,55,66,76,79,123,135]; (ii) works where only the effectiveness of the traditional didactic methodology was evaluated [57,126]; (iii) works comparing the traditional and the new didactic methodologies without introducing a measurement of each didactic methodology alone [89,92]; and (iv) works where the authors introduce a quantification of the effectiveness for each didactic methodology and also a comparison [34,64,84].

#### **8. Conclusions**

The followed research methodology allowed us to identify and analyze the papers addressing the teaching and learning of ordinary differential equations. We retrieved and reviewed 120 papers from 1970 to 2020 which are associated with Web of Science, Scopus, Qualis, ZbMath, and Scielo. We recognized the didactic methodologies pointed out in each paper. When doing this, the most explored concepts and topics associated to ordinary differential equations and the effectiveness of didactic methodologies reported by the authors were identified. We noticed an increase in research where the attention has been given to the design of new didactic methodologies which have also been strengthened by the development of digital tools. The research related to teaching and learning differential equations has transitioned from exploring elements associated to the teaching in traditional classrooms to the introduction of a qualitative and numerical approach, active learning methods, modeling, and use of technology, emphasizing the importance of student participation in their own learning. As a result of the nature of differential equations for describing several phenomena, it also stands out in research modeling and interdisciplinarity. It should be noted that the characterization presented is not unique and many papers could be organized in one or more category.

The most relevant features achieved of the present article are the identification of works that address the subject of teaching and learning of ordinary differential equations, the recognition of the most explored mathematical content, and the synopsis of teaching methodologies that have used to teach the topic over the years. However, through our review analysis, we have found that there are also some issues that have received little attention. For example, little evidence is found regarding the retention, in terms of learning and skills development, that students achieve after being involved in learning with a particular methodology, which requires considering the validation and improvement of the implemented methodologies. Another element to consider is the update of the university curriculum considering the research results that involve the new teaching methods and use of information and communication technologies (for instance, those indicated in Section 7.1.5) or the relevance of the processes involved in the transition from the learning of calculus to the learning of ordinary differential equations. In relation to the teachers who are normally in charge of teaching ordinary differential equations, the research does not give importance to the fact that in many cases they are engineers or mathematicians, without or a little knowledge of didactic. Then, it is necessary to pay attention to the desired knowledge (didactic, pedagogical and mathematical) that these teachers need to teach the subject, which will allow them to become aware of the learning difficulties that students may face. Teachers of ordinary differential equations still need to be encouraged to experiment and enrich their classes with different teaching methodologies to support the

students developing knowledge to respond the challenges that the academic or work field demands of them. Therefore, more research is currently needed in the classroom, in relation to the use of technology, development of simulations, resources for online teaching, and interdisciplinary projects.

The research on the teaching of differential equations is an active area with an increasing number of articles in the last decade. However, there is still much to do toward addressing the challenges in teaching and learning differential equations. We set out three issues that need more detailed exploration. Firstly, we found that some advanced topics of ordinary differential equations are incipient developed in the research. For instance the teaching of the existence an uniqueness Theorems for scalar equations of first order are treated only in [1,2,111,112] and an introduction to bifurcation concept is presented only in [31,135]. However, in the reviewed references, there is not a treatment of other relevant concepts, techniques, and classic results associated to the study of qualitative behavior of solutions, and some properties of the solutions deduced from the qualitative behavior. To name a few concepts, the teaching of linear and non-linear equations is implicitly treated by some articles. The teaching of concepts as autonomous and not autonomous systems and the concepts around stability in non-linear systems are still open topics to research. The teaching of advanced techniques and results to study non-linear systems like Lyapunov functions, topological degree methods, and the Hartman–Grobmann theorem, are still open. We did not find research regarding the teaching of analysis of equilibrium points for nonlinear systems, the periodicity of solutions, and the asymptotic behavior of solutions. Thus, briefly, there is still open the didactic transposition of several topics of ordinary differential equations theory. Second, in the teaching of modeling from physical and biological problems, the topic of existence of positive solutions is uncovered yet. For instance, in [10] the authors do not consider as part of the set of activities the basic aspect of the biological phenomenon: the existence of positive solutions of the system (5)–(8). Thirdly, regarding the systematic literature review, our short-term goal is to analyze the remaining 285 articles (*notes* and *curriculum*) which were found in the search of references given in Section 4. Since in our actual analysis some representative works were excluded, we plan to extend our search to other indexations including books, book chapters, and theses.

**Author Contributions:** Conceptualization, E.L. and C.G.-O.; methodology, E.L. and A.C.; investigation, E.L., C.G.-O., and R.M.; writing—original draft preparation, E.L. and A.C.; writing—review and editing, E.L., C.G.-O., and R.M.; supervision, C.G.-O. and R.M.; database searching, E.L. and A.C.; funding acquisition, A.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** Esperanza Lozada thank the support of the Universidad del Bío-Bío (Chile) and Universidad de los Lagos (Chile). Carolina Guerrero thanks to EDU2017-84276-R, España y Fondecyt/iniciación No. 11200169, Chile. Aníbal Coronel acknowledge the partial support of Universidad del Bío-Bío (Chile) and Universidad Tecnológica Metropolitana through the project supported by the Competition for Research Regular Projects, year 2020, code LPR20-06. Rigoberto Medina thanks to Agencia Nacional de Investigación y Desarrollo (ANID) through Proyecto Fondecyt Regular No. 1200005, Chile.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A. String Search Used in Web of Science and Scopus**

The string search used in Web of Science is the following

ALL FIELDS: ("differential equation\*" or "solution\* to differential equation\*" or "graphical interpretation" or "graphical solution\*" or "qualitative solution\*" or "numerical solution\*" or "analytic solution\*" or "first order equation\*" or "higher order

equation\*") OR ALL FIELDS: ("Laplace transform" or "power series method" or "variable separable equation\*" or "reducible to variable separable equation\*" or "homogeneous equation\*" or "reducible to homogeneous equation\*" or "exact equation\*" or "reducible to exact equation\*" or "Bernoulli equation\*") OR ALL FIELDS: ("linear equation\*" or "Ricatti equation\*" or "phase plane" or "isocline\*" or "slope field\*" or "equilibrium" or "stability of solution\*" or "initial value problem\*" or "boundary value problem\*" or "scalar equation\*" or "systems of equations" or " linear" or "nonlinear") AND ALL FIELDS: ("teaching methodologies" or "students' understanding and difficulties" or "interpretation of solutions" or "registers of representations" or "mathematical modeling" or "mathematical models" or "problem-based learning" or "problem solving") OR ALL FIELDS: ("error analysis" or "mathematics teaching practices" or "real world situation" or "computational resources" or "mathematical application" or "classroom discourse" or "didactic of differential equations" or "critical discourse analysis").

Meanwhile the string search for Scopus is given by

( TITLE-ABS-KEY ( "differential equation" OR "solution\* to differential equation\*" OR "graphical interpretation" OR "graphical solution\*" OR "qualitative solution\*" OR "numerical solution\*" OR "analytic solution\*" OR "first order equation\*" ) OR ALL ( "higher order equation\*" OR "Laplace transform" OR "power series method" OR "variable separable equation\*" OR "reducible to variable separable equation\*" OR "homogeneous equation\*" OR "reducible to homogeneous equation\*" OR "exact equation\*" ) OR TITLE-ABS-KEY ( "Bernoulli equation\*" OR "linear equation\*" OR "Ricatti equation\*" OR "phase plane" OR "isocline\*" OR "slope field\*" OR "equilibrium" OR "stability of solution\*" OR "initial value problem\*" ) OR TITLE-ABS-KEY ( "boundary value problem\*" OR "scalar equation\*" OR "systems of equations" OR " linear" OR "nonlinear" ) AND TITLE-ABS-KEY ( "teaching methodologies" OR "students' understanding and difficulties" OR "interpretation of solutions" ) OR TITLE-ABS-KEY ( "registers of representations" OR "mathematical modeling" OR "mathematical models" OR "problem based learning" OR "problem solving" OR "error analysis" OR "mathematics teaching" ) )

#### **Appendix B. List of Journals from Qualis, zbMATH, Scielo, WOS, and Scopus Datbases**

**Table A1.** List of journals from Qualis, zbMATH, and Scielo database. The notation A1, A2, B1, B2, B3, B4, B5, and C are the classification of Qualis. The notation AA, AB, and AC (or BA, BB, and BC) are used for journals considered in the Serie A (or Serie B) and types A, B, and C (or A, B and C) in the classification given by [46]. The "Journal code" is a abbreviated reference code of the corresponding journal which is introduced by citation convenience.



#### **Table A1.** *Cont.*

International journal of engineering education 0949-149X A1 1991–2020




#### **Table A1.** *Cont.*

**Table A2.** List of journals associated to WOS and Scopus databases which appear when we search articles related with the teaching and learning of ordinary differential equations by applying the strings given in Appendix A and are not included in the list of Table A1.


#### **References**


### *Article* **Formative Assessment of Pre-Service Teachers' Knowledge on Mathematical Modeling**

**Jhony Alexander Villa-Ochoa \*, Jonathan Sánchez-Cardona and Paula Andrea Rendón-Mesa**

School of Education, Universidad de Antioquia, Medellín 050010, Colombia; jonathan.sanchezc@udea.edu.co (J.S.-C.); paula.rendon@udea.edu.co (P.A.R.-M.) **\*** Correspondence: jhony.villa@udea.edu.co

**Abstract:** This document reports how formative assessment strategies promote the knowledge of modeling of pre-service mathematics teachers. This knowledge is understood from content and vehicle points of view. Formative assessment strategies were designed and experimented with 14 participants in a mathematical modeling course offered to pre-service teachers in a Colombian university. Thematic analysis was conducted on lesson plans built by pre-service teachers. In those plans, they evinced knowledge of class management, mathematics teaching, problem solving, and modeling teaching. Finally, the collective construction of assessment rubrics is highlighted. Its contributions and limitations as a formative assessment tool are reported. The role played by the advisors' feedback and support to pre-service teachers is also presented.

**Keywords:** formative assessment; mathematical modeling; teacher education; teachers' knowledge

#### **1. Introduction**

Research on mathematics teachers' knowledge has produced models regarding characteristics, dimensions, components, and facets of teachers' teaching knowledge have emerged. Pino-Fan, Assis, and Castro [1] explored some dimensions and theoreticalmethodological tools suggested by the didactic-mathematical knowledge (DMK) model for the analysis, characterization, and promotion of teacher's knowledge, intended to efficiently develop their teaching practices. Carrillo-Yañez and his team [2] presented the mathematics teacher specialized knowledge model (MTSK); the authors proposed a framework that considers mathematical-knowledge specialization as a model-inherent property which extends to all subdomains. Such models are ways to investigate, understand, analyze, and evaluate teachers' mathematics knowledge. Some models transcend a descriptive dimension and offer tools for intervention in training programs that promote teacher knowledge development. In those cases, continuous evaluation of teachers' knowledge becomes a tool to study and promote the evolution of such models.

In a complementary perspective, assessment of teachers' knowledge is associated with the knowledge they have developed to accredit, certify, or get promoted in their profession. To this end, research methods have been developed to measure teachers' knowledge and produce valid and useful results for policy formulation [3]. Mesa and Leckrone [3] offer an overview of six types of processes, methods, and components to be assessed regarding mathematics teachers' knowledge.

In another perspective, training programs are concerned not only with determining teachers' knowledge, but also have the objective of promoting it. In this regard, assessment of teachers' knowledge can be considered both summative and formative. Accordingly, a course to promote teachers' mathematical modeling knowledge was developed and related formative-assessment strategies were implemented. To analyze the contribution of these strategies, a study was developed to answer the question: how can pre-service teachers' knowledge on mathematical modeling be assessed in a formative way?

**Citation:** Villa-Ochoa, J.A.; Sánchez-Cardona, J.; Rendón-Mesa, P.A. Formative Assessment of Pre-Service Teachers' Knowledge on Mathematical Modeling. *Mathematics* **2021**, *9*, 851. https://doi.org/ 10.3390/math9080851

Academic Editors: Francisco D. Fernández-Martín, José-María Romero-Rodríguez, Gerardo Gómez-García and Magdalena Ramos Navas-Parejo

Received: 13 March 2021 Accepted: 9 April 2021 Published: 14 April 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

#### **2. Theoretical Background**

#### *2.1. Teacher's Knowledge on Mathematical Modeling in Mathematics Education*

International research on modeling in mathematics education has revealed the opportunities that modeling offers for learning and development of students' competencies, supporting of institutional needs, and fostering of teacher training (ICTMA collection). Blum [4] pointed out that the integration of modeling in school implies open and demanding environments which require complex teaching abilities and, consequently, ways of evaluation capable of facing those requirements. Certainly, mathematical and extramathematical knowledge is also required, as well as some familiarity with the selected modeling tasks. Research has also highlighted that teachers require experiences to transcend the use of routine and stereotyped tasks, so they can promote in their students' critical views and help them to solve real-life problems, to use mathematics in society [5–7], and connect mathematics and other STEM areas [8]. In their research, Romo-Vázquez, Barquero, and Bosch [5] point out that teachers require to transcend the rigidity of the curriculum, strict time schedules, lack of adapted assessment devices, problems in the use of ICT, multidisciplinary challenges, among other aspects.

Cetinkaya, Kertil, Erbas, Korkmaz, Alacaci, and Cakiroglu's literature review [6] reported that teachers have limited professional knowledge about the nature of mathematical modeling and about how to use it in mathematics teaching and learning. These authors suggested to pay greater attention to modeling-related learning opportunities for pre-service and in-service teachers through training programs. In their research, theses authors grouped a significant part of the modeling research into the following topics: (i) knowledge of the cognitive demands of certain modeling activities in order to select tasks and appropriate curricular materials for promoting specific concepts in students; (ii) knowledge about how to manage tasks and organize speech during modeling activities; (iii) knowledge on how to promote adaptive activities, make strategic interventions and foster independence as a form of scaffolding and promotion of the principle of minimal teacher assistance; (iv) knowledge of productive modeling ways (contrasted with less productive ones) to help students differentiate between more and less useful ideas, as well as to make connections between them; (v) recognition of unexpected solving approaches to modeling and development of strategies to deal with crises in the modeling process; (vi) mathematical and extra-mathematical knowledge and abilities to use information and communication technologies (ICT) effectively during the modeling processes.

Teacher's knowledge on mathematical modeling must also include at least two intersecting dimensions, namely: conceptions about the nature of modeling and students' training purposes [7]. In this study, the nature of modeling involves a conception of the object and the tool [9]; regarding training purposes, it is assumed that future teachers should not only learn mathematics, they should also learn to use modeling in their professional practice; that is, teachers should promote mathematical thinking as well as modeling skills and competencies. In Figure 1, this perspective of the teacher's modeling knowledge is represented.

In this framework, the intersection between the conception of the tool and the purpose of mathematics training implies the design of learning environments that allow future teachers, through modeling, to conceptualize, to solve problems, and to generalize mathematical concepts. The intersection between the conception of modeling as a teaching tool and as a professional tool suggests the need to promote the development of knowledge in which the (future) teacher uses mathematical modeling in the design of tasks, classes, and environments for mathematics learning, considering all the facts that this implies (students learning, curriculum, context, among others). The intersection between the object and mathematics teaching perspectives implies the design of environments in which (future) teachers can "learn to do modeling"; this also implies the development of a sensitivity to identify and delimit problems, to select relevant variables, techniques, procedures and ways to build models, to solve problems using mathematics, to validate the results, etc. Finally, in the intersection between modeling as an object and as a professional practice, knowledge

about the nature of modeling for teaching can be considered, including the type of tasks, type of environments according to contextual and institutional needs, among others.

**Figure 1.** Representation of a perspective of the teacher's modeling knowledge.

It is expected that for each of the abilities that the literature suggests for mathematics teachers, these perspectives and purposes can be identified, so each one can fit in some of the intersections shown in Figure 1. Figure blocks do not represent disjoint compartments in the teacher's knowledge, but analytical categories for the design of learning environments for those professionals. Due to the nature of the question that motivated this study, the intersection between tool and object perspectives will be used to train future teachers.

#### *2.2. Formative Assessment for the Teacher's Knowledge*

For Black and Wiliam [10,11] formative assessment or assessment for learning demands from teachers and students an active interpretation and use of evidence about their performance to make decisions during the processes. This is a practice that seeks a constant improvement of teaching and learning, tracking students' development in order to make decisions and reformulate tasks according to the observed results [10,11].

In this study, teacher educators and pre-service mathematics teachers were considered key actors in the process of formative assessment. According to Black and Wiliam [11], formative assessment involves several stages, namely: the establishment of training goals or purposes, information gathering about students' thinking and knowledge, and a plan proposal (methods, strategies, environments). Pre-service teachers were allowed to participate in the stage planning, that is, they participated in the delimitation of the evaluation criteria and the procedures and strategies to achieve compliance with this purpose. Black and Wiliam [11] argued that five principles can be recognized in the design of environments for formative assessment, namely: (i) clarify and share learning intentions and criteria for success; (ii) design effective classroom discussions and other learning tasks that provide evidence of student understanding; (iii) provide feedback to helps students progress; (iv) promote students interaction to improve learning; (v) mobilize students to empower themselves in their learning.

Formative assessment, as a means of supporting the development of teachers' knowledge, considers the strategies, media, environments, and roles of teachers as learners and of teacher educators as teachers. In a synthesis of contributions from a special issue on formative assessment and professional learning of teachers (Teachers and Teaching, Vol 19, No 2), Tigelaar and Beijaard [12] found that in the context of professional learning, teachers can be considered as learners, given that the evidence of learning that is being collected during formative assessment processes provides them with an idea of how are

they performing, where do they need to move, and what can they do to get there. Regarding strategies, the authors highlight the presence of heuristic diagrams, self-evaluations combined with co-evaluation, formative feedback, negotiated evaluation, among others.

#### **3. Methodology**

#### *3.1. Context and Participants*

This study was carried out during the second semester of 2019, in a mathematical modeling course for pre-service teachers. The course was part of a Bachelor program offered by a school of Education at a public university in Medellín, Colombia. In Colombia, mathematics teachers are prepared in Bachelor programs offered either schools of Education or Mathematics Sciences or both (more details about the Colombian Mathematics teacher preparation system see Guacaneme-Suárez et al. [13]).

Throughout the course, the pre-service teachers had to develop modeling tasks [14] and analyze their own experience based on theoretical and empirical constructs studied during the process. They also participated in workshops and discussed with modeling researchers and with in-service teachers who had modeling experience. During the course, students had to develop a modeling project [14] and design a lesson plan.

The course was distributed in 16 sessions of 4 h each. In the first session, the objectives of the course, the methodology, and the evaluation products were presented. The meaning of mathematical modeling and their experience in previous courses were also discussed. In sessions 3, 8, 15, and 16 oral presentations about their progress in the projects and lesson plans were developed. Both the teachers and the pre-service teachers could comment, suggest and argue about the progress of their classmates. Based on the approach of Black and Wiliam [11], the course followed the phases and roles for teacher educators and preservice teachers. The main aspects of formative assessment during the course can be found in Table 1.


**Table 1.** Aspects of formative assessment adapted to this research.

Fourteen pre-service teachers (11 female and 3 male) participated in the course and were informed of the ethical protocols, signing an informed consent. The names used in this article are pseudonyms. The mathematics education program was a five-year BSc program, the students (pre-service teachers) were selected according with their scores from university

entrance examination. 7 of the participants in their fourth year, and 7 were in the fifth year of the program. The pre-service teachers' ages ranged from 19 to 23 years. All participant had completed mathematics courses (e.g., geometry, arithmetic, mathematical analyses), mathematics education courses (e.g., Didactics of algebra, geometry, statistics), and a part of pedagogical courses (e.g., curriculum, educational politics, culture and education). Only six participants reported that they were coursing *practicum*. None of them reported work experiences as teacher.

#### *3.2. Data*

The pre-service teachers committed themselves to the development of the modeling tasks, the projects, and the lesson plans. The collective construction of the rubrics was made around the seventh-class session, after studying theoretical aspects of mathematical modeling and developing related tasks. Each session of the course was videotaped, therefore, for the lesson plans developed by the students, videos of the discussion sessions and of the evaluation rubric agreements were recorded.

Each workgroup participated in at least one advisory space with the teachers. A video that records the interaction between the trainers and the pre-service teachers was recorded. There were four work teams in the course. Each one developed a class plan that was reported in a written document and video-recorded while presented to classmates and teachers.

#### *3.3. Data Analysis*

To analyze the data (videos and documents), a category system with its respective coding was developed in an iterative process of going back and forth between predefined concepts (see the second section of this article) and data. The three researchers reached a common understanding on the codes and categories, later, the second author of this article organized and coded the data. He performed the first analysis of each lesson plan separately. The three researchers were regularly meeting to discuss and negotiate agreements and disagreements about the evidence, and data interpretations in light of the theory.

With the data from each lesson plan, a thematic analysis was carried out [15,16], the information was organized by themes, and points of convergence and divergence were sought. This allowed the emergence of other categories of analysis in light of the theoretical aspects described above. Then, the entire team of researchers conducted a cross-sectional analysis of the four lesson plans. The final system of topics and categories is detailed in Table 2. In the results section, the meaning of the categories is illustrated in greater detail with fragments of conversations extracted from the videos and the lesson-plans documents.

**Table 2.** Category and code system.


#### **4. Results**

The results of this study are presented in two sections: the first one presents the results of the analysis of each lesson plan; in the second one, an analysis of the formative assessment of the knowledge of pre-service teachers is made from a joint interpretation of the four lesson plans.

#### *4.1. Analysis of the Four Lesson Plans*

#### 4.1.1. Lesson Plan 1: Clash Royale. Mathematical Modeling Experience in the Classroom

This team designed a class based on the use of the Clash Royale video game. The objective of the class was "To record and interpret numerical data from the environment offered by the Clash Royale video game" (Document 1—Class Plan). In their report, the students argued their design on the need for learners to build and compare representations, and to solve arithmetic problems that involve calculation and estimation strategies [17].

Pre-service teachers argued that the need to know a game and build winning strategies enables students to face a challenge. The class design was structured in three stages, each one one-hour long. The first stage was based on the recognition of the video game, its components, rules, players, etc. The second stage involved the delimitation, collection, and organization of data; according to the pre-service teachers "the students will have to extract different numerical data from the game environment: elixir production, cost (in elixir), attack speed, resistance and damage produced by the characters of the cards. The data obtained will be recorded in tables ... " (Document 1—Class Plan). The third stage was organized through questions about the strategy to play the game efficiently.

This team proposed an evaluation of the class with scores according to the following game criteria: exploration and systematization of numerical data (10 points), analysis of situations (10 points), development and implementation of strategies (20 points), and communication of proposals by the students, during the dialogue spaces in each stage (10 points).

An analysis of this lesson plan allows to infer students' understanding of mathematical modeling as the solution of problems using mathematics; in the context of the video game, mathematical modeling was represented by the construction of a strategy to improve performance. Despite this, aspects such as mathematical work and validation of results were absent. During the modeling process, pre-service teachers took into account elements such as data collection and its organization, identification of variables to reach the solution, reasoning, and communication. In a broad understanding of mathematical modeling, these processes are part of modeling learning. Additionally, considering Colombian curricular guidelines, this team proposed to promote in students the creation of representations to solve problems. These aspects are key in modeling processes as a tool to achieve some curricular goals.

The lesson plan included considerations about assessment related to professional knowledge. For the team, the assessment was present in the three stages of the class. It was based on criteria to assess what students can do; however, it was not in line with the proposed objective or with the stated standards of the class. In this case, knowledge on the assessment during the modeling process is a key aspect in the professional training of pre-service teachers and is related to the intersection between this component and the modeling-as-an-object perspective presented in Figure 1.

#### 4.1.2. Lesson Plan 2: Impacts on a Person's Life Expectancy Caused by Tobacco Use

The team designed a class to promote reflection on the consequences of tobacco use and the understanding of linear functions. In this case, pre-service teachers relied on Colombian curricular guidelines [17]. From this document, they extracted the notion of "learning evidence" that guided the assessment proposal.

The lesson plan was structured in four stages. In the first one, students became familiar with the context, identified a smoker, and interviewed her/him to obtain data on their age, habits, and motivations for smoking. In the second stage, students were invited to

deepen in the context understanding; To do this, teachers proposed to observe a video and to answer three questions about the consequences of tobacco use, life expectancy, and its decrease due to tobacco. In the third stage, the students used the rates of change and percentages included in the video (years of life per amount of tobacco use) and, based on the data obtained in the interview, they concluded on the life expectancy of the interviewed person. In the fourth stage, students constructed tables of values and other representations of the obtained data set. After constructing Cartesian graphs, students were asked to "Show your model below, and explain how you got there" (Document 2—Lesson Plan).

An analysis of this lesson plan shows the intention of pre-service teachers to design a modeling task to promote reflections on health care. This purpose is within the scope of the socio-critical perspective of modeling that was studied during the course. In the class plan, there is also an interest in delimiting stages and tasks that students perform, which are gradually designed for the development of the activity. There is an interest in using change ratios to interpret data tendencies and construct linear functions; all of this describes a perspective of mathematical modeling as a tool to understand a situation, to mathematize it through linear functions, and to reflect on the impact of tobacco consumption.

On the other hand, the ordering of data, its organization in tables, and the identification of trends in generated graphs was encouraged. These elements are important for the learning of modeling as an object. Aspects such as experimentation, delimitation of a context, validation, and communication of the results were not observed in this lesson plan. Nor was it observed the creation of a space for reflection on the learning process by students or the promotion of actions or campaigns for health care, aspects that could strengthen the socio-critical scope of the modeling process.

4.1.3. Lesson Plan 3: Get Oriented and Take Tours inside the University of Antioquia

This team proposed a class to study spatial location, including direction, distance, position in space, and representation of space. These themes were based on Colombian curricular guidelines [17].

The class plan was designed based on a fictitious situation in which school children would visit the university facilities, the place where pre-service teachers carry out their studies. The tasks were organized in four stages. The first stage consisted on tracing a path through a 6 × 6 squared mesh; only horizontal and vertical displacements were allowed. The second stage involved a tour of several places of the University. In the third stage, in the classroom, students must mark on a map the most significant places during the tour. Finally, in the fourth stage, a plenary session was proposed in which they describe what they learned about the more meaningful, faster, and shorter routes. This team considered that evaluation should be used at every stage. They consider, as pre-service teachers, to be attentive to what children do and say, so that they could make timely recommendations. They would pay attention to the way they communicate, during the fourth stage, their actions, and recommendations to other classmates.

In the analysis of this lesson plan, knowledge on modeling was identified as a vehicle to promote spatial location skills in students. Although it was a possible scenario for mathematical work, the activity was not conceived to build mathematical models as representations, but to use notions of laterality and their mental representation. Students supported their choice in the course bibliography. In the class plan, modeling in primary school was described differently as conceived in higher grades; modeling was understood as "a mathematization of reality", according to Parra-Zapata and Villa-Ochoa [18]. Stages were planned so the children gradually gained experience, represented their knowledge on maps, and communicated them to their peers. Regarding modeling as an object, opportunities to explore, position one-self, and move inside the environment are worth noting.

Unlike the first two teams, in this lesson plan, no evaluation rubrics were identified, but there was a continuous effort to be attentive to students' actions and reflections to offer feedback; this evinces comprehension of formative assessment as a permanent activity throughout the modeling process.

#### 4.1.4. Lesson Plan 4: Mobile Operators in Colombia

Unlike the previous ones, this lesson plan focused on solving a problem through an authentic context, supporting students to understand the phenomenon of mobile phone consumption in the country. The design was supported by the course bibliography. The pre-service teachers determined the topics that would include the process, namely: directly proportional magnitudes, conversion of measurement units, collection, and interpretation of data; however, they reported that such topics should emerge as part of the solution, but they were not the main objective of the designed task. Like the other teams, design criteria were justified in the Colombian curricular guidelines [17]. Unlike the other teams, in this lesson plan, the pre-service teachers provided information about what they considered a classroom environment should be: they described the way they conceived the active role of the students, the role of the teachers as helpers, and how to promote collaborative work and good use of resources by the students.

The lesson plan included five class sessions. In the first session, they created a fictional character (Carlos) who needed a mobile phone and wanted to purchase a plan. To help him, the team proposed to the students to inquire about operators, plans, costs, and other relevant facts. They would also assess Carlos' needs and determine how each plan could or could not satisfy his needs. In the second session, students were invited to fill out a table containing information about Gigabytes, prices, duration, among others. Based on the table, students should generate proposals to solve Carlos' needs. The third session focused on Carlos' need to use the internet to upload photos. Students should offer responses according to the number of files to upload and the number of messages received and sent. The fourth session was called "decision making", students were invited to determine Carlos' internet consumption and, based on that, offer him recommendations to make a decision.

In the analysis of this lesson plan, the pre-service teachers created a fictitious character as a way of delimiting the activity so that it became semi-open, that is, it had intentionality and facilitated the knowledge of the phenomenon, the identification of variables, and some simplification according to the initial intention. It is worth noting the effort of pre-service teachers to create not only a working guide for students but also to consider criteria to consolidate a participatory learning environment. That way they, as teachers, could regulate their actions while following and supporting students' performance. This course of action is related to what Cetinkaya et al. [6] call spaces that promote adaptive interventions.

In this lesson plan, opportunities offered by "experimentation" with the phenomenon are highlighted. Pre-service teachers propose to students to identify variables, obtain and organize data, and make inferences about them. The construction of models was guided by the identification of patterns in the data and inductive reasoning. Nevertheless, little emphasis was put on promoting communication of the results to the fictitious character and offering mathematical generalization of the generated algebraic model. All these elements are related to the perspective of modeling as an object.

In these four lesson plans, pre-service teachers show their knowledges on mathematical modeling. These knowledges include several understandings about modeling (process, problem solving) and purposes (introduce a content or developing critical and other skills) [19,20]. It also notes several of types and uses of contexts for the development of modeling (e.g., realistic, authentic [14,21]). The inclusion of tasks and phases was a common aspect in the lesson plans; assessment strategies were also included in all plans. The following section reports how the formative assessment strategies of the course promote knowledge on mathematical modeling.

#### *4.2. Analysis of Formative Assessment of Pre-Service Teacher's Knowledge on Mathematical Modeling*

The lesson plans provided information about the knowledge that future teachers developed about teaching (through) modeling, that is, modeling as a teaching vehicle and modeling as a teaching content or object [9]. However, in the context of a teacher-training course, it is not only interesting to identify the generated knowledge, but also how it was promoted; in other words, it is important to consider a formative assessment.

As showed in the previous section, the four lesson plans were guided by a similar framework. This framework included title, class objective, alignment with Colombian curricular guidelines, class development, assessment, and bibliographic references. Additionally, lessons included student's work guides and a justification of the design based on the course's theoretical references. This structure of the four lesson plans included a guide for the student. The similarity in the structure of the lesson plans is due to the agreements reached for the construction of the rubric.

As reported in the methodological section, the pre-service teachers participated in the construction of the rubric, where the components of the lesson plans and evaluation criteria were established. As an example, Amelia pointed out that "A class must have a clear objective, which is expected to be achieved in one or more sessions. In every class that we have had, they presented an objective, the development of the class and the methodology, and, well, the evaluation" (Video, 4 July 2019, negotiation of the guide). Also, Carlos pointed out that "In the tasks that we have read, we see that the authors always state their purpose and establish the tools to measure the achievements of the modeling tasks" (Video, 4 July 2019, negotiation of the guide).

An analysis of the video of the rubric-construction session allowed us to infer the main guidelines on which students relied to consolidate the rubric and the structure of the lesson plans. These results are presented in Table 3.


**Table 3.** Lesson-plans elements and supports.

Rubrics are instruments designed to help assessors, teachers, and students to judge the quality and progress in student's performance [26]. These instruments are used for both summative and formative assessment. The participation of pre-service teachers in the design of the rubric was intended to promote formative assessment about their modeling knowledge. This participation produced the structural components of the lesson plans (components to be evaluated) and detailed criteria for evaluating them (descriptions of student's performance). The consolidated rubric is presented in Appendix A.

The participation of pre-service teachers in the construction of the lesson-plan structures and its corresponding rubric offered them opportunities to anticipate what would be the evidence of their learning about the use of modeling in teaching; in the words of Black and Wiliam [10,11], this participation contributed to the principle of "clarifying learning intentions". As shown in the previous section, in the lesson plans, certain knowledge became evident: knowledge about the management of the class (lesson plan 4); knowledge on the use of modeling to teach mathematical content (lesson plan 1, 2, 3) and knowledge on problem solving (lesson plan 4). To a lesser extent, knowledge about the teaching of modeling was evidenced, including subjects such as: knowledge of the context (lesson plans, 1, 2, 3, and 4); exploration of conditions and variables (lesson plans, 1, 2, 3, and 4); construction of a model (lesson plans 1 and 4) and use of mathematical information to

understand the implications of a situation (lesson plan 2). Despite this, processes such as reasoning and communication, which are fundamental in modeling, were not noticeable in all the designed plans. Table 4 summarizes the knowledge evidenced in the lesson plans designed by pre-service teachers.



In these results, participation in the construction of the rubric played a normative role. In this study, it was observed that the rubric offers guidance on what will be evaluated and how it will be evaluated; also, it seems to promote the appearance of other modeling knowledge not directly declared in the rubrics, but which can be valuable for pre-service teachers. This result recalls the criticism that Panadero and Jonsson [26] have called standardization and reduction of the curriculum. According to the authors, it is questionable the way rubrics standardize assessments by providing simple lists of criteria for complex skills and by creating a tendency on students and teachers to guide their actions exclusively towards those criteria.

Another characteristic of the pre-service teacher's formative assessment was the continuous feedback achieved. During the course, in all class activities (workshops, homework, readings, discussions), there were reflections on: What was learned? Why was it important? And how could this be integrated into their future profession? Additionally, spaces for continuous advice were created in class and extra-class times. During the class, oral presentations were made about progress in the lesson plans; both teachers and pre-service teachers could comment and criticize each team. In extra-class spaces, pre-service teachers dialogued with teachers about their progress. Teachers permanently invited pre-service teachers to reflect on: why to do what is proposed? What does the literature say about it? etc. This allowed a reflection on the nature of modeling in mathematics school teaching. As an example, Josefina, a member of the lesson plan 2, indicated:

*Josefina:* We want to propose our class for third grade children, we liked the document we read about geometry and modeling in primary school, so we would like to do something similar with the children.

*Teacher Educator*: But, how is modeling conceived there (in the document)? What is the most relevant thing the authors talked about? What is different from other ways of modeling?

*Josefina:* Well, what most caught our attention is that the authors showed that modeling allows students to establish a relationship with space, in such a way that geometric notions become a means of decision.

*Teacher Educator:* And what does that mean? How did the authors propose it? Is it a matter of getting the students to move in space or is there something else that requires planning?

In response to these questions, in their lesson plan document, the team described in greater detail the arguments they extracted from that bibliographic reference to design the four stages of the plan and the transition between the real displacement and the map location activity. A similar situation happened while giving advice to the team of class 3.

*Alexander:* Teacher, we don't know how to integrate the assessment part into our lesson plan, we don't want the assessment to focus only on mathematical concepts; we don't want the assessment to scare students either.

*Teacher Educator*: Alexander, but according to what we have experienced in the course, how do you think your processes have been assessed? What tools and forms of assessment have we used or studied? Ideally, everything we have developed in the course contributes to the construction of your lesson plans.

*Alexander:* Teacher, you have accompanied us with questions that guide us or questions that make us realize the errors or weaknesses we have.

*Teacher Educator*: Accordingly, how should assessment processes be included in your lesson plans?

*Alexander:* Teacher, then it would be like not even telling the students that they are being assessed, but teachers should be very attentive and assess what the students are doing and try to redirect what may not lead them in the right direction. But in that scenario, don't we have to apply an exam or a rubric or a final assessment?

*Teacher Educator*: The idea is that you make the decision about how you will carry out the assessment process and, in general, how you will build your lesson plan. But what is clear is that you do not have to use the rubric as an evaluation instrument, you can use other resources or instruments. What is necessary is that you indicate how the evaluation process would be developed in your lesson plan.

The third team's lesson plan showed that the elements discussed in advise sessions offered clarity to the students (pre-service teachers). In particular, this work team integrated, during the four stages of the lesson plan, feedback processes, and support to the students and made possible an assessment that facilitated orientation and success of the students.

Feedback can be considered a key strategy within formative assessment [10,11]. In the case of the present study, the feedback was conceived as a continuous dialogue and questioning about what pre-service teachers were proposing, thereby offering them opportunities to reflect on their proposals and helping them to improve their arguments and actions. Pres-service teacher's arguments were based on the reviewed literature and also on the projection of other variables present in the institutional context. According to Romo-Vázquez et al. [5], teacher training should not only be based on the design of tasks and its implementation in class, but also on knowledge of the curriculum and other institutional considerations. Despite these reflections, no important evidence of the presence of such knowledge was included in the lesson plans. This can be justified by the fact that pre-service teachers had not yet had contact with school environments and, therefore, were unaware of the diversity of institutional conditions that may be present in daily school life.

#### **5. Conclusions**

In the first part of this article, conceptions about the notion of teachers' knowledge assessment were presented. Those conceptions are aligned with the notion of measurement and certification of teachers' knowledge and abilities. It also debated the need for this notion to transcend into a formative assessment of teacher's knowledge in the context of training courses and professional programs.

This article offers evidence that, in the context of a course, the notion of formative assessment of pre-service teachers' knowledge requires a conceptual delimitation of the knowledge that is expected to be achieved and the strategies to achieve it. The courses, by their nature, are delimited in space and time; therefore, their purposes, methodologies, and scope are also conditioned. In the case of this study, a conceptualization of two broad categories of modeling knowledge in teaching was offered: modeling as a tool and modeling as a learning object. In this framework, this study offers evidence of the knowledge showed by pre-service teachers in their lesson plans and on the contributions and limitations of rubrics and feedback in the strengthening of this knowledge. In this regard, this study highlights two important results.

The first result that stands out is the local character of the knowledge that is achieved in a course for pre-service teachers about teaching of (and through) modeling. The literature has shown the complexity involved in integrating modeling into everyday school life and the high demands that it implies for teachers. Faced with this panorama, the scope of a course is only part of that knowledge; the teaching practice will be conditioned by the opportunities and limitations that pre-service teachers have about school practice. It will also depend on the environment and strategies implemented during the course. In this sense, the second important result derived from this study is related to the opportunities and limitations offered by continuous advice and participation in the construction of rubrics. As argued in this study, some research supports the use of rubrics for student learning, academic performance, and self-regulation; however, rubric design requires care. In this study, participation in the rubrics contributed to the development of pre-service teacher's knowledge about "teaching by and through modeling" and conditioned the appearance of other important knowledge in this category. Regarding advise sessions, its contributions to continuous feedback were important, but it also became clear that these contributions may be conditioned by the possible existence of other knowledge, for instance, the institutional context. These results can be used by mathematics teacher educators as an insight to the opportunities and limitations of the formative assessment for developing preservice teacher knowledge on mathematical modeling. Some formative assessment strategies would need to be reworked to afford a generation of other knowledges on mathematical modeling among pre-service teachers.

One limitation of the study is that pre-service teacher knowledge was analyzed through lesson plans. Other studies could analyze pre-service teacher knowledge in professional authentic situations (for instances, practicum) that might provide more differentiated descriptions of their prospective professional work; but as our interest was in the knowledge on modeling as both object and content we found lesson plans more appropriate. The variety of knowledge found in the participants informs about contributions of rubric and feedback, but we cannot generalize all our findings to other formative strategies uses or mathematics teacher education programs. In this sense, this study suggests the need for new research that accounts for the contributions of other strategies to the development of pre-service teachers' knowledge. New studies on the design of rubrics are suggested, to address the participation of pre-service teachers and the formative/normative tension described in this article.

**Author Contributions:** Conceptualization, J.S.-C., P.A.R.-M. and J.A.V.-O.; methodology, J.A.V.-O. and J.S.-C.; formal analysis, J.S.-C., P.A.R.-M. and J.A.V.-O.; investigation, J.S.-C. and P.A.R.-M.; data curation, J.S.-C.; writing—original draft preparation, J.A.V.-O. and J.S.-C.; project administration, J.A.V.-O.; funding acquisition, J.A.V.-O. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by CODI of Universidad de Antioquia, grant number 2018-22989. Project: "Fundamentación y desarrollo de una propuesta de formación STEM para futuros profesores de matemáticas".

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of CEI-CSHA of Universidad de Antioquia (protocol code 22GO-19, 13 August 2019).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.


**Table A1.** Rubric built collaboratively with pre-service teachers. Rubric for classroom assessment of mathematical modeling experiences.

**Appendix**

 **A** 


**Table A1.** *Cont.*

#### **References**


### *Review* **Influence of Game-Based Learning in Mathematics Education on Students' Affective Domain: A Systematic Review**

**Peter Vankúš**

Faculty of Mathematics, Physics, and Informatics, Comenius University in Bratislava, 84248 Bratislava, Slovakia; peter.vankus@fmph.uniba.sk

**Abstract:** In modern education nowadays, the use of game-based learning as a teaching and learning method is popular in all school subjects, including mathematics. There are numerous studies dealing with the influences of this teaching method on the students' achievements. Modern teaching theories consider an important effect of education on the development of students' affective domain, connected with the subject and its teaching. In this work, the author studies journal articles that the use game-based learning in mathematics to assess its effects on the students, with the aim to analyze its impact on students' affective domain. To achieve this, a systematic review with the use of a PRISMA statement is applied. The data sources are 57 journal articles from the area of interest listed in the Web of Sciences and Scopus. The results indicate that 54% of the articles consider the affective domain in the measurement of the effects of game-based learning in mathematics education. These articles report mostly (84%) the positive influences of game-based learning on students' motivation, engagement, attitudes, enjoyment, state of flow, etc. The rest of the articles show mixed results, with the authors' conclusions possibly affected by flaws in the research instruments, selection of study groups, and game design, therefore, stressing the importance of these elements in future research on this topic.

**Keywords:** game-based learning; affective domain; mathematics education; systematic review

### **1. Introduction**

Based on the fast progress of sciences and technologies, continuous innovations of the content, methods, and goals of school mathematics education are needed. One of the promising methods to achieve the active participation of students in learning activities and their higher motivation is game-based learning [1,2].

Game-based learning encourages active learning and engagement by providing students with possibilities to place problem-solving within the context of play [3].

The idea of games as an educational tool is not a new one, it was originally devised by Hellenic philosophers, Plato and Aristotle. In more recent history, game-based education has been part of the educational theories of important figures in this scientific area, such as J. A. Comenius (1592–1670), J. H. Pestalozzi (1746–1827), F. W. Fröbel (1782–1852), H. Spencer (1820–1903), K. Groos (1861–1946), M. Montessori (1870–1952), J. Piaget (1896–1980), L. S. Vygotsky (1896–1934), and J. Dewey (1859–1952) [4].

The impetus for the vast integration of game-based learning nowadays is driven by other factors, including the inclusion of digital technologies in education. Digital games support learning by giving students an opportunity to develop knowledge and cognitive skills, to learn by problem-solving, and to experience situational learning [5,6].

With this huge increase in game-based learning applications come natural questions about their effects on students. Many studies and reviews of existing research in this area have been conducted, mostly focused on the effect of games on students' performance compared with that of traditional classroom instruction [7].

Randel et al. [8], in their review, compared the effect on student's performance of games with that of traditional teaching in 68 studies up to 1991. The results were mixed;

**Citation:** Vankúš, P. Influence of Game-Based Learning in Mathematics Education on Students' Affective Domain: A Systematic Review. *Mathematics* **2021**, *9*, 986. https://doi.org/10.3390/math9090986

Academic Editor: Francisco D. Fernández-Martín

Received: 28 March 2021 Accepted: 25 April 2021 Published: 28 April 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

285

the beneficial effects of games were mostly found when specific content was targeted. The review study of Hays [9], including 105 instructional gaming articles, also reported that the use of games in specific areas can provide effective learning, but the general conclusion was that there is no evidence that games are a preferred instructional method in all situations.

In the area of mathematics, educational games were identified as suitable to promote mathematic achievements in various domains, e.g., problem-solving and algebra skills [10], strategic and reasoning abilities [11], geometry skills [12], arithmetic [13,14], and critical thinking [15]. These studies are mostly focused on game-based learning's influences on mathematical achievements in the form of knowledge. But other important parts of mathematic education are affective factors such as students' motivation, beliefs, and attitudes towards mathematics and its teaching, as these factors can have a big impact on students' mathematical skills and their future mathematic learning [16–18]. Therefore, the question arises of how and to what extent game-based learning in mathematics education influences students' affectional dimension.

This question has already been discussed in some studies and reviews. The research of Garris et al. [19] found out that games could improve the engagement of students, advocating that games could influence outcomes including attitudes. These games were used in the setting of school education, but also in adult training. The study of Vogel et al. [20] reported positive effects of games vs. traditional teaching methods for both cognitive gains and attitude. However, the authors of the study considered the reliability to be low. The study of Ke and Grabowski [21] dealt with the effect of game-based learning on fifth-grade students' mathematics performance and attitudes. The results indicated that the integration of games positively influenced attitudes towards mathematics, mostly in cooperative structured groups. Based on a review of previous research studies, Vandercruysse [22] suggests that educational games positively affect students' attitudes, though just three journal articles support this suggestion.

The above-mentioned studies imply possible influences of game-based learning on students' affective factors connected with mathematics and its teaching process. However, these indications are fragmented and do not give a general overview of the influences of game-based learning on students' mathematics. Therefore, there is a need for a systematic review of the journal papers dealing with game-based learning in mathematics, focusing on the present state of research on the influences of game-based learning upon student's affective domain.

Based on this need, in the current paper, the following questions are investigated:

Q1: To what extent do the research studies dealing with the effects of game-based learning in the field of mathematics education address the influence of this teaching method on students' affective domain, connected with mathematics and its teaching?

Q2: Which journals have published scientific articles in this field?

Q3: What have been the influences of game-based learning on students' affective domain? Q4: What research instruments were used to measure the influences of game-based learning on the affective factors?

To present answers to the research questions, the paper has the following structure. The next chapter details the materials and methods used for this systematic review. The third chapter states the result of the systematic review concerning the research questions. The last chapter discusses the results, comparing them with the outcomes of other studies, summarizes the limitations of the study, and proposes ideas for future research on this topic.

#### **2. Materials and Methods**

To answer the research questions, a systematic review was selected as the most appropriate research method, as it was developed for identifying and synthesizing research evidence by taking a systematic approach, following transparent and rigorous processes [23]. As the protocol for this systematic review, the author used the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) framework [24]. The reason behind the choice of PRISMA over other existing protocols was its comprehensiveness, its use in

several disciplines worldwide beyond the medical field for which it was originally developed, and its capability to increase consistency across reviews. The PRISMA checklist used in this paper reflects the fact that it is a systematic review, not a meta-analysis. Therefore, checklist items 12–16 and 19–23 were not included [24] (p. 3). This is in accordance with the recommendations on PRISMA applications for systematic review studies [25].

The eligibility criteria for the papers included in this systematic review are specified in Table 1.


**Table 1.** The criteria for eligibility.

According to the criteria, only published versions of journal articles are selected. This means that books, chapters in books, conference proceedings, etc., are excluded. This particular criterion is based on the higher scientific validity of peer-reviewed published journal articles when compared with other types of reports. For the journal articles, both empirical data articles and review articles are included, to cover the biggest possible range of data. The second selection criterion is the language; journal articles are accepted that are written in English. This is to avoid any confusion due to problems with translation and misunderstanding. The third inclusion criterion is that the open-access version of the article is used, to enable the extraction of all relevant data during the data collection process. The timeframe of selected articles is not limited, again, to achieve the biggest collection of relevant sources.

As information sources for this systematic review study, the author selected two major databases, namely Scopus and Web of Science. This selection was based on the broad range of the themes and journals covered in these databases, and their high scientific recognition. The items in these databases are systematically structured and search algorithms are provided that offer the possibility of covering all three eligibility criteria.

The search of the selected databases was implemented on 16 March 2021. The search terms were 'game-based learning' or 'games' combined with a Boolean operator AND with the terms 'mathematics' or 'math'. The exact search strings and limits used are listed in Table 2.

**Table 2.** The search strings and limits used for the electronic search process.


#### **3. Results**

The initial search highlighted 68 document results in the Scopus database and 37 results in the Web of Science database. The search results including the abstracts were exported for further screening in MS Excel. The next stage was the identification of duplicates, which led the author to find 20 duplicate records; those were excluded. The remaining records' abstracts were carefully studied to judge their relevance. A total of 27 articles were excluded as they did not focus on the effects of game-based learning in mathematics education. The remaining 58 papers were retrieved in the full text form. They were screened to further inspect their relevance. During this process, one paper was excluded since it was only partially written in English. This process left 57 articles; those are included in the qualitative synthesis described in the next part of the paper. A graphical representation of the flow of citations reviewed during the systematic review process is presented in Figure 1.

**Figure 1.** Systematic review flow diagram. Generated by PRISMA Flow Diagram Generator. Available online: http://prisma.thetacollaborative.ca/ (accessed on 20 March 2021).

The 57 included articles were thoroughly studied to identify the results connected with the research questions of this paper. For this analysis, the software ATLAS.ti 9 was used, mainly because it enabled the analyst to solve a range of methodological challenges, such as working with large datasets and supporting deeper levels of analysis than are possible by hand [26]. During analysis, the author found that 26 articles [27–52] did not discuss the direct effects of game-based learning on the affective domain. So, to answer research questions Q2–Q4, the 31 remaining studies are investigated [6,53–82]. Table 3 summarizes the main characteristics of these game-based learning studies, with detailed information concerning the research questions.


**Table 3.** Study characteristics.

<sup>1</sup> Mathematics-related affective factors.

#### *3.1. Extent of the Studies Considering the Affective Domain*

As mentioned previously, of the 57 studies included in this systematic review, 31 (54%) addressed the affective domain in game-based learning in mathematics education, while 26 (46%) did not study the influences of game-based learning on the affective domain. The papers dealing with the affective domain were mostly from the last decade, and their number has slowly increasing trend, as can be seen in Figure 2.

#### *3.2. Journals Publishing Studies Considering the Affective Domain*

The 31 identified studies with research directly targeting the affective domain were published in scientific journals. Four journals (15%) contained two studies, while the remaining 23 journals (85%) included a single study. The scientific orientation of these journals was mostly Education and Educational Research, followed by Computer Science Interdisciplinary Applications, Psychology and Educational Sciences, and other similar scientific disciplines.

**Figure 2.** Publishing years of the source studies concerning the affective domain.

#### *3.3. Influences of Game-Based Learning on the Affective Domain*

The majority (26, i.e., 84%) of the journal articles dealing with the affective domain reported positive influences of game-based learning on students. These positive results related to the students' motivation (20 studies), engagement (eight studies), attitudes (seven studies), enjoyment (four studies), state of flow (three studies), and attention (one study). As for the number per study, one study reported four of the above-mentioned affective domain elements with positive results due to game-based learning, five studies reported three elements, seven studies considered two elements, and the remaining 13 studies focused on just one element of the affective domain.

The remaining five (16%) journal articles stated mixed results of game-based learning in mathematics education on students' affective domain. These results related to attitudes (three studies), motivation (two studies), and anxiety (one study). Of these studies with mixed results, one study addressed two elements of the affective domain and four studies just one element.

#### *3.4. Instruments Used to Measure the Influences of Game-Based Learning on the Affective Domain*

The most widely used research tool to study the influences of game-based learning on students' affective domain in mathematics education was a questionnaire, used in 18 studies (58%). The next most often used instrument was the analysis of video data (four studies), followed by an interview with the game participants (three studies), analysis of the data and metadata from the game (two studies), analysis of students' participation during game-based activities (two studies), and observation of students learning processes (one study). Three studies included literature reviews and three studies did not specify their research instruments.

Considering the number of instruments per study, one study used three of the abovementioned instruments, three studies use two instruments, and the remaining studies used just one instrument or did not specify the instruments used.

#### **4. Discussion**

This systematic review concerns the influences of game-based learning on the affective domain, as studied in 54% of the journal articles in the area of game-based learning in mathematics education. These articles are mostly from the last decade and there is a slowly increasing trend of their number per year. The articles are published in various scientific journals with a broad scientific scope. This underlines that researchers in this field understand the importance of the affective domain for effective teaching, as many include this important dimension in their studies. There is a trend of increasing research focused on the affective domain [16–18].

Considering the research instruments, the author found that the studies included mainly questionnaires (58%), interviews (12%), and analysis of video data (10%), or a combination of these instruments, which are standard in the assessment of the affective domain. This finding follows a general trend in research on this topic [83].

One very promising result of this review is the fact that the majority (84%) of the studied journal articles report positive effects of game-based learning on students' affective domain. These results mostly include increases in motivation and engagement, and improvements in students' attitudes related to mathematical content and its teaching. Although some of the articles report mixed results, none report a negative impact. The positive influences of game-based learning on the affective domain are in accordance with previous research in this area [19–22].

Those with mixed results note that playing the selected game did not have a discernible effect on students' motivation to learn math [57]. The authors conclude that this is because they incorporated into the game features that they believed would be entertaining, but that proved not to be the case for students. Alternatively, the questionnaire was not appropriately framed to allow the researchers to detect any effects on motivation. Another study [59] reports that in terms of learning anxiety, significant differences between students in the high-score and low-score groups may be a result of family factors, as most of the students in the high-score group were very familiar with computer operations, compared with students in the low-score group who had limited experience using computers. The other study with mixed results [72] shows that the use of mathematical games in math courses does not change students' attitudes towards mathematics courses in terms of the content that they are learning. However, in contrast, it was observed that students were much more active and had fun when learning; in addition, informal interviews with students showed that students had positive feelings and thoughts about their mathematics lessons. The authors conclude that this conflict in the results occurs since primary school students who are still in the concrete process period may not be able to fully internalize the scale items in the questionnaire used in the research. The article [76] states, as a reason for the mixed results, different play preferences and motivation, based on the content of the game and to what extent it motivates students and matches their preferences.

The limitations of this study concern the focus on journal articles, thus omitting other sources as books, book chapters, and conference papers. Moreover, this study focuses only on articles written in English. The author was also not able to include articles that were not open-access, which again limits the sample. However, the included databases Scopus and Web of Science are highly representative, and English is used in most of the scientific journals, therefore, these limitations should not influence the overall comprehensive nature of the study.

In conclusion, this systematic review indicates mostly positive influences of gamebased learning on students' affective domain (84% of studies). Studies with mixed results, according to their authors, are mostly influenced by inappropriate research instruments, mistakes with the game design, or special conditions within the study groups. Based on this, the author can conclude that if those limiting factors are not present, then it is likely that positive influences of game-based learning will be recorded on students' affective domain. Therefore, proper research instruments, selection of study groups, and game design are recommended for future research on the topic of the influences of game-based learning.

**Funding:** This research was funded by the Ministry of Education, Science, Research and Sport of the Slovak Republic; grant number KEGA 007UK-4/2020.

**Conflicts of Interest:** The author declares no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


### *Article* **EXPLORIA, a New Way to Teach Maths at University Level as Part of Everything**

**Pantaleón D. Romero 1,\*,†, Nicolas Montes 1,†, Sara Barquero 2,†, Paula Aloy 2,†, Teresa Ferrer 2,†, Marusela Granell 2,† and Manuel Millán 3,†**


**Abstract:** The main objective of this article has been to evaluate the effect that the implementation of the EXPLORIA project has had on the Engineering Degree in Industrial Design and Product Development. The EXPLORIA project aims to develop an integrated competence map of the learning process, where the subjects are no longer considered as isolated contents, by elaborating an integrated learning process where the competences and learning outcomes of the subjects are considered as a whole, global and comprehensive learning. The EXPLORIA project connects the competencies of the different STEAM subjects that make up the degree, designing a learning process as a logical, sequential and incremental itinerary. Through concepts on which the foundations of design are based—shape, volume, colour, space and structure—the competencies of the different subjects are defined in incremental learning levels: understanding, applying, experimenting and developing, all taken from Bloom's taxonomy. Mathematics is linked to the rest of learning through active learning methodologies that make learning useful. This new methodology changes the student's affective domain towards mathematics in which positive emotions are transformed into positive attitudes that will improve the learning result and therefore, the students' academic results. To validate it, at the end of the paper, the academic results compared with previous years are shown, as well as an ad hoc survey of the students' assessment of the new teaching methodology.

**Keywords:** EXPLORIA; STEAM; active methodologies; university level; afective domain

#### **1. Introduction**

Mathematics is described by the National Council of Teachers of Mathematics (NCTM) as "Maths for Life" ([1], p. 4). This means that mathematics is essential for life as it helps in decision-making, planning, mathematical thinking and problem solving, which are necessary in different professional areas and daily life [2,3]. In [4] they add that mathematics is related to other sciences, not only numerical such as engineering or statistics, but also to arts, drawing, commerce, medicine, and so forth.

#### *1.1. Affective Domain*

The affective domain is defined as a set of feelings, moods and states of mind, understood as something other than pure cognition, and among which three specific elements stand out: attitudes, beliefs and emotions [5,6]. In [5,6] it is explained that these factors interact in a cyclical way, in the way we perceive mathematics, as we can see in Figure 1.

**Citation:** Romero, P.D.; Montes, N.; Barquero, S.; Aloy, P.; Ferrer, T.; Granell, M.; Millán, M. EXPLORIA, a New Way to Teach Maths at University Level as Part of Everything. *Mathematics* **2021**, *9*, 1082. https://doi.org/10.3390/ math9101082

Academic Editors: Francisco D. Fernández-Martín, José-María Romero-Rodríguez, Gerardo Gómez-García and Magdalena Ramos Navas-Parejo

Received: 30 March 2021 Accepted: 7 May 2021 Published: 11 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

295

Beliefs can be understood as a knowledge or feeling of certainty acquired and determined by past situations, which gives meaning to its own world, and which generates typical reactions without being fully aware of it [7].

Emotions are affective and automatic responses that arise from a significant event for the individual, and which result from complex learning, social influence and interpretation itself [8].

Regarding attitude, there is no unified definition in the literature, however, most authors agree in defining attitude as a disposition or predisposition towards mathematics, as for example in [8].

Attitudes are considered as one of the variables that most explains performance in mathematics [9–13]. In [10] it is estimated that attitudes constitute 30% of the explanatory factors of performance, concluding that students who display a more positive attitude towards mathematics will obtain higher mathematical performance.

#### *1.2. Rejection towards Mathematics*

In [12], an in-depth study is carried out on the rejection and negative attitudes towards mathematics. In this study, the number of participating students was 3187, belonging to all education cycles, primary, secondary, high school and the first year of university. The study was carried out in Spain in 10 different autonomous communities. The first item analysed was the taste for Mathematics at the different educational levels. The results show a high taste for mathematics in the initial levels at 87%; however, the taste for mathematics decreases as students go up in level, with 57% when they reach the first year of university. In [12], other subjects were also analysed but this decrease in the negative perception of Mathematics was not detected. In [12] the students' self-perception of mathematics skills is also analysed when answering the question, do I consider myself good at mathematics? In this case, the decrease is higher, going from 80% in primary education to 24% at university level. Finally, in [12] the level of the teacher's responsibility is analysed. In this case, the responsibility is about 15% at primary school level, reaching its maximum in the first year of university with about 60%.

The results obtained in [12] were later corroborated in [14], which showed that 67% of the students dislike mathematics and stated that they did not fully understand it. On the contrary, only 38 % showed an interest and liking for this discipline.

Recently, in [2] a study has been carried out on attitudes towards mathematics in university students. In the study, 1293 students (830 women and 453 men) of different degrees, Agri-food Engineering, Biology, Food Science and Technology, Pre-school and Primary Education, IT and Tourism were analysed. As a result, the average percentage in attitude obtained was 54% which shows that, in general, men have a more positive attitude towards mathematics, agreeing with other existing studies in this regard, such as in [15,16]. In [2] it was also found that students taking engineering degrees showed a better attitude towards mathematics than the rest, agreeing with other studies such as [17]. These degrees tend to have a greater number of men than women.

#### *1.3. Trends in Learning: STEM, STEAM, STREAM*

STEM (Science, Technology, Engineering and Mathematics) is a curriculum based on the idea of educating students in four specific disciplines: science, technology, engineering and mathematics, in an interdisciplinary and applied approach. Rather than teaching the four disciplines as separate and discrete subjects, STEM integrates them into a cohesive learning paradigm based on real-world applications.

The implementation of STEM learning generated an in-depth debate on how the four disciplines should be integrated. Two different approaches were established: the traditionalist approach, in which the four disciplines are developed independently, and the integrative approach, in which the four disciplines are developed together [18]. Of the two approaches, the integrative approach is currently the most widely accepted where the four disciplines constitute a single teaching-learning practice [19]. Still, there are researchers who believe that a fair interaction is the right thing to do [20] and others place one discipline above the other [21]. In [19] they observed that, although the disciplines were treated jointly, there was no true connection among them and [22] considered that educational institutions did not agree on how to establish and connect the four disciplines. To solve this problem, in [18] it is proposed to include Art as a new discipline in the STEM context, which was renamed STEAM. In STEAM learning, Art, in addition to promoting interdisciplinarity, facilitates communication and understanding of reality and provides creative strategies and solutions [23]. The concept of Art proposed by [18], is a very broad concept that encompasses, in addition to the so-called fine arts, other fields such as language and social sciences. The combination of scientific and artistic disciplines, apparently opposed, provides "the variety and diversity necessary for innovative product design" [24] and they complement each other because "science provides a methodological tool in art and art provides a creative model in the development of science " [25]. The European Parliament [26] considers the inclusion of art essential as it leads to the acquisition of key competences. They consider that art in STEAM is primarily concerned with creativity and creativity includes divergent thinking [27] which leads to multiple solutions to a single problem.

STREAM incorporates another component to STEM and STEAM by integrating R (Reflective learning) into the equation [28].

The study developed in [29] revealed that the Greek elementary school curriculum in science and mathematics was lacking a connection to the real-life problems that the students have encounter outside of school. This disconnection with the real-life make STEAM projects of little value if they are not connected to the real problems and do not promote critically thinking citizens. Lessons that address issues of equity, gender, cultural diversity and in general, SDG(Suatainable Development Goals) are the key of R in the STREAM projects introduced in [28].

Whichever form of STEM education we are speaking of, STEAM, STREAM or other, it is a definite "plus" with respect to traditional education. The key principle is "integration": subjects and real society problems are not taught separately but form part of an integrated curriculum [30].

#### *1.4. Active Methodologies*

STEAM projects in general promote the use of so-called active methodologies, encouraging the active participation of the student, who becomes the protagonist of the teaching-learning process and develops his/her own knowledge. Active methodologies place students at the centre of this process and make them protagonists of the discovery, rather than passive recipients of information [31].There are different teaching strategies for creating an active learning environment and engaging students in it. The most common ones are project-based learning, problem-based learning, collaborative learning, and so forth [31].

Active methodologies, such as challenge-based learning, project-based learning, problem-based learning, collaborative learning or flipped classroom, are revealed as effective tools to generate meaningful learning and train critical and creative people who will be prepared to face current and future challenges and will be able to work in a team, communicate, discuss, evaluate.

One of these active methodologies is challenge-based learning, which, based on an initial and global question or challenge, sets out the objective of guiding the students' learning to focus them on an achievable and upcoming challenge, which allows them to get personally involved in the search for effective and plausible solutions. Learning is based on a complete process of research, ideation, documentation and communication, also enhancing personal skills such as teamwork, consensus, negotiation and leadership, as key elements of emotional intelligence. Challenge-based learning allows the process to be approached in a creative and innovative way, so that the process allows the detection of other challenges or problems to be solved. It therefore implies a broader vision than project-based learning.

Project-based learning starts from an initial question or challenge and raises the objective of generating a final product, generating learning through the tasks that are carried out to develop it. If any of these tasks, in addition to being part of the project, pose a new challenge or problem to solve, we will need to overcome these with techniques from another methodology, the problem-based learning. Both methodologies, project-based learning and problem-based learning, use the large methodological umbrella of cooperative learning and therefore for their implementation we need a new organizational structure of the classroom, a different way of managing times and evaluation systems as well as changing the role of teachers and their training.

These methodologies allow the development of practical knowledge, critical thinking, through formal analysis, creative thinking, through empirical analysis and complete active learning.

This type of learning emerges from university education and is, in turn, an active methodology that focuses on the student and generates a good dose of meaningful learning. Both methodologies, project-based learning and challenge-based learning, use the great methodological umbrella of cooperative learning and they need a new organizational structure of the classroom for its implementation, a different way of managing time and evaluation systems and changing the role of teachers and their training.

#### *1.5. Previous Works. STEAM Projects in Educational Systems*

The main limitation for the use of STEAM projects in compulsory education is their absence in the national curricula. For that reason, these type of projects are usually part of extracurricular activities and are not integrated into the normal functioning of the classroom. In our previous works [32–34], the curricula of 4th, 5th and 6th grade of Primary Education in Spain, in particular in the Valencian Community were analyzed to determine the areas of opportunity for STEAM learning projects. From the 11 detected opportunity areas, the opportunity area "Sustainability" was selected in [34], for the development of the STEAM project called "Sustainable City". The board in Figure 2 reproduces the block of a city that must be organized so that when the robot travels around its perimeter street, the different elements that make it a sustainable city are activated.

The central core of the platform is composed of nine tiles, six of which are robotic tiles that pose six sustainability and robotics challenges: (1) wind power, (2) sustainable roof, (3) photo-voltaic field, (4) mobility control, (5) selective collection, and (6) urban lighting. Students must work the tiles alternating cooperative and individual work. Subsequently, they begin the assembly and programming of the complete board.

**Figure 2.** "Sustainable City" STEAM project developed in [33].

The "Sustainable City" STEAM project was tested in [34] in a real classroom. The participants were 30 students (aged 10-11) from 5th year of Primary Education and the project consisted of 14 sessions in which different active methodologies such as project-based learning, collaborative learning and the flipped classroom were carried out. The project included a comprehensive and complete evaluation system with eight questionnaires covering three flipped classroom sessions, two group and two individual self-evaluations, an explanation to the base team, the presentation of the final product and a final test. The average rating of the questionnaires was satisfactory, obtaining an average of 7.23. Throughout all the sessions, a very high degree of motivation and interest has been observed in all the students who have felt highly identified with the project, have discussed among their peers,have solved problems in a collaborative way and have shared objectives. Maths are involved in the project as a part of everything, as a part of STEAM project, but with the stimulus of the "sustainable city" project, seeing the applicability of maths in the real world.

#### **2. EXPLORIA Project. A New Way of Conceiving the University**

The EXPLORIA project was born from the need to update university learning methodologies to the new trends, such us active methodologies and STEAM project based learning, among others.

In this sense, the CEU Universities (CEU San Pablo, CEU Cardenal Herrera and CEU Abat Oliva), are developing different pilot projects in degrees such as Advertising, Political Science, Business Administration, Journalism, and so forth, rethinking the processes of university student learning. A group of teachers was formed for each degree to rethink how to do it and which of all new trends in learning methodologies are better for each one. Among the pilot degrees, there is the degree of Engineering in Industrial Design and Product Development, a degree that integrates subjects that coincide with the STEAM classification. That is why STEAM learning process was selected as a way to improve the learning process in this degree.

#### *Project EXPLORIA in the Degree of Engineering in Industrial Design and Product Development*

In our previous works [32–34], it was shown that it is possible to transform national curricula into STEAM projects and to improve the learning process, where maths is learned through a project that show its value in real life, in a "sustainable city". In this context, STEAM projects applied to the university environment can be the way to generate positive emotions in the students that change their perception of mathematics and improve their academic performance. There are no STEAM experiences in the literature, based on the authors' knowledge, integrated into the curriculum at university level to improve the understanding and perception of mathematics.

The EXPLORIA pilot project in the Industrial Design and Product Development degree aims to develop STEAM learning process an integrated competence map based on National curricula, in which the subjects are no longer considered as isolated contents, by elaborating an integrated learning process where the competences and learning outcomes of the subjects are considered as a whole, global and comprehensive learning.

In this way, active learning allows students to make the necessary connections to address the resolution of various challenges and problems that require the integration of knowledge from various disciplines. Active learning also enhances motivation, the need to discover, and the autonomy of learning, placing the students at the centre of their development. It transforms the passive attitude, from receiving knowledge and instructions, to an active attitude, in which searching, inquiring, creativity and innovation are present throughout the process.

The pilot project makes use of integrated learning, of temporal sequences focused on different learning objectives linked to Bloom's taxonomy: understanding, applying, experimenting and developing. In this way, through active methodologies, the student addresses all levels of learning, learning by doing. Students develop critical and creative thinking, through formal and empirical analysis, they develop creativity and innovation, and the capacity for global and multidisciplinary analysis, essential in the current context.

The teacher assumes the role of a learning guide, a teacher who accompanies students in their personal and professional development process. The teacher abandons the role of instructor, encouraging students to discover, the motivation to learn and the awareness of the need to learn from each challenge, stage or new situation that may arise. In this way the student is prepared to respond to complex problems, in changing, unstable and equally complex contexts.

#### **3. Research Objectives**

As shown in the previous section, the perception and predisposition of students towards mathematics is low when entering university, mainly motivated by the beliefs that the students have about mathematics, which come from previous training cycles. The stimuli that the students receive and their emotions can worsen their results even more, generating negative attitudes that increase the failure of students in this subject, see Figure 1.

The objective of our research is to develop an EXPLORIA pilot project in the Industrial Design and Product Development degree using STEAM learning based on the competences of the Spanish law. The EXPLORIA project connects the competencies of the different STEAM subjects, designing a learning process as a logical, sequential and incremental itinerary. Through concepts on which the foundations of design are based: shape, volume, colour, space and structure, the competencies of the different subjects are defined in incremental learning levels: understanding, applying, experimenting and developing, all taken from Bloom's taxonomy.Each of the learning periods of the fundamentals of design ends with a Milestone based on the Challenge-Based Learning methodology, where students actively and autonomously, and working in teams, integrate the skills acquired, using the learning to propose their solutions.

The goal of the present paper is to analyze the effect of the EXPLORIA pilot project has in Maths in the Industrial Design and Product Development degree. Mathematics is linked to the rest of learning through active learning methodologies that make learning useful, generating positive stimulation and emotions, which lead to positive attitudes of the students and improve their academic performance in all subjects, but especially in mathematics.

#### **4. Materials and Methods**

*4.1. Research Design and Data Analysis*

An experimental design was carried out, following the experts in this field [35]. A qualitative, quantitative and mixed analysis was also carried out, following the experts in this field [36].

The students were classified into two different groups in order to be assessed. On the one hand, the control groups followed the traditional methodology. On the other hand, an experimental group followed the EXPLORIA pilot learning as a methodology. The methodology was defined as an independent variable. Both groups share course, content and teachers, so it is established that there is no prior significant difference between the control and experimental groups.

R was selected as the data analysis language. Descriptive statistics on graphics, mean and standard deviation were used for this analysis. The effect size measure was obtained using Kruskal-Wallis, where a *p* < 0.05 is established as a statistically significant difference. Cronbach's alpha test is also used to see if multiple-question Likert scale surveys are reliable. In that test, the reliability is achieved if *p* > 0.7.

#### *4.2. Participants*

The participants in the study were the students of the degree in product design engineering from the courses 2018–2019, 2019–2020, 2020–2021, where the course 2020–2021 is the experimental group in which STEAM learning was applied and the other two courses are the control courses that followed the traditional methodology. The number of students in each sample group was 23, 27 and 31 respectively.

#### *4.3. Scope of Application*

STEAM learning has been planned and applied to the first four-month period of academic year 2020–21 in which the following subjects are included, see Table 1.


**Table 1.** First year subjects.

Where the syllabus of the mathematics subject is as follows, see Table 2.

**Table 2.** Syllabus of the mathematics course.


#### *4.4. Instrument*

Data collection was obtained through an ad hoc questionnaire. The design of this tool was carried out following other validated methods found in the scientific literature, such as [37]. There are 9 items in the questionnaire. A type of scale is followed depending on the question, some of the questions had the option of YES, NO, others allowed to enter comments in an open format and the rest followed a Likert-type format with a range of five points (from 1 = Strongly disagree to 5 = Strongly agree).

The final grades obtained by the students in each of the courses were also used. In the case of the control courses, 2018/2019 and 2019/2020, they were carried out with a final standard exam while in the experimental group, course 2020/2021, an evaluation by projects and acquisition of skills was carried out. The ways of evaluating, although different in each of the groups, seek to measure the level of acquisition of competences by students.

#### **5. Design and Implementation of EXPLORIA Pilot Project**

The EXPLORIA project was born from the need to update university methodologies to new trends and market needs. In this sense, the University CEU has started different pilot projects in degrees such as marketing, law, political science in order to rethink the way of teaching at university. Among the pilot qualifications, there is the degree in design engineering in which the formative character of the subjects coincides with the STEAM classification.

The EXPLORIA project connects the competencies of the different STEAM subjects, see Table 3, where the standard subjects disappear, designing a learning process as a logical, sequential and incremental itinerary. In this learning process, teachers do not have a fixed weekly schedule but rather their schedule is based on the designed itinerary.


**Table 3.** First-year subjects of design engineering degree.

The EXPLORIA project has been designed based on the specification and synthesis of the specific and general competencies of each subject included in the study plan of the Degree in Industrial Design and Product Development, it was estimated that, aiming at obtaining a significant and integrated learning result, it was appropriate to group these skills according to a learning process based on Bloom's Taxonomy relating to the verbs understand, apply, experiment and develop.

On the other hand, and according to the learning objective established by the degree for the student who completes the 1st year of the Degree in Industrial Design and Product Development, we decided to include five concepts that will articulate the itinerary of this course, making them coincide with the basic fundamentals of design: shape, volume, colour, space and structure. In order to adjust to the academic calendar that divides the course into two semesters, we divided the learning itinerary of design fundamentals into two modules. These in turn are divided into three acts as shown:

MODULE I


In addition, to strengthen the objective of each of the fundamentals worked on and obtain a global vision of the related competences, we decided to introduce a milestone at the end of each Act. This milestone is a challenge-based methodology in which students, actively and autonomously, and based on a general topic raised by teachers, respond to their own concerns through a challenge. This challenge is formalized and sustained through the application in a project of the skills and learning acquired by the student during the weeks that have made up each act. In this activity, the role of the teacher is to accompany and guide the student according to the needs required by each phase of the project, being flexible when intervening and adapting to the requirements of the teams depending on their specialization. Since one of the pillars that sustains the EXPLORIA program is the creation and consolidation of the learning community, it is therefore appropriate to develop the milestone within a team. It is in this way that transversal competences such as decisionmaking, communication, critical thinking, and so forth, are integrated. In addition, the group is changed for each Act, which allows the students to vary their role depending on the idiosyncrasy of the team and obtain different experiences. The project developed based on the challenge is exposed by each team to the community (other teams and teachers) and evaluated on the one hand by the teaching staff, who will determine the cohesion of the acquired competencies and the learning results established for the Act through a rubric designed for this activity. The other teams, using the Post Motorola tool, will qualitatively evaluate what items worked or not, what can be improved and what we have learned, determining a quantitative score based on the responses. Finally, the team itself, and based on an attitudinal and aptitude rubric, carries out a self-evaluation and co-evaluation. The weighting of all these results will be the final grade of each student.

#### *5.1. Mathematics in EXPLORIA*

Mathematics, as a basic subject in a first year of Engineering, is part of this project, which has required analysing the role of the subject and how to connect it with the students' learning. Mathematics is a core element not only in the necessary knowledge for the learning of other subjects, but also necessary for thought processes that allow solutions to problems of various kinds to be achieved. In the specific case of the EXPLORIA project, mathematics is essential for understanding physical principles, understanding concepts such as proportion, harmony, present in nature, the objects that surround us and art. The mathematical calculation has a "utility" that can be perceived by students when integrating it and needing it to tackle other type of learning.

Mathematics sessions have a general structure that covers 1 h of theoretical concepts (lectures) and 1 h of practice (seminars). The contents taught are distributed according to the theme of the act in which the subjects take part in a systematic way that determines which curricular concepts should be emphasized. The Milestone makes it possible to evaluate the acquired mathematics competencies, applied to real problems in relation to other competences developed in the other subjects.

#### 5.1.1. Description of Sessions and Timing

We detail the sessions and subjects involved in Table 4 in which you can see a summary of the sessions:

Session 1


Session 2

1. In mathematics we develop the concepts of vector and matrix as set of coordinates of an object. The isometries.

2. In Basic Design, the matrices associated with the shape are applied and also to modular structures.

Session 3


Session 5: MILESTONE I. Sport: Students develop a design project related to sport in groups where they must apply the knowledge acquired in sessions 1,2,3,4. This activity lasts one week and concludes with a defence of the project before a panel of teachers. During the presentation, the students must explain how they have applied the knowledge acquired in the project.

Session 6


Session 7


Session 8


Session 9: MILESTONE II. Light: Students develop a design project related to light in groups in which they must apply the knowledge acquired in sessions 5,6,7,8. This activity lasts one week and concludes with a defence of the project before a panel of teachers. During the presentation, the students must explain how they have applied the knowledge acquired in the project.

Session 10


Session 12


Session 14: MILESTONE III. Well-being: Students develop a design project related to well-being in groups where they must apply the knowledge acquired in sessions 10,11,12,13. This activity lasts one week and concludes with a defence of the project before a panel of teachers. During the presentation, the students must explain how they have applied the knowledge acquired in the project.


**Table 4.** Sessions, timing and subjects involved.

#### 5.1.2. Assessment Methodology

The assessment methodology is one of the most relevant factors introduced in the EXPLORIA project because the traditional exam is replaced by the activities developed in each session as well as the marks obtained in each milestone.

Each session has a theoretical part and a practical part where the student must apply the maths in an exercise related with other subjects (Physics, Basic Design, etc.). The exercises must be done in class. The most important part is that the exercises for each student must be original and different between them. Therefore, the students needs to learn maths to solve their own exercise.

Milestone projects are developed in groups and the mark obtained by the group is the same for each other. However, the teachers check in class the implication of each student in the activity. At the end of the semester, 14 exercises and 3 Milestone projects are used to evaluate the students. The final mark is obtained by 75% exercise sessions and 25% Milestone projects.

#### 5.1.3. Some Milestone Project Examples

Appendix A shows some project examples presented by the Students in Milestone II and III. The projects are:


In these projects, mathematical concepts are applied in different parts of the project. For instance Pappus-Guldin is used to compute the volume and the surface area of the prototype. Both are important for the prototype design. The first one to know the material required if you want to develop the prototype with Polispan. The second one is necessary if you want to develop your prototype with a 3D printer. Students compute both and choose the way to develop their prototype. Color are selected using cosine law and doing linear combinations in order to represent it. In Figures A1 and A2.

#### **6. Results**

#### *6.1. Student Perception Survey*

To carry out an evaluation of the new teaching methodology, a Microsoft Forms form has been made, in this form the student indicated whether the previous year they had taken the mathematics subject or not. They were also asked what their perception of mathematics was before starting the course and whether that perception had improved after or not. Finally, there were more specific questions about this experience, such as the degree of satisfaction with the activity carried out, perception of learning and appreciations about the educational model. Finally, there is an open question for the student to comment on the experience. The questions asked in the form are shown below in Table 5.

**Table 5.** Questions asked in the questionnaire for the students.


**Table 5.** *Cont.*


Table 6 shows the results of question 1, YES/NO answers. Table 7 shows the answers to the question about prior perception of mathematics. Table 8 shows the answers to the Liker-type questions. Finally, Table 9 shows us which of the links with the rest of the subjects have been more useful in understanding mathematics. Cronbach's alpha tests of the multiple-question was performed to assess internal reliability of the questionnaire about perception of mathematics (*p* = 0.7791).

**Table 6.** Answers to YES/NO questions of the questionnaire.


**Table 7.** Student questionnaire responses related to the previous perception of mathematics.


**Table 8.** Student questionnaire responses, Liker-type questions.


**Table 9.** Answers to the question about which link was the best.


If we focus on the answers given by the students of the 2020–2021 course, regarding the evaluation of the EXPLORIA educational model, the survey was answered by 60% of the enrolled students.

The data show that practically all the students have studied mathematics in the previous year. Regarding the perception of mathematics before starting university, 60% show a neutral or negative perception of mathematics, but 85% acknowledge that their perception has improved thanks to the EXPLORIA project.

In the question about if this way of working was more fun than the traditional way, 95% of the students who took the questionnaire completely agreed or agreed with it. In the same way, practically all the students believed that their perception of learning had improved significantly with this new methodology.

Finally, practically all the students expressed the opinion that they would like it to continue in later courses.

Finally, 57% indicate that the connection that has helped them the most to understand mathematics has been with physics, while 36% indicate that the most useful connection has been with basic design. This result is indicative that the applicability of mathematics is key to understanding.

These are some of the answers given to the open question:


As a final result, the vast majority of the students have improved the student's belief about mathematics, specifically, 86% of the students, corroborating what is shown in Figure 1 and in the research by [5,6], a positive stimulus or emotion generates positive attitudes that allow the student's beliefs about mathematics to be changed.

#### *6.2. Comparison of Academic Results in the Last Three Years*

Table 10 shows the academic results obtained in the last 3 years.

**Table 10.** Results of Ordinary Exam Call.


In order to determine whether there are significant differences in the average marks obtained, we have used the Kruskal-Wallis test giving a *p*-value of 0.02168, which allows us to conclude that there are significant differences among the average final grades in the different courses and therefore the methodology introduced in EXPLORIA for learning mathematics has had a positive impact. Figure 3 shows the distribution of grades in the different courses.

As we can see, the distribution pattern of grades, as well as the percentages in the control groups, years 2018–2019 and 2019–2020, are similar, 17% vs. 15% of fail grades, 30% vs. 26% of pass grades 31% vs. 37% of B grades. However, the pattern changes significantly in the experimental group, in the academic year 2020–2021, we can see that the percentage of A grades has increased significantly, reaching 29%, and there is a 3% of grades with honours and finally, the percentage of fail grades has disappeared, reaching 0%.

**Figure 3.** Distribution of grades.

#### **7. Discussion**

The EXPLORIA project implemented in the degree of Engineering in Industrial Design and Product Development produce a great impact in the learning process. The vast majority of the students have improved the student's belief about mathematics as well as to understand why it is necessary to learn maths. The present study scores in control and experimental groups and, although it is true that the assessment methods are different because in the control group there are no exams, in the experimental group, the teacher has 14 activities developed in class where the most important part when the teachers design these exercises is that the exercises for each student must be original and different between them. Therefore, the students needs to learn maths to solve their own exercise and then the teacher could evaluate if the student understand the concepts. this is as if an examination exercise of the classical methodology were done in each session of the new methodology. Moreover, the exercise levels are higher than in a traditional exam because the teacher is focused in one concept.

The main limitation for the use of STEAM projects in compulsory education is their absence in the national curricula. For that reason, these type of projects are usually part of extracurricular activities and are not integrated into the normal functioning of the classroom. The first STEAM project developed taking into account national curricula was developed in our previous works [32–34] and in the same way, developed for university level in the present study. Therefore, it is not possible to compare with similar experiences in other universities.

One of the important limitation of the present methodology is that the teachers' schedule is not fixed and it is determined by the learning sequence. If some students fail one part, it is not easy to recover a single part because all the learning process is intertwined between subjects. This could generate organizational problems for the university.

The construction of the learning process could also be a problem for the university because requires a deep effort for the teachers. If an effective learning process is to be achieved, the teachers of the different subjects must act as a single teacher and must know what the other teachers want to achieve from their students.

#### **8. Conclusions and Further Developments**

This article shows the design and evaluation of the EXPLORIA project, based on STEAM learning in the degree of product design engineering. The development of an integrated competence map of the learning process, where the subjects are no longer considered as isolated contents, by elaborating an integrated learning process in which the competences and learning outcomes of the subjects are considered as a whole, a complete and global learning, this has allowed a change in students' perception of mathematics, increasing their motivation and their commitment to discovering. The results obtained by the students have improved substantially, both due to the almost absolute decrease

in the drop-out rate of the subject, as well as the very low rate of students who failed, as well as the average grades that have increased substantially. In addition to the results, the most significant aspect of the implementation of this project is the change in students' perception of mathematics, which has resulted in a change in attitude and, therefore, in an improvement in academic results. Generating positive emotions through active methodologies in which the students can see the application of mathematics, improves their taste for mathematics and their understanding.

In future works, we intend to implement the EXPLORIA methodology in other degrees related to STEAM learning, such as Architecture. In addition, we are going to transform our STEAM projects into STREAM projects, including Critical reflective learning, mainly focused in the SDG (Sustainable Development Goals).

**Author Contributions:** Authors contributes equally in the development of the present research. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by CEU-Santander Precompetitive Project Grant.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data supporting reported results can be found in the present paper.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**

**Figure A2.** YOUMOOD. A product with a bottle of water where each box or bottle has a color related with art and emotions.

#### **References**


### *Article* **Improving the Teaching of Real Valued Functions Using Serious Games. Binary Who Is Who?**

**Sagrario Lantarón 1, Mariló López 1,\*, Susana Merchán <sup>1</sup> and Javier Rodrigo <sup>2</sup>**


**Abstract:** The study presented makes an original, new and exhaustive analysis of the adaptation of a classical board game which has been named Binary Who is Who? This proposal shows a very useful tool for the consolidation of mathematical concepts related to the study of real-valued functions that are treated in the different levels of the teaching of mathematics (first and second year of superior secondary studies and the first years of some university degrees). The use of games as a means for learning is the authors' proposal. The aim is to offer teachers the chance of using the games as a method of teaching mathematical concepts, as well as a motivating instrument for them. This game has been created to be played face-to-face in the classroom and it has also been programmed to create a video game which allows the students to play virtually.

**Keywords:** gamification; mathematical teaching methodologies; educative innovation; learning through video games; real-valued functions

#### **1. Introduction**

The objective of this proposal is to use the educational value of certain games for the presentation and consolidation of the knowledge of some concepts of the subjects of mathematics that students must know, and proficiency related to logical–mathematical reasoning. We want to offer students the opportunity to play and beat challenges as a way of working with the concepts of daily lessons in the classroom, as well as a way of training their reasoning abilities.

There have been studies that confirm that people are naturally playful and, thus, we are open to all proposals that are related to games and competition. The Dutch historian, Johan Huizanga, showed in his book Homoludens [1] that making tends to include games in culture and society. Thanks to works like this, game-based learning and gamification are being incorporated in sectors like education, business, and digital commerce, among others [2–10].

The basic ingredient of gaming consists of the challenge and what it can represent for the individual.

If the users consider that the game that is presented to them puts their abilities to the test, they will show interest in it and see how far they can get [11].

Focusing on the use of games for the teaching and learning of mathematics, it should be noted that a good game, a game that has well-defined rules and has an approach rich in logical content, needs to include, on one hand, certain mathematical concepts, and on the other hand, a type of analysis whose characteristics are very similar to those that are needed to solve typical problems of this science. Mathematics is, to a large extent, a game and the game can, in many cases, be analyzed by means of mathematical instruments. In games, we look for fun, the possibility to quickly perform actions, and competition.

**Citation:** Lantarón, S.; López, M.; Merchán, S.; Rodrigo, J. Improving the Teaching of Real Valued Functions Using Serious Games. Binary Who Is Who? *Mathematics* **2021**, *9*, 1239. https://doi.org/10.3390/ math9111239

Academic Editors: Francisco D. Fernández-Martín, José-María Romero-Rodríguez, Gerardo Gómez-García and Magdalena Ramos Navas-Parejo

Received: 11 April 2021 Accepted: 26 May 2021 Published: 28 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

This can be used to establish interest of the students in mathematical concepts, and secure the learned concepts. In this work, we take advantage of stimuli and the motivation that the spirit of games can infuse in students [12] and they are used to introduce and reinforce certain concepts that form part of the curriculum of this subject in different levels of education. What is intended, as stated by Marín [13], is not to "ludify" education, but "to promote learning processes based on the use of games for the development of effective teaching-learning processes, which facilitate cohesion, integration, motivation for content, and enhance the creativity of individuals".

Some questions that are posed about proposals of this type are: Can games really be used in the teaching of mathematics? How? What games? What objectives can be achieved through the games? [14,15]. In the present article, these questions are answered through an exhaustive analysis of a concrete proposal made by the authors.

We support game-based learning in mathematics [16–23]. Iriondo-Otxotorena [24] already did in an experiment in which an introduction to algebra was proposed through the solving of puzzles and enigmas. Our experience allows us to affirm that it is necessary to incorporate new methodological tools that are attractive for students. It is very common for students to find the subject of mathematics difficult to understand, boring, and impractical, so they get discouraged, stop paying attention to the teachers' explanations, and neglect their studies.

The incorporation of games into the mathematics classroom can be done one of two ways: through traditional games and through digital ones [25–27]. This article presents the possibility of combining these two modalities, by creating a game that is based on a traditional one to be played in the classroom, but which also has been adapted digitally to be played online. This combination has made it possible to work using both perspectives, taking advantage of the benefits provided by both.

#### *Statement of the Problem*

The study of real functions of real variables is taught in high school programs and the first courses of scientific and technical degrees. The proposal that has been developed adapts the foundations of a classic game such as "Who is Who?" with the objective of using its educational value, mainly for the consolidation of knowledge related to the study of real functions of real variables. We want to offer students the opportunity to play and overcome challenges, for which their mathematical knowledge on this subject will be of great help, while providing teachers a tool that they can adapt to their needs and to help them improve performance in their classes. The adaptation of the game has been proposed so it can be played not only in the classroom with other students but also as a video game that allows students to play individually as many times as they want.

In most games, participants play to win, and to overcome the challenges that the game presents. To win the game that is presented, it is necessary to resort to skills that are related to mathematics, which students should know or are learning in their classes. In addition, they will have to observe the possibilities, deduce, generalize results, plan future options, etc., all elements necessary for their academic training.

Thus, with this proposal, we want to offer teachers and students the opportunity to use and create games with the purpose of working within the daily lessons in the classroom, as well as carrying out a training method for their reasoning abilities. This way, the presented proposal aims to answer the following questions:


We believe in the need for another form of education, in which the emphasis is placed on the essential skills of people, promoting creativity, personal initiative, and self-learning, and where educational innovations and initiatives are integrated, in which games have an important role to carry out. Guzmán [28] pointed out that the factor with the highest

influence is the teaching method. The work to come is framed in this line: games as an adequate instrument to learn. In this case, we are focused on students of different ages.

#### **2. Materials and Methods**

#### *2.1. Theoretical Framework*

Undoubtedly, one of the teaching fields where more work remains to be done is that of mathematics. While in other scientific disciplines, there has been great advances in their teaching in recent years, mathematics classes are moving in the same direction but much more slowly. Still, they often remain unrelated to daily reality, becoming a cluster of mechanical and disconnected exercises. Therefore, we consider it important to develop new tools or adapt some existing ones that are useful and easy to implement in the classroom, and always related to the curriculum.

According to Brull and Finlayson [29], game-based learning allows students to participate and learn, enjoying the freedom to experiment and fail in an enjoyable environment. Learners have the opportunity to interact with experiences that keep them motivated. There is evidence that students involved in gamified environments improve their learning, and increase their motivation and engagement [30].

Werbach and Hunter's proposal [31] suggests that in order to create a successful game, three fundamental elements must be introduced: game components, mechanics, and dynamics. However, before choosing these elements, six steps must be followed: defining objectives, defining the desired behaviors, describing the type of players at whom the game is aimed, choosing the activities to be carried out, including an element of fun in the activities, and developing tools (Figure 1). Thanks to its structure, this methodology is flexible and can be adapted to almost any context, particularly mathematics. These steps, as explained below, have been followed in this work for the development of the proposal.

**Figure 1.** Steps to follow for the start-up process.

We have made several innovative educational proposals through games that allow us to understand and apply different concepts and mathematical theories.

The methodology applied in the project presented in this paper has been put into practice in different stages:

• Search for board games existing in the market that can be suitable from a mathematical point of view and adapted for teaching and learning of various topics. The choice of games which are familiar to students means that the game mechanics are already familiar to them.


The educational courses and levels at which the project is directed include:


Concepts tackled:


These steps are outlined in Figure 2. A further development of them, specifically for the game developed, can be found in Section 2.3.

**Figure 2.** Steps to follow for the implementation of the game.

#### *2.2. Procedure: How Do You Work with Students?*

A session of "Games in the Classroom" will be presented to the students. For this, a group of students must be selected who will be involved in the realization of the game and in the organization of the game. The rest of the students will be those who will attend the game day as participants.

We will work with the group in charge of carrying out the proposal in various guided sessions.

When the day of the games session arrives, it will be adjusted to the agenda developed in class, and a competition will be proposed to the students in the classroom.

After the competition, the most appropriate mathematical strategy for the game played is explained. Mathematical contents that propose these strategies are remembered and students are allowed to analyze and think about them.

In this paper, we will focus on one of the games presented to the students: Binary Who is Who? We develop its design below.

#### *2.3. Methodological Design*

The design of the game is based on the classic game Who is Who? whose rules are the following:

Each player has a series of cards with different characters. One of the participants chooses a card with a character and places it without the other player seeing it. The other player performs the same operation.

The objective of the game is to guess which character the other player has chosen.

In each turn, questions are asked about the features of the character the player wants to discover.

The other player answers these questions with a yes or no: if the player answers yes to a question, the cards of the characters that do not have that feature are removed; if the player answers no, the tiles of the characters that do have that feature are removed.

#### 2.3.1. Adaptation of the Game

We wanted to adapt the idea of this game to the framework of the study of real functions of real variables, also including some other concepts such as numerical systems, specifically the binary system, which is of great use in computing (the same could be done with other topics).

For this, instead of characters, the idea is to work with defined functions both analytically and graphically. The first step is to choose the characteristics (traits in the classic game) about which the player will ask in order to discover the function chosen by the opponent. In our proposal, the selected ones are: sign of the function, range, monotony, continuity, derivability and existence of a vertical asymptote. Therefore, the six questions to be asked about the function, from which the desired function should be discovered, would be:


Each of these questions will be answered with a yes (1) or a no (0). With this, each of the functions that are part of the game will have a definition in binary code that will consist of a vector of six digits, zeros or ones, depending on the answers to each of the questions to be asked for that function (Figure 3).

**Figure 3.** Binary vector that is updated each moment with the known data.

The number of possibilities that exist in the game will be the number of variations obtained with repetition of two elements {0, 1} taken from six in six: 26 = 64 (Table 1).


**Table 1.** Possible game responses.

They seem like too many functions to generate. We will see in the next section that, thinking a bit, this number is largely reduced.

#### 2.3.2. Creation of the Game and Its Collaborations

The first work to be done will be with the group of students who have agreed to collaborate in the realization of the game. They will be asked:

1. Carry out the necessary study to limit the number of functions that the game will offer. They must think what affirmative or negative answers to certain questions necessarily imply a certain response to other questions. For example, a non-continuous function implies that it will not be derivable either. With this, vectors of Table 1 with elements (-,-,-,0,1,-) must be removed from the list. Carrying out this work supposes an exhaustive review of all the theories of functions that are studied in the classroom from a new perspective. Students are also familiarized with the concept of binary numbers (in base 2) and, in general, with the different bases for expressing numbers.

A manageable number of possibilities, say 28, is selected from Table 1 (see Table 2). This is a suitable number to play.


**Table 2.** Possible feasible game responses.

2. Define the 28 functions that will represent each of the 28 resulting vectors and which will be the game cards. This definition will sometimes be carried out through a formula (analytical definition of the function), and other times through a graph.

Once the functions are obtained, the chips of the game will be produced physically or virtually (if the game is going to be made with the support of a computer). Each of these cards will have an assigned definition of the function and the number in the decimal system that corresponds to the binary number of the vector that gave rise to the function. For example, the function associated with the vector (0,1,0,0,0,0) will be written with the number 16 = 0 × 25 + 1 × 24 + 0 × <sup>2</sup><sup>3</sup> + 0 × <sup>2</sup><sup>2</sup> + 0 × 21 + 0 × 20 (Figure 4).

**Figure 4.** Image of an analytical game sheet.

This corresponds to a function that is not positive, that is bounded and that is not monotonous, nor continuous, nor derivable and it does not have vertical asymptotes.

The function associated with the vector (1,0,0,0,0,0) has the written number 32 = 1 × <sup>2</sup><sup>5</sup> + 0 × <sup>2</sup><sup>4</sup> + 0 × 23 + 0 × <sup>2</sup><sup>2</sup> + 0 × 21 + 0 × <sup>2</sup><sup>0</sup> (Figure 5).

**Figure 5.** Image of a graphic card of the game.

This corresponds to a function that is non-negative and not bounded and is not monotonous, nor continuous, nor derivable and that has no vertical asymptotes.

#### 2.3.3. Development of the Competition

There are several ways to introduce this game in the classroom. We have chosen one that allows all students to play simultaneously, so that no student is unable to participate. Each participant will have a template with the 28 cards corresponding to the functions of the game and a box of six positions to complete. In addition, the cards can be made in a digital format and will be shown to the participants through a projector. On this screen, the cards are projected on the side of the functions in an arrangement of rows and columns, and students are told that they are numbered from left to right and top to bottom. (Figure 6).

**Figure 6.** Projection in digital form of the functions selected for the game.

Steps to follow:


The search for a good game strategy makes the participants reflect broadly on the concepts related to the real functions of real variables:

A good knowledge of the theory allows the participant to ask the right questions that lead to a quick resolution. In the same way, one can make beneficial use of the questions asked by colleagues and the answers received. For example, if a student asked about the existence of vertical asymptotes, receiving an affirmative answer, it is supposed, for someone familiar with the subject, that the answers to continuity and derivability of the function will be negative. One can complete the sequence of the binary number without asking questions.

#### *2.4. Video Game*

It is an assumed fact that the current generation of students feels a great attraction to computer games. This can be used to motivate the teaching of mathematics [34,35]. In this way, we have adapted the proposed game to the virtual environment by programming a videogame that recreates the "Who's That Function" challenge. This allows the students to become acquainted with the concepts related to the study of functions in a proactive way and to improve their performance through the obtained marks in the online game.

The instructions are displayed in Spanish and the game has free access by the following link: https://flyingflamingo.itch.io/whos-that-function, accessed on 11 March 2021 (Figure 7).

**Figure 7.** Screenshots of the online game.

Technical Description:

• Programming language:

The game has been programmed by using Godot Engine, whose scripting language is GDScript (similar to the Python language). To execute it in the navigator, the system allows for creating an executable file translated to HTML5.

• Operative environment:

Any standard desktop navigator can serve as an operative environment. Therefore, the online game can be played on platforms such as GNU/Linux, Windows or MacOS. It has been tested in Firefox 82 and Chromium 85.

• Flow chart (see Figure 8):

**Figure 8.** Flow chart for the "Who's That Function" online game.

#### *2.5. Educational Content*

The game to be implemented has been designed to be able to assess the students' knowledge once the analysis unit has been completed. The main contents of this didactic unit are:

Limits of functions Calculation of asymptotes of a function Continuity Derivability Growth and decay

Concavity and points of inflection Domain and range Monotonicity Intervals of constant sign, regions Graphical representation

As explained above, the objective of the game is for the students to find out the function that has been previously selected either by the teacher if the game is played in the classroom, or by the computer, by answering six key questions.

It is essential that when given an analytical representation of a function, a student can produce its graphical representation, and also know how to specify its characteristics. This first objective is achieved by a student matching the function cards with their analytical and graphical representation. A student who is unable to match both representations will not receive a good result and will not score a mark.

The decision of which questions should be asked responds to the need to assess the maximum amount of key knowledge of the didactic unit. It is essential for the student to evaluate whether a function is continuous or has discontinuities. As such, one of the questions that may be asked is: Is the function continuous? After the game gives the answer to this question, the player must be able to eliminate the incorrect functions by applying their knowledge of continuity. Similarly, for the concept of the derivability of a function, there is the direct question: Is it derivable?

The calculation of the limits of a function and the concept of asymptotes are addressed by the question: Does it have a vertical asymptote? To correctly respond, the student must be able to manage these concepts both graphically and analytically.

The range of a function is reviewed with the question: Is it bounded? Growth and decay and monotonicity are analyzed with the question: Is it monotone?

To put the student's knowledge about regions of constant sign for a function into practice, they are asked the question: Is it a non-negative function?

Additionally, the game works with the theoretical implications of function properties. For example, a derivable function is continuous, a function with a vertical asymptote is not continuous in R, a discontinuous function is not derivable, a function with a vertical asymptote will not be bounded, etc. The student must put this knowledge into practice. When playing in the classroom, using this knowledge will save time and allow the student to respond more quickly to the challenge. In the case of the video game, the format is designed to give the highest score to the player who gets the correct answer with the fewest questions asked.

The concepts of domain, concavity and points of inflection are not directly included in the questions asked in the game, but are addressed in a tangential way. For example, a common misconception among students is to think that, at a point where the function has a vertical asymptote, the function is necessarily undefined, or that a point of discontinuity does not necessarily belong to the domain of the function. Additionally, some students make the mistake of thinking that an inflection point marks a change in the growth of the function (e.g., from increasing to decreasing). The graphic cards in the game, as well as the teacher's guidance as the game is played, can be helpful in correcting these misunderstood concepts.

Additional content that is worked on in the game is the representation of a number in different bases (positional notation). This content is part of the number theory unit, which is useful for preparing students for technical degree programs such as computer engineering or mathematical engineering. Even though it is not present in the game cards, it is evaluated at the end of the game, since the students have to convert a number in base 2 to base 10 to obtain the final result.

Table 3 shows the most relevant information on the key mathematical concepts included in the game, their relationship with the game design and the score given to the players.


**Table 3.** Concepts tackled in the game.

#### **3. Results**


After the experience ended, in order to carry out an assessment of the opinions of the students and the results, a survey was carried out for all the participants (to both those who contributed to the creation of the game and those who played it). The survey and its analysis are attached in the Appendix A. It is worth mentioning the good evaluation of the students who were involved in the design of the game who, in a major way, valued very positively the learning that their accomplishment gave them.

Regarding the students who played the game, around 90% of them agreed or strongly agreed that the game helped them to understand the unit and be motivated by mathematics:

92% say it has helped them to review the concepts.

84% consider it appropriate or very appropriate for their level.

85% would use more of these types of games in the course.

86% would find it appropriate to use these types of games in other subjects.

87% have increased their interest in mathematics.

90% would recommend the game.

We reached the objectives that we set ourselves and can be summarized as:


The achievement of the aforementioned objectives entails the facilitation of the students' learning process, which can be done through an enjoyable, playful, flexible, dynamic and interactive mechanism, which is expected to attract students and promote their involvement in the subject.

#### **4. Discussion and Conclusions**

We are sure that the actions proposed in this project will contribute to the approach of students towards understanding essential basic subjects in their studies. The application of games is highly motivating and is a good reinforcement if applied to mathematical subjects. In addition, it allows students to integrate and relate to each other since it promotes team actions.

The groups of students to whom we have presented the activity have shown great receptivity and an excellent understanding of the content presented, which has improved their perception of mathematical topics.

These good results are supported by the surveys presented to the participants (see the Appendix A). We emphasize that the students who collaborated in the realization of the game showed greater satisfaction with the experience and what it gave them.

As future actions, we want to involve students in the design or invention of new proposals (new games) where they must use the contents they have learned or are learning in the subjects related to mathematics. In this way, we want to stimulate creativity and mathematical knowledge.

#### **5. Patents**

The software shown in Section 2.4 has been registered in the Intellectual Property Registry. File number: 09-RTPI-08550.8/2020.

**Author Contributions:** Conceptualization, S.L., M.L., S.M. and J.R.; methodology, S.L., M.L., S.M. and J.R.; software, S.L. and M.L.; formal analysis, S.L., M.L., S.M. and J.R.; investigation, S.L., M.L., S.M. and J.R.; resources, S.L., M.L., S.M. and J.R.; data curation, S.L., M.L. and S.M.; writing—original draft preparation, S.L., M.L., S.M. and J.R.; writing—review and editing, S.L., M.L., S.M. and J.R. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research has been carried out with the help of the project of educative innovation IE1920.0403: "From Game to Theory" (UPM).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The third author has been partially supported by project PDI2019-110712GB-100 of Ministerio de Ciencia e Innovación, Spain.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **Appendix A**

Survey conducted with students who participated in one way or another in the proposal.

Survey on the activity "Binary Who is Who"

1. Rate from 1 to 5 your level of interest towards the Mathematics subject, where 1 is nothing and 5 is a lot.

Mark only one box.

12345

2. Do you think that this game has helped you to better understand the functions? Rate from 1 to 5, where 1 is not useful and 5 very useful.

Mark only one box.

12345 3. Do you think this game has helped you to review the contents of the subject? Rate from 1 to 5, where 1 is not useful and 5 very useful.

Mark only one box.

12345

4. Do you think the difficulty of the game is appropriate to your level? Rate from 1 to 5, where 1 is not adequate and 5 is very adequate.

Mark only one box.

```
12345
```
5. Would you like to be able to have games of this style for other subjects of Mathematics? Rate from 1 to 5, where 1 is I would not like it and 5 I would like it very much.

```
Mark only one box.
```
12345

6. Would you like to be able to have games of this style for other subjects? Rate from 1 to 5, where 1 is I would not like it and 5 I would like it very much.

```
Mark only one box.
```
12345

7. Do you think this game has contributed to increasing your interest in Mathematics? Rate from 1 to 5, where 1 is nothing and 5 a lot.

Mark only one box.

12345

8. Would you recommend this game for students of your same level? Rate from 1 to 5 where 1 is not recommended and 5 highly recommended.

Mark only one box.

12345

Attached is the analysis of the surveys made to the students of the 1st year of Civil and Territorial Engineering degree of the UPM (2018-2019 academic year):

(a) Students who participated in the making of the game.

Number of students who participated: 6.

**Figure A1.** Results of the survey of the students who collaborated in the making of the game.

(b) Students who have played "Binary Who is Who".

#### Number of students surveyed: 100.

**Figure A2.** Results of question 1.

**Figure A3.** Results of question 2.

**Figure A4.** Results of question 3.

**Figure A5.** Results of question 4.

**Figure A6.** Results of question 5.

**Figure A7.** Results of question 6.

**Figure A8.** Results of question 7.

**Figure A9.** Results of question 8.

#### **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Mathematics* Editorial Office E-mail: mathematics@mdpi.com www.mdpi.com/journal/mathematics

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18

www.mdpi.com