1. Introduction
In recent times, due to a world in constant evolution, with great technological advances [
1], and the effects of the recent pandemic [
2,
3], researchers seek to build novel learning environments to develop lifelong skills in students [
4,
5]. Additionally, educational institutions seek to implement strategies and methodologies that offer continuity in the students’ learning and motivate them along the teaching–learning process, allowing a balance in the effort of teachers to adapt their courses. Some researchers propose the use of active learning methodologies for the development of such skills, such as in [
5]. The active learning process has the student learning at its center. It focuses on how students learn, not just what they learn. Students are encouraged to ‘think hard’, rather than just listen and take notes.
Active learning is important because of its positive impact on learning and because it can be incorporated into a course in numerous ways [
6] and there are several works on this. For example, [
7] analyzed the impact of the implementation of an active methodology, namely inquiry-based learning. In the academic context, the authors associated self-efficacy with students’ performance, goals, and motivation. They analyzed students’ motivation as a key element in the teaching–learning process. Additionally, according to [
8], the existence of active learning helps students improve emotional intelligence. When self-efficacy and emotional intelligence are achieved, student learning outcomes will improve.
On the other hand, the need to continue experimenting with pedagogical strategies has increased as student groups are heterogeneous, with students from different economic, social, and cultural backgrounds, or students who have had negative or frustrating experiences in their learning; more significantly with students who require more motivation and guidance to acquire knowledge. In this sense, an educational approach based on the combination of blended learning and active learning is presented in [
9]. Due to the flexibility offered, the authors argue that it is a great prospect for the enhancement of knowledge and skills acquisition.
These differences are accompanied by the challenges that teachers have and that are derived from the pandemic. For example, [
10] mention that keeping students motivated and the possibility of applying traditional strategies (and even e-learning for some disciplines) are a complication that increases professors’ workload and creates stress. Additionally, job market demands are changing rapidly, and professors must prepare students for elements that have not yet been discovered. The introduction of technology in the teaching and learning process, and the introduction of new learning methodologies, present important advantages for students, such as enhancing access to educational resources, flexible education, gaining self-learning abilities, increasing learning opportunities, and lifelong learning as presented in [
11]. However, technology integration is another challenge to the teaching–learning process as stated by [
12]. Technology is not always accessible for every professor or for every student at any moment. As such, learning strategies must be easy to implement given all the variables in this context.
As an alternative to traditional teaching, in [
13,
14,
15], the flipped learning pedagogy is mentioned. The flipped learning approach works by requesting students to do the passive learning activities outside of the classroom while the active learning activities take place inside the classroom [
16]. It involves students reviewing learning materials before coming to class; the content attainment is shifted to outside. Class time is used to explore topics in greater depth via problem-solving and peer instruction. This pedagogical approach has gained popularity in the quest to improve student learning outcomes because it presents advantages for students and professors. For example, they can watch a video at their own pace, they can pause, fast forward or rewind it, and watch it as many times as they need. Another advantage is that the professor can use class time to assist the students while working on activities where they may need the professor’s expert help.
Additionally, analyzing the results obtained on flipped learning revealed that not all students consulted the technological resources or the theoretical contents in books, magazines, tutorials, or articles [
17]. This led to the establishment of another strategy to support flipped learning, which would guarantee or make it possible to detect those students who did not consult the information outside the classroom. This strategy is known as gamification.
Gamification is a learning technique that has shown to have a positive effect and has been used in different branches of education to encourage student motivation and learning. It consists of incorporating game design elements into the educational context. The most used elements that have generated positive results are the points, badges, and classification tables or leaderboards; this generates commitment and engagement to the activity, and create positive expectations that help in achieving significant learning. For this reason, gamification is frequently used to increase student motivation required to obtain better results in their learning process [
18,
19,
20]. As stated before, gamification and flipped learning strategies have been reported as an efficient methodology combination for learning [
21]. For example, in [
22], the author argues that the advantages of game-based learning combined with flipped learning improve students’ intrinsic motivation towards learning goals achievement.
The analysis of the application of gamification in online environments when implementing flipped learning is presented in [
23]. The study used mixed-methods sequential explanatory design to collect and analyze quantitative and qualitative data. The study seeks to determine if students achieve better learning outcomes by increasing interaction with online resources using gamification. According to the findings reported by the authors, the experimental group got higher scores in terms of interaction data, participation, and achievement compared to the control group.
The methodological contrast between gamification and flipped learning to determine the most influential methodology in teaching–learning processes is described in [
24]. The results of the analysis carried out by the authors show that gamification is better valued in participants at an early stage of the process, while more experienced participants value the flipped learning methodology better.
Light teams and gamification are explored in [
25] to enhance students’ learning and social experience. In addition, team-based gamification was documented in this study. The experiment consisted of a design and implementation of a Moodle plugin for gamification to reward students for practicing “good” study habits (such as turning in assignments early and retaking quizzes for extra practice). The results documented showed that the students adopted better study habits. However, these good practices did not generate better grades in the final exams.
Another experience of the application of the combination of flipped learning and gamification is described in [
26]. Experience documents that the results obtained by students improve when applying both pedagogical strategies.
Therefore, it can be summarized that the combination of these two techniques has been used in various areas of knowledge and at different educational levels. A challenge that has been reported by authors is to define the game elements to be used to scaffold the flipped learning strategy to meet learning objectives [
17,
23,
24,
25,
26]. Control over the competition environment must be applied to maintain and increase the quality of the teaching and learning process.
In this work, a quantitative study is presented. The objective of the experiment was to test the learning gain of the students through the separated and combined use of the flipped learning and gamification strategies. The study was applied to 414 students in basic programming groups. The students were divided into four groups: in group 1, the pedagogical strategy used was flipped learning, group 2 exclusively applied gamification, group 3 combined both strategies, and group 4 was a control group. Tests were applied to the students at the beginning and at the end of the analyzed courses, deploying a pretest–posttest methodology. Statistical analysis was applied to generate results on the learning gain obtained in each of the groups studied. According to a statistical study, students that went through flipped learning or a gamification strategy perform 9% better than those in a traditional class. Moreover, students subject to a combined strategy of gamification and flipped learning have a lower learning gain (by about 10%) than those that were included in only one of the strategies. Results are presented in subsequent sections and a graphical abstract can be found in
Figure 1.
This document is organized as follows:
Introduction: In this section, a general scene for learning strategies is presented.
Hypothesis: Objectives of this study regarding the relationship between learning gains for flipped learning, gamification, mixed, and traditional teaching–learning strategies.
Materials and methods: Description of following the methodologies of flipped learning, gamification, and the combination of both.
Results: Statistical analysis over normalized learning gains of the pretest–posttest and focus–control groups process.
Discussion: Analysis of the results under the light of its strengths and limitations and contrast to other studies.
Conclusions: Summation of the presented work.
Data Availability Statement, Author Contributions and Acknowledgements appear at the end of the document.
4. Results
In this study, 414 undergraduate students participated in a basic programming course of engineering programs at Tecnologico de Monterrey. They were divided into 16 groups. The duration of these courses was 16 weeks. 68 students in 3 groups received the flipped learning methodology suggested in [
27]. 97 students in 4 groups received the gamification methodology suggested in [
28]. 206 students in 7 groups receive a combination of both methodologies. Additionally, 43 students in 2 groups received a traditional methodology and were considered as a control group. A quantitative experimental research methodology with primary data has been used to compare these samples using pretest–posttest and control–focus group processes. In this section, we will describe the variables used to compare pretest and posttest results. Learning methodologies were applied along the whole duration of the course (one semester).
Thirty-four videos were created to cover all the concepts in the course. Some of the videos were focused on step-by-step solved exercises to complement the theory parts. Slides, having the same content as the videos but in text and static images, were published at the same time. These elements were used by the students to explore contents outside the classroom as part of the flipped learning strategy. For the first part of the in-class section and to explore students’ learning, quizzes with one or two questions were prepared. Additionally, exercises, with no grade assigned and like those in the videos, were given to the students to practice and solve doubts with the professors’ help. As an evaluation, a set of programming problems (two to four, depending on the difficulty of the topic) were used in class. A grade was assigned for these activities.
A gamification system based on points, leaderboards and badges was also implemented. A points system was deployed to support the flipped learning methodology and to motivate students to check videos and material before class. Points were awarded based on the grade obtained in quizzes. A leaderboard was also publicly displayed on the learning management system of the class.
Badges were used to provoke different positive behaviors in students for basic activities in the course, such as arriving on time, improving their grades from partial to partial, being the first student to solve homework, among others. A specific badge was integrated into the system to reward students having the highest grade in the group for each quiz.
Additionally, a system with 8 levels was introduced. Students reached a level by accumulating a certain number of points. Students were awarded points in their final grade for the course if they reached levels 6 (1 point over 100), 7 (2 points over 100) or 8 (3 points over 100).
The flipped learning cycle was applied for each topic in the course while the gamification system was active from the beginning of the semester and was closed in the last class, updating and publishing results each week.
Pretest and posttest were the same test. They contained 96 true or false questions, covering all the topics of the course. Pretest was applied in the first session of the semester while the posttest was applied in the last one. Students had one hour to work on them. They were allowed to leave blank questions if they did not know the answer. Support material was not allowed. Students were not previously notified about the application of the tests, so they had not had the opportunity to study before it. The maximum possible grade was 100 points.
To progress with the results analysis, a normalized learning gain (as stated by [
29]) was obtained using the following variables:
Pretest result for the student
i:
Posttest result for the student
i:
Students’ normalized gain (1) for the student
i. This is a measure of the actual gain that the students achieved during the course (
Posti−
Prei) over the maximum gain that they could have obtained (100−
Prei), as a percentage.
Normalized gain mean for the group (2). N is the number of students.
Table 1 and
Table 2 show the summary of these variables for the four samples along with statistical values.
For the control group (N = 43), mean value = 41.013; std. deviation = 16.3918702, and std. error mean = 2.4997368. For the focus group that used only flipped learning (N = 68), mean value = 50.02, std. deviation = 13.7088726, and std. error mean = 1.662445. For the focus group that used only gamification (N = 97), mean value = 50.92, std. deviation = 16.2649151, and std. error mean = 1.6514519. For the focus group that used both methodologies (N = 206), mean value = 40.339, std. deviation = 14.9450952, and std. error mean = 1.0412741.
Figure 8 shows these values for the four samples.
Figure 9 shows the density charts for flipped learning, gamification, mixed methodology, and control groups’ learning gain variable.
Ref. [
29] defined the three ranges to characterize normalized learning gains: low for a value below 30; medium for a value between 30 and 70; and high for a value above 70. Additionally, Ref. [
30] say that interactive engagement courses (that use methods for hands-on activities with immediate feedback, such as in gamification or flipped learning) present a normalized learning gain in the medium range.
Table 3 shows this categorization for the four samples. As we can see, most of the samples present a medium normalized learning gain value. This agrees with the idea that these groups received an interactive engagement learning methodology.
To check if these results are comparable, we applied the Shapiro–Wilk Normality Test [
31] and the Levene’s Test for Homogeneity of Variance [
32] to the data.
Table 4 shows the results of these tests.
Given that these results allow us to continue the analysis, a one-way ANOVA test [
33] was performed to know if the means of these groups present a difference.
Table 5 shows the results of the test. As it can be seen, the results point to the fact that at least one group differs from the others with a 99.9% confidence level.
To analyze the pattern of difference between means, Tukey’s Honestly Significant Difference (HSD) test [
34] has been applied to data. Results of the test are shown in
Table 6.
Figure 10 shows the 95% family-wise confidence level for the test and the learning gain comparisons chart.
As it can be seen from the results of the HSD test, flipped learning (alone) and gamification (alone) groups have a significant difference of mean with the control group. Both present a higher normalized learning gain of around 9% (flipped learning diff = 9.0051422%, p = 0.0135752; and gamification diff = 9.9115396, p = 0.0023937). This means that both learning strategies have a better performance than a traditional course.
On the other hand, the group that applied both methodologies had no significant difference of mean with the control group. Additionally, there is no significant difference between the flipped learning (alone) and gamification (alone) groups. This result means that we cannot agree to hypothesis 1.
In contrast, we can see that there is a negative significant difference of mean between the flipped learning (alone) and gamification (alone) groups compared to the mixed group. The mixed group presents a lower normalized learning gain of −10% (flipped learning diff = −9.6790857%, p = 0.0000427; and gamification diff = −10.5854831, p = 0.0000002). This result means that, alone, each learning strategy increases the impact on students’ learning gains but when mixed, the result is similar (or even a little lower) to the effects of a traditional learning strategy. These results refute hypothesis 2 and 3 and prove that following this combined methodology of flipped learning and gamification strategies have the opposite effect on students’ learning gain.
5. Discussion
As we can observe, results point to the following elements:
Flipped learning strategy has a greater learning gain compared to a traditional class.
Flipped learning strategy has a greater learning gain compared to a class that applies the combination of gamification and flipped learning.
Gamification strategy has a greater learning gain compared to a traditional class.
Gamification strategy has a greater learning gain compared to a class that applies the combination of gamification and flipped learning.
The deployment of a flipped learning and gamification mixed strategy does not have a significantly different effect compared to a traditional class.
Results 1 and 3 were expected. There are several studies that confirm this. For example, [
35,
36,
37] found efficacy in the flipped learning strategy. On the other hand, [
18,
19,
38] found that gamification increases the learning gain in students. These findings are important because, given the composition of the used flipped learning and gamification methodologies, their advantages can be extended to other contexts. For example, such a flipped learning strategy can be implemented as a tool for environments with online elements (either completely online courses or in a hybrid learning way) to help professors tackle complicated concepts and to apply authentic assessment by observing students’ behavior and performance directly in the implementation phase. However, it is important to note that this work implemented flipped learning by means of videos and the use of a learning management system to store and publish them. It is possible that these elements are not available for all classes. In these cases, other tools (such as readings) can be used. On the other hand, the positive results obtained from the gamification experience can be extended to other learning environments because the learning strategy is flexible, as it focuses on behaviors instead of concepts or knowledge domains.
Outcomes 2, 4 and 5 of this study were not expected but it agrees with what was found in other studies, such as in [
25]. Authors in that study found that the combination of flipped learning and gamification indeed foster motivation on a computer science basic course (domain of this study), but there was no difference regarding the students’ final exam scores when introducing gamification to flipped learning. The present study can be extended to include motivation aspects; however, if this combination of strategies does not show an improvement in the students’ learning, such extension should be aimed to analyze a more effective arrangement of the combination points during the teaching and learning process. For example, the design of the mixed methodology in the present work was focused to engage students to be prepared for the start of the topics in the class by consulting prepared material outside the classroom. Hence, the combination points were directed to reward this behavior and it was not aimed to increase students’ performance (the measured variable). A new design can be created to support and increase learning performance additionally.
A similar experience can be found in [
39]. Authors observed that the introduction of gamification to a flipped learning class had a direct positive effect on learning process performance and that this aspect was a significant positive predictor of application-oriented knowledge, thus proving that gamification has an indirect influence on application-oriented knowledge. This result contrasts with the results in the current study where there was no positive effect on learning process performance (according to the type of the pretest–posttest design). It is possible that this difference comes from where the gamification elements were inserted to the flipped learning methodology. Ref. [
39] added them to in-class activities while this study placed them around pre-class activities. This contrast could suggest that a mixed approach presents better results when applied to elements of direct work of the professor with the students. Maybe the influence and guidance of a professor is needed in the first stages when learning new contents.
The gamification of pre-class activities and the results presented in this study contribute to increase the analysis of these learning strategies given that, as stated by [
21] in their systematic review, in-class elements of flipped learning are more frequently gamified compared to the out-of-class components.
Alternatively, we find other studies that found that the combination of these learning strategies resulted in a greater learning gain. For example, [
23] reported that the inclusion of gamification in a flipped learning environment had a small positive effect in the grades of the students compared to those of students in a control group with only a flipped learning experience. Despite this finding, the difference of mean between these groups is approximately 1 or 2 points. Ref. [
26] confirm this outcome reporting a difference of means below 1 point over 10 points between focus and control groups. Additionally, [
40] have similar results. They find a difference of means around 2 points over 10 points. It is important to note that these studies use pure quizzes grades to measure learning benefits instead of learning gains or normalized learning gains. This could suggest that a mixed strategy has positive results when applied to reinforce short-term learning elements (as for quizzes) instead of being applied to long-term learning elements (as for a complete course).
Another variable to consider for this 5th finding is the complexity added to the course when combining elements from both strategies. Ref. [
41] says that a flipped learning and gamification mixed strategy increased motivation on students but presented an increased complexity for the students to be prepared for classes and activities. Additionally, the author states that preparing and deploying the course represented a bigger workload for professors. This is confirmed by the professors that participated in the current study. It is possible that this complexity or difficulty preparing the course and following its journey, and that is applied to both students and professors, interferes with the learning gain results. According to this, we suggest having a separate deployment of strategies initially while students get used to the pace and increase the combined elements towards the end of the course. This demands a long-term course and an increased workload for professors.
Lastly, it is important to say that gamification is a strategy directed to enhance a specific behavior by means of the application of extrinsic motivation. As stated by [
42], extrinsic motivation could inhibit players’ autonomy and diminish competence needs satisfaction. This is because, if players do not internalize this type of motivation, there is no meaning for them in performing the activity or changing their behavior. This, in turn, prevents motivation from being sustained. This effect can spill over to the other strategy and result in students losing interest.
On the other hand, an important thing to include as part of the gamification system design is the players’ characteristics and goals, as stated on [
22]. The present study, as stated before, used a general leaderboard, points and badges gamification system without any specific element related to aspects outside the learning environment (i.e., students’ age or hobbies). A gamification system with elements as avatars or a game-like story could be designed to explore if this variable can scaffold the teaching–learning process. However, we could suppose in advance that this will not be related to the learning gain even if it could have a positive impact on motivation. Moreover, a design like this could require technological resources to be deployed.
Table 7 summarizes the results of the studies analyzed in this section.
6. Conclusions
In this study, we presented a statistical analysis on learning gains between groups of students that received classes with a flipped learning, a gamification, a mixed, and a traditional methodology. A total of 414 undergraduate students in a basic programming course that were part of engineering programs at Tecnologico de Monterrey, participated in this study. A normalized learning gain was obtained by means of a pretest–posttest process based on focus (flipped learning, gamification, and mixed strategy) and control groups (traditional class).
It was found that flipped learning and gamification control groups presented a greater normalized learning gain compared to the control group and the one that received the mixed strategy. Additionally, no statistically significant difference was found between flipped learning and gamification samples, nor between the control group and the mixed strategy group.
It is important to state that these results were obtained by implementing the strategies in a specific area (basic programming). The application of the suggested methodology must be done in other fields to surpass the limitation of generalization.
Future work over motivation elements must be applied to know the impact of the combination of both learning strategies over the scaffolding of students’ positive behaviors. One possible path is to measure changes in both extrinsic and intrinsic motivation. This would help to know the degree of success in internalizing the meaning of the activities carried out by the students. Another possible path would be to use other game elements beyond the basic ones used in this work (points, badges, and leaderboards). For example, the use of avatars or virtual reality could be explored to know if these technologies increase students’ performance.
Additionally, as stated before, the proposed mixed methodology is based on outside-class gamified elements. Research over the points of insertion of gamification elements over the flipped learning structure is required to identify if students’ learning gains depend on how and when the combination of these elements occurs and if there is a change when applying in-class gamification in a mixed methodology. This suggests, for example, the use of gamification elements during the discussion phase of the flipped leaning strategy by assigning roles to students to explain what they have learned by doing the outside-class activities.