Effectiveness of Using ChatGPT as a Tool to Strengthen Benefits of the Flipped Learning Strategy

Abstract

In this study, we evaluate how ChatGPT complements and enriches the traditional flipped learning strategy in higher education, particularly in engineering courses. Using an experimental design involving 356 students from basic programming courses in undergraduate engineering programs, we compared the normalized learning gain between groups that used the ChatGPT-assisted flipped learning strategy (focus groups) and those that followed a traditional video-based flipped learning methodology (control groups). The intervention lasted ten weeks, with two sessions of two hours each week. A pre-test–post-test analysis revealed that the focus groups showed significant improvement in normalized learning gain values compared to the control groups. These results confirm that incorporating ChatGPT into the flipped learning strategy can significantly enhance student performance by providing a more active, interactive, and personalized approach during the teaching–learning process. We conclude that the flipped learning strategy, upgraded with the assistance of ChatGPT, provides an effective means to improve understanding and application of complex concepts in programming courses, with potential to be extended to other areas of study in higher education. This study opens routes for future research on the integration of artificial intelligence into innovative pedagogical strategies with the goal of scaffolding the learning experience and improving educational outcomes.

1. Introduction

Currently, the world is facing extraordinary changes. Mainly in the field of education, institutions are confronted with challenges ranging from understanding the different ways in which students learn, up to making decisions about what innovative technology needs to be used to deliver learning content [1,2], and how to increment the motivation of students [3,4]. This perspective proposes two paths to follow: faculty can explore ways to enhance teaching and learning with tools that have existed for several years, or faculty can look for new tools to achieve the goal of improving the teaching–learning process. The challenge is not only the selection of the technology, but also how to integrate it into the current didactic strategies that play a fundamental role in enhancing learning in higher education and developing in students lifelong learning skills.

Some didactic strategies are based on active learning, which is generally defined as any method of instruction that involves students in the learning process, asking them to think about what they are doing [5]. The core elements of active learning are student activity and participation in the learning process. Active learning is often contrasted with traditional lecturing where students passively receive information from the instructor.

Active learning allows students to actively participate in the teaching–learning process, improving their motivation and attention. Some of the techniques are project-oriented learning (POL), problem-based learning (PBL), challenge-based learning (CBL), and flipped learning. The latter one will be addressed in the next section due to its acceptance and flexibility to integrate tools in it.

1.1. The Traditional Flipped Learning Methodology

The Flipped Learning Network defines flipped learning as follows: ‘Flipped Learning is a pedagogical approach in which direct instruction moves from the group learning space to the individual learning space, and the resulting group space is transformed into a dynamic, interactive learning environment where the educator guides students as they apply concepts and engage creatively in the subject matter’ [6].

In flipped learning, students consult material (videos, books, or websites, for example) before class. Professors explore what students have learned through instruments such as quick surveys or interaction reports with study tools to adapt the group space to specific students’ learning needs. During class, students work on active learning tasks with the professor’s help to solve any questions. It is a student-centered technique because the students are more active than the professors during class [7]. The professor is a facilitator to guide and give feedback on their performance to students during class. The use of this technique allows students to use technology, such as videos, podcasts, or applications, to access material before class.

The main idea behind the flipped learning methodology is that the students do not need the professor to give them content (i.e., students can obtain the content on their own), but they do need the professor when they get stuck [8].

Because the revision of material takes place before class, it frees up substantial portions of time during class that can now be spent on the activities where students typically need the most help, such as applications of the basic material and engaging in deeper discussions and creative work with it, while professors can focus on the required outcome [9,10,11].

One important benefit of the flipped learning model is to lengthen professor-to-student and student-to-student interaction during class time. Educators using the flipped learning strategy declare that the best benefit is that, for the first time in their teaching careers, they have some one-on-one contact with every student during every class period [12].

In the flipped learning model, out-of-class activities facilitate the acquisition of lower-order thinking skills (understanding concepts, definitions of terms, etc.), and in-class activities facilitate the acquisition of higher order thinking skills [13]. Applying Bloom’s taxonomy [14] to flipped learning, students engage in the lower levels of cognitive work (knowledge and comprehension) outside of class and focus on the higher levels of cognitive work (apply, analyze, evaluate, and create) during class, where they benefit from the stimulation given by the interactions with their peers and the instructor [15]. The root of flipped learning lies in social constructivism and is connected to problem solving, active learning, inquiry learning and interpersonal communications, as [16] reported.

As we have already mentioned, the flipped learning strategy has gained momentum in its use, and there are studies and articles that support it. Paryani and Ramadan-Jradi [17]’s results lead to positive impacts due to increase of student engagement. Karabulut et al. [18] mention that flipped learning gained popularity amongst engineering educators after 2012, but it is imperative to understand the current practices in order to shed light on future implementations. Jensen et al. [19] highlight the importance of active learning, a constructivist approach, and the purpose of learning gains regardless of whether flipped learning is used. Fulton [11] mentions the reasons why flipped learning should be used and mentions an important fact about the acceptance of parents as a preferred technique for instructing their children. Kozikoglu research [20] finds an increase in articles on flipped learning and mentions that the model has a positive effect on the students’ academic achievement, motivation and metacognitive awareness.

Zainuddin et al. [21] found that one of the biggest challenges, when applying to a flipped classroom, is the student’s lack of motivation to consult the material prior to class, whether reading a text or watching a video. This makes it necessary to incorporate new strategies in the flipped classroom practice to increase student participation in class and so that the professor can really see and provide feedback on their performance. In [22,23], Huesca et al. implemented a flipped learning strategy mixed with gamification, in undergraduate basic programming courses, pointing to elements that must be taken care of to have positive results.

Finally, innovative strategies integrating AI tools, such as ChatGPT, could be implemented to strengthen the teaching–learning process. Faculty and institutions have a great responsibility to carry out research on how these tools can be applied to enhance students’ learning gains. Also, it is important to explore if AI assistance could help solve the problem of students’ low motivation to review material before class.

1.2. The Artificial Intelligence in Education and the Need for the Flipped Learning Plus ChatGPT Methodology

Modern technology uses artificial intelligence (AI) intensively. AI is employed in several sectors such as health, transport, economy, or communications. Higher education cannot be left behind. Some applications are already present in personalized learning, assessment, and assistance for professors. For example, ref. [24] states that the following AI techniques can be applied as tools into the learning process: adaptive learning method and personalized learning approach, academic analytics, image recognition, computer-vision, prediction systems, data mining or Bayesian knowledge interference, intelligent teaching systems, learning analytics, face recognition, speech recognition, virtual labs, A/R, V/R, hearing and sensing technologies, edge computing, virtual personalized assistants, and real-time analysis. This is because these tools enhance comprehension skills, facilitate teaching effectiveness, and support collaborative learning [25].

An example of the use of AI in education shows how robots can be built to collaborate with professors (cobots) to capture the attention of students in the classroom, monitor their progress, identify those who need extra help, keep them engaged and interested, and answer any questions the student may have [26]. AI has also been used to make adaptive learning more effective and enhance web-based educational systems as presented in [27].

The debate remains open regarding the relevance of using AI technologies in education, focusing primarily on student learning, development, and assessment. A common point among the authors is their suspicious view towards AI; however, optimism and hope emerge that with proper supervision and policy formulation, AI can contribute to enhancing teaching and learning [28]. Also, institutions and faculty have foreseen that the inclusion of AI in education is unavoidable, as [29] indicates.

In this sense, faculty needs to explore how AI tools can be applied to improve the teaching–learning process and how it can be used to extend the positive characteristics of the learning process. While AI helps students to learn, it does not replace learning [30]. This is also true for professors: while AI helps professors to teach, it does not replace teaching.

An AI instrument that has created a revolution in the latest years is ChatGPT. It is a generative pre-trained transformer developed by OpenAI. This means that it is an AI model that interacts in a conversational manner with the user to answer questions, create new information, or find patterns. The dialogue format makes it possible for ChatGPT to answer follow-up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests [31]. ChatGPT has been available for public access since November 2022, gaining significant interest since then. This tool is the result of the evolution of natural language processing techniques. Other similar models are BERT [32], XLNet [33], T5 [34], RoBERTa [35], and BLOOM [36].

ChatGPT stands out for its ability to generate high-quality text in real time as an answer to complex problems; it is currently one of the leaders among artificial intelligence chatbots. This has become a great choice for teaching and learning [37]. This specific characteristic concerns faculty because students might reduce their critical thinking skills by becoming dependent on this tool to solve their assignments instead of generating a complete solution on their own. There is a latent fear of considering ChatGPT as a high-risk tool in terms of its potential use for cheating.

However, this is possibly provoked because ChatGPT is not intended for traditional instruction types. That is, the inclusion of ChatGPT requires new designs for learning and assessing activities. It is also important to evaluate the implications of its use according to each activity context. Faculty cannot forget that it is still an imperfect tool that makes mistakes. Therefore, it is essential to design and implement teaching and learning strategies that integrate AI tools consciously and that evaluate their effectiveness for students and professors.

ChatGPT has been used in various ways in some courses; for example, ref. [38] used it in a programming learning course, to compare the professor’s solution with the automatically generated solution. This allows for comparisons on best practices for learning programming with ChatGPT. However, it was found that ChatGPT is not as effective in these courses because, despite providing immediate responses, it does not necessarily aid the student in developing their creativity and problem solving skills. Human intervention becomes necessary, as it offers guidance for problem solving rather than providing the exact code.

Ibrahim et al. [39] made an interesting finding and stated that ChatGPT provides good answers in introductory courses, but it is not satisfactory in advanced courses. In that study, professional graders (teaching assistants and professors), were asked to blindly evaluate work generated by both students and ChatGPT in two introductory courses, English Composition I (EC) and Intro to Computer Sciences (ICS), and in two advanced courses, Discrete Mathematics (DM) and Computer Networks (CN). ChatGPT’s mean grade was substantially greater than that of professional graders in EC (85% versus 55%) and was comparable in ICS (80% versus 90%). As for the more advanced courses, ChatGPT lagged far behind in DM (42% versus 85%) and in CN (24% versus 83%).

Sangzin Ahn [40] uses this approach to encourage active learning and critical thinking. Instead of traditional lecture-style teaching, students are assigned lessons in a chapter to research and present in small groups. These presentations become the core of classroom activities, delivering basic knowledge, and stimulating interactive discussions and question-and-answer sessions to enable deeper comprehension of the material.

The study of [41] provides new valuable insights into how ChatGPT is utilized to support human-computer conversations and how a ChatGPT-based flipped learning guiding approach (ChatGPT-FLGA) promotes students’ performance and perceptions in the flipped classroom. The findings revealed that ChatGPT-FLGA can significantly improve students’ project performance, self-efficacy, learning attitudes, learning motivation, and creative thinking tendency, compared with the T-FLGA. The interviewed students’ records provide some evidence that the ChatGPT-based flipped learning system can meet students’ learning needs, such as instant feedback, personal guidance and intelligent reminders, so that students’ learning interest is stimulated and self-efficacy is promoted, which results in improved learning achievement and learning attitudes.

Nataliya Hristova Pavlova mentions in [42] that not all students have an interest in or desire for independent thinking. One solution is the implementation of this approach with the help of appropriate homework and projects. Formulating interesting topics and assignments based in the interaction with ChatGPT is a challenge for the professor, and it is very important to teach the student that technology must be used responsibly.

Mo Wang et al., in [43], investigate the impact of prompt engineering, a key aspect of ChatGPT, on college students’ information retrieval in flipped classrooms. The experimental results provide evidence that proficient mastery of prompt engineering improves the quality of information obtained by students using ChatGPT. Consequently, by acquiring proficiency in prompt engineering, students can maximize the positive impact of ChatGPT, obtain high-quality information, and enhance their learning efficiency in flipped classrooms.

Currently, there is a constant search to find strategies that maintain students’ motivation in both face-to-face and online environments. These strategies should focus on students and facilitate the work of professors to improve the learning environment. In addition to this problem, there is the emergence of multiple tools that can impact students’ learning directly or indirectly. These types of tools can either accelerate or hinder the learning process if not properly directed. For example, using ChatGPT can be motivating for the students knowing that they are using the latest technology tools for their learning. ChatGPT can significantly enhance the flipped learning experience by providing students with a personalized and interactive learning resource that could help to scaffold motivation and foster learning. Therefore, this study also aims to demonstrate that it is better to use and harness the benefits of such tools rather than risking students using them without channeling their potential effectively.

Furthermore, it is of great importance to continue the research regarding the impact of using ChatGPT to complement the flipped learning strategy, since it is an area with a lot of potential, and we must confirm the findings that may appear such as those of the present work.

Due to what was previously mentioned, we propose the use of ChatGPT to enhance the flipped learning didactic strategy, trying to guarantee a good development environment for our students.

In this work we explain in detail the methodology used when merging ChatGPT with flipped learning and present a quantitative study with the aim of verifying whether the normalized learning gain is enhanced, by means of a pre-test-post-test process using control and focus groups. The results will be presented in the following sections.

This document is organized as follows:

• Introduction: An overview of the weighted learning gain applied with flipped learning.
• Flipped learning strategy and the use of generative AI: Description of the strategy to include the use of ChatGPT to scaffold the flipped learning methodology and a comparison to the traditional flipped learning methodology.
• Hypothesis: Objectives of this study regarding the relationship between learning gains for flipped learning and flipped learning extended by ChatGPT.
• Methodology: Processes, tools and material to collect and analyze normalized learning gain data.
• Results: Presentation of the statistical analysis of the normalized learning gain obtained in the application of pre-tests and post-tests for control and focus groups.
• Discussion: Exploration of the results and contrast to other studies.
• Conclusions: Summary of the main findings in the study and limitations statement.

2. Flipped Learning Strategy and the Use of Generative AI

As mentioned before, this study analyzes the use of ChatGPT to support the flipped learning methodology. In this section, both methodologies are described, the traditional one and the one supported by ChatGPT.

2.1. The Traditional Flipped Learning Methodology

The traditional flipped learning methodology used as a basis in this research is the one proposed by Huesca et al. in [22]. This methodology defines two phases: preparation and implementation. Next, these two phases are described.

2.1.1. Preparation Phase

The preparation phase includes activities that professors must complete before the course begins. This consists of the following steps (Figure 1):

• Identify expected outcomes and topics. The professor identifies the course topics, the expected students’ outcomes for the course, and the characteristics of the students (i.e., study program).
• Select or create materials to explain concepts and how to apply them. It is recommended to have different kinds of materials, such as videos or readings, to meet different students’ learning styles and preferences [15,44].
  Huesca et al. [22] also state that it is helpful to look for two types of materials:
  • To illustrate concepts and
  • To demonstrate how to complete exercises step-by-step.

This will help students to acquire the theory and then apply it.

2.1.2. Implementation Phase

The implementation phase has a list of steps that will guide the students to perform the pre-class activities, help professors in their teaching process and to assess students’ performance (Figure 2). This phase has the following steps:

• Study pre-class materials. Before each class session, students must prepare themselves by exploring the contents and materials the professor gave to them. It is recommended that professors explain the importance of this first step to students at the beginning of the course and that students know that there will not be a lecture about the topic in the class sessions so they need to take notes and write questions that arise [8].
• Apply a test at the beginning of the class. Professors apply a test to explore what students have learned in the pre-class activities. This test must be just about basic knowledge that students should have obtained while studying the pre-class material.
• Group discussion to solve student’s misunderstandings. The purpose is to develop an interactive learning environment where students feel comfortable asking questions about their assumptions to clarify concepts and their application [9].
• Provide activities to practice student’s new learning. Ask the students to work on tasks, exercises, and activities where they can put their newly found information into practice. In this moment, professor should be observing students’ work with the purpose of identifying difficulties and providing expert feedback [8,11], while students acquire higher-order thinking skills [13].
• Apply evaluation activities. Students need to know that their learnings are according to the class expected outcomes, and the professor needs to know if the topic is fully comprehended by students. Therefore, it is recommended to have performance assessment for each topic, and those assessments shall be of different types with the same complexity level as the class activities. This assessment process is aligned to the general and global course assessment process.

2.2. The Flipped Learning Methodology Supported by ChatGPT (AI Enhanced Flipped Learning)

ChatGPT is causing a lot of expectation among faculty and students. Faculty is interested in increasing motivation on students and scaffold learning strategies, like flipped learning, to increase learning gains of students. On the other hand, students use technology in daily activities and are aware of its progress. Given this, we present in this work a methodology to implement the flipped learning strategy supported by ChatGPT. In this section, the changes made to integrate ChatGPT to the traditional flipped learning methodology, described in the former section, are presented.

The proposed methodology is also divided into two phases, preparation and implementation.

It is important to preserve the preparation phase as it guides the professor’s previous work to carefully prepare how students will receive their first exposure to the topic and to ensure it is of the appropriate difficult level. For the implementation part, it is emphasized that with the use of ChatGPT the students will have new tools that represent different opportunities to learn.

It is important to respect the two phases of the flip learning methodology because this proposal is to support the original methodology with the use of ChatGPT.

2.2.1. Preparation Phase

The preparation phase includes the following steps. Figure 3 marks those steps that are inherited from the traditional flipped learning strategy and those that are added to enhance the process with ChatGPT.

• Identify expected outcomes and topics. As in the traditional methodology, it is necessary to identify the course topics, the expected students’ outcomes for the course, and to know the characteristics of the students (i.e., study program). This will be useful to design a set of guiding questions for the students to use in the AI tool, and to define the learning level that students must develop in each topic.
• Create a list of guiding questions to use with ChatGPT. This list will be used by the students in the pre-class activities to introduce themselves to the topic. With this activity they are expected to understand concepts and comprehend their main application (lower order thinking skills). It is important to make sure to include questions about:
  • The concepts
  • Examples of how to apply these concepts

In Figure 4, there is an example of a pre-class activity shared with students in a basic programming course. In this description, a presentation of the expected outcomes and the list of guiding questions that students are expected to use when chatting with ChatGPT are included. As can be seen, there are questions about the concepts (e.g., What is a loop in a programming language?) and about the application of them (e.g., Ask ChatGPT to show and explain to you an example of how a while statement works.).

3.

Prepare a ChatGPT prompting guide to be shared with students in the first session. It is necessary to help students to know the basics about making prompts to use ChatGPT. The guide must emphasize that it is not just about typing prompts and accepting the answer, but to keep a conversation with ChatGPT in which they can ask for different ways to define the same concept or different ways to present an example. In this sense, ref. [45] suggests the following elements: to give some context to the tool, and to establish a role to be played by the tool. For example, in this case, communicate to ChatGPT to act as a student that has a beginner level in the topic. This role playing will be useful so that ChatGPT generates answers according to the learning state of the student.

This guide and the list of guiding questions for each topic will act as a starter pack for students. The guide is a useful resource for students to apply in transforming the guiding questions into prompts to be used with ChatGPT. The success of this part of the process will be in the relationship that the students make of the prompting guide and the guiding questions so to create suitable prompts to interact with the tool and explore the topics in depth. This, in turn, will support the achievement of learning objectives. It is also recommended to explain to the students that a lecture will not be displayed in class, so it will be relevant for them to take notes and to validate the correctness of the answers given by ChatGPT.

2.2.2. The Implementation Phase

The implementation phase consists of a list of steps to guide the students to have their first exposure with the concepts, to help the professor in the teaching process and to evaluate students’ learning gains. As can be seen in Figure 5, it consists of the following steps:

• Analyzing the guiding questions. Professors publish the list of guiding questions prepared for the topic. The students need to know exactly which topics they need to learn before the next session to interact with ChatGPT.
• Ask ChatGPT to explain the concepts and provide examples. Students interact with ChatGPT to study the concepts and to obtain examples about their application. Using the starter pack (the guiding questions for each topic and the prompting guide), students interact with ChatGPT to begin their learning path.
  In the traditional methodology, students come to class with questions of concepts they did not understand. In this proposed version supported by ChatGPT, students have the opportunity to ask those questions to the AI tool. This is an advantage because it is expected that this process will help to decrease the number of questions a student can bring to the class session. Another advantage is that if the student asks the right questions of the AI tool, they can come to the class even with better understanding of how to apply those concepts.
• Apply a test at the beginning of the class. As in the traditional methodology, professors apply a test to explore what students have learned in the pre-class activities.
• Engage in group discussion to solve student’s misunderstandings. Professors start an interaction with the group to clarify misunderstandings about the concepts.
• Provide activities to practice student’s new learning. Ask the students to work on tasks, exercises, and activities where they can put their newly found information into practice. In this step, students collaborate to help each other to solve the class activities with the professor’s help and feedback. When the professors identify misunderstandings on the part of several students, they should explain the concept to the group.
• Apply evaluation activities. Students’ performance assessment shall be of different types with the same complexity level as the class activities.

Figure 6 and Figure 7 summarize both strategies comparing them face to face and highlighting that the flipped learning strategy extended by ChatGPT enriches the educational experience, benefiting both professors and students. During the preparation phase, professors engage in comprehensive planning, which pays off as students come to class better prepared, armed with a deeper understanding and examples ready for discussion in the next session. This approach enhances the learning journey, making classes more interactive and productive.

Additionally, in the implementation phase, students are encouraged to take a proactive role in their learning, moving beyond the passive notetaking of traditional methodologies. They are now enabled to actively engage with the material through pre-class activities. This shift prompts them to ask insightful questions and engage in meaningful conversations with tools like ChatGPT, ensuring a thorough understanding of the topics at hand. The proposed strategy not only deepens students’ comprehension but also fosters a more dynamic and participatory classroom environment, benefiting everyone involved.

3. Hypothesis

The main objective of this research is to know if the use of ChatGPT, enhancing a flipped learning strategy, improves students’ learning gains when compared to a traditional flipped learning strategy that uses videos. According to this research question, the following hypothesis is stated for this work:

H1: 

A flipped learning strategy that uses ChatGPT is as effective as a traditional flipped learning strategy that uses videos when applied to basic programming courses for undergraduate programs.

4. Methodology

This study applies a between-subjects analysis in an experimental research design to results of 356 students in basic programming courses for undergraduate engineering programs. Students were organized into 14 groups taught by 9 professors. Randomly, groups were designed as focus and control groups. Focus groups received a flipped learning strategy scaffolded with ChatGPT while control groups received a flipped learning strategy based on videos. The exposition to both strategies lasted 10 weeks with two two-hour sessions per week. This is the whole duration of the course. Samples were: 8 focus groups that had 214 students and 6 control groups that had 128 students.

A pre-test–post-test process was used to analyze learning gains in both samples. A 20-question test, covering all the topics in the course, was originally created. Some questions had true/false answers, and other questions were multiple-choice with only one answer being correct. Conceptual and code analysis questions were included in the test, including topics like algorithms, control structures (sequential, decision, and iteration flow), nesting, variables and keywords, arithmetic operator precedence, functions, and lists. Informed consent was obtained from all subjects involved in the study. This consent was collected in the first class of the course after professors explained the research objectives to students and before they answer the pre-test.

An analytical strategy was applied in three stages: (1) instrument validation of the tool used as pre-test/post-test, (2) analysis of mean difference of pre-test results, (3) analysis of mean difference of normalized learning gain.

For the instrument validation, expert consultation and a statistical process were applied to validate this instrument. Results of all students that presented the pre-test were used in the statistical process.

A second step in this analysis, a mean difference analysis of the number of correct answers in the pre-test, was applied to both samples to know if students presented differences in their knowledge acquired before the start of the course. The results of all students that presented the pre-test were used in this analysis.

For the learning gain analysis, we used the normalized learning gain as in [46] using the following variables:

• Pre-test result for the student i: ${Pre}_i$
• Post-test result for the student i: ${Post}_i$

Students’ normalized gain for the student i (1). This is a measure of the actual gain that the students achieved during the course (Post_i − Pre_i) over the maximum gain that they could have obtained (# of items in test − Pre_i), as a percentage.

$$g_i=\frac{{Post}_i-{Pre}_i}{\# \mathrm{o}\mathrm{f} \mathrm{i}\mathrm{t}\mathrm{e}\mathrm{m}\mathrm{s} \mathrm{i}\mathrm{n} \mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}-{Pre}_i}\times 100,$$

(1)

Normalized gain mean for the group (2). N is the number of students.

$$〈g〉=\frac{1}{N}\sum _{i=1}^N\left(g_i\right),$$

(2)

Then, we evaluated if there was any initial difference between the two samples over their normalized learning gains by means of a statistical exploratory analysis. Afterwards, to analyze the validity of this work’s hypothesis, a two-sided t-test for independent samples was applied to the normalized learning gains of students in both clusters. Students who did not take the pre-test or who did not take the post-test were removed since the normalized learning gain could not be calculated. Additionally, two outliers (normalized learning gain ≤ −100%) from control group and three outliers (normalized learning gain ≤ −25%) from focus group were removed.

5. Results

This study applies analysis to the normalized learning gain of students in basic programming courses for undergraduate engineering students. A total of 356 students participated in a pre-test–post-test process applied to focus and control groups. This process was divided into three stages: (1) instrument validation of the tool used as pre-test/post-test, (2) analysis of mean difference of pre-test results, (3) analysis of mean difference of normalized learning gain.

5.1. Instrument Validation

A 20-question test, covering all the topics in the course, was originally created. Expert consultation and a statistical process were applied to validate this instrument. Five experts were consulted about the content, the scope, and the strength of the test. In this process, two new questions were added. Also, 5 redaction revisions were applied. Finally, 9 modifications or specifications in the codes were integrated into the test. As a second validation step, Cronbach’s alpha [47] and the composite reliability [48] coefficients were calculated for the new 22 question test to measure its internal consistency. 337 records from the post-test application were used. Results can be found in

Table 1. Cronbach’s alpha and composite reliability coefficient for test validation.

Cronbach’s Alpha			Composite Reliability
Value	Confidence Interval		Value
	Lwr.	Upr.
0.718	0.672	0.76	0.668

.

5.2. Difference Mean Analysis for Pre-Test Results

Randomly, groups were divided designed as focus and control groups. A total of 8 focus groups had 214 students, while 6 control groups had 128 students. A mean difference analysis of the number of correct answers was applied to these sets to know if students presented differences in their knowledge acquired before the start of the course. This test was applied in the first session of the course.

Table 2. Exploratory analysis according to the number of correct answers in the pre-test application for control and focus samples.

Value	Focus Group	Control Group
N	214	128
Min.	3	6
1st Qu.	9	10
Median	11	11
Mean	10.95	11.43
3rd Qu.	13	13
Max.	19	17
Standard deviation	3.15	2.78
Standard error of the mean	0.22	0.25

shows the summary of the characteristics of both samples along with statistical values.

For the control group (N = 128), mean value = 11.43; std. deviation = 2.78, and std. error of the mean = 0.25. For the focus group (N = 214), mean value = 10.95, std. deviation = 3.15, and std. error of the mean = 0.22. Figure 8 shows raincloud plots for both samples.

To check if these results are comparable, we applied the Shapiro–Wilk normality test [49] and Levene’s test for homogeneity of variance [50] to the data. The Shapiro–Wilk test showed that the distributions departed significantly from normality (focus group: W = 0.979, p-value = 0.00244; control group: W = 0.974, p-value = 0.0133), and Levene’s test indicated equal variances (F = 0.49, p = 0.484).

Given these results, we applied a non-parametric test (Wilcoxon–Mann–Whitney test [51]) over the difference of mean between both groups. Results (z = 1.09, p = 0.277) do not show evidence for a difference of mean over the number of correct answers in the pre-test application between focus and control groups. Given this, we cannot state that there is a difference between these groups, suggesting that both groups come from the same students’ previous knowledge.

5.3. Difference Mean Analysis for Normalized Learning Gain

To progress with the results analysis, a normalized learning gain (as stated by [46]). The post-test was the same test as the pre-test and was applied at the end of the course (last week) after finishing the course contents. Given that not all students answered both tests, for this analysis, 254 records were used.

Table 3. Exploratory analysis according to the normalized learning gain in a pre-test-post-test application for control and focus samples.

Value	Focus Group	Control Group
N	140	114
Min.	−12.50	−28.57
1st Qu.	38.22	21.63
Median	56.35	44.10
Mean	52.85	41.29
3rd Qu.	72.73	61.54
Max.	91.67	90.91
Standard deviation	24.27	26.84
Standard error of the mean	2.05	2.51

shows the summary of the characteristics of both samples along with statistical values.

For the control group (N = 114), mean value = 41.29; std. deviation = 26.84, and std. error of the mean = 2.51. For the focus group (N = 140), mean value = 52.85, std. deviation = 24.27, and std. error of the mean = 2.05. Figure 9 shows raincloud plots for both samples.

Three ranges to characterize normalized learning gains are described by [46]: low range for values below 30%; medium range for values between 30% and 70%; and high range for values above 70%.

Table 4. Classification of the number of students that obtained a low, medium, or high normalized learning gain.

Strategy	N	Low Gain	% Low Gain	Medium Gain	% Medium Gain	High Gain	% High Gain
Focus Group	140	24	17.14	79	56.43	37	26.43
Control Group	114	40	35.09	58	50.88	16	14.04

shows this categorization for both samples.

Furthermore, ref. [52] outlined that interactive engagement courses using hands-on activities that provide immediate feedback, such as flipped learning, generate a normalized learning gain in the medium range. As we can see, both samples present a medium normalized learning gain value. This supports the idea that these samples received an interactive engagement learning methodology.

We applied the Shapiro–Wilk normality test and Levene’s test for homogeneity of variance over these groups. The Shapiro–Wilk test showed that the distributions departed significantly from normality (focus group: W = 0.957, p-value = 0.000244; control group: W = 0.974, p-value = 0.0249), and Levene’s test indicated equal variances (F = 1.90, p = 0.169).

We applied a Wilcoxon–Mann–Whitney test over the difference of mean between both groups. As it can be seen from the results (z = −3.42, p = 0.000584; lwr. = −17.9, upr. = −4.71) of the Wilcoxon–Mann–Whitney test, students that used ChatGPT along with the flipped learning strategy have a significant greater mean on normalized learning gain over students that used videos along with the flipped learning strategy with a 99.9% confidence level. The confidence interval is lwr. = 17.9 and upr. = 4.71, p = 0.000584.

6. Discussion

Previous work has shown the effectiveness of flipped learning (FL) versus the traditional classroom. In this work, we compared the effectiveness of the combination of FL and ChatGPT as a source of pre-class students’ preparation. The comparison was made based on focus groups using both techniques (FL + ChatGPT) and control groups where FL alone was applied. The combined use of FL + ChatGPT showed a higher normalized learning gain on students from experimental groups over the students of control groups which used FL without ChatGPT support.

The use of ChatGPT by students varies from improvement in writing academic assignments and problem solving and as a virtual assistant to support them in understanding problems [37,53]. Among these different uses, the one that stands out as having the highest accuracy and the best impact on learning is its application as a virtual assistant. ChatGPT serves as a tutor for domain-specific problems and supports students in pin-pointing the essential knowledge underlying each problem and enhancing their understanding of conceptual knowledge through problem solving. This tutoring capability is particularly important as students struggle to decide on relevant concepts and formulas by analyzing the problem’s statement [54].

According to [39], the use of ChatGPT as a student assistant serves better in introductory courses than in advanced courses where the concepts are complex and the relationships between them are more complicated. This supports the positive results obtained because our application was carried out in an introductory course on programming using the Python language for first-semester undergraduate students.

On the other hand, ref. [55] states that the quality in the introduction of prompts is proportional to the usefulness of the response obtained. In our study, students were given a curated set of prompts associated with each of the topics covered through FL. In this way, we managed the level of quality of the content reviewed by the students. We were also able to observe that students generated or improved such prompts for the refinement of the study topics.

Prompt engineering is an area of great importance, since it allows for the proper use of tools like ChatGPT and limits the possible failures it may have. Future work on this topic is needed to explore strategies and perspectives in depth. A work that can be useful as a start for this is that of [56].

Adding to this, the traditional flipped learning strategy provides a specific space where professors and students interact to verify and validate the knowledge acquired in the students’ self-directed activities. These steps are preserved as an advantage in our methodology (elements 3 and 4 in the during the class stage of the implementation phase, Figure 5) and strengthened because they serve to verify the accuracy of the concepts and knowledge provided by ChatGPT. This is a very important task to be implemented by professors because it is well known that ChatGPT is fallible.

There is empirical evidence between the positive relationship between the use of generative artificial intelligence tools, such as ChatGPT, and the level of academic achievement by students during a learning course [57]. Our study reinforces these outcomes by finding that the combined use of FL + ChatGPT allowed students in the experimental groups to obtain a higher normalized learning. These results are in line with Hai-Feng [41], who found that the use of FL and ChatGPT promotes students’ performance.

However, ChatGPT as a learning tool is new, and there are few research studies that analyze its effectiveness in depth. Nevertheless, this is an important aspect to explore to know if the results found in this work have a positive greater impact in students’ learning.

Table 5. Summary of works that apply learning strategies supported by technology (like AI tools) compared to the results of this work.

Reference	Strategy	Number of Students	Application Context	Discipline	Normalized Learning Gain
This work	Flipped learning supported by ChatGPT	140	Undergraduate–Engineering	Basic programming	52.85
[58]	Use of ChatGPT to clarify questions	36	Undergraduate–Engineering	Physics	−34.67
This work	Flipped learning	114	Undergraduate–Engineering	Basic programming	41.29
[59]	Flipped learning	40	Bachelor’s degree in early childhood education	Quantitative Research	37
[60]	Flipped learning	70	Secondary school	Basic physics	46
[61]	Flipped learning applied to creative problem solving	23	Physics education undergraduate study program	Physics	60.5
[22]	Flipped learning	68	Undergraduate–Engineering	Programming	50.02
[22]	Gamification	97	Undergraduate–Engineering	Programming	50.92
[22]	Flipped learning mixed with gamification	206	Undergraduate–Engineering	Programming	40.34
[62]	Intelligent tutoring system	58	Computer science and computer engineering majors	Recursion	−13.5
[63]	Conversational-like intelligent tutoring system	16	Software developers	Linear regression	33.99
[64]	Use of visuo-Haptic Simulations	111	Undergraduate–Engineering	Electric forces	47.2
[65]	Use of plasticine wax media	44	Automotive engineering	Right-aligned lathe chisel geometry material	57.34
[66]	Use of 3D surface visualization tools, virtual environments in an online project-oriented learning approach	210	Undergraduate–Engineering	3D surfaces	46
[67]	Use of a computer-based simulation learning tool	123	Secondary level	Free fall motion	42

summarizes works that have measured interventions to the teaching–learning process by means of normalized learning gain. These references use a flipped learning strategy, technology to scaffold learning, or other resources.

For example, ref. [58] found that the use of ChatGPT as a tool to clarify concepts gives a negative normalized learning gain. Moreover, this value represents a negative medium learning gain as classified by [52]. This finding highly contrasts with the results in this research even if both strategies are similar. It could be possible that this difference comes from the application of the structured methodology (e.g., the use of the starter pack) presented in this work to integrate ChatGPT to flipped learning that focused students on the use of the tools.

On the other hand, it is important to compare results to other flipped learning experiences. Refs. [22,59,60,61] present results of using traditional flipped learning in different contexts. On average, these works have a mean of normalized learning gain of 48.38 that is lower than the 52.85 value found in this research. This agrees with our results, in which it was found that the control group presented lower normalized learning gain.

A work worth highlighting is that of [61]. Authors combined flipped learning with a creative problem solving strategy finding a normalized learning gain of 60.5, greater than this work’s value. This difference is possible because of the combination of these two pedagogical elements. However, ref. [22] present results of the combination of flipped learning and gamification pointing to statistically significant lower learning gains that can be compared on magnitude to those of a traditional teaching–learning process. Further research about the combination of flipped learning with other pedagogical methodologies is needed.

As stated before, ChatGPT can be conceived as an intelligent tutoring system (ITS) for its customization and tracking features that can be obtained by interacting with the tool. In this sense, Refs. [62,63] present results of the use of ITS having a normalized learning gain of −13.5 and 33.99, respectively. These values are lower than the value found in this work. Potentially, this difference is due to the interaction features that ChatGPT has and that go beyond the conversational interface and that can be a motivational element. It is worth highlighting that [63] uses a conversational-like ITS, similar to the way in which someone can interact with ChatGPT, with a greater learning gain between both implementations.

Finally, refs. [64,65,66,67] present research on the integration of other computer-based tools into learning activities. All their results are below the learning gains obtained in this work. This shows that the integration of ChatGPT into flipped learning produces better results than other technological interventions. The exception is the work in [65], which has slightly greater effectiveness. However, even if this implementation has some computer-based elements, the strategy is mostly based on elements to support learning with physical tools.

An important question that arises is why a professor would prefer our methodology integrating FL + ChatGPT over established flipped learning strategies even if its effectiveness has been proven. In addition to the already established advantages of flipped learning, our methodology has an important advantage in that the introduction of ChatGPT (instead of using videos) reduces the workload for professors during the material preparation stage for the course. This has been an obstacle for the adoption of flipped learning, because creating videos, or finding and curating them, are highly time-consuming tasks. Using ChatGPT for this purpose eliminates this burden and makes it easy to flip a complete course. Furthermore, our methodology provides students with self-directed and professor–student contact workspaces that give assistance to the process of validation of the knowledge acquisition. This is relevant since the AI tool can provide erroneous or inaccurate information. However, on the other hand, it is appropriate to say that this methodology was tested in the context of an initial course for undergraduate engineering students where the learning process includes basic cognitive elements. Future work on its application in more advanced contexts is necessary to understand its impact.

The integration of FL + ChatGPT framework introduces a novel dimension to self-directed learning, enabling students to pursue inquiries beyond the standard curriculum. This capacity for personalized exploration, adopted in our methodological approach, enhances the FL model by allowing learners to tailor their educational journey to their individual interests and pace, and not limiting it to the materials generated or suggested by the professor. Flipped learning supported by ChatGPT methodology promotes an exploratory behavior through the analysis of their interactions with ChatGPT, demonstrating a proactive approach to learning that extends beyond passive consumption of information.

7. Conclusions

The search for pedagogic strategies to scaffold the teaching–learning process has been accelerated by the advance of technology. In recent years, AI developments have arrived to help in this quest. The integration of ChatGPT into the flipped learning strategy has proven to be a significant pedagogical innovation in undergraduate engineering education.

From this perspective, we presented a statistical analysis on learning gains comparing groups of students that received classes under a flipped learning strategy supported by videos (traditional flipped learning) to groups of students that received classes under a flipped learning strategy supported by ChatGPT (AI-enhanced flipped learning). A total of 356 undergraduate students in a basic programming course, part of engineering programs at Tecnologico de Monterrey, participated in this study. A pre-test–post-test process using focus and control groups was used to obtain a normalized learning gain measurement. It was found that AI-enhanced flipped learning showed a greater normalized learning gain compared to the traditional flipped learning strategy, supporting the hypothesis of this work.

Additionally, a methodology to insert ChatGPT into flipped learning was presented as a tool to help professors adopt the strategy in their courses.

This comparative study has shed light on the potential of generative artificial intelligence tools to enhance the effectiveness of existing pedagogical strategies. The results show that AI tools can serve as powerful facilitators of learning, providing an interactive and personalized experience for students.

Comparison of our intervention with existing literature suggests that the personalization and interaction offered by tools such as ChatGPT can address some of the limitations of traditional flipped learning strategies, such as the lack of motivation to review material before class and the difficulty in adapting learning resources to the individual needs of students. In this context, ChatGPT acts not only as an additional study resource, but as a facilitator that guides students through the material effectively, stimulating their interest and active participation in the learning process.

The implications of this study extend beyond improving grades or learning gains. They open the door to a reconsideration of how educational content can be designed and delivered in the digital age, especially in technical and complex fields such as engineering. ChatGPT’s ability to provide detailed explanations, generate relevant examples in real time, and respond to specific student queries presents an invaluable opportunity to enrich active learning and deep conceptual understanding.

Going forward, it would be prudent to further explore how different AI modalities, including but not limited to ChatGPT, can be integrated into various pedagogical strategies and academic disciplines. Future research could focus on optimizing the interaction between students and AI tools, further personalizing the learning experience, and evaluating the long-term impact of these interventions on students’ skill development and career readiness.

To continue analyzing the effects of the ChatGPT enhanced flipped learning strategy over the scaffolding of students’ positive behaviors, future research on motivation factors must be implemented. Measuring shifts in internal and extrinsic motivation would be useful to measure how this strategy helps students to internalize the course contents.

On the other hand, it would be beneficial to investigate the potential challenges and limitations of implementing these technologies on a large scale, including ethical considerations and the risk of over-reliance on AI tools for learning.

From the standpoint of cognitive biases, employing a flipped learning strategy augmented by ChatGPT as a problem solving tool might unintentionally lead to an overestimation of one’s own abilities, an effect encapsulated by the Dunning-Kruger phenomenon. This bias highlights a paradox where individuals with limited experience in a specific domain may overestimate their proficiency, mistakenly believing themselves to be more adept than they truly are. This presents a critical need to embed a variety of assessment techniques and activities (that might or not involve the use of ChatGPT) into the learning process.

Additional to this, an important aspect about the ethical use of the AI tool arises. This overestimation of skills and the apparent easiness of using ChatGPT can project an artificial environment in which students could believe they do not need any type of personal work to develop their competencies. In turn, this can lead to students committing breaches of academic integrity. Or trust the tool’s answers, without carrying out any verification of their veracity. Further research is needed to know the ethical impact of using ChatGPT as a supporting tool for learning.

Also, it is important to state that the span of this research was conducted within the fields of engineering and programming, indicating a need for further research across different disciplines and scopes to transcend the limitation of generalization. Links to this further research must be created to know if the methodology can be applied in other levels other than undergraduate programs. This is important because ChatGPT requires specific cognitive skills to interact with it. It is possible that not all students at all levels can follow a conversation with the tool or can carry out analysis and synthesis work to extract knowledge from the tool’s responses.

These limitations highlight how new learning strategies need to be adjusted, analyzed, and monitored so that they can mature, leading to an improvement in learning outcomes.

Finally, a limitation of the methodology presented in this work comes from the use of the AI instrument. This tool requires access to a computer and to internet. For certain sectors of society, the application of the methodology may not be possible due to not having access to these tools.

Given the impact that artificial intelligence has had across various environments and industries, it is plausible to assume, with caution and through serious experimentation, that its use is becoming a cross-cutting strategy for different aspects of everyday life. This study contributes significantly to the understanding of how AI tools can enhance traditional pedagogical strategies, offering promising prospects for educational innovation in the digital age. The adoption of emerging technologies in education not only can improve learning outcomes but also transform the educational experience, making it more engaging, accessible, and tailored to 21st century learners’ needs.