The Impact of a Combination of Flipped Classroom and Project-Based Learning on the Learning Motivation of University Students

Abstract

Our study investigated the effects of combining project-based learning (PBL) and flipped classroom (FC) methods in university education on the learning motivation and learning outcomes of students at the bachelor (BSc), master (MSc), and higher vocational education and training (HVET) levels. We aimed to explore how these modern teaching methods can influence students’ performance and motivation at different levels of education. The research used cross-sectional analysis and quantitative methods to evaluate the impact of FC and combined PBL and FC methods. This study followed groups of students for two academic years, comparing their results with control groups that did not benefit from the combined teaching method but were studied in a same FC environment. The results showed that students at the BSc and HVET levels significantly improved their learning motivation and achievement by combining PBL and FC methods (BSc: p = 0.0001248, HVET: p = 0.0485), while at the MSc level, this effect was not significant (p = 0.1000). These data support that an effective combination of PBL and FC methods can improve learning motivation and outcomes at certain levels of education, but further research is needed better to understand the effects for students at the MSc level.

1. Introduction

1.1. Matching Labour Market Needs and Education Innovations

1.1.1. Quality of Education

Discourse on the current state of the education system highlights that students often graduate without the labour market competencies that are essential for a successful career. According to Sharma and Shree [1], university students educated through traditional educational methods often need more skills to enter the labour market successfully.

This problem can be addressed by improving the quality of education in the issuing educational institutions. However, the quality of education is difficult to define because it is a complex, culturally differentiated entity at almost all levels of the education system, from early childhood education to the postgraduate level [2]. In our own definition, we define the quality of education as the added value of an individual’s personal and career success. Consequently, defining quality is a complex task that requires predicting future success in the present. This forecast must be based on social and economic research and projections describing the structures of future labour markets.

1.1.2. Changing Needs of the Labour Market

Flores et al. [3] emphasize the importance of preparing universities for future industrial challenges, focusing on human capital and the necessary competencies in the era of Industry 4.0. This approach can be understood as proactively preparing students for the ever-changing labour market needs and technological challenges.

The design of an education strategy must consider the changes by Industry 4.0, including new interactions in the workforce structure, concepts for future human capital, and the typology of competencies required by Industry 4.0. These considerations can help educational institutions provide relevant and timely training that prepares students for future challenges.

This is also the case in the work of Bratianu et al. [4] who call for a paradigm shift in business education to meet the changing needs of the labour market. According to them, education needs to shift from a knowledge-transfer- to a competency-based approach, which involves integrating knowledge, skills, and attitudes. According to them, the new approach should focus on active learning and developing applicable skills that increase students’ employability. The research results show that students are aware of the need for a paradigm shift, are willing to invest work in their learning to meet the needs of the new environment, and are engaged and motivated in this respect.

Considering the paradigm shift in Industry 4.0 or business education, we find that the proposed competency-based education directions are desirable and fit for purpose. The motivation of students’ interest in this direction needs to be met, which requires redesigning the educational environment of universities.

The research results of Kipper et al. [5] highlighted the need to develop a wide range of university education competencies to adapt to the challenges of Industry 4.0. These competencies include leadership skills, strategic vision, self-organization, the ability to give and receive feedback, proactivity, creativity, problem solving, interdisciplinary thinking, teamwork, collaboration, initiative, communication skills, innovation, adaptability, flexibility, and self-management. In addition, Industry 4.0 professionals should have a background in information and communication technology, algorithms, automation, software development and security, data analysis, general systems theory, and sustainable development theory. Universities are encouraged to integrate these competencies into their curricula, emphasizing interdisciplinary learning and the development of problem-solving skills. Developing the competencies demanded by Industry 4.0 is not only about expanding technological knowledge but also involves fostering a culture of critical thinking, innovation, and collaboration.

1.1.3. Transition to Education 4.0

These expectations are a “mandate” for universities to create an economically relevant, inspiring, and adaptive learning environment. This is the expectation of the future, which students recognise and expect when judging the quality of university education. This process can be called the transition to Education 4.0.

Williams et al. [6] have already pointed out that the quality of education is determined by the content of the curriculum and the way it is delivered, as well as by teacher–student relationships, curriculum design, and the learning environment. In examining the relationship between motivation to learn and quality education, they emphasise the importance of the diversity of inspiring teaching and learning environments.

Education 4.0 systematises these methods and provides a framework to measure their effectiveness in developing competences. As practising educators, educational designers and researchers, we are adapting and combining these methods in different experiments and testing how they meet the challenges of today’s world. Education 4.0 aims to deliver more effective, accessible, and flexible educational programs. We are developing the use of technologies and the best pedagogical principles, strategies, styles, and practices to achieve these goals. New teaching and learning methods will take different forms to optimise knowledge generation and information transfer.

Following Miranda et al. [7], the learning environments often used in Education 4.0 include face-to-face learning, which is primarily based on active learning, and online distance learning, which uses current technology platforms to implement remote processes in a digital, virtualised, and connected way, with synchronous and asynchronous activities as well as hybrid learning, which uses techniques such as blended learning (BL) or flipped classroom (FC), optimising learning processes and resources.

Muca et al. [8], in their work on Education 4.0, investigate the effectiveness of FC and peer-assisted learning (PAL) methods in veterinary education, particularly in the teaching of equine nutrition. This research shows that such methods can promote positive student learning and increase participation even in specialised subjects. The findings highlight the broader applicability of these approaches to address the challenges posed by Education 4.0.

1.2. Design of Experiment

1.2.1. Flipped Classroom as a Learning Environment

Our present experiment was carried out in a university where the innovative teaching approach of the FC method is implemented at all levels of all courses. By inverting the traditional classroom dynamics of the theory, we have achieved better learning outcomes, deeper student understanding, and an increase in the effectiveness of the learning process. We apply the organisational elements of prior home preparation, interactive class activities (group work, problem-solving exercises, discussion formats, etc.). The teacher, as a mentor (facilitator role), allows for personalised learning (pace and style) and encourages students to learn actively. It also promotes group work and peer-to-peer learning, i.e., collaborative learning. We integrate online technologies with the Teams and Moodle platforms to deliver pre-learning materials in digital format (videos, online readings, and interactive exercises). In this learning environment, feedback and assessments are built into the process, helping students to correct and improve themselves immediately.

In this dynamic and interactive learning environment, we have already achieved better results in terms of student performance compared to the previous, typically frontal environment. In measuring motivation to learn, we have considered student feedback, collected electronically and in written form, during courses and at the end of semesters. Based on students’ feedback, an experiment in teaching methodology was launched in this educational environment.

1.2.2. Problem-Based Learning as an Innovative Learning Method

The systematic work of Miranda et al. [7] highlights new learning methods based on the active participation of students and the use of new technologies to improve teaching-learning processes. They identify Challenge-Based Learning, Problem-Based Learning (PBL), Learning-by-Doing, and Gamification-Based Learning as new innovative learning methods. Of these, we used the PBL approach in the design of our experiment. Our experiment, therefore, was a student-centred learning approach in an FC environment, focusing on the innovation and effectiveness of the winemaking enterprises in our region using PBL methodology. A real practical problem is identifying and analysing the need for innovation and its enterprise constraints. Identifying the problems that hinder innovation was the starting point for the learning process.

1.2.3. Learning Objective

The learning objective was to develop critical thinking and problem-solving skills by encouraging independent learning. Through collaborative learning, students worked in groups to solve problems, promoting communication skills, cooperation, and teamwork. The groups were formed vertically at the MSc–BSc–HVET level with students from the economic and agricultural disciplines. This supported the integration of knowledge, as solving real business situations required knowledge and methods from several disciplines. Students were actively involved in their learning process, conducting research, collecting and evaluating information, and developing solution strategies. A strong emphasis was placed on reflection, which allowed students to evaluate their learning process and choices and the dynamics of group work. Students took on different roles in the problem-solving process, which helped them understand different perspectives and develop collaboration. The FC environment also provided the role of teacher mentor and continuous assessment for the pilot group.

The method described above is also a hallmark of quality education because it increases students’ motivation and active participation by confronting them with real challenges and problems. It encourages students to apply knowledge and find creative solutions.

Motivated students tend to do better in their studies; motivation, therefore, plays a crucial role in improving student performance. Overall, a strong correlation exists between students’ employment status and their prior motivation to learn [9].

1.2.4. The Quality of Education and Students’ Motivation

The quality of education, therefore, is a crucial determinant of the success of fresh graduates’ entry into the labour market and in their early years. Moreover, the quality of education can be measured in terms of student engagement and motivation. It logically follows from these two statements that an increase in students’ motivation also affects the success of their entry into the labour market, i.e., their employability.

In summary, our educational experiment focused on increasing student motivation. We investigated which methodological innovations can be used to increase student motivation further within our university’s flipped classroom (FC) educational environment, which is based on Education 4.0 expectations.

Our hypotheses along these lines are as follows:

Hypothesis 1.

The PBL methodology will further enhance student motivation and performance in an FC environment.

Hypothesis 2.

Applying the PBL methodology in an FC setting will lead to a different increase in learning outcomes and motivation in all disciplines and learning levels.

1.2.5. The Logical Structure of the Research

The logical structure presented in Figure 1 was derived from the labour market critiques of higher education as a research observation and the aspects that underpin the implemented experiment in higher education pedagogy. The theoretical evolution goes through the competence requirements of Industry 4.0 and the new demands of economic life, which can be used to draw up a competency map of the future labour market. We identify that students also understand and know the expectations of the economy and are therefore motivated to actively participate in educational scenarios that they identify as having future benefits. On this plane, we select the theoretical thread based on the relationship between motivation and the quality of education. The response of education to this “order” is sought after in the paradigm of Education 4.0. This is why adapting educational environments and methods is the focus of Figure 1. The educational environment chosen in our experiment is the flipped classroom (FC) because all courses at the University of Tokaj, where the experiment is located, are taught in this environment. Therefore, FC is also called traditional education in our experiment. This experiment aims to investigate whether, based on this traditional training in combination with PBL, we can induce any effects on student motivation and performance.

The second string of ideas on the right in Figure 1 focuses on student motivation as a critical factor in learning processes and labour market success. This strand justifies motivation as a measure of quality education, better outcomes for motivated students, and better labour market entry opportunities, drawing on the literature review presented.

Thus, this illustration not only presents the current challenges of higher education and labour market expectations but also offers an integrated framework for understanding the future directions of education, where the development of educational environments and methods is focused on student motivation. Figure 1 reveals the logical links between the issues described in the introduction, providing a comprehensive picture of a multidisciplinary approach to the educational paradigm shift.

2. Materials and Methods

2.1. Methodology for the Selection of Participants

The experiment was carried out within the framework of the University of Tokaj Master of Science in Business Development (MSc), Bachelor of Science in Viticulture (BSc), and Higher Vocational Education and Training in Viticulture and Enology (HVET). HVET stands for “Higher Vocational Education and Training”. It refers to post-secondary education that focuses more on practical skills and vocational training, often leading to vocational qualifications that are less academic and more job-specific than a bachelor’s or master’s degree. At the university where the experiment was conducted, FC as an educational environment is applied in all disciplines at all levels of training in the entire vertical range. In our case, FC can, therefore, be considered a “traditional educational” environment. The experiment was repeated in two academic years, with a group of students (the control group) learning in a traditional (FC only) environment at each level of education and a group of students (the experimental group) learning in a project group in both academic years who were taught using the PBL method combined with an FC environment. The data from the two academic years were analysed and treated separately by the level of education and as a whole. At the three training levels, the students acquired knowledge related to business innovation with the same content but with a different educational objective. Experiments were conducted in six groups over two academic years (2021/22; 2022/23). Figure 2 shows the correlations between experimental and control groups and training content.

When repeating the experiment, the sizes of the control and experimental groups do not necessarily have to be the same. The choice of groups has no role in the experiment; simply two or two groups of students per level are required, so one should be designed as FC only (control group) and the other as a vertically and interdisciplinarily organized experimental group where the PBL method is applied in an FC set. The sample sizes are shown in

Table 1. Determination of sample sizes.

Names of Different Groups	Group Features	Number of Students in the Two Academic Years (n)
MSC	control group	10
BSC	control group	7
HVET	control group	17
MSC	project group	10
BSC	project group	8
HVET	project group	17
MSC	all	20
BSC	all	15
HVET	all	34
project group	all	35
control group	all	34
total sample size	all	69

, where it can be seen that there is a difference in the number of BSc students in the experimental and control groups, which does not affect the analyses.

2.2. Methodology for Data Collection and Analysis

On the one hand, the data collection is based on students’ subject performance (grades) as defined by national legislation (on a scale from 1 to 5), obtained from the electronic registration system used at the university.

On the other hand, we assessed their motivation to learn by analysing their self-reflections and various documents they had produced (on a scale of 1 to 5). The teachers carried out the research and evaluation. The data were not evaluated by the project group, as there were no project groups in the control group, but by the level of training.

Students’ motivation to learn was measured using a four-pillar method to obtain discrete data, also rated on a scale of 1 to 5 (1 = worst, 5 = best).

Based on data from students taught in a traditional FC-only environment as a control group and the experimental group supplemented with PBL, statistical methods were used to investigate the effectiveness of the learning process and changes in motivation to learn.

T-tests were used at all three educational levels to compare differences in learning outcomes and motivation to learn.

However, due to the small number of elements, it was essential to carry out an effect size test to determine the extent of the difference between the two methods.

We then conducted a text analysis and answered the hypotheses.

Measuring the students’ motivation was the most challenging part of the experiment. We developed a four-pillar method that integrates features of the Intrinsic Motivation Inventory (IMI), the Ryff Psychological Well-Being Scale (RPWS) [10], and the Short Flow State Scale (SFSS).

Sequencing was also important because the four pillars can be interpreted as an evaluation process, as follows:

• Self-reflection: Self-reflection allows learners to assess their intrinsic motivation and well-being. This method combined features of the IMI and RPWS measurement tools to help students reflect on their motivation and feelings, thus encouraging them to learn. Our experience has shown that self-analysis has been beneficial; it helps students to learn. The method involves students completing a self-analysis task to analyse their soft and communication skills. Through this analysis, students understand their strengths and weaknesses [11]. The method’s difficulty is that the self-analysis task has to be completed separately for each learning objective.
• A 360-degree peer assessment: Student interactions and group dynamics were assessed using RPWS and SFSS analysis methods. This feedback helped us obtain a comprehensive picture of students’ interactions and motivation.
  • The methods used include the following:
  • Group analysis: the observation and evaluation of learners’ ability to cooperate in group tasks, including the level of learners’ commitment, contribution, and cooperation in achieving common goals, conducted by MSc students as group leaders;
  • Peer evaluation allows learners to evaluate their peers’ contribution to the group work objectively and constructively. MSc students carried out the peer evaluation because they had an overview of the whole process and the actors involved;
  • Self and peer assessment: it allows learners to reflect on their own and their peers’ progress in the affective domain [12].
• Teaching experiences: Student performance and behaviour were monitored in line with IMI and RPWS methods. This allowed for the identification of students’ motivational patterns. In line with the IMI and RPWS methods, we attempted to track students’ performance and behaviour through a two-phase structured process [13]. In the first phase, students were categorised according to predefined criteria (educational attainment, age, and previous learning performance). In the second phase, students participated in an additional assessment process, during which the most recent performance data were collected and analysed. Based on the results of these analyses, students were given a final ranking (i.e., a score from 1 to 5) reflecting their motivation to learn and their behavioural patterns. We followed students’ progress in real-time (every two weeks), comparing their initial scores with their subsequent performance. This allowed us to evaluate the effectiveness of our method and the changes in each student’s learning motivation patterns.
• Workflow observations and participation monitoring: elements of the SFSS methodology were used to identify the “flow” states and active participation. This helped to track changes in students’ motivation levels and engagement over time. Knowing the nine components of Csikszentmihalyi et al.’s [14] “flow” states, we adapted them here in our research and assessed the “flow” states based on only five components: attentional focus, the loss of self-awareness, the sense of control or competence, inner pleasure, and the perception of clear goals. Thissen et al. [15] investigated the “flow” states in the context of digital games. The study results showed that the “flow” state significantly positively affects participants’ sense of comfort and performance, supporting the psychological theory of “flow.” This means that the “flow” state plays an essential role in entertainment, education, and learning, where deep immersion and positive experiences contribute to increased motivation and performance. These components helped us to observe and assess the state of “flow” in students’ submitted work and the accompanying comments and cover letters in their online presentations. The quality of the students’ comments and the quality of their work can be used to identify which of these “flow” components were experienced during the learning process.

Due to the limitations of using a simple self-assessment questionnaire, we decided to use this more complex multidimensional approach. Although the self-assessment questionnaire offers a quick and simple solution, it was justified to combine it with the above methods due to respondent bias.

The results were obtained by quantifying student ratings generated during the learning process on a discrete scale of 1–5. Of course, for the control group, we used tools 1, 3 and 4, since tool 2, which was the 360-degree assessment, only made sense to use in the case of group-based task solving. In their case, only the observations and notes made by the teacher as a mentor were available.

As you can see, we have carried out a qualitative analysis that goes beyond numbers. This helped us understand the impact of the two methods on the teaching process, motivation, and learning experience.

2.3. Ensuring the Authenticity and Validity of Our Data

A complex data-cleaning protocol was applied during the motivational data analysis to ensure validity and reliability. This protocol consisted of several steps based on best practices in scientific research methodology. The main steps were data filtering, the identification of outliers, the analysis of outliers [16], data deletion or separate analysis, and finally, data validation. This involved checking the consistency and completeness of the data and assessing their consistency with the research objectives. To ensure the validity of our data, we applied a complex data cleaning protocol during the motivational data analysis process and paid particular attention to checking inter-rater consistency, i.e., inter-rater reliability. In order to ensure the consistency of the evaluations, we measured the standard deviation of the evaluators using Cronbach’s alpha coefficient. All student written materials, including essays, weekly reports, and peer reviews, were analysed by four raters (teachers and researchers) who scored the materials on a scale of 1 to 5 on the abovementioned four dimensions.

Cronbach’s alpha coefficients were used to test consistency and reliability between raters. Based on the ratings the raters gave, we first calculated the correlation between all possible pairs of raters. The average of all correlations was then taken (i.e., the average correlation between raters), which provided a measure of the consistency of the raters.

The resulting average correlation was used to calculate the reliability of the assessment method using Cronbach’s alpha coefficient. The Cronbach’s alpha coefficient was calculated using Formula (1):

$$\alpha =\frac{N·\overline{r}}{1+\left(N-1\right)·\overline{r}}$$

(1)

where N is the number of raters (4 in our case), and r is the average correlation between raters. The obtained Cronbach’s alpha value, which was above 0.75, indicated that the evaluators applied the evaluation criteria consistently, thus ensuring a high level of reliability of the evaluation method. This meant that the assessment method used reliably reflected the motivation and performance of students based on their written materials. The Cronbach’s alpha score above 0.7 obtained in this process confirmed that our assessment method was reliable and that the assessors applied the criteria consistently. This success was achieved by having the evaluation team (4 people) jointly involved in the experiment’s design and giving regular evaluation presentations to each other, discussing their decisions and criteria.

3. Results

3.1. Group Training

The experiment was conducted among students of the so-called correspondence course at the University of Tokaj. A characteristic feature of the correspondence course is that each student studied while working. It is typically an intensive course, taking place every Friday and Saturday.

3.1.1. Experimental Project Group Characteristics

During the experiments, project teams led by MSc students were formed and worked with BSc and HVET students to develop innovative project plans based on their own data collection and research to improve the innovation capacity of viticulture and wine enterprises. The project teams, already presented in Figure 2, consisted of five to six people: one MSc student, one to two technical coordinators who were BSc or HVET students, and three to four HVET students who collected data. The training was, therefore, project-based in a “traditional” FC educational setting. Before the contact hours and throughout the semester, online training was continuously conducted using Teams and Moodle. Students had to develop data collection and analysis strategies to complete the projects successfully. This required visiting businesses and investigating their capacity to innovate. Although the MSc students had an adequate theoretical background, they needed to gain knowledge of viticulture–wine technology or markets. This gap was filled by the BSc and HVET students, who contributed to the knowledge of each project group. Based on the data collection and analysis, the groups prepared essays on the innovations observed, compared them with the theory they had learned, and also conducted comparative analyses between the enterprises visited. Each group presented their work to each other through project presentations. The other groups evaluated the work of their peer groups from a methodological and content perspective. Ongoing evaluations of the work and progress of individual members were also carried out within the groups. These evaluation documents were written in a predefined structure, where consistency rather than structure was important to simplify and clarify the implementation of the meta-analysis.

3.1.2. Control Group Characteristics

During the two academic years, three control groups were subjected only to the FC environment without project-based problem solving (PBL). These were traditional-year groups operating separately within courses, such as Enterprise Development MSc, Viticulturist-Winemaker BSc, and HVET students. These classes did not know or interact with each other. Their training was conducted in the same online Teams and Moodle environment, with accessible course material and the same theoretical lessons as the PBL pilot groups. The FC environment required considerable independent online preparation by the students to ensure effective contact hours. To complete the semester successfully, students were required to present at least one business innovation capability and activity in the form of a case study, with content and format appropriate to the level of training. The requirements are listed on the right-hand side of Figure 2. The exams were traditionally conducted using online tests and examination boards. At the end of the course, students reported on their progress and engagement according to a university student review system, which gave us insight into their motivation and suggestions for the teaching method and content.

In summary, both groups received the same theoretical training in the FC environment. The students reported on what they had learned in the form of case studies and demonstrated their knowledge through online and face-to-face exams, joint project presentations, and peer review papers.

3.1.3. Description of Data Sources and Data Analysis

The sample sizes for each group have already been presented in Table 1. The statistical analysis process is illustrated in Figure 3. After cleaning and sorting the data, the calculations were performed in three stages, followed by a meta-analysis to verify and interpret the results.

• Data cleaning and validation: The data are cleaned and validated after data collection to ensure accuracy and reliability. This step included the removal of anomalies, duplicates, and missing values.
• Data organisation in Excel: The data were imported into Excel and organised into different groups for efficient analysis (e.g., project-based and traditional education students). The first and second phases can be followed in the left column of Figure 3.
• Calculating basic statistics in Excel: Group averages, standard deviations, and sample sizes were calculated in Excel. There was no reason to use any other system for the task, so ease of access and use was the key. This may help the replicability of the experiment for other teaching and research groups.
• Conducting a t-test using online software: The data were uploaded to an online t-test calculator (https://www.omnicalculator.com/statistics/t-test accessed on 3 January 2024), where the software algorithm calculated the t-value and p-value. The calculation was based on the two groups’ mean, standard deviation, and sample size. Formula (2), the t-test formula, as follows, was used:

$$t=\frac{{\overline{X}}_1-{\overline{X}}_2}{\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}}$$

(2)

where ${\overline{X}}_1$ and ${\overline{X}}_2$ are the means of the two samples, $s_1^2$ and $s_2^2$ are the standard deviations of the two samples, and $n_1$ and $n_2$ are the sample sizes.

The wide reproducibility of the experiment is helped by the fact that we used simple software and accessible on-line tools.

3.2. Statistical Results

3.2.1. Statistical Evaluation of the t-Test Results

I. Between the MSc and MSc Control groups:

• T-statistic: −1.746;
• p-value: 0.1000;
• The p-value is higher than the usual threshold for significance (usually 0.05), indicating that the difference between the two groups is not statistically significant.

II. Between BSc and BSc Control group:

• T-statistic: −5.555;
• p-value: 0.0001248;
• The very low p-value indicates that the difference between the two groups is statistically significant. The negative t-statistic indicates that the BSc group achieved better average results than the BSc control group.

III. Between HVET and HVET Control group:

• T-statistic: −2.051;
• p-value: 0.0485;
• The p-value is close to the conventional threshold of significance, suggesting that the difference between the two groups is statistically significant. The negative t-statistic indicates that the HVET group achieved better average scores than the HVET control group.

In summary, there are significant differences between the BSc experimental–BSc control and HVET experimental–HVET control pairs, with the experimental groups performing better in both cases; However, no statistically significant difference was found between the MSc Experimental–MSc Control pairs.

3.2.2. Impact Magnitude Analysis

Using the t-test results, we calculated Hedges’ g, Cohen d, and Glass delta using another online tool (https://www.socscistatistics.com/ accessed on 3 January 2024). Hedges’ g was chosen for the evaluation because it is the most appropriate way to correct for small sample sizes.

Hedges’ g analysis provides an accurate and corrected effect size estimate when comparing experimental and control groups, especially to avoid possible biases due to small sample sizes. Although the t-test determines statistical significance, it does not provide insight into the magnitude or practical significance of the difference. Hedges’ g, however, quantifies the magnitude of the difference and compensates for the bias caused by small sample sizes, thus offering a more reliable means of assessing the true significance of research effects.

The Hedges’ g formula, Formula (3), is as follows:

$$g=\frac{{\overline{X}}_1-{\overline{X}}_2}{s_p}\left(1-\frac{3}{4\left(n_1+n_2\right)-9}\right)$$

(3)

where ${\overline{X}}_1$ and ${\overline{X}}_2$ are the group averages, $S_p$ is the pooled standard deviation, and $n_1$ and $n_1$ are the sample sizes.

MSc Group I (learning outcomes and motivational change):

·

Learning outcome: there is no significant difference between the MSc group’s results for project-based (mean: 4.875) and non-project-based (mean: 4.700) teaching.

·

Change in motivation: although there is a difference in motivation, it is not significant between project-based (mean: 4.600) and non-project-based (mean: 4.4286) education.

BSc Group II (learning outcomes and motivational change):

·

Learning outcome: there is a significant difference between the BSc group’s results for project-based (mean: 4.700) and non-project-based (mean: 3.8571) education.

·

Change in motivation: there is a significant difference in motivation between project-based (mean: 4.2500) and non-project-based (mean: 2.8333) education.

HVET Group III (learning outcomes and motivational change):

·

Learning outcome: there is a measurable but not significant difference between project-based (mean: 4.6471) and non-project-based (mean: 4.1176) education for the HVET group.

·

Change in motivation: the difference in change in motivation between project-based (mean: 4.1765) and non-project-based (mean: 3.4706) education is measurable but not significant.

These results suggest that project-based training was efficient for the BSc group regarding learning outcomes and motivational change. The differences were less significant for the MSc and HVET groups. The effect size test also confirmed our post-test t-test results, with Hedges’ g-analysis confirming and eliminating the lack of validity due to the small number of items.

In the analysis based on Hedges’ g data, we were unable to fully account for the large amount of written material recorded for the control group, so we would be evaluating the effect sizes of the study groups without being able to account for all of their observations (compared to our other analyses). We conducted a meta-analysis to provide a more complete analysis of our results (which we also consider to be the basis for scientific discourse). This phase allows us to put the results in a broader context, assess the effects on a more analytical basis, and increase the conclusions’ reliability and generalisability.

These three phases (three, four, and five) can be traced in the three middle columns of Figure 3.

3.2.3. Meta-Assignment

Organising the data into a CSV file: the calculated statistical values, including group averages, standard deviations, sample sizes, and Hedges g-values, were organised into the following CSV file format.

The CSV file in

Table 2. Content and structure of CSV file.

Treatment	Trt_Class	Average	Std.dev	n
MSC	result group no project	4.7000	0.4583	10
BSC	result group no project	3.8571	0.6389	7
HVET	result group no project	4.1176	0.9630	17
MSC	result group project	4.8750	0.3307	10
BSC	result group project	4.7000	0.4583	8
HVET	result group project	4.6471	0.5882	17
MSC	motivation change no project	4.4286	0.7284	10
BSC	motivation change no project	2.8333	0.6872	7
HVET	motivation change no project	3.4706	1.1437	17
MSC	motivation change project	4.6000	0.4899	10
BSC	motivation change project	4.2500	0.4330	8
HVET	motivation change project	4.1765	0.6169	17

was uploaded to another online tool (https://smuonco.shinyapps.io/Onlinemeta/ accessed on 3 January 2024), where the software performed graphical analyses, producing forest charts, ranking tables, and SUCRA graphs. These visualizations provided a clear representation of the effects and rankings between different groups.

This procedure strengthens the validity and reliability of the research, allowing for an accurate and comprehensive analysis of the data.

These two phases can be followed in the right-hand column of Figure 3.

3.3. Interpreting the Results in Terms of Motivation to Learn

The impact of the combination of FC and PBL on students’ motivation to learn is particularly significant at the BSc and HVET levels. The combination of project-based tasks and teaching with FC motivates students, especially when MSc students take an active role in the learning process. Students showed higher interest and engagement in the course material, which resulted in increased motivation to learn. The opportunities for practical applications and project work were inspiring for the students, which played a role in making the learning process more experiential and engaging.

In light of these results, the further integration of FC and PBL in university education could significantly improve students’ motivation to learn. This integrated approach allows students to participate actively in the learning process, linking theoretical knowledge with practical application. The results of our experiment confirm that this combined teaching method is an effective way to support student motivation and improve the quality of education and, ultimately, their employability.

4. Discussion

4.1. Theoretical Background to Motivating Higher Education Students

Inegbedion and Islam [17] proposed several theoretical methods to increase students’ motivation in higher education. Three of them are discussed below:

• Encouraging positive behaviour and emphasising the value of work;
• Supporting autonomy;
• Facilitating imagination and result orientation.

We kept these three principles in mind when designing the tasks for each educational project, communicating them to groups of students, and testing the impact’s magnitude. Understanding the work processes and performance of real wineries helped objectively assess the work’s value. Recognising the problem of disconnection from reality in university education, an alternative “learning by doing” approach was used to help students to understand the interactions between theoretical knowledge and real processes at the student level [18]. Students were required to visit the enterprises independently, and the groups themselves carried out data collection and analyses. Autonomy also implies taking responsibility according to the roles within the project. Understanding and reflecting on innovations required considerable imagination, co-creation, and thinking. They had to define their project goals in a goal-oriented rather than autonomous way.

One of the main aims of improving pedagogical methods in higher education is to increase student participation and motivation. Our experiment took place in an educational environment where FC is commonplace and, thus, already a traditional teaching method. Our experience in this field was based on continuous measurement. These experiences were mainly related to careful course design to optimise efficiency; however, we are looking for related methods to enhance the learning experience and, thus, motivation.

Other researchers [1] have investigated the comparative effectiveness of face-to-face, online, and blended learning modes in higher education. They confirm our observation that the quality of content remains consistent across these modes but that there are significant differences in facilitation, perceived value, and learning effectiveness. This suggests that although the delivery method of content may vary, the key to successful learning outcomes lies in the facilitation of courses and the perceptions of students.

4.2. Challenges to the Concept of Teaching

Our experience shows that despite its many advantages, this teaching concept still poses significant challenges in today’s university environment. One of the main difficulties in applying FC is the infrastructure requirements, such as complex software like Symbaloo, Khan Academy, Doceri etc. [19], which can be costly or technically challenging.

UNESCO’s Global Education Watch 2023 highlights the rapid changes in education technology and related challenges directly affecting higher education and education systems. The report highlights the critical role of technology in sustaining education during the COVID-19 epidemic but also draws attention to inequalities in access, the lack of diversity in educational content, and the negative impact of technology overuse on student achievement. Based on these approaches, the University of Tokaj hosted the experiment and introduced the FC environment in all its courses. We were aware that educational technology can improve specific learning processes, for example, by increasing access to teaching and learning resources, but it is essential to focus on learning outcomes and not just digital inputs. We have had to adapt to new technological developments such as artificial intelligence and the use of big data. These developments and pedagogical approaches have informed our experiment, as they require a continuous revision of teaching methodologies to ensure that university students can achieve sustainable learning in an uncertain future [20].

We have prioritised the development of digital skills, but we have found that many students and teachers feel they need to prepare to use digital technologies in education; however, our university must face and adapt to the rapid changes in these technologies.

4.3. Education Strategies

However, the changing educational technology of our FC-based educational environment is also opening up new platforms for students and pupils and social and economic actors. In their comprehensive research, Kipper et al. [5] used a systematic literature review and scientific mapping methods to investigate what competencies are needed to meet the requirements of Industry 4.0. The research was conducted in the Scopus, Web of Science, and Science Direct databases between 2010 and 2018 and aimed to identify the competencies that the literature considers essential for professionals working in the Industry 4.0 era. The researchers highlight the need for universities to work closely with companies to create appropriate education and training programs. These programs provide students with practical experience to prepare them for the real workplace challenges of Industry 4.0. The appropriateness of our methodology in this regard is supported by the fact that “the incorporation of project-based tasks allows for the creation of real-life experiences, which further encourages the development and growth of real-life competencies” [21].

In the framework of the PBL, the student groups solved tasks where they learned about and evaluated the processes of vineyard and wine businesses in the region. In this way, they also underwent a self-preparation process, which increased their motivation to learn. In this context, students dropping out was still observed in the control group, but all students in the PBL group successfully completed their training.

4.4. Finding Solutions to Attract Students Who Drop Out

One way to reduce dropouts is to increase motivation to learn; the student project groups have been an effective way to do this. The factors influencing learning motivation include interest, commitment, optimism, and a suitable learning environment. Intrinsic motivation is significant because students’ intrinsic motivation, goals, and ambitions are essential for their learning success [22]. Therefore, our experimental course design and implementation can be used as an effective tool to reduce dropout.

The PBL method reduced dropout by improving collaborative learning, subject knowledge, iterative learning, and authentic learning, increasing students’ engagement in learning [23].

The use of PBL, therefore, can also bridge the gap between students’ lack of motivation and passivity. Active learning with planned group performance helps to motivate learning.

It should be pointed out that without a proper assessment process and a typical group organisation, the phenomenon of “stowaways” can develop, which means that some students hide their underperformance behind the group’s excellent performance. In our experiments, we found that many students felt that PBL encouraged them to collaborate and negotiate within the group, but some students reported a lack of motivation to work in groups [24].

This phenomenon was eliminated when, in a 360-degree evaluation, the peer and group leader evaluations became relevant.

4.5. Horizontal and Vertical Comparisons between Groups

Our findings on the extent of comparisons between groups at different educational levels are as follows:

• The MSC group scored higher average scores in project-based and non-project-based teaching, indicating that they performed better than the BSC or HVET groups.
• For the BSC group, the average score of project-based learners was higher than that of traditional learners, which may indicate that project-based learning is more effective for them.
• The HVET group also had a higher average score for project-based education than the non-project-based group, demonstrating project-based education’s effectiveness.

Overall, the results suggest that project-based education positively impacts students’ attitudes and preparedness for innovation management. At the same time, besides the level-by-level evaluation, a positive outcome of the experiment is that project-based learning in university courses significantly impacts students’ academic success and contributes to developing their professional judgment. We also confirm previous research findings that PBL-supported learning allows students to experience real-life situations that promote interdisciplinary collaboration and teamwork [25].

4.6. Tests on the Effectiveness of the Experiment

Figure 4 is a ranking graph evaluating the effectiveness of each training group in our experiment.

·

MSc Group: In the graph, the score of the MSc group indicates that this group is the best post-ranking, which indicates that this group performed the best. The position of the dot and the interval line on the right-hand edge of the graph indicates that the MSc group is the most likely to have achieved the most favourable result.

·

BSc Group: The BSc group is in the middle of the field regarding points and interval lines, with the second-best result. The width of the interval indicates uncertainty, but the central position of the point shows that the BSc group performs stably in the middle, i.e., for them, this training method is ideal.

·

HVET group: The point and interval line of the HVET group is at the bottom of the graph, showing that this group was the least efficient. However, the end of the interval extends into the middle range, which may indicate uncertainty in the evaluation of the results, and it is advisable to extend and fine-tune the proxy method in the future.

Overall, the ranking graph shows the experimental groups’ relative performance concerning the educational program’s effectiveness. The experiment had no measurable effect on the relative performance of the groups, nor was this the intention. For example, it was not assumed that the relative performance of the BSc group would exceed that of the MSc group. The graph confirms that the “division of labour” within projects was consistent with expected performance, and the positive effect of PBL could be further enhanced by the fine-tuning of the HVET group.

4.7. The Role of Trainers in the Effectiveness of Experiments

Teacher trainers also have a significant role to play in fine-tuning. Sanchez-De Miguel et al. [26] studied students’ motivation, resilience, and perceived competence in different classroom settings, highlighting the impact of instructor profiles. They find that differences by training level can only be interpreted with knowledge of the work invested by instructors. This is important to note here because the involvement of instructors in the process should be seen as a critical factor, and it will be necessary to incorporate their experiences (which we now have as a result of the experiment) into further evaluations to fine-tune the process. Teachers’ deep understanding of the teaching-learning process is an essential characteristic that helps them evaluate their profession and the teaching process. These colleagues consistently argue that learning is a transformation that occurs through acquiring new knowledge, understanding empirical regularities, or a change in mindset [27]. This reinforces the importance of developing conscious teaching competence in the pilot project.

4.8. Summary of Results from a Hypotheses Perspective

Our analysis also confirms that the BSc students who participated in project-based learning achieved the most substantial change and benefited from this method. The HVET students also showed a positive impact, indicating that both practical fieldwork and report writing were beneficial exercises for them. The MSc students’ results were slightly different, indicating that the impact of the leadership role they experienced still needs to be clarified. In the control group, the scores of students taught using traditional teaching methods were generally lower, suggesting that project-based teaching was more effective in preparing students for innovation management.

Our analysis concludes that project-based education had an overall positive impact on students’ perceptions of their preparedness and their actual innovation management skills.

Thesis 1:

Based on our results and reflecting on our first hypothesis, we state: Combining flipped classroom (FC) and project-based learning (PBL) significantly increased students’ motivation to learn and improved learning outcomes, especially at the BSc and HVET levels. This combined teaching approach proved beneficial for students’ active participation and developing their independent learning skills.

Thesis 2:

Based on our results and reflecting on our second hypothesis, we state: At the BSc and HVET levels, we observed significant differences in the effectiveness of supplementing the traditional FC method with PBL, while at the MSc level, this difference was not significant. This suggests that different educational approaches produce different results at different levels. We believe that further methodological experiments at higher levels of education are needed or that the fine-tuning of our PBL method is warranted.

Our experiment was successful because we demonstrated that the combination of flipped classroom (FC) and project-based learning (PBL) significantly positively impacts students’ learning motivation and learning outcomes, especially at the BSC and HVET levels. At different levels of higher education (HVET, BSc, and MSc), the effectiveness of supplementing the traditional FC environment with PBL varied, with greater effectiveness at the BSc and HVET levels and less significance at the MSc level. Overall, the hybrid method effectively addresses the three levels of education together, showing that specific approaches at different levels effectively support students’ academic performance and motivation.

Our experiment draws attention to the fact that online and face-to-face learning environments, widely used in educational settings, need to be revised to provide an effective learning experience and, therefore, must be combined with other innovative methods.

4.9. Presentation of the Limitations of the Research and Further Research Opportunities

Our study did not show how we dealt with the “stowaway” phenomenon, which will be discussed in a later article because our other experiment aimed to deal with this phenomenon. This in itself does not affect our present conclusion.

The tools used in the PBL also influenced FC activities, so we cannot claim that the study and control groups were learning from the FC methods under the same conditions. This effect is not detectable but instead shows that the combination of methods continuously develops and transforms educational concepts, their interaction being one of the keys to progress.

Accurately measuring students’ motivation to learn is a complex but vital task in educational research. Although many methods are available for this purpose, many are more objective than our methodology but involve higher costs. For example, some researchers have developed sophisticated scales to measure motivation’s positive and negative aspects. An example is the reading motivation scales developed by Wigfield et al. [28], which measure the evaluation of reading activities. These scales effectively predict reading achievement, highlighting their usefulness in educational research [29]. However, in our experiment, we chose a different approach due to resource limitations. We decided to measure student motivation changes using a more straightforward instrument. Although this method may offer a more limited interpretation compared to more comprehensive scales, it allowed us to detect a significant effect. This decision is consistent with the notion that motivation and its measurement are inherently subjective and situated in a specific cognitive and social context. As noted by Wigfield [30], task scores can vary widely across individuals, suggesting that the same educational task may be scored differently depending on the learner. Therefore, although more objective and costly methods for measuring motivation to learn can provide more detailed insights, our experiment shows that even these more straightforward tools can yield significant and valuable results on learner motivation. This approach, while perhaps more limited in scope, can significantly contribute to understanding how learners relate to and value their learning experiences [29].

Given that we have studied almost identical groups of students at several levels of education, and the statistical methods used have been carefully chosen, we would obtain the same results with a larger sample. However, three years have passed since the first group, and we have observed a secondary phenomenon that we could not present in this study. This is a significant difference (deterioration) in students’ digital skills and online learning existences between those still in secondary or higher education during the COVID pandemic and those who entered higher education afterward.

The extent to which the results of MSc and BSc students are influenced by prior education and the discipline studied needs further investigation. Therefore, we have indicated that the non-significant effect for MSc students cannot be considered a generalisable result.

5. Conclusions

Industry 4.0 imposes novel demands on education systems, compelling the adoption of new solutions. The Education 4.0 response incorporates scientifically supported methodologies that may resolve the issue, but this is untrue. The continuous development of educational environments and methodological innovation remains indispensable. A robust educational environment, such as the FC, is crucial, but more is needed. Educational outcomes can be enhanced by transcending methodological boundaries.

Our study has shown that for less experienced students, these combinations offer a substantial entry into higher education, particularly for HVET students. For BSc students with prior learning experiences, project-oriented methodologies like PBL provide valuable practical insights. However, these combinations yield slight improvement in learning motivation for seasoned MSc students, who are often already in the workforce and understand the value of learning. Nonetheless, their presence is vital for the success of PBL, as they serve as an organizing force and experienced examples for project groups.

Incorporating mixed learning levels into project groups, where younger students benefit from the coordination and motivation of more mature peers, has proven beneficial. Such an approach presupposes an educational environment where the institution, particularly the faculty, provides ongoing support and mentorship. The effects can be further amplified by directly involving corporations in the mentorship process, ensuring that positive feedback is received from both educators and businesses.

It is essential to foster self-reflection and group dynamics, aiming for the state of flow in learning, which our study has repeatedly confirmed as beneficial. Beyond motivation, the FC and PBL combination at every educational level demonstrates higher learning outcomes, indicating an improvement in educational effectiveness and, thus, quality. This elevation in educational standards ultimately contributes to individual employability and economic development.

We hope our experiment inspires other universities, educators, and researchers to conduct similar experiments across various disciplines, potentially unlocking the complete impact mechanism of the FC-PBL combination.

We are eager to share our research experiences and methodologies and remain open to participating in similar university experiments. Our goal is to inspire other educators and researchers to engage in such pedagogical explorations, further uncovering the complete impact mechanisms of the FC-PBL combination across various disciplines.