**Incorporation of Sustainability Concepts into the Engineering Core Program by Adopting a Micro Curriculum Approach: A Case Study in Saudi Arabia**

#### **Muhammad Waqar Ashraf 1,\* and Faisal Alanezi <sup>2</sup>**


Received: 12 March 2020; Accepted: 18 March 2020; Published: 5 April 2020

**Abstract:** Higher education institutions are increasingly pursuing sustainable development goals in engineering and technology education. The concepts related to operations, production, and consumption continue to gain importance and significance for engineering students. In the engineering profession, the incorporation of sustainability means integrating environmental, economic, and social factors into the evaluation of design processes, products, and services. Therefore, it is necessary to develop an engineering program that along with the technical content, also fosters a critical sense regarding the social and environmental aspects of the field. The current status of sustainability education in engineering programs offered in Saudi universities is not very promising. In this paper, we explore the use of existing university curricula to incorporate sustainability elements into engineering education and training. Sustainability concepts were introduced into selected courses by using a micro-curriculum approach. Moreover, a standalone course is also introduced. We observed that this approach has been successful in integrating sustainability into the engineering curriculum. We recommend that such an approach be used to develop sustainability awareness in engineering programs.

**Keywords:** sustainability; education; engineering; micro-curriculum; PMU

#### **1. Introduction**

Humans control the earth's resources and use them for fulfilling their needs. These resources, however, are limited in capacity and are consumed in increasingly massive quantities as a result of globalization and expanding economic development within the span of recent decades. At present, the problem to be addressed is the lack of regenerative capabilities to satisfy needs and ease the burden that is being put on the planet by industrialized civilization. If no solution is found, these limited resources will diminish, with catastrophic results for the global ecosystem.

An effective solution is to incorporate processes with sustainability so that current needs are secured without compromising future generations. This can be achieved only by incorporating sustainability's three essential pillars: environmental protection, social development, and economic growth. The idea of sustainability assumes that nature and the environment are not inexhaustible resources and therefore, must be rationally used and protected. Sustainability promotes social development by seeking coherence between cultures and communities to achieve satisfactory levels of quality of life, health, and education. Moreover, sustainability seeks equal economic growth that maintains wealth for all without harming the environment. Implementing the ideas of sustainability,

however, requires consideration of numerous objectives such as poverty, environmental degradation, climate, inequality, etc., thereby making its widespread adoption challenging [1,2].

Of course anyone who glances at the news is aware of this, but it is necessary to rehearse the consequences of the discrete, rather than integrative analysis of sustainability, because there is evidence that engineering programs, and indeed other programs in the STEM set of disciplines, have a limited experience of incorporating sustainability into their curricula [3]. At the international level, the concept of sustainability was first introduced in the higher education system by the UNESCO-UNEP (United Nations Educational, Scientific and Cultural Organization—United Nations Environment Program) International Environmental Education Program [4]. A number of international and national declarations about the integration of sustainability issues in higher education (HE) institutions have been developed in subsequent years [5]. In Europe, the early initiative was the Co-operation Program in Europe for Research on Nature and Industry through Coordinated University Studies (COPERNICUS), which was established by Conference of the Rectors of Europe (CRE) to collaborate on common environmental issues. In this context, CRE developed the COPERNICUS charter for Sustainable Development in 1994. On the Global scale, another important declaration is the Ubuntu Declaration on Education, Science and Technology for Sustainable Development in 2002. This declaration was signed by major academic institutions worldwide like the Third World Academy of Sciences (TWAS), United Nations University (UNU), the International Association of Universities, and the Science Council of Asia, amongst many others [6].

In the engineering profession, the incorporation of sustainability means integrating environmental, economic, and social factors into the evaluation of designs of processes, products and services. Generally, the 'processes' consists of all kinds of activities by which the human civilization advances itself. These include tasks that start from extracting resources and transforming them into consumer products. Considering energy resources, fossil fuels must be transported and then processed to generate electric power, heat and light. All substantial human activity makes use of these resources, which are thus eligible for consideration in light of sustainability. Engineers are responsible for organizing and optimizing these processes by visualizing all aspects of the system and their collective interplay. Their tasks include designing supply chains to deliver components to manufacturers within deadlines, determining the optimal combination of raw materials to formulate finalized products, etc. [7,8]. The endeavors of United Nations in adopting sustainable development goals (SDGs) for the 21st century have triggered incorporation of sustainability concepts in the curricula of primary to higher education. UN-SDGs are focused on the environment (climate change, life on land, life below water), social (gender equality, peace) and economic (no poverty, decent work, and economic growth) domains of sustainability. In addition to that, SDGs go beyond three basic spheres of sustainability by emphasizing quality education (#4) and partnerships for the goals (#17). The National Academy of Engineering (NAE) have defined 14 grand challenges for engineering based on cross-cutting themes, which include sustainability [9].

Given these job descriptions, future engineers are eligible to address the need for incorporating sustainability in systems and overcoming the challenges that come as a part of its implementation. Industrial engineers can examine the requirements of a system and alter different parts with the motivation to achieving the system's ultimate goal while including sustainability. For example, they could consider minimizing resource consumption to foster environmental protection. Furthermore, engineers are trained with the characteristic of working towards maximum economic gain. While achieving this goal, the tools of the field can also be used to reach additional goals which were unlikely to be considered in the past. These additional goals such as minimizing resource consumption or maximizing population health can be combined with the objective of achieving profits to attain an overall balanced system, which achieves both its sustainability and economic goals [10,11].

The best way to accomplish this transformation in the mindset, and to prepare engineers to implement sustainability in their careers, is through their education. Modern higher education institutions are increasingly striving to educate the future professionals who are market ready. Mere technical training is no longer enough to meet the needs of the society. Therefore, it is necessary to develop engineers who along with technical knowledge, also have a critical sense regarding the social and environmental aspects of their profession [12,13]. Accordingly, sustainability needs to be central to the design of curricula in higher education. Many international declarations are regarded as landmarks for including sustainability into higher education, especially EESD (Engineering Education for Sustainable Development) and the Barcelona Declaration. Recognizing the variety of possible applied situations, the apprehension about incorporating sustainability into engineering courses has become an even more serious concern for curriculum development [13].

In the recent decades, the exponential growth of universities in the Kingdom of Saudi Arabia [14] has clearly indicated its intentions towards sustainable national development and qualified human resources across all provinces of the Kingdom (Figure 1). In a recent study, it was revealed that academic courses relevant to sustainability are lacking in Saudi Arabia [15]. Most Saudi universities need to integrate sustainability concepts in the curriculum and promote research and scholarship in this direction.

**Figure 1.** Growth of public and private universities in Saudi Arabia.

There are various ways to introduce sustainability concepts and disseminate information on how to achieve it through engineering practice; it is especially advisable to take these steps early in an engineer's education [16]. This gives time for these concepts to take root and be fully absorbed into the budding engineer's professional consciousness. Through a pilot program at Prince Mohammad Bin Fahd University in the Eastern Province of the Kingdom of Saudi Arabia, we endeavored to achieve this objective of introducing sustainability concepts into the engineering curriculum through the implementation of sustainability elements into the core curriculum. This paper presents our findings and conceptualizes them to suggest a process by which sustainability concepts can be efficiently incorporated into an engineering curriculum.

#### **2. Methodology**

To initiate the process of introducing sustainability concepts into the University Core Program, we first defined the objectives we sought to achieve through the enhanced curriculum. We aimed to offer engineering students multiple experiences that would demonstrate what it means to have a

sustainable attitude and to enable the development of both the desire for and the skills to use with engineering approaches to sustainable practices.

The broad goals of the project were defined as follows:


We then framed the scope of our work by defining the specific learning outcomes that we aimed to achieve through curricular enhancements. The course learning outcomes (CLOs), we defined could be measured through both qualitative evaluations and quantitative assessments. Students should be able to:


To select the courses most suitable for the incorporation of sustainability elements, we first sorted the courses offered by engineering departments at our university, into two groups: those that teach strategies and critical thinking in problem solving, e.g., "Manufacturing Methods & Designs", and those that introduce new perceptions along with the suitable approaches, e.g., "Engineering Economy." A senior design project is a problem-based learning opportunity for the students that helps them to understand sustainability issues in relation to the engineering industry. The main emphasis is on cleaner production and appropriate solutions from knowledge gained from taught courses.

In addition to this process, we developed a stand-alone course entitled 'Introduction to Sustainability' (SUST 1311). This course has been offered to all students in the university's Core Curriculum since 2017. The course description from the Prince Mohammad University (PMU) catalogue is as follows:

"This course familiarizes students to the theory, principles, and practices of sustainability. It will include discussions on sustaining ecological and environmental wellbeing, creating economic prosperity, and safeguarding social justice."

The main focus of this course is sustainable development. Our ultimate goal was the successful implementation of both approaches—inclusion of sustainability concepts into the core engineering program and the introduction of stand-alone sustainability courses in degree programs.

We also identified a list of sustainability notions, we targeted to address through a micro-curriculum. These areas were mapped with United Nations Sustainable Development Goals (UNSDGs) [17,18], as shown in Table 1.


**Table 1.** Sustainability concepts mapped with United Nations Sustainable Development Goals.

These concepts were assessed by using specifically designed rubrics with a Key Performance Indicator (KPI) set at 80% (Figure 2).

**Figure 2.** Assessment of targeted sustainability concepts.

Students who enroll in university studies often have little experience with inquiry, research, or scholarly discourse. They expect instead that they can satisfy academic requirements simply by restating content provided by their teachers. Thus, many students are poorly prepared to assume the responsibilities associated with university-level scholarship. The University Core Curriculum Program at PMU consists of additional courses required of all University students. These consist of communication courses in designated competencies as well as the sciences and Mathematics. Critical thinking experiences in the Core Curriculum stress reasoning as a means of discovery and a tool for increasing understanding in both university courses and the student's personal life. Throughout the Core Curriculum, all assignments involve a set of learning outcomes, and expectations are articulated by examples and models. We examined each course in the Core Program in detail to determine the most suitable curriculum adjustments to make. Table 2 presents the mapping of different concepts within the core courses. For competency courses, specific assignments are designed to incorporate

the sustainability concepts. For example, a short synopsis on water conservation, landscaping in the desert, harvesting solar and wind energy, oil related pollution problems, dynamics of the Arabian Gulf, etc. are introduced to the students to complement the defined learning outcomes. In leadership (UNIV1213) and critical thinking (UNIV1212) courses, UNSD goals and policies are introduced to the students in oral presentations. For the engineering curriculum, we considered synergies between existing course content and relevant sustainability content (for example, introducing the design of green buildings and sustainable transportation), and focused on the new ideas. This process allowed us to build tangible links between courses available and content suitable for incorporation into them. Some other examples include CVEN3331 and GEEN4311 which introduce students to the fundamental principles of environmental conservation and ethics that lead to sustainability for humans and the ecological systems in the Arabian Gulf.


**Table 2.** Micro-curriculum concepts incorporated into the Core Courses.

#### *Assessment Strategies*

Assessment strategies included both qualitative and quantitative methods. To test the effectiveness of our approach, we used several methods. We assessed students at the end of the course with quizzes on the material familiarized. In the stand-alone SUST 1311 course, with 156 students, we incorporated assessment strategies such as field reports, volunteering, outreach, and a final written exam. Almost all students scored 80% or higher, on the completion of the course. Although these results are limited by the small sample sizes, they clearly demonstrate that students exposed to the components did absorb a significant amount of their content.

Given that our primary objective was to introduce concepts of sustainability to engineering students, the qualitative assessments give a better picture. To carry out this assessment, the students were given end-of-course surveys at the end of the semester. The following are representative responses:


Most promisingly, students seemed excited about the possibilities that SUST 1311 would enable them to make an impact on society with respect to sustainability. Their responses demonstrate their mindfulness to carry out work in a sustainable manner, and to use their education to achieve the objectives of a sustainable and protected future. One student wrote:

*By taking this course, I realized that I can make a difference by saving the environment. As an engineer, I can think of reducing the use of fossil fuels and prepare for a non-oil based economy in Saudi Arabia. I am using social media to educate my friends and relatives about the importance of a sustainable economy.*

These and many other responses suggest that, if properly applied, the incorporation of sustainability concepts into the study plans can make a big difference in affecting students' perceptions—and, ultimately, in their skills and willingness to apply the concepts in their professional lives.

Other qualitative assessments were made by conducting student's feedback surveys, peer reviews, preceptor views for senior design projects, and employers' surveys. Students were given the opportunity to rate the performance of other senior design projects in terms of sustainability elements. Also, for the senior design, projects evaluators included both academic supervisors and practicing engineers working in industry. The comments by the evaluators were employed to incorporate sustainability concepts in the project. In some cases, a seminar on sustainability was included in the assessment to gauge the student's learning. Employer surveys served as a good opportunity to judge our students' edge over other university graduates in the market. A survey has been completed by a total of 16 employers in 2019. The results showed that our graduates were superior in three domains: sustainability awareness, critical thinking, and innovation.

Quantitative assessment, written quizzes on the materials introduced, and a writing assignment were also used. For example, students are required to write a three page paper on engineering practices to combat climate change in GCC countries. Students were also provided with a relevant rubric to grade the assignment. Although the results are limited by sample size, they clearly demonstrate that students exposed to teaching materials, did absorb the significant quantity of the concept. Environmental Sustainability was the best learned topic, followed by personalized learning. Students showed great interest in environmental issues related to impact oil production and tanker traffic in the Gulf and suggested interesting mitigation measures.

#### **3. Discussion**

#### *3.1. Challenges in Implementation*

The major hurdle in injecting the micro-curricula in the existing program structure comes from the diverse nature of sustainability issues. The work load, in terms of student's credit hours, is already heavy and it is a challenge to incorporate additional volume of sustainability related materials. The addition of sustainability topics might result in loss of other essential materials from the course. In our university, a student centered model is adopted whereby focus is on problem solving, decision making, working in multidisciplinary groups, technical and professional presentation skills, and broader exposure to different topics throughout the program. Therefore, it is possible to integrate sustainability concepts in different courses of the program. The development of micro-curricula was made in such a manner that sustainability concepts were introduced in a basic manner and students are made aware of the significance of environmental importance in any design development. In senior design projects, innovative solutions that require less expenditure of natural resources and work with natural environment are encouraged.

#### *3.2. Stand-Alone Courses*

Stand-alone courses (SUST 1311) also played a pivotal role in achieving the objectives. With such courses offered, students have opportunity to learn sustainability at different extents of instructional intensity. It is also noted that such courses should be offered at the junior/senior levels of the educational path. This will help the students to complement the early learned facts with sustainability concepts. Most importantly, these elective courses should help students address the intersection of sustainability with engineering concepts.

#### *3.3. Sustainability Co-Curricular Activities*

As an additional step, engineering departments can choose to expand their focus on sustainability through engagement with industry and research activities. Sustainability education could be more effective through capstone courses, undergraduate research programs, design projects and strong support by university management. Extra-curricular activities like posters, essay competitions, speeches, etc. can also augment this education. Organizations like Association for the Advancement of Sustainability in Higher Education (AASHE) support and publish such activities and initiatives.

#### *3.4. Gradual Process*

The process of developing and integrating sustainability micro-curriculum in all engineering majors cannot occur in one go. If such changes are too fast and massive, they might disrupt program objectives. We believe in a step-wise approach like starting with a micro-curriculum, then a stand-alone course followed by a full fledge sustainable engineering program.

#### *3.5. Consistent Focus*

In order to make sustainability education meaningful and effective, the effort must remain steady and focused. This focus is achieved by establishing a Sustainable Campus office at our university headed by a senior administrator. This office supports activities and initiatives related to sustainable education on campus. For example, in our university the theme of the year for 2018 was Sustainability with a slogan "together towards a sustainable campus".

#### *3.6. Comparison with Other Approaches*

We also consulted relevant recent research in other parts of the world for modifying existing engineering curricula. Galamboski and Ozelkan [7] have published an overview of curricula modification in both engineering and management study plans. Nazzal et al. [11] have used a modular approach to introduce sustainability concepts in industrial engineering curricula in the University of Central Florida. Wilson D. [12] have explored in detail the link between sustainability and engineering education. Murphy et al. [19] have published an overview on how much sustainability content is present in different programs of the university versus traditional content. Allenby et al. [20] have reported an overview of the topic, with emphasis on added content, funding, and relevant

educational techniques. They also talk about usual controversy between sustainability as a fuzzy subject and engineering an abstract science. Pierre et al. [21] used an alternative approach of using modules as stand-alone curricula for short courses. This is in agreement with our approach of introducing micro-curriculum. We also conducted a review of numerous papers discussing sustainability contents in different engineering disciplines. More generally, authors from world over suggest that there is increasing need for this integration across the engineering disciplines [22–26].

#### **4. Recommendations & Conclusions**

To assimilate sustainability elements into the curriculum, existing course syllabi must first be surveyed to identify the best candidates to afford the incorporation of diverse sustainability concepts. At this stage, the faculty member leading the initiative must be actively involved with instructors to further modify classroom and lab situations in which sustainability ideas can be most efficiently introduced.

Stand-alone courses focused on sustainability as the core subject are most effective ones. By offering such courses, students have the choice and opportunity to access sustainability directly. This achieves the objective of making sustainability an integrated element of their education. The departments can also choose to focus on sustainability through research and increased interaction between industry and university.

These suggestions, based upon our findings, are meant to serve as a basic outline for the progress of sustainability micro-curricula in connection with existing education programs. Although our work is narrow in scope, our accomplishment in meeting our goals and the encouraging responses we received from learners is quite encouraging, and demonstrates that our approach is effective and applicable.

**Author Contributions:** M.W.A. was responsible for collecting necessary data, evaluation of different core courses and development of learning outcomes. He led the development of stand-alone course materials and syllabus (SUST 1311). F.A. was responsible for literature survey, qualitative assessments and drawing conclusions from surveys. He was responsible for implementation of new stand-alone course in colleges degree plans. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Acknowledgments:** Authors are grateful to Department of Campus Life for supporting and helping in co-curricular activities. Support from PMU higher management and Students Sustainability club is also highly appreciated.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **A Design-Based Learning Approach for Fostering Sustainability Competency in Engineering Education**

#### **Zhiliang Huang, Annan Peng, Tongguang Yang \*, Shuguang Deng and Yuexia He**

School of Mechanical and Electrical Engineering, Hunan City University, Yiyang 413002, China; huangzhiliang@hnu.edu.cn (Z.H.); 1043808305pan@sina.com (A.P.); shuguangdeng@163.com (S.D.); h17670328830@163.com (Y.H.)

**\*** Correspondence: yangtongguang1@163.com; Tel.: +86-1338-737-1899

Received: 21 February 2020; Accepted: 4 April 2020; Published: 8 April 2020

**Abstract:** This paper provides and illustrates a design-based learning (DBL) approach for fostering individual sustainability competency in engineering education. We performed two studies with engineering students in typical educational activities. The first study helped students perform a topic-specific design task in the practicum unit of a sensor technology course, which compared the performance of the DBL approach and conventional passive learning approach. The second study guided students to develop innovative projects for participating in the "Internet Plus" Innovation and Entrepreneurship Competition (IPIEC). To validate the proposed approach, stakeholder questionnaires and performance evaluations were implemented. The results show that the DBL approach was viable for sustainability competency teaching in terms of learning demand and teaching procedure. We found that students in the DBL group gave more prominence in the individual competencies, such as system-thinking, multidisciplinary applications, and collaboration. These findings suggest that applying the DBL approach to train sustainability competency in engineering education is beneficial for promoting students' abilities in dealing with challenges involved in sustainability practice.

**Keywords:** sustainability education; design-basedlearning; individual competency; engineering education

#### **1. Introduction**

The emerging academic field focused on sustainability has been engaged in the studies of pedagogies to foster practitioners' competencies for corporation sustainable development. Global leaders integrated 17 goals into the Sustainable development agenda: 2030 [1], such as poverty, education, sustainable economic growth, renewable consumption, production, etc. These goals are rooted in exploring solutions to extensive social challenges around the world. Maximizing the contribution of science, technology, and innovation is critical to achieving the sustainability goals. Considering the value-oriented, outcome-motivated nature of sustainability, previous literature [2–6] has established a link between sustainability education and transformative learning. Its basic idea is to motivate change that addresses complex problems involved in the sustainability goals [7]. Strategic and operational decisions are generally taken at the individual level or the team level of individuals. Thus, individual competency of practitioners is essential for achieving the sustainability goals. Wiek et al. [8] suggested a framework of key individual competencies for sustainability, including system-thinking, embracing diversity and interdisciplinarity, interpersonal competence, action competence, and strategic management. Wesselink et al. [9] linked these competencies to core tasks in business sustainable practices and figured out research directions for developing individual sustainability competency in both academic and business communities. Consequently, there is a call for studies on exploring practical pedagogical approaches to promoting students' sustainability competency in engineering education.

Design-based learning is a constructivist pedagogy that inspires students to solve real-life challenges and reflect on the learning process by applying design activities. It has proven to be an effective approach for acquiring engineering expertise [10]. Building on theories in constructivism, the integration of principles and skills required for complex problems (e.g., sustainable practices) can best be learned by doing. The transition towards more learner-centered curricula has become a worldwide trend in engineering education [11]. As a result, an increasing interest emerged in both problem-based learning and project-oriented learning [12,13]. The design-based learning was developed from the two active learning approaches, borrowing the principles of learner-centered pedagogy [14]. Similar to dealing with sustainability challenges involved in a design task, a well-designed design-based learning activity guides students through the stages of orientation, communication, implementation, and productization, which are highly relevant to the students' competencies. Therefore, the design-based learning seems to be a promising approach for developing ISC in engineering education.

In contrast to previous efforts, this paper focuses on proposing a design-based learning (DBL) approach to fostering students' sustainability competency, which includes not only teaching procedures, but also performance evaluations. We conducted two studies with engineering undergraduates by applying the proposed design-based learning (DBL) approach in two educational programs, a sensor technology course and an innovation project. This paper will explore three primary research questions, as follows.


The paper is organized as follows. The second section gives a background on sustainability competency and DBL. The third section proposes the DBL approach, including materials, procedures, and measures. The fourth section outlines the findings. The fifth section discusses the three research questions based on results from our studies. The sixth section presents the conclusion and further research directions.

#### **2. Background**

#### *2.1. Key Competencies in Sustainability*

The business community is playing a pivotal role in sustainable development, as enterprises increasingly acknowledge the importance of responsible and sustainable practices to their legitimacy and competitiveness [15]. Enterprises voluntarily combine their economic benefits with environmental and social concerns when formulating business strategies [16]. Such voluntary action has the potential to enhance their business competitiveness. However, the problems like environmental protection, employment promotion, and industrial upgrading cannot be addressed in a unilateral way. That is, a company's sustainable development usually faces various challenges, where each problem should be analyzed in its specific context and time frame [17]. The complexity involved in sustainable practice further increases due to the often conflicting values and standpoints between multiple stakeholders, such as enterprises, governments, and non-governmental organizations [18].

In a company, practitioners (e.g., managers, engineers) address these complex problems in sustainability through innovative efforts. Hesselbarth and Schaltegger [19] called them "change agents", and they emphasized the importance of individual competencies for advancing the flexibility and adaptability of business operations to meet changing challenges in sustainable development. Over the past few years, ISC training has received increasing attention in both academic and business practices. A series of sustainability competencies [8,20] is summarized as system-thinking, embracing diversity and interdisciplinarity, action, interpersonal communication, strategic management, etc. Considering that these competencies find their origins in educational literature, Wesselink et al. [9] empirically explore the abilities as to which of them facilitate practitioners to implement core tasks in

a specific context and time frame. Lans et al. [21] identified the competency frame constituting the heart of sustainable entrepreneurship, and revealed the feasibility of developing these competencies in higher education. Hermann and Bossle [22] proposed a pedagogical frame based on bibliometric analysis to cultivate students' entrepreneurial competencies in business education.

Current studies mainly concentrated on the conceptualization of sustainability competency and the content of sustainability in business education. As promoters of technology research and development, engineers and business managers are equally important for achieving a corporation's sustainability. However, a research gap still exists in developing practicable pedagogies to foster students' ISCs in engineering education.

#### *2.2. Design-Based Learning Pedagogy*

Design-based learning (DBL) was initially proposed by Gijselaers in 1996, based on a problem-oriented and project-based learning model [23]. Wijnen et al. [24] highlighted six characteristics of a well-designed DBL activity, including professionalization, activation, cooperation, authenticity, creativity, integration, and multidisciplinary. This work suggested directions for further developing and integrating the DBL within educational programs. Puente et al. [25] framed the DBL characteristics in five dimensions, thereby also improving its definition. Table 1 gives a summary of the DBL characteristics.

These characteristics of DBL were summarized from various empirical studies on DBL-similar practices in engineering education. Consequently, these defined characteristics were taken as a theoretical construct to create a DBL activity, and a few studies began to emerge. Swan et al. [26] explored a collaborative, design-based approach to improving the learning effects in core courses of an online program. Gómez Puente et al. [27] boosted teachers to use the DBL theoretical framework to create teaching activities and proposed an immersive learning model to enhance teachers' professionalism. Baran and Uygun [28] outlined eight DBL principles to foster the competency of translating technological, pedagogical, and content knowledge into action in teacher education contexts. Royalty [29] presented a design-based teaching framework and cataloged a series of relevant variables gathered from senior engineers and design-thinking teachers. Qattawi et al. [30] discussed the DBL implementation by combining engineering design theories with cooperative learning skills.

Existing research exhibits the potential benefits of DBL for promoting the competencies of system-thinking, multidisciplinary applications, and collaboration. However, it is still not enough to prove that the DBL is an appropriate and practical tool for teaching sustainability competency in engineering education. Firstly, it requires further measuring of the teaching effects on the ability to handle complex problems in sustainability. Secondly, there exist challenges in integrating DBL activities into traditional engineering courses, such as conflicts with existing instruction objectives and schedules. Furthermore, to the best of our knowledge, previous research on exploring practical DBL approaches to developing the sustainability competencies of engineering students is very little.



#### **3. Methodology**

Considering the benefits of design-based learning (DBL) and the lack of relevant research, the purpose of this study is to explore a teaching approach for the integration of DBL activities in engineering educational programs. Specifically, we present the three research questions outlined in Section 1. To answer these questions, we implemented two separate studies with engineering undergraduates in typical educational activities, which are a sensor technology course and an open-topic innovation project. Stakeholder questionnaires and performance evaluations are used to measure the DBL approach. The research sketch is shown in Figure 1.

**Figure 1.** A sketch of the research content and implementation.

#### *3.1. Participants and Educational Activities*

#### 3.1.1. Study\_1 in a Sensor Technology Course

The participants in Study\_1 were 129 full-time junior students majoring in mechanical engineering from four classes at an application-oriented university that provides undergraduate and continuing education programs covering engineering, natural science, economics, literature, etc. We implemented Study\_1 in the required course of sensor technology, which is offered to undergraduates of various engineering majors, such as mechanical, electromechanical, electrical, etc. As one of the critical technologies in advanced information technology, sensor technology [31,32] has become a theoretical method and design tool that engineers must master. From the view of developing students' sustainability competencies, the reasons for selecting this course over others were (1) the presence of a practicum that is appropriate for integrating a DBL activity, (2) the requirement of the students to consider product performances from chip level to system level, and 3) multidisciplinarity, covering micro-electromechanical systems, precision machinery, material engineering, data analysis, etc.

The sensor technology course contains two teaching units: Theory teaching (32 hours/8 weeks) and concentrated practicum (2 weeks ). We proposed an image sensor design problem at the board level as the design task for this practicum. It comes from a real-world challenge [33] that requires students to explore a way to improve the imaging quality of the sensor under inevitable uncertainties. As shown in Figure 2, the image sensing module with an ultra-low-noise image sensor was developed for surveillance cameras under extremely low-light conditions. In this module, the image sensor and other components (e.g., codec, converter) were assembled on a printed circuit board (PCB), and the module can be fixed in a device through the mounting holes at the four corners. Due to the mismatch in the thermal expansion coefficients of the various materials, thermal deformation occurred on the sensor of this module under the combined action of self-heating and thermal environment. To acquire more image information under low-light conditions, the sensor with a large-format die was selected, resulting in an imaging quality of the module that was more susceptible to deformation. In addition, we provided students with the context of the design task, including the specifications, power dissipation test results of the components (e.g., senor, converter, and codec), and material properties of the PCB. Note that the power dissipation and material properties (e.g., elastic modulus, expansion coefficient) were uncertain; hence, we provided the sample data.

**Figure 2.** The image sensing module with an ultra-low-noise image sensor. (**a**) Top surface of the module; (**b**) bottom surface of the module.

The students were required to perform this task in random groups. Each group contained three members, and every eight groups shared one instructor. At the end of the practicum, each group needed to submit a design report, which should contain at least the following items: (1) Problem statement, (2) performance measurement, (3) normative model, (4) graphic representation, and (5) design validation. We defined these evaluation items based on the design elements listed in Table 1. The five items were evaluated on a five-point scale and given the same scoring weight.

#### 3.1.2. Study\_2 in an Innovation Competition

The activity in Study\_2 refers to instructing students to participate in the 5th China College Students' "Internet Plus" Innovation and Entrepreneurship Competition (IPIEC) [34]. The participants in Study\_2 were actively seeking our guidance, and the total enrollment was 78. These students were sophomores or juniors from the majors of mechanical engineering and electromechanical engineering. The instructor team in Study\_2 included four members with expertise in electromechanical system design and manufacturing.

Having been held successfully four times since 2015, the IPIEC has gradually grown into the largest innovation and entrepreneurship award for university graduates in the world. Student members of participating projects must be currently registered graduates or graduates who graduated from university within the past five years. A three-level evaluation system is adopted: College preliminary, provincial rematch, and national final. Each project team needed to prepare a business plan and a five-minute presentation. According to the entrepreneurial stage and equity characteristics of the participating projects, the racing tracks of the IPIEC consist of a creative group, start-up group, growth group, and teacher–student co-creation group.

In the preliminary round for our university, there were a total of 1449 participating projects, of which the participants in Study\_2 provided eight. These projects were registered in the creative group because its entry prerequisites emphasized good ideas, such as an innovative product prototype or original service model. After all, our purpose was to develop and measure the DBL approach for sustainability competency training, rather than to start a real business. From this perspective, the evaluation rules were very appropriate for verifying the teaching effect. The evaluation items of the creative group include five aspects, as listed in Table 2.


**Table 2.** The evaluation items of the Creative group in the "Internet Plus" Innovation and Entrepreneurship Competition (IPIEC).

#### *3.2. Procedures*

#### 3.2.1. Implementation of the Sensor Design Task

The sensor technology course was scheduled for a two-week practicum after eight weeks of theory teaching. Before the practicum, the instructors delivered the task statement (see Section 3.1.1) to students for background reading. The instructor guided the students to focus on outcomes as they performed the task. Specifically, we divided the implementation procedure into three phases: Exploration, formulation, and productization. The timeline is shown in Figure 3.

**Figure 3.** Timeline of the sensor design task in Study\_1.

The outcome exploration refers to converting a real-world challenge into an easy-to-understand problem for computers. For example, the challenge of this task is to improve the imaging quality of the sensor under inevitable uncertainties, and the anticipated outcomes are mathematical models of the imaging quality and uncertain parameters. The exploration phase consisted of three steps: introduction, discussion, and summary. For aiding students to integrate concepts from earlier courses, the contents of introduction covered: (1) the description of the image sensor design task, (2) the flowchart of thermal–mechanical coupling analysis based on the finite element method (FEM) [35], and (3) the concepts in uncertainty modeling and reliability analysis, including the statistical moments, six-sigma principle, and safety factor [36]. The contents had been used many times in a similar practicum offered to students majoring in mechanical engineering, which were compiled by a teaching team with expertise in FEM and structural reliability analysis. After the two-hour introduction, the instructors organized the group discussion. To guarantee outcomes, we encouraged students to use a structured expression; that is, to first express the opinion in one sentence, then provide evidence to support the opinion, and finally give a practical suggestion. At the summary step, each group needed to state the outcomes of their discussion, including current doubts, anticipated difficulties, and different views and consensus on the performing strategy of the task. In the end, each group was required to submit an action plan outline covering the labor division and timeline to enable completion of the task within two weeks.

The outcome of the formulation was defined as building a normative model, which can be solved by existing tools to obtain a design solution. The instructors took a series of measures to train the students' sustainability competencies during task implementation. (1) In system-thinking, the instructor guided the students to create a systematic design framework consisting of controllable variables, uncontrollable parameters, objectives, and constraints. When members proposed different design objectives, the discussion was encouraged on the topics of transforming some objectives into constraints, or introducing weights for integrating different objectives into the one. (2) In embracing diversity and interdisciplinarity, we encouraged the use of existing tools and concepts taught in other courses, such as the FEM solvers [35], optimization toolkit [37], and statistical analysis tools [38]. We emphasized the establishment of data interfaces between existing tools for efficient integration. (3) In interpersonal competence, the instructors suggested each member provide a quantified consequence of the work undertaken for embedding the personal efforts into the team outcomes. Quantifying outcomes has proven to be an effective way to strengthen the openness and trust in cooperation. Clarifying everyone's contribution to an overall goal could help to reach consensus in communication. (4) In action competence, each group was required to report the current progress and conduct self-assessment based on the submitted action plan at the end of each day. We used QQ group [39] as an online group-chat tool for students' feedback in task implementation. QQ group is a free cellphone application, which is useful for team communication and sharing files. In this manner, the instructors viewed the students' daily reports and text conversations about the task, and gave just-in-time teaching to motivate students to take further actions.

Productization refers to embedding the solution into practice. To provide students with a productive experience, they were required to not only answer challenging questions from a practitioner, but also, finally, to convince the practitioner that their solutions can work in practical engineering. We invited an engineering expert with expertise in design and manufacture to participate in the final phase of the practicum. Since this class was four hours in length, this allows adequate time for communication. This process enabled face-to-face interaction between the theoretical researchers and engineering expert, and hence provided an opportunity to embed the concept that productization brings value to design deeply into students' cognition. For example, a solution to enhancing sensor robustness through controlling the power consumption fluctuation may be of no value for device providers, but is instructive for integrated circuit designers. The students needed to respond to the expert's comments one by one and to improve the design solution in the final design report.

#### 3.2.2. Preparation for the IPIEC

To evaluate the effectiveness of the DBL for teaching sustainability competency in engineering education, we utilized mentorship of students to participate in the IPIEC as a case study. In this process, we purposefully selected a specific innovation field and racing track to train students' sustainability competencies. Firstly, since the instructors were specialized in the design and manufacture of electromechanical systems, the selected innovation field was advanced manufacturing [40], including smart factories, intelligent hardware, internet of things, environmental protection, etc. Secondly, the racing track was selected for the creative group because its rules place more emphasis on the individual competencies demonstrated by an innovative project. Thus, the results of the competition may verify our students' competence for all participants.

The DBL activity ran a full semester with eighteen weeks. The teaching procedure adopted a framework similar to that of Study\_1. The flowchart and outcomes are shown in Figure 4. Differently from the topic-specific task in Study\_1, the participating teams needed to choose their entrepreneurial projects. Considering that the participants in Study\_2 were engineering undergraduates, they first were exposed to an online business course describing entrepreneurial strategy and sustainability as well as change management. The first and most crucial step in entrepreneurship is to clarify the market boundary through data analysis, that is, the opportunities and challenges of the industry. The primary reading material was a practical handbook [41], which addresses how to achieve business value from data analysis. The students were required to collect the primary data from interviews with practitioners and literature surveys. The outcomes included unmet user core demands and corresponding target market capacity. QQ group was used as an online communication tool where the discussion was carried out and all material was submitted.

**Figure 4.** The flowchart and outcomes of the DBL in Study\_2.

In the next formulation phase, the project team needed to create a product prototype or service process to address the industry challenge. For the defined entrepreneurial theme, we invited an expert from the corresponding industry to join each project team as a consultant. Under the guidance of mentors and consultants, the students developed a project outline. The description of the project outline is detailed in Table 3. Included in the project outline were the project theme, target users, industry challenges, product descriptions, competition analysis, milestones, and development planning. The project outline is the essential project management tool used to capture the clear deliverables and timeline. As they developed and updated the project outline, the instructor and consultant followed three guiding principles: Autonomy, authenticity, and visualization. (1) Autonomy principle: The instructor and consultant motivated students to autonomously develop the project outline through a lecture-by-demand strategy, challenging questions, online questionnaires, and formative feedbacks. The students learned from updating the project outline and reflecting on lessons to reinforce just-in-time action awareness and strategic management skills. (2) Authenticity principle: The proposed innovation points should focus on users' real needs, rather than pursuing so-called sophisticated technologies and leading advantages. We emphasized that practical efforts to support sustainable development are solving problems in engineering practice, not talking about the concepts of sustainability. (3) Visualization principle: We asked the students to use more graphics and diagrams in the project outline to avoid large paragraphs of text. The students gained excellent communication

skills and expression habits by creating logical, beautiful pictures and charts, and their understanding of the project was further improved.


**Table 3.** The description of the project outline in Study\_2.

In the final productization phase, each student team transformed the project outline into a final business plan and prepared a five-minute presentation as a deliverable for participating in the IPIEC. We organized a mock oral defense, and the committee members included all instructors and consultants. The consultants who failed to arrive at the site participated in the meeting via video link. Afterward, the students supplemented materials and analysis to integrate the comments and suggestions into the business plan and improve their presentation. Consequently, they developed eight innovative projects for the IPIEC through the DBL approach. These projects involved various fields, such as fitness assistance, home medical, smart home, intelligent building, agricultural processing, and environmental management. Table 4 provides a feature summary of the participating projects.


**Table 4.** The feature summary of the participating projects.

#### *3.3. Measures*

#### 3.3.1. Stakeholder Questionnaires

At least three major stakeholder groups are related to the pedagogical research, that is, students, instructors, and practitioners. We utilized three questionnaires to gain insight into measuring the proposed DBL approach. The specific information of these questionnaires is listed in Table 5.

The first questionnaire asked the students in Study\_1 and Study\_2 questions, including their experience in learning sustainability competency, self-perceived importance of this learning, and their perception of the DBL approach. The second questionnaire was delivered to 35 practitioners for investigating their background and attitudes towards training sustainability competency. These practitioners covered a variety of occupations, and included engineers, patent agents, project managers, sales, and business owners. We provided them with the design reports submitted by two groups. The students in Group\_1 were the participants in Study\_1. The students in Group\_2 were the previous students (Group\_2) in a similar course with the passive learning approach [42]. Note that we randomly selected the current and previous reports, and the sample size for both groups was 40. The two groups of design reports were delivered to the practitioners for blind evaluation. Based on the provided reports, these practitioners intuitively judged whether they would be willing to work with these students in the further.

The first two questionnaires contained six closed-ended questions with the same scale of "Yes" or "No". Unlike these questionnaires, the third questionnaire contained two open-ended questions about the gains and challenges of integrating the sustainability competency teaching into engineering education from the instructors' perspective. Six instructors responded to the third questionnaire, and all of them were involved in Study\_1.


**Table 5.** The feature summary of the participating projects.

#### 3.3.2. Performance Evaluations

Two performance evaluations were performed based on the submitted design reports in Study\_1 and the deliverables in Study\_2. To compare the proposed DBL approach with the passive learning approach [42], the design reports of the two groups were blindly evaluated based on the same scoring criteria as described in Section 3.1.1. Group\_1 was the DBL group in Study\_1, while Group\_2 contained 100 random samples of previous students in a similar course with the lecture-based approach. The graders include not only the instructor, but also the invited expert in Study\_1. The instructor and expert respectively gave the grades for each report, and the final grade was the average of the two.

For another evaluation of Study\_2, the results were naturally generated from the three-level evaluation system of the IPIEC, as mentioned in Section 3.1.2. From the IPIEC's evaluation items, as listed in Table 2, it can be found that students' entrepreneurship is a key element to ensure their outstanding performance in this competition. The literature [22] pointed out that there are commonalities between the competencies of entrepreneurship and sustainability, such as system-thinking, complex problem-solving, and interdisciplinarity. Therefore, it makes sense to use the IPIEC results to measure the sustainability competencies of students in our DBL group.

#### **4. Results**

#### *4.1. Questionnaire Results*

In Questionnaire\_1, We received 123 responses for Study\_1 (response rate: 95.3%) and 78 responses for Study\_2 (response rate: 100%). We observed that most students had no experience in learning sustainability competency. Only three respondents in Study\_1 and two respondents in Study\_2 reported having prior experience from elective business courses or off-campus internships. Most respondents agreed that it is important to learn sustainability competency, and the proportions were 107/123 in Study\_1 and 78/78 in Study\_2. Students' ratings on the willingness of extending the DBL to other courses showed that 81 respondents (65.8%) in Study\_1 and 62 respondents (79.5) in Study\_2 answered "Yes".

In Questionnaire\_2, 32 respondents provided valid responses (response rate: 91.4%). More than half of the respondents (20/32) reported having no experience in learning sustainability competency. Almost all respondents (31/32) stated that they had plans to attend a course related to sustainability competency. Figure 5 shows the ratio of answering "yes" to Question (6) for each respondent based on the design reports from Group\_1 and Group\_2. The ratio means are 71.4%. Thus, the results show the practitioners favored working together with the students in Study\_1.

**Figure 5.** The ratio of answering "yes" for each respondent on the third question in Questionnaire\_2.

Striking in Questionnaire\_3 was the perceived overlap between the six respondents. The overlap of gains in teaching sustainability competency was found in two aspects. Four out of six respondents reported that performing the DBL approach in the course provided an opportunity to build the relationships between the school and corporations. These relationships can bring potential benefits for developing and reforming engineering curricula as well as placing students in internships and jobs. Five out of six respondents mentioned another gain, that teaching sustainability competency with the DBL approach also improved their own competencies. For example, through various perspectives on the design task exhibited in the discussions between the students and invited practitioners, the instructors perceived that their competencies of system-thinking and embracing diversity were improved to some extent. The overlap of challenges reported by the respondents was threefold. Firstly, all respondents noticed that a small number of students were still less engaged in the DBL activity, although the well-designed course had intrinsically motivated most students. Secondly, four out of six respondents mentioned possible obstacles to promoting DBL teaching, in that not all faculty believe in its benefits and are willing to change, since classroom learning through a traditional lecture mode based on passive learning techniques is still the norm. Thirdly, half of the respondents mentioned a significant concern about identifying corporations who are interested and have budgets for school–enterprise cooperation.

#### *4.2. Evaluation Results*

Figure 6 details the mean of students' scores for Group\_1 and Group\_2 in Study\_1. We verified these mean values by statistical hypothesis testing at the significance level of α = 5% [43]. The results showed that the mean total score (3.76) of Group\_1 was significantly higher than that (3.03) of Group\_2 (*p* < 0.01). By comparing the mean scores of the two groups in the five evaluation terms, we found that Group\_1 had significant advantages (*p* < 0.01) in terms of (1) problem statement, (2) performance measurement, (3) normative modeling, and (4) graphic representation. A significant difference was not found in (5) design validation (*p* < 0.36).

**Figure 6.** The mean of students' scores for the two groups in Study\_1.

Since the evaluation results of Study\_2 were generated from the evaluation system of the IPIEC, we present the system backgrounds in the college preliminary and provincial rematch. A total of 120 universities, 88,496 projects, and 215,827 students signed up to participate in the preliminary round of Hunan province. For our university, there were 1449 competing projects, including our eight projects in Study\_2, as shown in Table 4. A total of 19 projects won from the preliminary round in our university and were approved for the provincial rematch. The organizing committee of the IPIEC organized experts to blindly review the deliverables of the 915 projects recommended by the universities for the first round of the provincial rematch. As a result, 341 projects were approved for the second round. For the eight projects in Study\_2, five out of eight projects entered the first round, then four out of five projects entered the second round, and, finally, the four projects won the third prizes. These projects were the Smart Yoga mat, portable ventilator, home health monitoring system, and bamboo mat automated production line.

#### **5. Discussions**

In this section, the three research questions presented in the introduction section are discussed based on results from our studies.

• Is the DBL approach viable for fostering students' sustainability competency in existing educational programs?

The questionnaires show that students participating in two studies had little experience in training sustainability competency through educational programs or self-study. Although they lacked experience, the students agreed on the importance of learning sustainability competency within undergraduate courses. Furthermore, our questionnaires for practitioners found similar knowledge gaps. The practitioners reported a high likelihood of taking sustainability competency courses in the future. These findings indicate the need for undergraduates and practitioners to foster sustainability competency with practical pedagogy.

On the other hand, we applied the DBL approach to the two typical educational activities in engineering education: The topic-specific course with a strict timeline in Study\_1 and the open-topic project with a loose deadline in Study\_2. The results show the practicability of the DBL approach in the existing educational program. Therefore, the viability of the DBL approach is verified for training sustainability competency in terms of learning demand and teaching procedures in engineering education.

• What are the stakeholders' perceptions of using the DBL approach for training individual competency?

The questionnaires show that more than half of the students in the two studies developed positive attitudes towards the DBL approach. A higher percentage of students in Study\_2 were in favor of extending the DBL approach into other engineering courses. The reason may be that the students in Study\_2 were active learners, while the students in Study\_1 were passively involved in the educational activities. Active learners may prefer the DBL approach and be more aware of its advantages in developing individual competencies. We also found the potential benefits of the DBL approach in enhancing student employment. The questionnaires show that the practitioners were more willing to work with the students in Study\_1 because of better individual competencies exhibited in their submitted design reports. Furthermore, although several gains from applying the DBL to teach sustainability competency occurred, the instructors reported challenges from students, teachers, and corporations.

• What are the effects of the DBL approach for the development of individual competency?

The previous studies [8,9] created the sustainability competency frame and established a direct link between the competency development and core tasks in sustainable business practices. Our findings in Study\_1 extend the previous studies by showing the advantages of the DBL approach for enhancing students' sustainability competency. Through comparing the outcomes of students in the DBL group (Group\_1) to those in the group of conventional lecture-based approach (Group\_2), we found that students in Group\_1 gave more prominence in problem statements, performance measurements, normative modeling, and graphic representations. The differences in the mean scores of the four evaluation items were 38.5%, 36.1%, 17.2%, and 37.1%, respectively. It shows that the DBL approach was more beneficial in fostering students' sustainability competency. For example, exploring problem statements and issues of measurement requires students to think systematically about real-world challenges and to formulate action strategies by embracing diversity and interdisciplinarity. Logical graphic representation and normative modeling have proven to be an efficient communication tool in engineering practice.

Our findings in Study\_2 qualitatively show the contribution of the DBL approach to developing students' sustainability competency. Promotion in the IPIEC involving tens of thousands of competitors was a challenging task, which required participants with strong sustainability competencies. Through the DBL activities, our students developed several competitive entries covering the fields of smart hardware, intelligent building, and advanced manufacturing. The implementation of this open-topic task consisted of identifying industry challenges, defining innovative products, building collaboration, and developing sustainable plans. The results of the IPIEC showed that such experiences had contributed to students' sustainability competency, such as system-thinking, complex problem-solving, and interdisciplinarity.

#### **6. Conclusions**

When dealing with the challenges involved in corporate sustainability practice, business decisions are made at the individual level or at the team level of individuals. The academic community increasingly acknowledges the importance of practitioners' sustainability competency. This paper presents a DBL approach for fostering students' sustainability competency in engineering education. This pedagogy was tested in two typical educational activities: A topic-specific design task with a strict timeline and an open-topic project in an innovation competition. We performed two studies to evaluate the viability and stakeholders' perceptions of the proposed approach, and also compared its performance to a conventional passive learning approach.

In answer to the first outlined research question, the DBL approach was found to be viable for teaching sustainability competency in terms of learning demand and teaching procedure in engineering education. The findings of our questionnaires answered the second research question about the stakeholders' perceptions of using the DBL approach for training sustainability competency. It shows that students and practitioners both agreed on the importance of integrating ISC training into an educational program, and were in favor of the proposed approach. In addition, we summarized the perceived overlaps between instructors about the gains and challenges in teaching ISC with the DBL approach. Finally, we found that the DBL approach was more beneficial in fostering students' sustainability competency compared to the lecture-based approach. In the process of performing the design task and developing innovative projects, students guided by our approach gave more prominence in system-thinking, embracing diversity and interdisciplinarity, interpersonal competence, action competence, and strategic management.

However, using this approach for teaching sustainability competency may face challenges from three aspects. Firstly, the type and level of students' motivation for participating in a DBL activity may have a moderating effect on the effectiveness. For example, passive students were less engaged, resulting in a low level of gain. Secondly, the feasibility of incorporating the DBL approach for teaching sustainability competency in engineering education is significantly affected by teacher resources. It could be challenging to organize enough teachers every semester to run the practicum unit in all engineering courses similarly to that of sensor technology.

The limitations of this research are relatively clear. This study focuses on the specific mechanical engineering domain in undergraduate education, though it does not explore the effects of the proposed DBL approach in other engineering or educational fields, such as electrical engineering or continuing education. Therefore, the practicability of this approach to these domains cannot be confirmed without further research. This study also involved limited samples with the data analysis at a course level. Despite the presence of the qualitative analysis at an institutional level, a quantitative analysis could provide more insight into the features of using the proposed approach for training sustainability competency.

In the future, we will apply the discussed DBL approach to other engineering courses, such as electrical engineering and civil engineering. In addition, extending this approach to continuing education will be considered. Furthermore, we could conduct additional studies for each element of this proposed pedagogy. More efforts should be made to develop teaching materials and performance measuring tools.

**Author Contributions:** Conceptualization, Z.H.; methodology, Z.H.; software, A.P.; validation, A.P.; formal analysis, A.P.; investigation, Y.H.; resources, Y.H.; data curation, Y.H.; writing—original draft preparation, Z.H.; writing—review and editing, T.Y.; visualization, A.P.; supervision, S.D.; project administration, Z.H.; funding acquisition, T.Y. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by TEACHING REFORM RESEARCH PROJECT OF EDUCATION DEPARTMENT OF HUNAN PROVINCE (grant number 2018593).

**Acknowledgments:** The social science association of Yiyang City, through Project 2019YS015, provided administrative and technical support for this work.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

*Article*

## **Learning Science in Primary Education with STEM Workshops: Analysis of Teaching E**ff**ectiveness from a Cognitive and Emotional Perspective**

#### **Milagros Mateos-Núñez, Guadalupe Martínez-Borreguero \* and Francisco Luis Naranjo-Correa**

Department of Science Teaching, University of Extremadura, 06006 Badajoz, Spain;

milagrosmateos@unex.es (M.M.-N.); naranjo@unex.es (F.L.N.-C.)

**\*** Correspondence: mmarbor@unex.es; Tel.: +34-924-289-300

Received: 26 February 2020; Accepted: 10 April 2020; Published: 12 April 2020

**Abstract:** Several international institutions emphasize the need to develop a comprehensive education in STEM areas (Science, Technology, Engineering, Mathematics) to improve learning, competences and student perception of these subjects. The general objective of this study was to analyze the teaching effectiveness, from a cognitive and emotional perspective, of a STEM workshop versus an academic-expositional methodology in the science classroom in primary education. The research design was quasi-experimental with a control group, an experimental group, a pre-test and two post-tests. By means of a randomized probabilistic sampling, 256 students between 10 and 12 years old participated. Cognitive, emotional, attitudinal and gender variables were analyzed according to two teaching methodologies, an expository academic methodology for the control group and an active methodology based on the development of a practical STEM workshop for the experimental group. The results reveal that both methodologies are equally effective in short-term learning, but statistically significant differences are found in long-term learning, in favor of STEM workshops. Likewise, the STEM workshop mainly generates positive emotions and attitudes in the students compared to the transmission-reception methodology applied with the control group.

**Keywords:** STEM education; primary education; science teaching; active teaching methodologies

#### **1. Introduction**

The new educational perspectives focused on scientific literacy seem to agree on the importance of science and technology teaching to consider not only conceptual content as educational objectives. There is also a need for those other objectives that have to do with the processes of science, its implications for technology and society (STS), or the shaping of personally and socially important attitudes, values, and rules [1,2]. However, currently, science education poses great challenges for teachers, who must not only respond to the demands of how to teach and bring to the science classroom the curricular proposals, but also find the most adequate way to connect with students so that they learn meaningfully and develop the skills, attitudes, and values that they will need in the world they will face [3,4].

More specifically, changes in society demand science education that is consistent with new realities where people know how to access acquired knowledge and produce new information using the knowledge they have gained [5]. However, the lack of practical work-based teaching strategies in primary schools does not contribute effectively to the acquisition of scientific or technological skills, which is reflected in students moving on to secondary education [6–8]. This also leads to a decline in students' scientific vocations, since according to some authors [9], there is a strong positive relationship between students' experiences with science in school and their choice of future studies in STEM disciplines (Science, Technology, Engineering and Mathematics).

Some studies [10,11] explain that education should not be reduced only to basics such as academics, information gathering and processing, or strictly cognitive development, but should also include the emotional dimension, since both dimensions affect the teaching and learning process. In fact, one of the main causes of the lack of interest in scientific disciplines is due to a negative attitude towards science, as indicated by some authors [12,13]. In this line, other researches argue that scientific activities that are attractive to students can generate positive emotions and social interactions, as opposed to teaching methodologies that are mainly focused on the acquisition of theoretical knowledge of a certain complexity and little connected to real life [14,15].

In order to address this problem, in recent years, the traditional academic-expositional models, based on the transmission of theoretical knowledge, have been complemented by other active teaching models in which the experiences that give rise to the active construction of knowledge take precedence [16]. However, recent studies [17] show that the use of traditional teaching methods continues to predominate in science and technology education, even though it is known that these strategies induce students to adopt a passive role, do little to foster student interest and produce high levels of academic failure, especially in scientific subjects such as physics and mathematics [15,18]. On this basis, it is considered necessary to create and study new resources and methodologies that facilitate and motivate student learning in scientific and technological areas in the early stages of their education [19,20].

In line with these approaches, educational programs focused on STEM education have aroused the interest of politicians, researchers, teachers and students concerned with improving and gaining access to better scientific literacy [21]. Today, STEM education is widely accepted as a method that synthesizes mathematics, engineering, technology, and science for critical thinking, creativity, innovation, and real-world problem solving [22,23]. Specifically, some authors [24] define STEM education as an approach in which students are taught content in science, technology, mathematics and engineering across disciplines in contexts involving real-life problems to enrich their learning and scientific literacy. Thus, research suggests that schools that focus on STEM education have a positive effect on student learning and STEM skills. For example, findings from other studies [25] confirm that students who participate in STEM programs perform better on math and science tests than those who do not participate in such programs. These authors also conclude that students in STEM programs are more likely to specialize in STEM subjects in higher education and even to choose careers in these areas. In addition, other research [26] found that attending STEM programs increases the likelihood that students will improve their math and science proficiency in high school, enhances participation in extracurricular STEM activities, and increases interest in science careers and aspirations for higher degrees in these areas. Likewise, some studies [27] have shown that STEM schools have a positive effect on the average grades of students in STEM subjects and, therefore, can significantly influence academic performance in later years.

Given this scenario, the vision that teachers have about such aspects takes a great role. It is essential to know the trends in the beliefs of science teachers in initial training regarding this type of challenge, and their willingness to incorporate them into science and technology teaching [4]. The position that science teachers in training take on the aspects covered by teaching methodology will influence their innovative capacity and their willingness to create favorable contexts capable of promoting learning in the terms established by current trends [28]. However, according to other authors [29], the beliefs of future teachers regarding active teaching models are strongly influenced by their life experiences as teachers and students. For this reason, guiding teachers in initial training to recognize the need to innovate in science and technology teaching in the direction mentioned above, requires that during the training process there be an adequate relationship between theory and practice so that the future teacher can make didactic decisions based on this new educational paradigm [4].

On the other hand, several studies recognize that the affective dimension must be considered and encouraged in science education and, therefore, must be part of its educational background [30], since it has been shown that the affective and cognitive domains are mutually conditioned. Emotions influence learning while learning outcomes influence emotions [31].

Considering the relationship between the affective-emotional dimension and learning, numerous studies have shown that one way to generate positive emotions in students is to implement hands-on activities [15,32,33]. Along these lines, we agree with [34] that STEM education programs can be decisive not only in learning, but also in the attitude and commitment of students towards STEM subjects.

As a result of the need to promote the development of scientific literacy and to involve students in the learning of STEM areas, in this research, a STEM workshop was designed and implemented with primary school students in order to analyze its teaching effectiveness in the face of academic-expositional teaching from both a cognitive and emotional perspective. In this sense, we consider the concept of teaching effectiveness in the framework of this research as the usefulness of the intervention carried out in the classroom. Specifically, an intervention is effective or useful from a didactic point of view, if it contributes to student learning (cognitive dimension) or to improving students' emotions and attitudes towards the subject being taught (affective dimension). To understand the term cognitive dimension or cognitive ability, it is necessary to consider the development of the cognitive theory of learning. Cognitive theory argues that knowledge is constructed from the student's immediate environment in an active and meaningful way, since learning involves cognitive processing of information rather than mere mechanical memorization of information [35,36]. The emotional dimension is linked to the previous dimension since, according to some authors [37], emotions are also closely linked to the teaching-learning of concrete knowledge. There are a variety of taxonomies for referring to emotions [13]. One of the most accepted in the field of didactics of experimental sciences is that provided by [38], who indicates that emotions are not only reactions to the stimuli of the present, but are also produced by the memory or evocation of events that happened in the past or by the anticipation of possible future situations. Consequently, we assume that emotions have a psychobiological part [39], but they are also a social construction [40] interconnected with context and culture [41].

#### **2. Materials and Methods**

The research design was quasi-experimental with a control group, an experimental group, a pre-test and two post-tests. The didactic methodology used was selected as an independent variable, and the learning achieved by the students at the end of the didactic intervention, the emotions and attitudes expressed by the participants, as dependent variables. With the control group, a more traditional teaching methodology was used, based on the expository academic model for the explanation of the contents under study. Specifically, it was based on presentations and theoretical explanations. During these sessions, the students intervened and argued their ideas by asking the questions they considered appropriate and discussing the situations posed by the teacher. However, with the experimental group, a more active methodology was used, focused on the development of a practical STEM workshop following the indications of previous works [42]. Specifically, a STEM workshop was designed to learn about primary education issues related to forces and movement. Specifically, the selected contents of the science curriculum were forces, motion, deformations of bodies and Newton's laws. The aim was to analyze the influence of the use of two teaching methodologies, on the one hand, in the learning of the selected STEM contents, and on the other hand, in the affective and attitudinal domain of the students. Likewise, the aim was to verify whether the concepts learned through the different teaching methodologies used persist over time, or whether, on the contrary, they are forgotten.

#### *2.1. Objectives*

The main objective of the research carried out was to analyze the teaching effectiveness, from a cognitive and emotional perspective, of a STEM workshop versus an academic-expositional methodology in the primary education science classroom.

This general objective was broken down into the following specific objectives.


Based on the proposed objectives, the following hypotheses were formulated:

**Hypothesis 1 (H1).** *The participating sample presents a low level of initial knowledge in the contents under study.*

**Hypothesis 2 (H2).** *There are statistically significant di*ff*erences in the level of knowledge of the students after the implementation of the didactic interventions compared to their initial level of knowledge, regardless of the methodology applied.*

**Hypothesis 3 (H3).** *The implementation of STEM workshops facilitates meaningful, long-term learning for primary school students compared to expository academic intervention in the science classroom.*

**Hypothesis 4 (H4).** *The implementation of the STEM workshop with the experimental group mainly generates positive emotions in primary education students compared to the transmission-reception methodology applied with the control group.*

**Hypothesis 5 (H5).** *The implementation of the STEM workshop with the experimental group mainly generates positive attitudes in the primary education students compared to the transmission-reception methodology applied with the control group.*

**Hypothesis 6 (H6).** *Participating male students show a higher level of knowledge of the selected STEM content than participating female students throughout the study.*

#### *2.3. Sample of Research*

The participating sample in the research was selected based on a random and probabilistic sampling. Specifically, it consisted of 256 students belonging to the academic levels of 5th and 6th grade of primary education, aged between 10 and 12 years. These students were divided into two groups, control and experimental, which were homogeneous and equivalent in terms of ability, discipline and academic performance in previous years. Table 1 shows the distribution of the participating sample according to group and gender.


**Table 1.** Distribution of the sample by gender.

#### *2.4. Instrument and Procedures*

A pre-test and two post-tests were designed as measurement instruments, to assess the variables referred to both the cognitive and affective domains based on previous research [15,43]. First, the pre-test was implemented as a previous step to the teaching-learning process of the selected contents. Later, the implementation of the didactic methodologies was carried out both in the control group and in the experimental group. At the end of the didactic intervention in both cases, post-test I was performed. Finally, in order to know the long-term learning results, post-test II was carried out by the students months after the explanation of the contents. The basic strategy for the application of the instruments consisted of giving the tests to the participants personally, making clear the anonymous and voluntary participation and the confidentiality of the information. The measurement instruments were the same for the control group and the experimental group.

The pre-test was composed of multiple-choice questions with four options for the answer, where only one was correct, referring to both conceptual and procedural contents related to the contents selected for the research. Specifically, the contents of the selected science curriculum were forces, motion, deformations of bodies and Newton's laws. Similarly, the post-tests, both post-test I and post-test II, consisted of a section with 12 multiple-choice questions of a theoretical and procedural nature to analyze the level of knowledge of the students after the didactic interventions developed. Additionally, a section was included in these instruments to measure the emotional and attitudinal variables of the subjects. Specifically, and based on previous research [13,15,43], 8 emotions were included, 4 positives and 4 negatives. The selected emotions were curiosity, fun, confidence, satisfaction, disgust, boredom, worry and anger. The students had to indicate whether they had felt each emotion during the teaching interventions. Finally, in order to assess the students' attitudes, 10 statements were included on methodological, learning and self-efficacy aspects related to what was discussed in the classroom. The difference between post-test I and post-test II was that post-test II was passed on to the students several months later, to check whether they remembered the contents learned or had forgotten them over time.

It should be noted that the process of measuring the teaching effectiveness of the interventions carried out with the students was the following, based on the definition of teaching effectiveness considered in the framework of this research (i.e., the usefulness of the intervention carried out in the classroom). The teaching effectiveness of the interventions carried out in the classroom is measured based on two variables, one cognitive and the other affective. On the one hand, the variable level of knowledge was quantified before the intervention and on the other hand, it was quantified after the didactic intervention, at two different moments, using a pre-test and two post-tests. The increase in the level of knowledge variable indicates the learning achieved by the students in the different interventions carried out. It is considered that the didactic intervention was useful or effective if there are statistically significant differences between the initial and final state of the student's level of knowledge in post-test I, and if the content learned by the student is not forgotten over time (results of post-test II). That is, if meaningful and long-term learning takes place in the students. Likewise, it was considered in the framework of this research that a didactic intervention is effective if it produces an improvement in the affective dimension of the students. To this end, the emotions that the student expresses before and after the didactic interventions were measured. Thus, if an increase in positive emotions or a decrease in negative emotions is produced, the intervention is considered to be effective from a didactic perspective in the affective domain.

As an example, Table 2 shows some questions to assess the level of knowledge of the students.


**Table 2.** Examples of questions to assess the level of knowledge of students.

#### *2.5. Validation of the Evaluation Instrument: Calibration Indexes*

To validate the questionnaires concerning cognitive domain, several psychometric tests were carried out based on various studies [44–46]. Specifically, statistical tests were conducted focusing on the assessment of questionnaire items such as difficulty index and discrimination indexes. Correlations were estimated using the point biserial coefficient and Ferguson's Delta. Finally, the reliability of the instrument was calculated by means of the Kuder-Richardson Formula 20. For all these calculations, the formulas specified in the previous studies were used. Table 3 shows the values obtained and the recommended values [44–46] of the calculated indexes. All values are within the recommended range.


**Table 3.** Psychometric analysis of the questionnaire developed.

The mean difficulty index (P) indicates the degree of difficulty of the questionnaire. This index was calculated for all the questions, obtaining values in all of them within the established ranges. We can see in Table 3 that an average value of P = 0.77 is obtained, so the degree of conceptual difficulty of the instrument is adequate for the research.

With respect to the discrimination indexes (D), the discrimination index 1 (D1) was calculated, which measures the discriminatory power of the questionnaire. That is, it indicates whether the questionnaire can distinguish those subjects with a more solid knowledge who answer correctly, from those subjects whose understanding is weaker. The value obtained was D1 = 0.44 which indicates a correct discrimination index. The discrimination index 2 (D2) indicates the proportion of successes in the group of students with better grades in relation to the total number of successes. It can be considered satisfactory if it is at least higher than 0.50 and in this case, this fact is fulfilled in all questions. Specifically, a value of D2 = 0.65 has been obtained, considered as good by the literature.

The point biserial coefficient (r) reflects the correlation between the scores of the subjects on one item with the scores on the whole test, and its range is [−1, +1]. If an item is positively correlated with the entire test, it means that subjects with high total scores are more likely to respond than subjects with low total scores. The average point biserial coefficient of the questionnaire is r = 0.46, so it also meets the recommended criterion.

Another source of evidence about the discriminatory power of the questionnaire calculated was Ferguson's Delta (δ). The literature indicates that a test that offers good discrimination power will have values of δ greater than 0.90. In this case, as can be seen in Table 3, the value obtained was δ = 0.92, so the questionnaire offers good discrimination power.

Finally, as shown in Table 3, the value obtained for Kuder-Richardson 20 coefficient was 0.67. According to the literature, this indicates an adequate reliability value for the instrument used.

#### *2.6. Design of the STEM Workshops*

STEM workshops have many positive aspects in the educational process [42,47] but students also have an important role as being responsible for their own learning. For the design of the workshops with the experimental group, the following guidelines were considered:


Once the general guidelines were set out, the workshops held with the students are briefly explained. In the STEM workshop, two models were built. The first model, "Action-Reaction Car", was mainly used to experience Newton's laws with the students. With the second model, "Elasticar", mostly contents related to deformations caused by forces are explained. Figure 1 shows an image of the models made by students belonging to the experimental group.

**Figure 1.** Photographs of some models built in the STEM workshops.

#### **3. Results**

This section shows the results obtained in the research. IBM SPSS Statistics 20.0 software was used for data analysis and subsequent interpretation of the results. Two types of analysis were performed, a descriptive-exploratory analysis and an inferential analysis. In the case of inferential analysis, the Student's t-parametric test was used, considering the appropriate tests of normality. It should also be noted that in all the tests, a significance level of 0.05 was considered.

#### *3.1. Results Obtained in the Pre-Test*

The realization of the pre-test was based on all those studies that indicate that to address the conceptual errors of students, teachers must first know their previous ideas and conceptual schemes [48–50].

The results obtained in the pre-test suggest that the students were familiar with the contents set out in general terms, as the average score achieved was above the minimum average. The selected contents were studied in previous years by the students. Table 4 shows the descriptive statistics referring to the control group and the experimental group extracted from the pre-test.


**Table 4.** Pre-test descriptives (Variable: Study group. Control group vs. Experimental group).

These results can be justified with the previous explanation of the contents by the tutor before the intervention with the STEM workshops, i.e., the respondents had previously worked on the curriculum in class. However, it is assumed that some previous erroneous ideas are still held by students, since, coinciding with [51], at the end of the syllabus, there is often evidence of misunderstanding of the most fundamental concepts, and errors of interpretation continue to be made in the study of physical phenomena, even when they are taught repeatedly. Thus, the descriptive statistical analysis carried out by questions determined that there was a clear lack of understanding in concepts such as friction force (question 4 of the pre-test), deformations produced by forces (question 6 of the pre-test) but above all, it was observed that the students did not know how to apply the theoretical contents to real procedural situations.

On the other hand, an inferential statistical analysis was carried out to check the existence of statistically significant differences in the initial level of knowledge between the control and experimental groups. The Student's t-parametric test was chosen when it was verified that the conditions required in that test were met. The results obtained are shown in Table 5.

**Table 5.** Student's t-test in the pre-test (Variable: Study group. Experimental Group vs. Control Group).


Table 5 shows that there are statistically significant differences between the control group and the experimental group in relation to the initial level of knowledge, since the significance obtained was 0.005. Specifically, there is an average difference of 0.67 points out of 10 in favor of the experimental group that will be considered in the analysis of the level of knowledge after the didactic interventions. The above data imply the rejection of Hypothesis 1 raised in the research "The participating sample presents a low level of initial knowledge in the contents under study" since the cognitive results have not been as negative as expected (that is, less than 5 points out of 10 on average).

To conclude this section, the analysis of the level of knowledge carried out according to the gender of the students is shown below. This observation arises from the numerous studies that indicate that women show less interest and obtain lower scores on tests and standardized tests of conceptual domain related to STEM areas [52,53]. In this regard, Table 6 shows the descriptive statistics by gender and Table 7 shows the inferential analysis carried out to check whether there are statistically significant differences between the mean scores obtained by the two sets.

**Table 6.** Pre-test descriptives (Variable: Gender. Women vs. Men).



**Table 7.** Student's t-test in the pre-test (Variable: Gender. Women vs. Men).

The results shown in Tables 6 and 7 indicate that there are differences between the level of knowledge shown by boys and girls and, moreover, these differences are statistically significant (Sig. = 0.037) in favor of the female collective. In this sense, we can reject Hypothesis 6 "Participating male students show a higher level of knowledge of the selected STEM content than participating female students" because the opposite has been observed.

#### *3.2. Results Obtained in Post-Test I*

The following are the descriptive results extracted from the post-test I carried out by the students at the end of the didactic interventions. Table 8 shows the descriptive statistics obtained by the sample participating in this questionnaire.

**Table 8.** Post-test I descriptives (Variable: Study group. Control group vs. Experimental group).


Comparing these results with those of the pre-test, both groups have significantly improved their mean score with respect to the pre-test. However, this statement is corroborated by performing a Student's t-test. Specifically, a significance of Sig. < 0.001 (in favor of post-test I vs. pre-test) is obtained in the case of the experimental group and a value of Sig. < 0.001 (in favor of post-test I vs. pre-test) in the case of the control group when comparing the means of post-test I with those obtained in the pre-test in each group.

These results reveal that both didactic interventions have been effective, since the students in both groups obtain average grades higher than those shown in the initial level of knowledge. Likewise, the analysis by question reveals an increase in the average scores with respect to those obtained in the pretest in the different questions, that is, there has been a cognitive improvement in the students after the explanations, by increasing the average scores, for example, in the questions referring to deformations, friction force or the more procedural questions.

In addition, an inferential analysis was performed between the average scores of the post-test I of the control group versus the experimental group. Several statistical tests were previously performed in order to choose a parametric or non-parametric mean contrast. The results suggest a choice of parametric tests so the Student's t-test for independent samples has been used for the comparison between groups. The results are shown in Table 9.

**Table 9.** Student's t-test in the post-test I (Variable: Study group. Experimental Group vs. Control Group).



As shown in Table 9, there are statistically significant differences between groups (Sig. < 0.001). However, this difference of 0.83 points out of 10 in favor of the experimental group is evident since the starting point or level of initial knowledge was not the same in the two groups, as obtained in the pre-test. Although the results suggest that the active participation of the experimental group during the explanation of the contents has facilitated, to a greater extent, their acquisition, the data seem to indicate that the two didactic interventions developed have increased, in the same way, the initial level of knowledge of the students, considering that both are equally effective from a didactic point of view. This allows us to accept Hypothesis 2 proposed in research: "There are statistically significant differences in the level of knowledge of the students after the implementation of the didactic interventions compared to their initial level of knowledge, regardless of the methodology applied". To check the effect size of the statistically significant differences found in post-test I of the control and experimental groups, we calculated Cohen's delta value, represented by d [54]. Specifically, a value of d = 1.77 was obtained. This result reveals an effect size classified in the literature as high.

On the other hand, to verify the possible influence of the gender variable in the results, the descriptive statistics by gender are presented in Tables 10 and 11, and the inferential analysis carried out to check if there are statistically significant differences between the average scores obtained according to this variable.


−1.244 248 0.215 −0.28982 0.23296 −0.74866 0.16901

**Table 10.** Post-test I descriptives (Variable: Gender. Women vs. Men).

Although the data shown in Table 10 indicate that girls scored higher in post-test I than boys, the inferential analysis in Table 11 confirms that this mean difference is not statistically significant (Sig = 0.215). This allows us to reject again, as it happened in the pre-test, the Hypothesis 6 "Participating male students show a higher level of knowledge of the selected STEM content than participating female students", since no difference in the level of statistically significant knowledge is found according to this variable.

#### *3.3. Results Obtained in Post-Test II*

Finally, in order to validate the long-term teaching effectiveness of the STEM workshops compared to the academic-expositional methodology, the students participating in the study carried out a third questionnaire (post-test II) several months after the intervention, since the didactic validity of both methodologies in the short-term was demonstrated with the results shown in the previous section (results in post-test I).

Table 12 shows the descriptive statistics obtained by the participant sample in the post-test II carried out by the students months after implementing the designed didactic interventions in the classroom, to check whether the methodologies used promote meaningful and lasting learning.

**Table 12.** Post-test II descriptives (Variable: Study group. Control group vs. Experimental group).


An inferential analysis was carried out to see if there were statistically significant differences in the results of post-test II depending on the study group variable. Table 13 shows the results of the Student's t-test obtained.

**Table 13.** Student's t-test in the post-test II (Variable: Study group. Experimental Group vs. Control Group).


The results shown in Table 13 reveal that there are statistically significant differences (Sig. < 0.001) between the mean scores obtained by the two study samples. Specifically, a mean difference of 1.42 points out of ten is observed in favor of the experimental group. To check the effect size of the statistically significant differences found in post-test II of the control and experimental groups, we calculated Cohen's delta value, represented by d [54]. Specifically, a value of d = 1.89 was obtained. This result reveals an effect size classified as high in the literature. These results seem to indicate that the intervention based on the STEM workshops has been more effective from a didactic point of view than the methodology used with the control group, suggesting the agreement of Hypothesis 3 proposed in the research (Hypothesis 3: The implementation of STEM workshops facilitates meaningful and long-term learning in primary school students in the face of an expository academic intervention in the science classroom). However, in order to firmly confirm this, and to validate the long-term teaching effectiveness of the implemented STEM workshops, it is convenient to show the level of knowledge acquired by students in the three tests: pre-test, post-test I and post-test II in both the control and experimental groups. Tables 14 and 15 show the inferential analysis made from the statistical One-way ANOVA with post-hoc Tukey HSD Test.

Table 14 indicates that there are statistically significant differences between the pre-test, post-test I and post-test II questionnaires in both the control and experimental groups. Table 15 shows among which questionnaires these differences exist in both the control and experimental groups.


**Table 14.** One-way ANOVA.



\* Sig. < 0.05.

With respect to the control group, the statistically significant differences (Sig. < 0.001) are between pre-test and post-test I (in favor of post-test I) and between post-test I and post-test II (in favor of post-test I). In other words, the student improves his initial level of knowledge after the intervention (in the short-term) but forgets the contents over time (in the long-term). If we compare the results of post-test II with the initial level of knowledge (pre-test), we can see that there are no statistically significant differences between the average scores of these two questionnaires in the control group (Sig. = 0.979). These results reveal that these students return to their initial level of knowledge and, therefore, it is accepted that these students have not adequately retained the contents over time due to rote learning.

However, if we look at Tables 14 and 15 concerning the experimental group, we can see that there are statistically significant differences in all cases. Contrary to the control group, the results of the comparison between the post-test II and the pre-test carried out by the experimental group show that these students have preserved the memory of the contents in the long-term, since the average grade reached in the post-test II has been higher than the one obtained in the initial pre-test, finding statistically significant differences between the average grades of these two questionnaires carried out by said experimental sample (Sig. = 0.001). These results allow us to confirm the Hypothesis 3 stated in the research "The implementation of STEM workshops facilitates meaningful, long-term learning for primary school students compared to expository academic intervention in the science classroom".

The above data confirm the importance of using active practical methodologies that include student hands-on workshops in order to achieve meaningful and long-term learning of STEM content worked on in the classroom. Likewise, these results complement previous research which shows that hands-on learning is a way of enhancing meaningful learning because it favors mental constructions of more abstract contents [15,42,55–58].

Finally, as in the previous sections, Table 16 shows the descriptive statistics by gender and Table 17 shows the inferential analysis carried out to check whether there are statistically significant differences between the average scores obtained according to this variable.


**Table 16.** Post-test II descriptives (Variable: Gender. Women vs. Men).



If we look at Table 16, the descriptive analysis reveals again that girls show slightly better cognitive domain in the contents under study than boys, but the inferential analysis shown in Table 17 confirms that this mean difference is not statistically significant (Sig. = 0.527). These results allow us to finally reject the Hypothesis 6 proposed in the research "Participating male students show a higher level of knowledge of the selected STEM content than participating female students".

#### *3.4. Results of the Emotional and Attitudinal Analysis*

In this section, we present the data referred to the emotional and attitudinal analysis of the participating sample with respect to the intervention received in each case. Table 18 shows the results obtained in the different emotions analyzed, distinguishing by study group.


**Table 18.** Percentage of students who have felt the emotion.

As can be seen in Table 18, the analysis of emotions determined that the majority of primary school students show positive emotions when faced with the learning of STEM content, as other authors have pointed out in previous research [15,42,59,60]. However, it was observed that the experimental group showed a greater proportion of positive emotions than the control group, and this difference was statistically significant (Sig. < 0.05) in positive emotions such as fun, satisfaction or confidence.

At the same time, it seems logical to find high values in the emotion of curiosity in both groups, since the simple fact of receiving a class session from a person outside the educational circle draws attention and generates uncertainty and curiosity. This is verified in the qualitative analysis of the data, where statements are obtained from students belonging to the control group such as "I was curious when the new teacher arrived" or "Because I didn't know what I was going to do". Likewise, a large part of the participating students showed curiosity when some contents that were new to them were introduced in the session, such as Newton's laws ("With Newton's laws", "With the explanation of the laws" or "Learning the laws" are some of the statements made by some subjects of the control group and "With the Newton's" and "To know what Newton's laws were about" are some of the statements made by some students of the experimental group. However, this same group of students also expressed curiosity about the model that they were going to make, finding arguments such as the following: "When we were shown the car we were going to make" or "To know how the car was made". Curiosity is an engine that helps generate intrinsic motivation, as opposed to repetitive, mechanical or memorized tasks [61]. Providing learning strategies based on curiosity can increase students' dedication to work and deepen their scientific literacy [62].

On the other hand, it is evident that 90% of the students in the experimental group showed the emotion of fun during the intervention compared to 60% of the students in the control group. As several authors indicate, manipulative activities not only favor learning, but also make the teaching-learning experience fun for both students and teachers [15,33,63]. In reference to the fun emotion, some comments extracted from the experimental group were "When the car was moving", "When I was making the car with my classmates" and "When we were measuring the distance our car travelled in the corridor".

On the other hand, it is observed that the students of the experimental group show more emotions such as confidence or satisfaction for having been immersed in the elaboration of a model and having made it adequately. In this situation, the students of the control group have not been involved and therefore, most of these students have not experienced moments in which they have manifested such emotions.

Regarding negative emotions, it should be noted that emotions such as boredom contribute to the progressive loss of attention [64]. Considering this statement, it should be noted that almost 20% of the students in the control group indicated that they had experienced boredom at some point during the session. This may be due to the fact that the academic-expositional methodology applied to this group did not promote a relationship between the content and daily life, and did not encourage reasoning and active participation by the students as the STEM workshop did with the experimental group. For this reason, it is necessary to consider more active and practical learning strategies in the classroom,

since, as some studies point out [13], negative emotions are more often expressed during theoretical learning, and positive emotions through practical learning.

On the other hand, it seems logical that up to 19% of the students in the experimental group expressed greater concern than those in the control group. As observed in the qualitative analysis of the data, the latter were under pressure of not making the model well and that, for this reason, it could not work in the end. Opinions such as "When the car didn't work at first", "In case the car didn't work" or "Because we didn't know how to place the balloon properly" were some of the proposals made by the experimental group regarding the emotion of concern. The same thing happened with the emotion of anger. 8% of the experimental group expressed this emotion when they encountered obstacles during the STEM workshop. On the part of the control group, some students indicated that they felt this emotion because they had received a mere explanation of the scientific content dealt with.

Based on the results exposed in the analyzed emotional variables, we can accept the Hypothesis 4 proposed in the research "The implementation of the STEM workshop with the experimental group mainly generates positive emotions in primary education students compared to the transmission-reception methodology applied with the control group".

On the other hand, the attitudes of the students towards the intervention carried out were analyzed. Ten questions were posed to each group of students, adapted to the session received. However, in both cases, the questions were related to the acquired learning, the interest shown, the methodological preferences and the self-efficacy of the students. Students had to choose between two options (YES and NO) based on their considerations. The results are shown in Table 19.


**Table 19.** Analysis of student attitudes (percentage of Yes).

The results shown in Table 19 indicate that the students generally show interest in the teaching-learning of scientific-technological contents, since more than 85% of students in both groups expressed interest in the session received (statement 1) and more than 80%, in both groups, indicated having learned new concepts during the intervention (statement 7); concepts that, in addition, were not difficult to understand (statement 8). However, although in both cases, the students consider what they have learned, it is interesting to highlight that practically the totality of the experimental sample indicated that they had learned the contents compared to 87% of the students in the control group (statement 2).

In reference to methodological issues, it is noted that there is a strong preference for hands-on learning methods by the participating sample. 73% of students in the control group indicated that they would have liked to make a model related to the contents explained and more than 95% in the experimental group would like to make more models in the science and mathematics classes (statement 3). These results are linked to those obtained in statement 4, where it is observed that more than 80% of the students, in both groups, consider that the practical activities facilitate the learning of the contents, but they can also be related to the contributions of the experimental group in statement 6 that indicate that they will remember the contents better thanks to the workshop. Likewise, 50% of the students in the control group would have liked to make a supplementary model to the contents explained in the intervention (statement 5). Finally, the information extracted on the level of self-efficacy suggests that there are great differences between groups, since almost 70% of the experimental group considered themselves to be capable of making models only as opposed to 46% of the control group. Evidently, this is due to the technology and engineering skills and knowledge incorporated in the STEM workshop and acquired by the students in the experimental group.

The previous results allow us to accept the Hypothesis 5 proposed in the research "The implementation of the STEM workshop with the experimental group mainly generates positive attitudes in the primary education students compared to the transmission-reception methodology applied with the control group".

#### **4. Discussion**

After analyzing the data, it is assumed that students in the last levels of primary school show problems in remembering content about forces and movement that they have already worked on in previous years. This leads to the formation of conceptual errors that may have negative repercussions on the future learning of these contents [65]. In this line, we agree with other authors [2], that the main orientation of educational processes is usually based exclusively on the development of knowledge about concepts, principles and laws of scientific disciplines, forgetting or relegating other important areas of training.

In contrast, the data confirm that the application of constructivist and constructionist approaches has a positive effect on increasing the interest and involvement of participants [66]. The inclusion of simple experiences in the classroom to work on STEM content greatly favors learning and the consolidation of this content in the long-term [15,42], as can be seen from the results obtained in post-test II. In this sense, our results coincide with previous research linking hands-on activities with increased student knowledge and academic performance in STEM subjects [23,67,68]. In order to learn about a phenomenon in nature, it is necessary to experiment and explore how it manifests and, to do so, students need to become fully and actively involved with the phenomenon in order to understand it in depth [69]. Furthermore, the results found suggest that the use of pedagogical practices based on cooperative and authentic learning fosters a timely work and learning environment since the group work environment emphasizes student effort, improvement and mastery and helps students not only to feel safe but also to be competent in STEM [70].

Regarding gender differences, many studies show that girls tend to have a lower cognitive level in science than boys, regardless of their type of school and their age. They also suggest that boys show greater self-confidence in dealing with scientific problems than girls and, therefore, that the cognitive level acquired is higher in males than in females [71,72]. However, this study supports the opposite view, as it was found that girls showed a better domain of the contents discussed in all the cases analyzed. However, this cognitive difference is not statistically significant according to the data.

Furthermore, we agree with other researchers [73] that it is possible that participation in STEM activities may have a positive effect on student self-efficacy in STEM or even in a specific discipline related to STEM content, and therefore, continued participation in STEM activities will further enhance self-efficacy in these fields. Moreover, literature shows that self-efficacy predicts both academic and career-related choices [74], and therefore, improving self-efficacy within a specific content domain may increase the likelihood of choosing a career associated with that domain [75,76].

With respect to the affective variable, we agree with the latest contributions of neuroeducation that it is desirable for teachers at any educational level to pay attention to the emotions that are generated during the teaching-learning process [12,13]. After participating in the workshop, the students in the experimental group have developed an understanding of what science is and of the importance of scientific literacy, but they have also increased their motivation to study science and technology in the classroom [62]. Thus, it is necessary that the activities proposed to the students awaken their curiosity because students who are curious will focus their attention on the object that arouses it, improving their predisposition to learn and their desire to learn [77]. However, although it is essential to influence academic emotions through the choice of learning strategies because they can have an important effect on learning [78], we agree with other authors [79,80] that the simple experience of positive emotions is not enough to trigger interest and situational engagement. Enjoyment must be explicitly connected to the content of directed learning.

Teaching actions must take into account differences in the way learning is approached in order to adjust their action plan and thus optimize the learning of all students [81,82], but given the high number of scientific and technological contents that currently exist, the task of selecting them becomes increasingly complicated. However, in order to fulfil its function of helping the process of development and socialization of students, school education must consider the intrinsically constructive nature of the human psyche and build on it [83].

STEM teaching interventions aim to contribute to the improvement of science and technology education from an early age by providing teaching methodologies that deliver quality education. The STEM workshops were carried out with recycled material, thus promoting responsible consumption among primary school students. In this way, it is recognized that both scientific and technological knowledge should be part of citizenship worldwide, promoting sustainable attitudes and behaviors [84]. This type of methodology, which influences not only the cognitive but also the emotional domain of students, can contribute to the objectives of sustainable development within the educational context. In this line, the Sustainable Development Goals decided upon by the United Nations include an objective (SDG 4 "Quality education") focused on the acquisition by students of the knowledge and skills needed to promote sustainable development [85].

Today's education requires knowing what to do with information, that is, how to analyze it, cooperate with others to summarize it, apply it and communicate the results [86]. Therefore, quality education is no longer based primarily on the acquisition of knowledge but requires the development of skills and attitudes that ensure the proper use of information in accordance with the context. In this sense, we agree with [87] that promoting scientific literacy through STEM programs includes the development of skills that actively engage students in solving current societal problems such as changing stereotypes in education [88] or improving specific behaviors related to sustainability.

Today's social needs require that education be directed towards the development of sustainability and social justice [89]. However, working on these types of problems in the classroom to promote critical thinking, democratic values and the active search for solutions are aspects that imply the mastery of the necessary scientific knowledge [90]. Education for sustainability implies a different vision of the curriculum, pedagogy, organizational change and educational policies [91]. In this sense, we agree with other researchers [92] that the best way to develop these competencies in sustainability is through interdisciplinary work on scientific subjects. Consequently, STEM skills are part of the competence field required by 21st century citizens because they are oriented to develop a range of key competences that are essential for living and working in our society.

This research shows evidence that student learning can be improved if we incorporate a teaching style that connects scientific and technological content in the classroom [93,94]. In this sense, we consider that STEM programs help students to develop a better understanding of scientific and technological knowledge based on the results presented. Supporting communicative methodologies with STEM programs produces positive changes in students' cognitive and emotional development, fostering active participation and autonomy of students in science classrooms [42].

Likewise, there seems to be a high degree of agreement between the results of this study and the literature regarding the need to choose a common and reduced core of knowledge, of great intrinsic and educational value, related to the lives of students, that can be applied and be useful to them [19,20,95]. In this line, it is concluded that STEM teaching approaches offer students the opportunity to participate in practical and manipulative lessons that have a positive effect on students' knowledge, interests and skills, as well as on their decision to continue learning science and technology together.

#### **5. Conclusions**

To enhance long-term learning, it is important to use learning resources in a meaningful way, that is, connected and integrated within the thematic structure. Active methodologies are recognized as facilitating and promoting strategies for learning and critical thinking because they facilitate the transmission of messages and promote communication among students by relying on the application of participatory mechanisms. In this line, the contribution of the STEM workshops in the development of scientific-technological learning could be related to the stimulation of three important and significant processes that are generated in a coordinated way in this participatory field, namely, cognitive, attitudinal and socializing processes. The STEM workshops aim to enable students to develop epistemic, procedural and contextual knowledge and are geared towards developing the ability to act and acquire skills in relation to what is being learned. However, this educational paradigm is not only intended to develop intellectual skills, but also to encourage students to act in the experimental design itself and in the discussion about it, thus enhancing their ability to formulate hypotheses and to reason about them and to establish connections between some contents and others, thus causing an improvement in the scientific-technological vocations and in the students' attitudes towards the STEM areas.

Based on the results of the study, we believe that, in order to maintain attitudes and vocations throughout the educational process, it is necessary to rethink the possibility of applying this integration of content throughout primary education but also during compulsory secondary education, the baccalaureate and higher education, taking into account the comprehensive and functional nature that science teaching should have at those stages. However, the work of developing competences in others forces teachers to review their own competences. We consider the role of the teachers to be indispensable, since through their training and attitude they will be able to offer their students learning in a more playful and effective way. In this sense, strengthening the attention paid to specific didactics in general, and to the didactics of experimental sciences in particular, in permanent teacher training plans, becomes a challenge for education institutions to define the levels of achievement of competences and the processes of training and updating of future teachers in order to improve their knowledge, skills and attitudes to achieve greater efficiency in their professional practice [96].

Finally, we suggest that future studies explore the experiences of high school students in STEM workshops designed for these levels and further investigate the factors that may influence students' decisions to choose one educational path over another. Likewise, it would be interesting to assess the training of active teachers in order to gather information about the skills that should be strengthened in the training itineraries of these groups. In the meantime, the results of this study will be added to the growing literature on STEM learning environments and their influence on the cognitive and affective dimensions of students.

**Author Contributions:** Conceptualization, all authors; methodology, all authors; validation, all authors; formal analysis, M.M.-N. and G.M.-B.; investigation, all authors; resources, all authors; data curation, M.M.-N. and G.M.-B.; writing—original draft preparation, all authors; writing—review and editing, all authors; visualization, all authors; supervision, all authors; project administration, G.M.-B.; funding acquisition, G.M.-B. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Junta de Extremadura/Fondo Europeo de Desarrollo Regional, Research Project IB16068, Agencia Estatal de Investigación/Fondo Europeo de Desarrollo Regional, Research Project EDU2016-77007-R and Junta de Extremadura/Fondo Europeo de Desarrollo Regional, Grant GR18004.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Cultural Sustainability in Ethnobotanical Research with Students Up to K-12**

**Lorena Gutiérrez-García 1, José Blanco-Salas 1,\*, Jesús Sánchez-Martín <sup>2</sup> and Trinidad Ruiz-Téllez <sup>1</sup>**


Received: 16 June 2020; Accepted: 10 July 2020; Published: 14 July 2020

**Abstract:** In the achievement of the Sustainable Development Goals (SDGs), education plays a fundamental role. However, traditional methodologies do not favor the enrichment and personal development essential to promote global awareness. The use of active methodologies based on experiences improve the quality of learning. This work describes the design, implementation, and evaluation of the acquired knowledge of a didactic proposal for non-formal education as a support for regulated education based on botany content. Firstly, a workshop was held, where young people participated directly in developing field work with a real scientific methodology. Subsequently, a group of students was chosen to be interviewed to obtain a global vision of the learning they obtained. The motivation of the students was quite positive, which allowed us to obtain voluntary participation in the field work and also gave the students a participative attitude throughout the development of the workshops. Four months later, this positive attitude remained during their direct involvement in various activities, and the students still remembered the fundamental content discussed. Relating the didactic proposal to its immediate environment was shown to increase interest in learning and value in its own context. The results of this educational experience have been very positive, as knowledge was acquired, and interest in the preservation of the environment and the profession of a researcher was promoted.

**Keywords:** active methodologies; Sustainable DevelopmentGoals; non-formal education; Ethnobotany; learning assessment; STEM

#### **1. Introduction**

Today, society demands new educational methodologies that encourage the active participation of the student, unlike traditional methods where the teacher is the protagonist [1]. Another modern issue is sustainability. This concept is an indisputable need that must be integrated into multiple fields, including teaching [2].

Since 1992, UNESCO (United Nations Educational, Scientific and Cultural Organization) has recognized and promoted education as a basic right for the promotion of sustainable development, but it is in the new World Agenda for Sustainable Development 2030 that this vision is made evident. In this document, education appears both as a specific goal and as a means to achieve all the Sustainable Development Goals (SDGs) [3]. Despite its relevance, SDGs are still unknown by a large majority of the society; some examples of this have been recently published [4].

SDGs constitute not only a traditional agenda of "things-to-do", but a fully comprehensive plan for reaching a specific level and way of development, at global scale [5]. Due to that, it is not possible to achieve one of them and not another; the entire program must be pursued as a common and complex target [6].

Education is the basis for promoting development, but it is also essential for personal enrichment, endowing individuals with skills and values and enabling them to be more versatile, critical, and resolute. In other words, developmental education facilitates an understanding between globalization and development by helping promote awareness of "global citizenship" [7]. The success of education would entail improvements in employment and reduce depopulation—problems that mainly affect rural areas. It has no sense focusing only on cognitive aspects of education, as the education process itself has revealed to be an integral path of developing many other competences, skills and variables [8,9].

Here, the terms "education" and "sustainable rural development" intersect. Rural development refers to the need to establish a business fabric to make a territory strong and avoid problems such as population loss, emigration, aging, and poor economic diversity.

Initiatives are needed to adapt to the needs and constant changes that occur in society, thereby responding to the aforementioned problems, improving the standard of living of the inhabitants of rural areas, and helping to exploit and develop their sustainable environment [10].

The sociocultural context surrounding an individual is, in itself, a didactic technique that directly influences the learning process [11]. Students learn more effectively when using active methods and content similar to their daily lives and interests. Therefore, including traditional knowledge as an object of learning [12] can contribute to the perceived value of one's "own culture", which in turn is important for sustainable development, allowing people to harness their wealth to build their future, transforming it into processes of solidarity and popular economy [13].

Although many previous experiences already show the positive impact of non-formal education [14,15], there are few references that focus on an inclusive way to promote so-called integral sustainability [5]—that is, an integrative vision of sustainability beyond environmental and ecological aspects. Thus, the current work presents a didactic experience that is primarily related to three of the SDGs and some of their goals:


The scientific education of the population is essential for the transition to sustainability. Only in this way can we gain an understanding of the seriousness of the problems facing our planet and our way of life, as well as train future scientists capable of developing more efficient resources and to become citizens committed to achieving a sustainable society [17].

However, to accomplish the abovementioned goal, changes must be made in the traditional education system, moving from the present transmissionist tendency towards a more constructivist teaching pedagogy [18]. Teaching methodologies with the teacher at the center are still used in many disciplines, especially science. However, there is already evidence that the use of experience-based methodologies improves the quality of learning [19]. These principles are the basis of new science education approaches, such as the systematic integration of different science-related subjects in an entire knowledge construct as STEM or even including arts in STEAM.

Although human beings are born with the instinct to observe, discover and create, as we become adults, that desire to investigate becomes disconnected. Many young people have the preconceived idea that science is boring, difficult, and does not ensure a well-paid job in the future [20]. Among the main causes for a negative assessment of science is the organization of the educational system itself, as well as the poor relationship between the subject of study and daily lives of students [21,22].

As early as the 1980s, the need to develop the "scientific and technological literacy" of society was promoted to ensure that all citizens are able to make decisions on social issues in a democratic and responsible way [23]. Although actions have been taken since then, there is still a disconnect between what is planned and what is taught in the classroom. There is also a tendency to continue using traditional teaching methods [24]. An academic transformation at different educational levels is still needed, requiring new and more active and realistic teaching programs and methodologies.

Disinterest in science also affects the field of botany, showing a downward trend in botanical vocations in university. However, vegetables are an available and attractive teaching resource to promote a positive view of the sciences. This task is in the hands of educators. For students to develop an interest in science, the teacher must play a key role from an early age. To carry out this task, training in science is essential [25]. Sometimes, the scientific training received by teachers is limited, which represents an obstacle in the teaching of the natural sciences [26]. Direct collaboration between teachers and researchers is essential for the teacher's own training, as well as for the development of an integrated scientific curriculum [27].

The current article is focused on analyzing an educative experience that will be described below, with the following objective: To analyze the impact of a non-formal education activity on knowledge acquisition according to the framework for teaching Sustainable Development Goals (SDGs) conforming with an integral sustainability vision framework, as proposed by Zamora-Polo and Sánchez-Martín [5].

This general objective can be split into several research questions:

Question 1: Does a non-formal activity inserted into a standard academic space (a high school) have a positive impact on the knowledge acquisition of botanical and ethnobotanical concepts?

Question 2: Can a non-formal activity, if designed according to an integral sustainability vision framework, develop positive attitudes towards traditional knowledge preservation?

Question 3: Can a non-formal activity, if designed with close collaboration between secondary teachers and university researchers, promote the core concepts of the nature of science, such as what science is for and what kind of person a scientist is?

This research is structured in two phases. The first is the design and implementation of a didactic proposal that aimed to raise awareness among young people about the importance of preserving their ethnobotanical heritage and valuing it as a resource for sustainable development by involving the students in a realistic scientific study. The second phase examines, through interviews, the impact of this proposal on the achievement of the goals after a period of four months.

Consequently, this piece of research connects two traditional ways of scientific work: quantitative (stage 1) and qualitative (stage 2). This is made under the perspective of transversal research, which is quite common in ethnobotanical and athropological research [28].

#### **2. Materials and Methods**

#### *2.1. Relevance of the Location*

This work is part of a larger piece of research focused on the town of Hornachos, located in the autonomous community of Extremadura (Spain), adjacent to the protected area "Sierra Grande de Hornachos" included in the Natura 2000 Network [29].

The environmental richness of Hornachos is important [29] but is not the only reason we chose the area for this work. This environment brings together several features that make it especially interesting. On the one hand, it has great cultural richness resulting from its historical past featuring various cultures, such as Roman, Jewish, Arab, and Moorish. On the other hand, it has a great diversity of plants associated with traditional knowledge and uses [30].

However, this rich heritage is being lost due to the increase in intensive agriculture and globalization, leading to the abandonment of natural resources [31,32].

Many authors have already noted the importance of education as part of a developmental process to promote integral sustainability [33]. It is a challenge to include the conservation of rural areas and their ethnobotanical identities (understood as cultural and social heritage) in an educative proposal. This process requires the development of skills and experiences that enable the construction of learning based on an assessment [34].

#### *2.2. The Educative Center: Institute of Secondary Education (I.E.S), "Los Moriscos"*

The I.E.S., Los Moriscos, is located in the village of Hornachos, in the province of Badajoz (Extremadura) (Figure 1). This area was selected for its particular features, as Hornachos sits on the boundaries of the protected area of "Sierra Grande de Hornachos", which has several protection catalogs both regionally (Regional Zone of Interest (RZI)) and internationally (the Site of Community Importance (SIC), Special Protection Area for Birds (SPA), and Special Area of Conservation (SAC)), which are included in the Biodiversity Conservation Network of the European Union [35]. This area stands out for its cultivated varieties of wild plants with a rich historical–cultural legacy [36].

**Figure 1.** Location of the town of Hornachos in Extremadura, Spain. Source: Own elaboration.

It should be noted that the school in this area is a publicly owned rural secondary education institute created in 1997 and attended by students from various localities adjoining the protected area of "Sierra Grande de Hornachos", but the largest percentage is composed of young people from the town of Hornachos.

According to the corresponding data for the academic year of 2018–2019, the period of the first part of this study corresponding to the teaching workshops, the Center consisted of 368 students distributed at the Secondary Education levels, including those in a baccalaureate in the modalities of sciences, technology, humanities, and social sciences as well as those in the formative cycle of the middle degree, "Attention to people in situations of dependence". In addition, the Center had a staff of 48 teachers and 9 members of non-teaching staff working as computer scientists, administrators, concierges, social educators, etc.

This Center has a "low" socioeconomic level [37], and most of the students' families live off agriculture, livestock, or construction. In addition, the immigrant rate is low, and no significant differences were found in either socioeconomic conditions or cultural and religious ones.

#### *2.3. Participating Students*

The entire design of the educative intervention was done on the basis of a tight collaboration with the teachers. Based on a prior agreement with the Center's teaching and management team, K-9 students were distributed into 4 groups, totaling 72 young people between the ages of 13 and 16.

This level was chosen because of its near-adult cognitive development and proximity to the end of compulsory studies, as determined by the Spanish administrations [38]. Since it did not inconvenience the workshop objectives, the representation of students from locations other than Hornachos was not taken into account because of their small representation and also because this study was not interested in excluding certain students but in arousing interest in research, the value of one's own resources, and the promotion of sustainable development awareness in young people.

Following the ethics of data protection, identification of the interview participants is done using fictional names. In addition, all participants gave consent to be recorded and have their answers used as part of this academic research.

#### *2.4. The Educational Proposal*

A pedagogical proposal was made consisting of a workshop based on the scientific method and research applied to Ethnobotany called "Learning to research in our protected space—Sierra Grande de Hornachos". This proposal was part of the program "The Week of Science and Technology in Extremadura", an oriented program focused on K-8 to K-12 students from public or private centers, which has been held annually since 2017 and whose main objective is to promote scientific culture and its dissemination.

The didactic proposal was divided into 3 milestones, carried out over 3 days and lasting 2 h each, within school hours. The participation of the Center's director and faculty of teachers was necessary. Every day, the workshop was attended by all the selected students accompanied by a group of teachers (ranging from 3 to 4 teachers) who depend on shift work or whose schedules were dedicated to the workshop.

The content of the workshop focused on the significant interculturality that persists in knowledge linked to wild and cultivated plant species of enormous value [30].

The activities carried out are detailed below and summarized in Figure 2.

**Figure 2.** Graphic summary of the study process. Source: Own elaboration.

Milestone 1 (first session). Introductory talk on the work carried out by scientific staff, the importance of the promotion of science, and the scientific method, an exhibition of workshop objectives and programming, an explanation of fieldwork or practical workshops, and the delivery of data collection sheets and guidelines to follow.

Milestone 2 (second session). Collection of tokens and samples by students and resolving any doubts; practical activity: preparation of herbarium sheets with vegetable samples of *Arbutus unedo* L., a typical plant in the study area.

Milestone 3 (third session). Sharing and discussion of the results obtained from the collection of data carried out by students, highlighting the most significant plant species, and discussion and conclusions of the sessions.

Below is the field activity or hands-on workshops conducted by students outside the classroom. These tasks consisted of each student choosing one or two adults over the age of 70 to be treated as an ethnobotanical informant. The selected persons could be family, but they had to be linked to rural life and should, therefore, know the traditional uses associated with wild or cultivated plant species. Students had to ask about the traditional uses that local wild or cultivated plants might have in this particular area.

A model table was used for data collection. Each student was given two tabs (Figures 3 and 4) that served as a script and included informed consent informing them about the research and the purpose of the collection of the data, which all the interviewed adults signed. Each of these tables was identified with a different tab code, making it easy to identify later.


**Figure 3.** Data collection sheet provided to students (front).


**Figure 4.** Data collection sheet provided to students (back). Source: Own elaboration.

For the design of the data collection sheet, 10 subcategories of use were considered based on those described in the *Spanish Inventory of Traditional Knowledge Relating to Biodiversity* [39]. Each of these subcategories was quoted verbally during the workshops for each of the plants, while the interviewer (student) wrote the information given by the informants. The following subcategories relate to the types of uses that a plant can have for humans: 1. Human food; 2. Animal feed; 3. Medicine for humans; 4. Animal medicine; 5. Toxic or poisonous; 6. Fuel; 7. Used to make tools and other utensils; 8. Used for traditional parties or events; 9. Used as a building element; 10. Other uses.

The time spent on each interview was variable and depended on the work done by each student. The tokens, samples, and photographs provided were collected by the researchers during the second day of the workshop, and the data were digitized at the Botany Laboratory of the University of Extremadura to be exhibited and, in the last session, debated in the classroom.

#### *2.5. Data Analysis*

Both quantitative and qualitative mixed methodologies were considered [40]. In the analysis of the development of the workshop as well as the practical workshops carried out by the students, the quantitative logic was applied, while in the second part of this research, open interviews were conducted with a small group of student participants in the workshop whose results were qualitatively analyzed to assess the educational impact of the proposed activity. In this last case, the individuals were selected according to what the teachers suggested when they were asked for students that could comply the following conditions: (a) They attended the activity; (b) they were not especially brilliant students or had a clear and evident motivation towards the sciences; and (c) they came from the village and had the opportunity to connect with their own relatives in ethnobotanical research.

For the second part of this work, direct, unstructured interviews were used to avoid limiting the interviewee's responses [41].

Six students, 3 boys and 3 girls, were selected. All students had participated in the face-to-face workshop and also showed interest and involvement in the practical workshops, delivering truthful information on two plant species collected in the tokens provided to them along with a sample and/or a photograph of those plants as a sample.

The interviews were conducted individually in the month of March 2020 during school hours in the time dedicated to recess. This period was four months after the activity itself took place, so the responses and the whole conversation can be considered a memory of the practical workshop.

To facilitate comfort and conversation, 6 open-type issues were designed for obtaining information on the impact of the conferences held in the first phase and for determining their educational impact. These issues were the initial starting point for a fruitful conversation that was recorded and analyzed. Three fields of study or variables were thus established: 1. Learning about science, 2. Learning about botany, and 3. Emotional performance. Thus, the number of issues for each variable is 3. Over the course of the interviews, the interviewer used the pre-designed questions as a guide and was able to pose them in a different order or ask other secondary questions (but always keeping in mind the objectives).

The data were collected by using audio recordings and field notebook annotations. Two different experts analyzed each recording to identify the particular expressions for each variable. Afterwards, a third expert contrasted both codifications and made them reach a consensus for qualified data codification.

#### **3. Results and Discussion**

#### *3.1. Quantitative Data Analysis: The Workshop*

Curiosity and motivation are essential for learning. This is determined not only by the teacher's performance or the subject matter but also by the teaching methodologies employed [42]. Traditionally, methodologies have been promoted where the teacher is the protagonist and the student is a simple

recipient; however, more active methodologies involve the participation and direct intervention of the student in concrete or everyday experiences, which allows greater activation of knowledge [43,44].

During the development of the present workshop, the direct involvement of the students in the practical activities was varied. On the first and third days, where the methodology used involved speaking or discourse, the performance of the students was more passive, showing an increase in interest when the analysis and discussion of the results obtained from the fieldwork was carried out, suggesting that this work was a topic of interest to the students; in the second part, all the students became involved, although this was not an optional decision.

Moreover, in the workshop that took place outside the classroom, motivation was an important factor. Several authors argue that the choice of the subject of study is essential to achieve greater learning [45]. Further, not all students attending the workshop participated in the practical workshops held outside the classroom because such workshops were not mandatory. More than 29% (nearly a third) of students conducted interviews with their elders and collected the information in the pieces previously provided to them. Thus, a total of 21 students collected a total of 40 tokens and, therefore, almost all students collected information from two plants. Specifically, 19 of them collected information from 2 plant species, among which 4 students also delivered plant photographs and samples, and 2 students contributed a sheet with 1 species per person, one of which was accompanied by a photograph and vegetable sample.

Information was collected from 23 different plants—9 cultivated (*Aloe vera* (L.) Burm. F., *Citrus x sinensis* (L.) Osbeck, *Ficus carica* L., *Gossypium* sp., *Olea europaea* subsp *europaea* var. *sylvestris* (Mill.) Lehr., *Punica granatum* L., *Rosmarinus o*ffi*cinalis* L., *Solanum lycopersicum* L., and *Ziziphus jujuba* Mill.), and 14 wild plants (*Arbutus unedo* L., *Cistus ladanifer* L., *Crataegus monogyna* Jacq., *Cyperus* sp., *Daphne gnidium* L., *Erica* sp., *Eucalyptus camaldulensis* Dehnh., *Heliotropium europaeum* L., *Hypericum perforatum* L., *Quercus rotundifolia* Lam., *Retama sphaerocarpa* (L.) Boiss., *Rumex pulcher* L., *Thymus* sp., and *Urtica urens* L.), some of which appeared more frequently than others (Figure 5). Among the most represented were *Arbutus unedo* and *Thymus* sp., indicating that these species are of great interest to the ethnobotanical identity of Sierra Grande.

**Figure 5.** Frequency with which each species appeared in the collected tokens.

A total of 80 uses were obtained, the majority of which were, in order of majority of use: 2. Animal feed, 3. Medicine for humans, and 1. Human food (Figure 6). The maximum number of uses obtained for the same species was 8, and this figure was only achieved for *Arbutus unedo,* followed by *Thymus* sp. with 7 and *Crataegus monogyna* with 6. From the set of extracted information, we also observed that the most commonly used parts of the plant were the Leaves (L) and Stems (St), which were consumed from more than 13 of the 23 total plants. Least used were the Seeds (S), which were only used for three species (*Ficus carica*, *Retama sphaerocarpa,* and *Urtica urens*). Flowers (F) and Fruits (Fr) were also used for more than 40% of species (Appendix A).

**Figure 6.** Total uses obtained by subcategory.

The Autonomous Community of Extremadura is one of the Spanish regions with the fewest ethnobotanical studies [46]. Most studies focus on medicinal and veterinary plants [47–57]. However, few enclaves in the region have had in-depth studies on the traditional knowledge that communities possess about their plants. Two exceptions are the Monfragüe National Park, in the province of Cáceres [58], and Calabria and Siberia in Badajoz province [59]. For the Regional Zone of Interest (RZI), Sierra Grande de Hornachos, only one recent study has been done on edible plants [60]. Therefore, the results obtained by the students, which may seem modest, have high scientific interest since most of the uses collected are novel for the area and, in some cases, are novel for the region.

The results reveal that the students who participated did so with total involvement. In general, the group proved to be curious and interested in the plants and cultural richness of their environment, which is a very interesting source of knowledge.

#### *3.2. Qualitative Data Analysis: The Interviews*

Analysis of the six interviews offered interesting insights into the three study fields: nature of science, learning methods, and affective domain and emotional performance. Since a qualitative methodology was used, the most relevant results were obtained by examining samples from the recorded interviews. The entire text for these interviews is available upon request. The following section offers samples from the interviews.

#### 3.2.1. About Nature of Science (NOS)

These interviews included several specific questions on how the students understand some of the core concepts of the nature of science, such as: "What do you think science is? What is science for? What do you know about the Scientific Method?" and "What is a scientist for you?" Students had difficulties defining these concepts, which agrees with previous studies [61], but some understandings were clear:

#### "Science is important for society." (Individual 1)

"Science is knowledge, and there are many kinds of scientists." (Individual 3)

"Science explains reality." (Individual 3)

These students presented epistemologic statements about the focus of science and how it influences their lives. Furthermore, when asked about scientists and the scientific method, they said:

"Science method is a pathway, a procedure for researching." (Individual 2)

"A scientist has to have interest in details and be rigorous, careful ... Science cannot be made with fuzzy borders." (Individual 3)

"A scientist cannot be subjective, but objective." (Individual 4)

The interviewees easily identified some of the most relevant values of science in itself [62], such as objectivity, reliability, methods, and systematicity.

#### 3.2.2. About the Botanical Learning Method

An important insight of the current work is to determine how important the method is that we use to teach science, particularly botanical science. In this sense, the students again expressed ideas that were different from traditional ways of learning:

"I like plants. What I liked the most was the practical activity with the strawberry tree." (Individual 2) "[Making things directly by hand] is a very motivating way of learning. I felt ready to make new things and to learn." (Individual 3)

"Question: Is it easier to learn in this way? Answer: Yes, yes, much better. This is better than writing all the time." (Individual 5)

"The most boring part was the initial oral exposition." (Individual 6)

3.2.3. About Sustainability as An Integral Concept Involving Ethnobotanical Ideas

Cultural sustainability is not a very frequently taught subject when discussing sustainability concepts. Indeed, the relevance of cultural sustainability was only recently framed [5]; presently, it is not possible to talk about sustainability without mentioning cultural sustainability. In this sense, the interviews exhibited the acquisition of several interesting ideas:

"Question: Do you think it is important to preserve these uses for these plants (traditional medicine usages)? Answer: Yes, so in the future we will be able to know how the past was." (Individual 1) "I have learned some new information about plants and how to keep in contact with nature." (Individual 4)

"I have learned general culture about the plants that are around us in our town. We should be proud of living in such a beautiful place, with a lot of plants with interesting usages. This is part of our town and its history." (Individual 5)

3.2.4. Feelings, Self-Regulation, and Emotional Performance of the Learning Process

Lastly, the core dimensions of education, such as self-regulation of the learning process, and several emotional aspects emerged during the interviews, as can be seen in the following statements:

"The practical workshop helped me learn more vegetal species. I already knew some of them, but now I can identify more." (Individual 1)

"I felt happy; nothing made me feel bored." (Individual 2)

"I would recommend using these kinds of activities again during the course." (Individual 6)

Qualitative analysis of the interviews revealed effectiveness in the acquisition of many types of knowledge, not only knowledge related to botanical learning but also the understanding of ethnobotanical issues, the value of cultural sustainability, and even an understanding of various aspects related to the nature of science. In addition, since these interviews were carried out several months after the activity, they revealed a significant recall of related knowledge.

#### **4. Conclusions and Further Studies**

Considering the results obtained, we conclude that the use of training techniques and tools based on active methodologies provides an interesting resource to generate knowledge in symbiosis with traditional classroom teaching (i.e., for Ethnobotany in the "Sierra Grande" protected area of Hornachos).

This piece of research has revealed the significant and strong relationship between apparently disconnected disciplines such as ethnobotanical and science education. Further studies should be oriented in similar directions and can be guided by research questions such as "What relationship appears between talking about traditional knowledge and science education? Is it possible to raise the interest in science if we focus on scientific knowledge from our ancestors?" Hornachos and its environment, its richness and its potential natural resources surely could address these and other similar intuitions.

This work has shown that a non-formal activity inserted into a standard academic space (a high school) has a positive impact on the acquisition of botanical and ethnobotanical knowledge. Moreover, if designed according to an integral sustainability vision framework, students can develop positive attitudes towards traditional knowledge preservation. Further, if designed with close collaboration between secondary teachers and university researchers, this type of activity can promote the core concepts of the nature of science, such as what science is for, and what kind of person a scientist is.

**Author Contributions:** Conceptualization and methodology, J.S.-M.; software, L.G.-G.; validation, J.B.-S.; formal analysis L.G.-G.; investigation, J.B.-S. and L.G.-G.; resources, T.R.-T.; data curation, L.G.-G.; writing—original draft preparation, L.G.-G. and T.R.-T.; writing—review and editing, J.B.-S.; visualization, L.G.-G.; supervision, J.S.-M.; project administration, J.B.-S.; funding acquisition, T.R.-T. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Junta de Extremadura (Spain) and the European Regional Development Fund through grant number IB16003.

**Acknowledgments:** The authors would like to extend their sincere thanks to the town of Hornachos and the collaboration of the I.E.S. of "Los Moriscos" for their cooperation during the fieldwork.

**Conflicts of Interest:** The authors declare no conflict of interest.



#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

*Article*

## **Inter-Relations among Motivation, Self-Perceived Use of Strategies and Academic Achievement in Science: A Study with Spanish Secondary School Students**

#### **Enric Ortega-Torres 1,2,\*, Joan-Josep Solaz-Portoles <sup>2</sup> and Vicente Sanjosé-López <sup>2</sup>**


Received: 30 July 2020; Accepted: 18 August 2020; Published: 20 August 2020

**Abstract:** The relationship between motivation and the use of learning strategies is a focus of research in order to improve students' learning. Meaningful learning requires a learner's personal commitment to put forth the required effort needed to acquire new knowledge. This commitment involves emotional as well as cognitive and metacognitive factors, and requires the ability to manage different resources at hand, in order to achieve the proposed learning goals. The main objectives in the present study were to analyse: (a) Spanish secondary school students' motivation and self-perception of using strategies when learning science; (b) the nature of the relationship between motivation and perceived use of learning strategies; (c) the influence of different motivational, cognitive, metacognitive and management strategies on students' science achievement. The Motivated Strategies for Learning Questionnaire (MSLQ) was administered to 364 middle and high-school students in grades 7–11. For each participant, the academic achievement was provided by the respective science teacher. The results obtained from the Pearson product-moment correlations between the study variables and a stepwise regression analysis suggested that: (1) motivation, cognitive and metacognitive, and resource management strategies, have a significant influence on students' science achievement; (2) students' motivation acts as a kind of enabling factor for the intellectual effort, which is assessed by the self-perceived use of learning strategies in science; and, (3) motivational components have a greater impact on students' performance in science than cognitive and metacognitive strategies, with self-efficacy being the variable with the strongest influence. These results suggest a reflexion about the limited impact on science achievement of the self-perceived use of cognitive and metacognitive strategies, and highlight the importance of students' self-efficacy in science, in line with previous studies.

**Keywords:** motivation; learning strategies; science learning achievement; secondary school students; MSLQ questionnaire

#### **1. Introduction**

People's strategic efforts in learning and decision making are better understood when motivational and cognitive factors are considered together [1,2]. Recently, the neural bases of the interaction between motivation and cognitive control have been studied [3] (Yee and Braver, 2018), and physiological mechanisms by which motivation influences cognitive control have been described [4].

In their psychological approach, Ausubel proposed a particular relationship between motivational and cognitive factors as a pre-requisite for meaningful learning: the learner's personal commitment to put forth the required effort needed to properly process the learning materials, integrating the new information with prior knowledge and managing the resources at hand to foster comprehension. Involved in the commitment with learning, students' motivation seems to enable the activation of metacognitive and self-regulatory mental activity [5].

When students' motivational, cognitive and metacognitive factors are integrated, a significant effect on their academic achievement is observed [6]. Zimmerman [7–9] accounted for this fruitful motivation-cognition integration in their self-regulated learning model aimed at helping teachers to develop a more effective teaching.

In science learning, variated motivational components have proven to have considerable influence on science achievement [10–13]. Moreover, the relevance of the merging of motivation and learning strategy use was pointed out by Anderman and Young [14], Obrentz [15], and Bryan, Glynn and Kittleson [16]. Models integrating motivation and science learning in specific ways have also been proposed and tested. Miñano, Castejón, and Gilar [17] proposed a structural model aimed at examining to what extent the motivational variables interact with cognitive variables, such as intelligence or learning strategies, in predicting academic performance. The proposed model had a satisfactory fit, explaining 66% of the variance of academic achievement. Alpaslan, Yalvac, Loving andWillson [18] used structural equation modelling to explore the relations among personal epistemologies, self-regulated learning and achievement in physics. The model explained about 12% of the variance of students' achievement in physics. Sungur and Güngören [11] built a structural equation model, showing that students' perception of classroom environment concerning motivating tasks, autonomy support, and mastery evaluation were positively associated with motivational and cognitive components of self-regulation and science achievement. The model proposed by Lee, Lim and Grabowski [19] revealed that the combination of generative learning strategy prompts with metacognitive feedback enhanced learners' self-regulation, and this improved their recall and comprehension of the human heart system.

Summing up, emotional factors have to be considered as integrated with cognitive, metacognitive and management factors to understand and improve students' academic achievement. Among these factors, strategies play an essential role. Learning strategies can be defined as a sequence of specific activities that will enable the learner to gain new knowledge [20].

#### **2. Aims and Predictions**

In the present study, we aimed at obtaining data about Spanish secondary school students' motivation and the use of strategies when learning science in an integrated manner. We also aimed at relating students' integrated motivational and cognitive perceptions with their academic achievement in science. Up to our present knowledge, there are not similar studies in the Iberoamerican context. Our predictions were formulated in agreement with previous outcomes in different cultural contexts:

(P1) Motivation, cognitive and metacognitive, and resource management strategies will significantly influence students' achievement in science.

(P2) Motivation will act as an ability factor for the learning strategies: low motivation should be associated with the low use of learning strategies, (and then with a limited intellectual effort).

(P3) Due to the rational grounds of science knowledge, the cognitive and metacognitive strategies will have a greater impact on students' performance in science than motivation and management strategies.

Due to their subjective nature, emotions are difficult to assess in a direct way and thus, have to be assessed in an indirect way. One possibility is students' self-reports. The Motivated Strategies for Learning Questionnaire (MSLQ onwards), elaborated by Pintrich, Smith, Garcia and McKeachie [21], was proposed, with the goal of providing data about students' self-perception of motivation, cognitive, metacognitive and resource management strategies in an integrated way. This instrument has been used in about 150 research papers [22]. Its validity has been stated in diverse contexts using Turkish high school students [23], Uruguayan students of psychology [24], Mexican university students [25], and Hong Kong high school students [26].

Several studies relating some MSLQ dimensions to students' achievement in science and mathematics have been conducted in many countries with university students [27,28], high school students [12,29], and even with primary students [30,31].

#### **3. Methods**

#### *3.1. Research Design*

An ex post facto design was used in this study, since no manipulation of the variables occurred. The relationship between variables was retrospectively determined. Quantitative analyses were conducted using statistics when needed.

#### *3.2. Participants*

A sample of 364 male and female Spanish middle and high-school students enrolled in courses from 7th to 11th grade (12–18 years old) participated in the present study. In order to increase the students' diversity in the sample, the participants belonged to 7 high schools of different ownership (public, privately managed-cooperative, and privately managed-religious) located in the surroundings of Valencia. The socio-economic level associated to these areas is intermediate, the families having no important economic or social troubles. Although no participants showed unusual differential characteristics, the selection was random and thus, this sample does not represent the secondary students' population in Spain.

In each school, the classroom groups and the participants were not selected. Students were all invited to participate in educational research as anonymous volunteers, and informed that their collaboration would have no consequences on their marks. Informed consent was required from the families and school management. No previous selection of students was carried out, but all those included in the course groups participated in the present research.

#### *3.3. Instruments and Measures*

3.3.1. MSLQ: The Self-Perceived Use of Metacognitive Strategies and Motivation Was Evaluated with the Motivated Strategies for Learning Questionnaire (MSLQ)

The MSLQ contains three different sections:


Each item proposes an assertion, and the student has to assess it using a Likert scale of 7 points, where 1 means "Not at all true of me" and 7 means "Very true of me". The score of each scale is obtained by averaging the items involved.

#### 3.3.2. Academic Achievement

For each participant, the academic achievement was obtained directly from the science teachers in each secondary school. The traditional scale for academic marks in Spain ranks 0–10. These marks were the final ones that every school must provide to the government educational department, and are based on the same scale and the same published assessment criteria. Some non-controlled criterial differences among teachers/centers could appear, but of a limited extent, due to governmental control. In fact, globally the students' marks did not show significant differences in the different educational centers (Kruskal–Wallis: X2(6) = 6.849; *p* = 0.335).

The global mean value and standard deviation of students' marks were: M = 6.9; SD = 1.8. Participants' marks were not normally distributed (Kolmogorv–Smirnov test; *p* < 0.001); the skewness was non-significant (skew = −0.23; SEs = 0.13; skew/SEs = −1.77; *p* > 0.05), but the kurtosis was clearly significant (kurtosis = −0.073, SEk = 0.26; kurt/SEk = −2.9; *p* < 0.01). Differences with the Gaussian expected frequencies mainly appeared in extreme marks (more students than expected in 3.0–4.0 and 9.0–10), suggesting that teachers overestimated or underestimated students' achievement in some extreme cases.

#### *3.4. Procedure*

The MSLQ was translated into Spanish, and adapted to an electronic format, to facilitate the process of collecting responses from different educational centers. In each center, permissions were requested, and the science teachers were instructed on the administration of the questionnaire to their students.

The questionnaire was administered in a normal science class. Completion lasted less than 90 min, and typically took 65–80 min. One of the researchers was present in the sessions to clarify participants' doubts. Excel and SPSS 24.0 were used to collect and to analyze data, using descriptive and inferential statistics.

#### **4. Results**

#### *4.1. Self-Perceived Use of Learning Strategies in Science Learning*

The scores for the global questionnaire and for every section were obtained by averaging the corresponding items. According to the Kolmogorov–Smirnov test, the distribution of the global mean value (all the sections included) was not significantly different from a normal distribution (*p* > 0.20), with an average of M = 4.81 and Standard Deviation of SD = 0.64. Quartile values were placed at values 4.41; 4.85; 5.26. The scores for the components in the three sections of the MSLQ are shown in Table 1.

Participants reported using more motivational components than cognitive and metacognitive strategies or resource management strategies. The most used components in science learning (according to students' self-perceptions) were GO and EO (both motivational), and the less used were R (cognitive) and PE (resource management).

Table 2 shows the product-moment correlations between pairs of components obtained in the sample. The r values ranged from −0.07 to 0.74, and most were significant. The components with higher multiple correlation with the other components were MR (RMR = 0.85); E (RE = 0.81); and SE (RSE = 0.81); and the components with lower multiple correlation were TA (RTA = 0.46); SO (RSO = 0.50); and LB (RLB = 0.57).


**Table 1.** Mean values obtained for the The Motivated Strategies for Learning Questionnaire (MSLQ) components (scale 1–7).

**Table 2.** Pearson correlation between the MSLQ components in the sample.


The reliability (internal consistency) of the whole MSLQ questionnaire was high (Cronbach's α = 0.86), and so was the reliability of the motivational section (α = 0.82) and of the cognitive and metacognitive section (α = 0.84). However, the reliability of the resource management section was low (α = 0.50), probably because this section is only made up of four components.

#### *4.2. Relationship between Motivation and Perceived Use of Learning Strategies*

We analyzed the nature of the relationship between Motivation and the perceived use of learning strategies. First, the motivation components taken together (averaged) significantly correlated with the mean score of the cognitive and metacognitive section (r = 0.51; *p* < 0.001), and with the mean score of the resources management section (r = 0.41; *p* < 0.001). Thus, 26% of the variance of the perceived use of Cognitive and Metacognitive strategies and 17% of the variance of Management strategies can be explained by students' Motivation.

Second, the cognitive and metacognitive section and the management section were considered together (9 strategies) as a Strategy-use variable in an averaged score, as they share an important part of their variances (46%; Pearson-r = 0.68; *p* < 0.001). Third, different levels for Motivation and for Strategy-use were defined: Low level (score ≤ 3), Intermediate (3 < score ≤ 5), High (score > 5). The low levels were infrequent in both variables (7 cases in low Motivation, i.e., 2% of the sample, and 7 different participants in low Strategy-use). Therefore, in both variables, the low and intermediate levels were collapsed and considered together. Table 3 shows the distribution of participants when these variables are crossed.

**Table 3.** Relationship between the motivation level and the use-of-Strategies level, as perceived by the students themselves.


Careful observation of the data in Table 3 suggests that low Motivation is strongly associated to a low Strategy-use with a high probability, but high Motivation cannot be associated to any level of Strategy use. In the complementary analysis, a low level of Strategy-use cannot be associated to any level of Motivation, but a high level of Strategy-use is strongly associated to high Motivation.

#### *4.3. Influence of Motivation and Learning Strategies on Academic Achievement in Science*

Table 4 shows the components significantly correlated with the achievement scores of students in science. Only Task Anxiety (Motivation-type: r = −0.01), Critical Thinking (Cognitive and Metacognitive-type: r = 0.09), and Peer Learning (Managing-type: r = 0.08) had no significant correlations with students' academic achievement scores.

**Table 4.** Significant correlations between MSLQ component and academic achievement in science.


\*\*\*: *p* < 0.001; \*\*: *p* < 0.01.

When all the MSLQ components were considered together as predictors in linear regression, they explained 27 percent of the variance of participants' scores in science (F(15,348) = 8.509; *p* < 0.001; R = 0.52). When a step-wise method with forward selection was used to enter only the significant predictors (*p* < 0.05), the resulting equation (for normalized variables) was:

#### Science achievement = 0.497 SE + 0.176 SO + 0.171 TE − 0.198 GO − 0.112 CT

These significant predictors explained 25 percent of the variance of the students' scores (F(5358) = 23.711; *p* < 0.001; R = 0.50). Using a forward procedure, the predictor coming first was Self-Efficacy, with the greatest single contribution, 17 percent (see Table 3); next, the components Time and Study Environment (2.1%), and Support of Others (1.9%) added significant single contributions. Goal Orientation and Critical Thinking obtained negative coefficients in the regression, although these strategies had positive correlations with the achievement scores in science. Thus, the above equation did not reveal an inverse relationship between students' GO or CT and their academic achievement in science. Instead, the negative coefficients were due to corrections to avoid over-estimation and to improve the prediction. These corrections added 4.0 percent to the explained variance. The contributions of the remaining strategies not included in the equation were non-significant (*p* > 0.05), due to co-linearity.

Considering the type of components, motivational components explained 19.7% of variance of students' achievement in science; the strategies together explained 14.5% of that variance. Considered apart, the cognitive and metacognitive strategies explained 10.0%; and the resource management strategies explained 12.8% of the variance of students' achievement in science.

#### **5. Discussion**

In the present work, we used the MSLQ questionnaire to obtain information about students' self-perception of their use of different self-regulated learning components in science. The global average indicated that participants consider that they use variated self-regulated learning components with a moderate-high frequency. In our sample, 75 percent of students obtained an average above the central, neutral value (4 in a Likert-type 7-point scale). In fact, the 1st quartile was placed at a score of 4.4 over the central, "neutral" value 4.0.

Motivational components reached higher mean scores compared to cognitive and metacognitive ones, suggesting that participants were more aware of their emotional approaches to learn science topics than of their abilities to organize, structure, elaborate, relate, summarize information, or monitor learning obstacles. Resource management components were the least used, according to participants' self-perception, with Peer Learning (PE) being one of the components having a low mean value: only 40 percent of students declared using this strategy with some frequency.

The mean values for the fifteen components in MSLQ (see Table 1) obtained in the present study are similar to the ones obtained by Pintrich, Smith, García and Mckeachie [32]. In that initial study, motivational components obtained the higher average and resource management components obtained the lower average, as in the present study. In absence of the complete individual data of Pintrich et al.'s [32] study, we made a "clumsy approach" to compare our values to the ones obtained by Pintrich et al. [32] (p. 808, Table 1). First, we computed a t-test for the two sets of fifteen mean values. The t-value was low (t(14) = 1.00; *p* > 0.10), suggesting no differences between the mean values of components in both studies. In addition, both sets of mean values had a high correlation (r(15) = 0.67).

Therefore, although MSLQ was validated for college students, it seems usable with secondary students as well. However, differences in the evolutionary state in students between 12 and 18 years old have to be considered in future replications. Karadeniz, Buyukozturk, Akgun, Cakmak, and Demirel [33] conducted a validation study in a Turkish context, to adapt the MSLQ to male and female primary and high school students from 6th to 11th grades. Students in the sample were tested focusing on different academic subjects: mathematics, sciences, social sciences and language. In order to compare results from this Turkish study to the present Spanish study, we computed again a t-test and

a Pearson's correlation for the two sets (Turkish/Spanish) of fifteen mean values. The mean values were not significantly different, and varied in parallel across the different components (t(14) = 1.19; *p* > 0.10); both sets of mean values were highly correlated (r(15) = 0.79). However, additional replications are needed to increase the reliability of the present results for secondary students using the MSLQ questionnaire.

#### *Influence on Academic Achievement*

In their study with undergraduates, Komarraju and Nadler [34] obtained that Self-Efficacy, Effort Regulation and Help-Seeking together explained 18 percent of variance of the academic achievement. This value is comparable to the one obtained in the present study from these same three predictors together: 21 percent. In a similar analysis with Malaysian engineering undergraduates, Kosnin [35] reported that the MSLQ scales together predicted 35 percent of the variance of their academic achievement, not far from the 27 percent obtained in the present study. However, Test Anxiety was a significant predictor in Kosnin's study, but not in the present one, suggesting possible uncontrolled effects due to cultural factors.

As correlations between academic scores and the different components are concerned, the Pearson r-values obtained in the present study are comparable to the ones from other empirical studies. In their analysis on the reliability and predictive validity of the MSLQ questionnaire, Pintrich et al. [32] obtained r-values very similar to the values obtained in the present study for the components sharing the higher variance with the academic scores (data are offered in the order: their study/present study): Self-Efficacy (0.41/0.41), Effort Regulation (0.32/0.29), Metacognitive Regulation (0.30/0.26), Time and Study Environment (0.28/0.29), Elaboration (0.22/0.20). Yet, some differences appeared in some strategies: Rehearsal (0.05/0.26), Extrinsic Goal orientation (0.02/0.24), Task Anxiety (−0.27/−0.03), Support of Others (or Help Seeking) (0.02/0.18). The differences in Extrinsic Goal Orientation and Support of Others could be explained by the different academic level, university in Pintrich et al.'s [32] study, or secondary education in the present study. University students are expected to have more intrinsic and less extrinsic motivation towards the own goals and to be more autonomous in their learning work than teenagers. As Rehearsal concerns, the differences are difficult to evaluate as in the present study participants were focused on science learning only, whereas in Pintrich et al.'s [32] findings, participants proceeded from several branches including Foreign Language, Humanities, etc. However, some differences in the r-values for some strategies (as Rehearsal, for instance) could be due to differences in the teaching approaches.

In the same vein, Kitsantas, Winsler, and Huie [36] conducted a study with undergraduates in the first year at the university. They correlated some of the MSLQ scales (Task Value, Self-Efficacy, Test Anxiety, Metacognitive Self-Regulation and Time Management), with the GPA marks for two distant moments, 2nd and 5th semesters. Again, most correlations they obtained were very similar to the ones obtained in the present study (correlation GPA-2nd semester/GPA-5th semester/correlation Marks-present study): Self-Efficacy (0.37/0.44/0.41); Task-Value (0.30/0.32/0.28); Time and Study Environment (0.35/0.32/0.29) and Metacognitive Regulation (0.21/0.22/0.26). Differences appeared in Test-Anxiety (−0.20/−0.19/−0.03).

Self-Efficacy was the most strongly related component to academic achievement (see Table 3), thus having the greatest single contribution in explaining the variance of students' marks (17 percent). This is a well-known result from previous studies. For instance, Robbins, Lauver, Le, Davis, Langley, and Carlstrom [37] conducted a meta-analysis of factors influencing students' academic achievement. Factors considered in more than 100 studies included institutional commitment, perceived social support, social involvement, achievement motivation, academic goals, self-efficacy, self-concept, self-regulatory study skills, and contextual factors. The academic self-efficacy was the strongest predictor of students' academic achievement in this meta-analysis. Bryan, Glynn and Kittleson [16] conducted a study with 14–16-year-old students, focused on the influence of motivation on learning science. Using a structural equations adjustment, they found that intrinsic goals, self-determination,

self-efficacy and academic achievement were significantly associated, but self-efficacy showed the strongest association (r = 0.62). The highest correlation between students' performance and any strategy was observed for self-efficacy (r = 0.40), in a recent study conducted by Jackson (2018), with 258 undergraduates enrolled in STEM gatekeeping courses. A positive and significant correlation between the self-efficacy items of MSLQ and academic achievement (r = 0.45) was also found by Al-Harthy, Was, and Isaacson [38], in a sample of 265 undergraduate students enrolled on an educational psychology course. Lynch and Trujillo [29] also found that self-efficacy was the strongest and more consistent MSLQ factor associated with academic performance in a sample of 66 college students, in the second semester of organic chemistry.

However, it is expected that self-efficacy and academic achievement influenced each other, in both directions. Cheung [39] showed that students' science self-efficacy could be increased by using efficacy-enhancing teaching based on Bandura's theory of self-efficacy. Bernacki, Nokes-Malach, and Aleven [40] found this mutual and continuous feedback effective, while students performed a problem-solving activity. A similar mutual effect was also observed in a study conducted with Italian junior students, where self-efficacy beliefs interacted with personality traits to explain academic achievement [41].

Rehearsal was the strategy of the lowest perceived use in science in the present study: only 25 percent of students declared using this strategy with certain frequency. This is not a surprising result, as comprehension of science concepts, laws, principles and applications are hardly achieved by mere repetition and training. However, individual differences in this strategy had significant and positive impact on the academic achievement: the higher perceived use of this strategy, the higher the mark in science. This is not an expected result, as learning activities based on rehearsal are expected to have low importance in learning science and low influence in academic marks. The positive and significant influence of this strategy turns the attention to the level of cognitive demand of the tasks proposed by secondary science teachers in assessment. Among other important skills, Stiggins [42] proposes science teachers to assess students' mastery of content knowledge, where mastery includes both knowing and understanding, and their use of knowledge to reason and solve problems (see [43]). Rehearsal does not seem relevant to acquire such skills, but other cognitive and metacognitive strategies.

The Peer Learning and Support of Others (or Help Seeking) strategies are related to the ability of a student to obtain help from other students when they need it, asking them for information, collaborating in learning activities, or obtaining some help to overcome learning obstacles. However, in the present study, both strategies obtained low averages compared to the rest of MSLQ. This result suggests that science learning activities in secondary classrooms do not properly encourage collaborative learning, even though the socio-constructivist approach to science education claims that knowledge has to be elaborated and shared by the members of a learning community by means of collaborative work [44–46].

#### **6. Conclusions**

It should first be emphasized that the MSLQ questionnaire seems usable with secondary students. This approach is justified, because the main values for the fifteen MSLQ components obtained in this study were similar to other related studies with university students. Furthermore, the Pearson's r-values between components sharing the higher variance with the academic scores in the present study were very similar to the ones obtained in other surveys.

It has been reported that most components of the SRL dimensions have significant correlations with students' performance scores in science. In the present study, all the components of the MSLQ questionnaire were used to predict the students' scores in a multiple regression analysis. This analysis was significant, and explained 27 percent of the variance of participants' scores in science. In addition, separate regression analyses were carried out for each MSQL section—motivational, or cognitive and metacognitive and management strategies—and all of them were also significant. Motivational components together explained almost 20 percent of the variance of students' scores in science; the cognitive and metacognitive components jointly explained 10 percent; and resource management components explained 13 percent of that variance. In view of this, it appears that the first prediction (P1) "Cognitive and metacognitive strategies, motivation, and resource management will influence the science achievement significantly" has been supported by the data in this study.

As the second prediction (P2) concerns, the data in Table 3 suggested that a high level of Motivation is (almost) a necessary, but not a sufficient condition for using many strategies frequently when learning science. In this way, data seem to support P2 about the role of Motivation as an enabling factor for using strategies frequently when learning science.

Within the limits of the present study, cognitive and metacognitive strategies included in the MSLQ instrument explained less variance of the academic marks (10 percent) than motivation (20 percent), or than resource management strategies (13 percent). Thus, the third prediction (P3) was not supported by the data. It is surprising that the cognitive and metacognitive strategies have lower impact on performance in science than the motivational components, and a coherent explanation is still needed. It opens the question on what science is being taught and how it is assessed in Spanish secondary schools.

One of the consequences that could arise from the current study is the need to investigate the effects of different instructional methods focusing on self-regulatory strategies. To date, it has been proven that Inquiry Based Learning [47] and Problem Based Learning [48] methodologies were superior to traditional instructional approaches on various facets of students' self-regulated learning.

Moreover, Self-efficacy (a motivational component in the MSLQ questionnaire) was the largest contributor to students' science achievement. However, concluding from this outcome that self-efficacy has to be directly promoted in the school curriculum could be inappropriate. Due to the circular nature and mutual influence of self-efficacy and academic success, the one-way causal effect of self-efficacy on academic achievement is difficult to isolate; perhaps the data obtained here and in other similar studies are due to the opposite relationship: the success in school science causes an increase of the student's perception of self-efficacy. This is also a matter for further research.

**Author Contributions:** The three authors significantly contributed to every phase of the research and the elaboration of the manuscript. All authors have read and agreed to the published version of the manuscript.

**Funding:** The research was supported by the University of Valencia (Spain).

**Conflicts of Interest:** There are no conflicts of interest.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **STEM Education in Secondary Schools: Teachers' Perspective towards Sustainable Development**

**Thi Phuoc Lai Nguyen 1,\*, Thi Huy Nguyen <sup>2</sup> and Thanh Khiet Tran <sup>2</sup>**


Received: 23 September 2020; Accepted: 23 October 2020; Published: 26 October 2020

**Abstract:** Young people are the future of society and agents for social change, and so it is crucial to provide education that not only equips them with knowledge and skills but also changes their attitudes and behavior towards sustainable development. This study provides a review on how pedagogical approaches in science, technology, engineering, and mathematics (STEM) education can be deployed to teach concepts of sustainability. It also shows how secondary school teachers perceived STEM education and how they applied integrated STEM disciplines in designing projects to address development issues in Vietnam. Seventy-seven STEM teaching projects of teachers across the country were analysed, and interviews were conducted with 635 teachers who participated in the STEM program. Teachers valued STEM education and were willing to apply constructivist pedagogical methods to help solve the real-world problems. It is hoped that an integrated STEM approach can transform education into an innovative and inclusive education for social equity and sustainable development.

**Keywords:** real-world contexts; constructivist pedagogical methods; STEM teaching projects; sustainability; SDGs; Vietnam

#### **1. Introduction**

Humans are the key actors of most contemporary global and regional environmental changes. The changes, such as technological innovation, environmental disasters, climate change, and pandemics, have enormous impacts on present and future human well-being and socioeconomic stability. Human–environment interaction and the connection between elements in bio-physical, technical, and human systems are complex and dynamic, which creates challenges for humans to navigate a safe operating space for development. This opens a space for education—a key sector for preparing the young generation with the knowledge and skills to address present and future socio-economic and environmental challenges, such as global climate change, digitalization and globalization. As the world is connected, dynamic and complex, with global challenges, young people should experience a school education that reflects the reality of these challenges. As education can enhance people's lives and contribute to sustainable development, the need for education that supports a more sustainable world has become more evident [1]. Sustainable development is defined as "development that meets the needs of the present without compromising the ability of future generations to meet their own needs" [2] (p. 43). The goal of sustainable development is to integrate economic development with environmental concerns and social integrity, sharing fairly the costs and benefits to contribute to the well-being of the current generation without compromising the needs of the next generation [3].

Society expects the young generation to deal with these development challenges. In this context, education is the engine behind the development of the young generation. There has been growing international attention and recognition of the role of education in sustainable development. Innovative education empowers people to change their ways of thinking and work towards a sustainable society. The United Nations (UN) has launched several initiatives to promote the role of education in sustainable development, such as the UNESCO Global Action Programme on Education for Sustainable Development and the UN Decade for Education for Sustainable Development (2005–2014). The UN's 4th Sustainable Development Goal (SDG) was set with a specific focus on education for sustainable development [4]. Education is an important means of achieving the SDGs. Educational policies play a key role in the effective implementation of education for sustainable development and influence the development of curricula, teacher training, learning materials, and learning environments [1]. Every SDG requires education to empower people with the knowledge, abilities, skills and values to develop themselves and contribute to society. In this context, STEM (science, technology, engineering, and mathematics) education appears to be essential in preparing the young generation to address the challenges facing society.

STEM education is an innovative approach to education [5,6] and features extensively within the global landscape of educational policy and reforms. STEM education represents a multidisciplinary approach that combines the four disciplines of science, technology, engineering and mathematics [7]. This approach not only addresses the aims of policy reforms, such as ensuring competency in mathematics and science, but it also emphasises that it is no longer sufficient for modern citizens to understand science and mathematics; their knowledge must be integrated with technology and engineering [7]. STEM education uses a "learner-centred" approach to develop learners' self-direction, problem solving, collaboration and project management [8]. It also drives innovation through creating, designing and producing solutions to real-world problems [9] and uses real-world challenges as entry points for the integration of STEM disciplines [10].

Through literature review and a case study of STEM education in Vietnam, this study examined how sustainable development concepts and practices can be integrated into the pedagogical principles of STEM education and to discuss how an integrated STEM approach in secondary education could promote the role of education in sustainable development. This paper begins with a literature review on pedagogical approaches in STEM education and their relevance to "sustainable development" education. It is followed by a case study of the STEM education program in Vietnam in which we illustrate what the secondary school teachers perceive STEM education and how they use integrated STEM disciplines in designing projects to address development challenges.

#### **2. Literature Review**

The literature review focused on giving a summary of the knowledge base of STEM education principles and discussion on how these teaching approaches are relevant to teaching the concept of sustainability in the secondary schools. The integrative or critical review approach was applied to critique and synthesize the literature [11] on STEM teaching methods and their relevance to real-world contexts of sustainable development. Since this integrative or critical review approach does not require researchers to follow a specific standard of data analysis [12], we performed the review and interpretation through our conceptual thinking [13] to critically review the STEM education, its pedagogical approaches and their relevance to "sustainable development" education.

#### *2.1. STEM Education Principles and "Sustainable Development" Education*

#### STEM Education

STEM education is defined as an interdisciplinary approach to learning where rigorous academic concepts are coupled with real-world lessons as students apply science, technology, engineering and mathematics in contexts that make connections between school, community, work and the global enterprise enabling the development of STEM literacy and with it the ability to compete in the new economy [14]. Although many definitions of STEM education have been proposed [15], it is commonly agreed in both theory and practice that STEM education refers to the integration of the four disciplines (science, technology, engineering and mathematics) in various ways and levels [16]. STEM education emphasises the importance of making connections between academic knowledge and real-world problems as a foundation for integrating S-T-E-M subjects in teaching [10]. With the integration of S-T-E-M disciplines, STEM education can offer high-quality science and technology education and the relevant knowledge to understand scientific, technical and cultural interrelations [15]. It also provides students with important digital competences [17] and soft skills, such as problem solving, creativeness and critical thinking, that are essential for the workforce to sustain a country's economic growth [18].

There have been increasing demands for future citizens to be literate in STEM subjects and to have knowledge of socio–scientific–technical interrelations and their application in addressing real-world problems. Modern citizens must be able to employ the science and technology knowledge that they learn in school to deal with development challenges, including environmental pollution, unpredictable climate phenomena, the exhaustion of natural resources such as water and energy, and social and political conflict. As well as preparing students with STEM competencies, STEM education improves students' abilities to innovate—an increasingly important skill for openly embracing change and responsibly shaping the future. With these soft skills, modern citizens are able to follow sustainable lifestyles; promote human rights, gender equality and a culture of peace and non-violence; appreciate cultural diversity; and trigger culture's contribution to sustainable development.

Due to its interdisciplinary nature, STEM education is seen as both a curricular and a pedagogical approach [10]. In terms of curriculum, STEM education refers to the interdisciplinary framework, whereas pedagogical approaches centre around instructional practices such as enquiry through representations, problem solving and reasoning, challenge-based learning, design-based approaches and digital technologies. Many studies have shown that the following key principles are necessary for integrated STEM education in secondary schools to address sustainability: the integration of STEM content; enquiry based on the real world; and problem-centred, design-based, and cooperative learning approaches [19,20]. Hence, the interdisciplinary and real-world problem-based works are engaged in STEM pedagogical practices and curriculum and material designing [21–23].

Many instructional practices in STEM education are congruent with the underlying real-world problems and concepts of sustainability. It is widely agreed that the SDGs are a global responsibility, but their achievement requires local action. Due to their complexity, most of the SDGs require a multidisciplinary and interdisciplinary approach, and so the UN has been promoting education projects involving STEM education to achieve the SDGs (sustainabledevelopment.un.org). An integrated STEM approach is a promising framework for sustainable development education.

#### *2.2. STEM Pedagogical Principles and "Sustainable Development" Education*

#### 2.2.1. Integration of STEM Disciplines

A principle of STEM education is the application of knowledge and practices from multiple STEM disciplines to learn about or solve problems in the real world. STEM education was first introduced in the US to "raise mathematics and science achievement, improve economic competitiveness, increase job prospects for future generation workers, and support greater opportunities for low-income and minority students" [24]. STEM literacy and skills are essential to address sustainability. An important question is how education's contribution to sustainability will shape our future. The local and global challenges today cannot be solved within a single discipline. With comprehensive knowledge of STEM disciplines, however, schools will produce a young generation with the multidisciplinary expertise and, with the support of technology, the skills to examine the complexity of real-world challenges and to determine integrated solutions.

Through the integration of STEM disciplines with the support of technology and mathematics, all science disciplines can teach and discuss sustainable development in their fields, which enhances students' awareness of their roles in the achievement of the SDGs. For example, by exploring

agro-ecological systems, biology education, combined with other science disciplines such as chemistry and physics, can contribute to achieving the 1st, 2nd, 14th and 15th SDGs. Similarly, chemistry education can contribute to achieving the 3rd, 6th, 12th, 14th and 15th SDGs by improving students' knowledge of physio-chemical properties and their impact on the environment and human health.

Furthermore, while there is much demand in the global jobs market for people with the multidisciplinary expertise to address the problems of a complex, connected and dynamic world. Global youth unemployment and living in poverty is because the current labour markets need people with new skills. Basic skills are not adequate. Soft skills, analytical skills as well as knowledge of science, technology, engineering and mathematics are required for more dynamic and complex jobs. If school education can prepare young people with multidisciplinary expertise, this could facilitate the employment pathway of youth in the future. Improving the skills of young people can improve their employment prospects and improve the quality of their lives and well-being [25]. The UN states that STEM education "can remove poverty and reduce inequality in developing countries". Science and technology is a key element in economic and social development. STEM education will also encourage and recognise women who are making strides in the STEM arena and help to balance the employment rate of men and women.

#### 2.2.2. Real-World Context-Based Inquiry

Educational approaches based on real-world contexts help students see the relevance of science to their daily lives and enhance their interest and enjoyment in addressing the real-life situations around them [26–28]. Real-world context-based teaching and learning assumes that everyday situations that are familiar to learners can be enquired into and that STEM concepts relevant to these problems can be explored and deployed to explain the situations [27,29]. Some studies, such as George and Lubben [27], Gutwill-Wise [30] and King and Henderson [31], have shown that students' interest in science increases through teaching approaches that are based on real-world situations. Students can make connections between the real-life context and the concepts they are taught, and they are more interested in learning if they see how what they are taught in schools relates to what they do in their daily lives. This kind of learning and reasoning will change students' behaviour towards their environment.

It is well known that human activities have an enormous negative impact on our planet [32,33]. There has been increasing international concern about people's ecological behaviour towards protecting and preventing from global environmental degradation threats [34,35]. STEM education is an indispensable tool in addressing environmental problems [36,37]. Through teaching approaches based on the real world, STEM education can influence students' internal representations and understanding of the real world, and ultimately transform students' attitudes towards environmental protection.

#### 2.2.3. Problem-Based Learning Approach

The problem-based learning approach is another essential instructional practice in integrated STEM education [19]. Problem-based learning is a constructivist pedagogical approach in which students learn about science and develop their skills in critical thinking, problem solving and collaboration by solving real-world problems [38]. The core principles of problem-based learning emerged from cognitivist constructivism [39] and social constructivism [40]. This approach is premised on the theory that students' abilities are developed through social learning [41]. Through working together in a small group and being coached by teachers, students identify problems, formulate hypotheses, collect data, perform experiments, develop solutions and choose the solutions that best "fit" the problems. The problem-based learning approach encourages students to use and build on their knowledge and to work collaboratively in self-organizing small groups to make sense of new information, to solve complex problems, and to produce a solution [42].

The global socio-environment is becoming more complex and uncertain and is continuously evolving because of climate and environmental changes and global volatility. Furthermore, the development and increasing use of digital technology are transforming people's daily lives and societies. It is challenging to identify the kinds of jobs that will exist in the future and the expertise they will require. Thus, education must be adjusted to accommodate and equip students with the necessary skills, qualifications and flexibility to fill future jobs.

The problem-based learning approach enhances students' capacity for thinking and reasoning about problems and integrating previously assimilated knowledge and experience into a life-long learning process. This approach refers to instructional "scaffolding" which plans learning steps. Learning built on prior knowledge and skills and developed through the pathway for new knowledge to be acquired and applied to practice has proved to be effective in STEM education [43]. Through the problem-based learning approach, students' skills including reasoning, critical thinking, application of theory to practice, communication, reflection and teamwork are developed.

#### 2.2.4. Design-Based Learning Approach

An important part of STEM education is the engineering design process, in which scientific enquiry, artistic design, construction engineering, mathematical reasoning, and technology are used to solve real-world problems [44]. Design-based learning, in which problems are solved using design assignments, is a form of problem-based learning in which students are given hands-on experience of real-world problems [45,46]. It is an inductive teaching approach built and grounded in the enquiry and reasoning processes leading to the generation of innovative artefacts, systems and solutions [47]. This approach is centred on students' experience of designing a product or object, through which they develop their scientific understanding and problem-solving skills [48]. Design-based learning is considered a promising instructional method to enhance students' learning of and interest in science [49], and it is commonly applied in teaching science and design skills [45,48] and to engage secondary school students in engineering design tasks [47,50].

In terms of sustainability, the key competencies of a modern citizen include systems thinking, interpersonal competence, interdisciplinary study, the embracing of diversity, strategic action and management competence [51]. Design-based learning, as a form of project-based learning, offers students with creativity and innovative mind-set for sustainability practices. Many studies have shown the relationship between sustainability education and transformative learning [52,53]. A multidisciplinary approach is a core element in design thinking, and, due to the complexity of the SDGs, design thinking is essential in addressing and achieving them. A design-based learning approach begins by defining a problem from a real-world context and involves developing optimal solutions for social challenges, contributing to the achievement of the SDGs [54].

#### 2.2.5. Cooperative-Learning Approach

The cooperative-learning approach is another constructivist pedagogical method that involves students working in small groups to help one another learn. The aim of STEM education is to equip students with a broad mix of skills and interdisciplinary knowledge. Cooperative learning plays an essential role in helping to develop, spread and sustain the role of education in society. It has been shown to be an effective instructional method that provides a wide variety of outcomes and academic achievements of students [55].

The positive effects of cooperative learning include motivational aspects, such as the motivation to self-learn and to encourage and help groupmates to learn; social cohesion, as the goal of group leaning is to ensure that all the members of the group learn; and cognitive development, as the students' interaction with one another to complete specific tasks results in their mastery of critical concepts [56,57]. Cooperative learning builds students' collaborative skills and teamwork—important skills in the 21st century, as they are essential for addressing the complexity of present and future socio-economic challenges [58]. By building collaborative skills in school, students will develop positive attitudes and behaviour outside school to work together for the sustainable development of their community and their country. The collaborative attitudes and behaviour of citizens will greatly

contribute to the achievement of SDGs such as poverty reduction, peace and equity, conservation, sustainable consumption and production, social responsibility for development and democracy.

#### **3. The Case Study**

The Second Secondary Education Sector Development Program II (SESDPII) (sesdp2.edu.vn) in Vietnam was selected as a case study to give insights on how secondary school teachers perceive STEM education and how they develop their STEM teaching projects in practice. Within the case study, we collected and analysed data on what secondary school teachers perceived STEM education and how they applied integrated STEM disciplines in designing projects to address development issues.

#### *3.1. Vietnam and National STEM Education Program in Vietnam*

Vietnam is a lower-middle-income country. It is experiencing rapid demographic and social change, including rural out-migration and an ageing rural population. Industrialization, rapid economic growth, and a population boom have not been friendly to the environment and natural resources, causing challenges for the management of waste and pollution. Moreover, according to the Intergovernmental Panel on Climate Change, Vietnam is among the most vulnerable nations to climate change. The country suffers from sea level rises, typhoons, landslides, flooding and droughts, and other weather events.

Vietnam ranked highly in the Program for International Student Assessment (PISA) in 2012 and 2015, in which the performance of the Vietnamese students exceeded that of many OECD countries. Although Vietnam performs good on general education in terms of both education coverage and the level of learning, the teacher-centred approach and teachers' absolute rely on text books are common in most public and private schools. This approach prevents students from gaining the maximum benefit from contextualised lessons. Thus, the questions arise, however: how can the educational system in Vietnam develop students' academic and professional competences and ultimately contribute to the achievement of quality education for sustainable development?

Through the Second Secondary Education Sector Development Program II (SESDPII) funded by the Asian Development Bank, the Ministry of Education and Training has integrated STEM education into secondary schools. The objectives of the STEM education established by the SESDPII are (i) to enhance students' comprehensive education; (ii) to improve students' STEM literacy; (iii) to develop students' soft and academic skills, such as problem solving, creativity, critical thinking, argumentation, intellectual curiosity and collaboration; (iv) to connect schools to communities; (v) to guide students' career development; and (vi) to prepare for Industry 4.0.

The ultimate goal of the program is to promote the teaching of science integrated with math, engineering and technology. The program established an STEM research team at the national level whose members are from pedagogical universities in both northern and southern Vietnam. The results of the team's experiments on interdisciplinary teamwork, collaborative ideas and innovative approaches to teaching integrated STEM are available for all high school teachers in the country. The program provides school managers, educational policy-makers and implementers with professional training in STEM concepts, integration frameworks, pedagogical approaches for both lower and upper secondary schools and STEM education's role in development. Members of the STEM research team provide the training, and lecturers from several pedagogical universities in different regions of the country are invited as observers. The program encourages teachers to collaborate with their colleagues to develop their STEM teaching projects/topics and start experimenting with STEM teaching in their schools.

#### *3.2. Data Collection and Analysis*

Using SESDPII as the case study, we examined which teaching topics and pedagogical methods the teachers who participated in the STEM program of SESDPII have used to develop their STEM teaching lessons or projects and assessed their understanding of STEM education and their perception of STEM education's role in addressing development challenges. Two steps of investigation have been made:

(i) Step 1: Face-to-face interviews with secondary school teachers

Face-to-face interviews were conducted with 635 teachers who participated in the four professional training sessions organised by the SESDPII in 2019 in the Highlands, Central and Northern regions of Vietnam. The interview was conducted in Vietnamese.

The interview was composed of two parts:


In order to avoid influencing of an interviewee's response to another's as the interviews were conducted during the breaktime of the professional trainings, interviewed teachers were asked to write on a piece of paper how they defined STEM education and list the development challenges in Vietnam that they wanted to address through their STEM teaching projects/topics.

Content analysis was used as the main research tool to the interview transcripts. We firstly determined concepts about STEM education and real-world development topics, teachers wished to integrated into STEM teaching, expressed in written by teachers during the interviews. Thus, the presences of these themes and concepts were quantified.

(ii) Step 2: Analysis of STEM teaching projects of secondary school teachers

Before selecting participants for professional trainings of the SESDPII, secondary school teachers across Vietnam were called to submit their STEM teaching project proposals. They were encouraged to collaborate with their colleagues to develop their project ideas and start experimenting with STEM teaching in their schools. Seventy-seven STEM teaching projects across the country (15 provinces) were submitted to the SESDPII.

We fully accessed the database of these seventy-seven STEM teaching projects. Using content analysis, seventy-seven documents were read several times to get sense of the projects' contents and structures. We then divided these project documents into several sections: premises, objectives, class procedure/steps and learning outcomes. The main focus of the analysis was to see within each project (i) what was the entry argument of the project, (ii) what was the main teaching topic, (iii) which subjects and how many were integrated to design the lecture, and (iv) which pedagogical methods were used. Existing terms in the literature of STEM education and sustainable development were used to label information according to their significance in our research objective.

#### *3.3. Findings*

#### 3.3.1. Sociodemographics of Interviewed Teachers

Overall, 211 teachers participated in the SESDPII professional trainings in the North, 244 in the Central region and 180 in the Highlands were interviewed, providing a total interview number of 635 (Table 1). There was a significant difference in the number of male and female teachers that participated in the interviews among the three locations (χ2 = 10.41, df = 2, *p* < 0.01): more female teachers in the North (53.08%) and the Highlands (58.33%) participated in the interviews than in the Central region (43.03%).


**Table 1.** Sample demographics and comparisons among interviewees of three locations (North, Central and Highlands).

There was no difference in the number of interviewed teachers holding a bachelor's and master's degree among three locations (χ2 = 0.49, df = 4, *p* = 0.78). However, the significant difference was found in the numbers of interviewed teachers from different school types (χ2 = 242.55, df = 4, *p* < 0.01). The majority interviewed teachers were more likely to come from regular schools in the North (90.05%) and the Central (90.98%) region, while a large number of interviewed teachers in the Highlands were from boarding schools (58.33%). This can be explained by the fact that many boarding schools in the highlands have been established in the last decade under the Vietnamese policy on strengthening education development in ethnic minority communities and mountainous regions. There was also a difference in the proportion of interviewed teachers from different teaching subjects (χ2 = 31.85, df = 12, *p* < 0.01). Math and science (i.e., physics, biology and chemistry) teachers participated more than technical design and informatics teachers, as math and science are the core subjects in the secondary schools.

#### 3.3.2. Teacher's Perceptions of STEM Education

Through the interviews, we explored how the teachers perceived STEM education and what development challenges in Vietnam they wanted to use in their STEM projects. They were asked to write on a piece of paper how they defined STEM education and list the development challenges in Vietnam that they wanted to address through their STEM teaching projects.

Table 2 presents the teachers' perceptions of STEM education. The results reveal that the teachers hold positive perceptions of STEM education. Most considered STEM education to be an integrated learning approach that uses real-world problems to enhance students' competency in science and their soft skills.



3.3.3. Teacher's Perspective on Development Issues to Be Addressed through STEM Education

Figure 1 shows how the teachers responded when asked to list the development challenges in Vietnam that they wanted to address through their STEM teaching. More than 80% of the teachers preferred to use the topic of environmental pollution in their projects, to enhance students' environmental knowledge, to increase how much they value environmental protection and to improve their knowledge of the tools and methods to monitor and evaluate the environment. The second most common development topic was sustainable consumption and production. The teachers were interested in applying STEM teaching methods such as design-based learning to promote more sustainable daily use objects. The third-ranked issue was recycled materials. Renewable water and energy and sustainable agriculture and food production are also among the teachers' preferred topics. The teachers viewed STEM education as an innovative teaching approach that enhances their roles and the importance of secondary education in addressing the current development challenges of the country, which contributes to the achievement of the SDGs.

**Figure 1.** Teachers' preferences for development topics to be used in their STEM education projects (% respondents) (N = 635).

3.3.4. Teachers' STEM Teaching Projects on Socio-Economic and Environmental Challenges

The seventy-seven teaching projects were submitted by both upper (43%) and lower (57%) secondary schools. Most projects came from schools in big cities like Ho Chi Minh (16 projects), Hanoi (11 projects), Kien Giang and Can Tho (11 and 7 projects). However, there were few projects from remote and minority ethnic schools in the mountainous north such as Lai Chau, Tuyen Quang and central highlands in Lam Dong and Daklak provinces. These STEM teaching projects were led by 56% female and 44% male teachers in collaboration with her/his interdisciplinary teams for emerging collaborative ideas and methods to enhance students' science knowledge and application of theories to solve the problems around them. 45% projects were designed by full integration of four STEM disciplines: the combination of Physics—Informatics—Technical Design—Math (41%) and Physics—Biology—Informatics—Technical Design—Math (4%). Other 44% were the combination of three STEM disciplines, Biology, or Chemistry, or Geography with Informatics and Math (a few with Technical Design). The only 10% were the integration of two STEM disciplines.

Most of the projects used the combined inquiry and design-based learning (49%), combined problem and collaborative-based learning (22%), combined inquiry and experiment-based learning (20%) and real-world inquiry-based learning (9%) approaches. These constructivist and solution-based approaches help students in building their creativity and innovative mind-sets, which is essential for addressing current and future development challenges. Details of STEM teaching projects are found in Table S1 in the Supplementary Material.

In addition, Figure 2 shows the development themes were used by the teachers to develop their STEM teaching projects. Although no suggestions were given to the teachers about the types of topics for their projects, most of the projects were related to the SDGs that have a local impact as well as global relevance, such as sustainable consumption and production, water availability and quality, sustainability of energy, and sustainable agriculture and food production. For instance, 31% projects were designed to teach students to create simple sustainable daily life use objects or products, recycling materials and waste management. In total, 30% of projects focused on food safety and food security, water treatment and saving and renewable energy. These projects were also designed to teach health protection and education (10%) and flood prediction, drought mitigation and agri-environmental protection (8%). In short, the majority of projects (83%) were used real-world issues and phenomena such as the premise of the lecture objective, whereas only 17% of projects were purely scientific experiment and inquiry.

**Figure 2.** Distribution of STEM teaching topics of examined secondary school teachers to Secondary Education Sector Development Program II (SESDPII) (N = 77).

#### **4. Discussion**

#### *4.1. Connecting Teaching to Real-World Context*

The teachers who participated in the case study showed their interest and willingness to connect their teaching with real-world context through STEM projects. They seemed to believe that STEM education can be deployed to solve development issues. This finding is in accord with Steiner and Posch [59] that STEM education can be used to teach sustainable development concepts and practices in secondary schools by breaking out of the existing conventional teaching structures and processes that mainly rely on textbooks. Sustainable development is a complex concept that is not easy to teach in secondary schools, as different subjects and contents must be integrated into a class, and the class should be connected to real-world context and communities.

The above results also demonstrated that teachers were sensitive to the development issues surrounding them. Sustainable water, food and energy, as well as sustainable consumption and production are among the most common topics the teachers integrated into their STEM projects and/or they wished to teach within their STEM lectures. Indeed, in the last decades, Vietnam has experienced threats to its energy security [60], vulnerability of its water resources [61], and ground water contamination. It is a country that is particularly vulnerable to climate change, given its extensive coastline and river deltas and its vulnerability to typhoons and floods [62].

A challenge facing many secondary schools is the disconnection between the school and the community. The teachers in this study seemed to want to connect the teaching of science with real-world context through STEM education. Through the problem-based learning approach, teachers can bridge community-based knowledge and school-based knowledge, providing intellectual and meaningful science learning through practical experience. Bouillion and Gomez [63] conducted a case study of teachers who adopted an interdisciplinary approach to teaching science, mathematics, language, arts, and civics by having their students identify pollution problems in a river. Their study confirmed that such contextual scaffolding to connect science with the community can create bridging opportunities between community-based and school-based knowledge.

#### *4.2. Nurturing Secondary School Students as Agents of Social Change*

Our findings also show that interviewed teachers positively perceived and understood STEM education. Most of the examined STEM teaching projects are the integration of three or four S-T-E-M subjects and the application of constructivist teaching methods and the topics were around the development issues. The integrated STEM thinking facilitated teachers to design their STEM projects aiming at challenges of development with the sense of enthusiasm, viewing the development problems through the eyes of someone actually facing it. Teachers applied systems thinking in designing their projects. They used a problem identified from the local context and developed the optimal solution by applying the STEM pedagogical principles.

With this STEM approach, teachers seemed to break up the traditional didactic triangle—teacher, student, content—in which teachers indoctrinate passive students. The teacher-centered and textbook-based approach is common in most public and private schools in Vietnam, but it prevents students from gaining the maximum benefit from contextualised lessons [29].

These STEM contemporary constructivist pedagogical methods might help to reform education in secondary schools, encouraging students to take part in science and technology competitions, assume agentive positions in reconfiguring their own individual and collective futures and contributing to sustainable development [64]. These approaches promote "learning by doing" through lessons based on real life that allow experimentation, enabling students to make mistakes and learn from them. The integrated STEM approach also improves students' motivation to address real-world problems as they learn, "touch" and experiment with the real-world problems through the learning process.

According to Hoff and Hickling-Hudson [65], many international non-governmental organisations have promoted the teaching of sustainable development in adult education (e.g., environmental education and social justice education). The challenge of sustainable development, however, is complex and systemic and requires different ways of thinking, receptivity to new ideas, and the ability to navigate direction for innovation and transformation. Achieving sustainable development through education is a long-term process that requires the transformation of the young generation's thinking and actions. Motivated teachers with STEM teaching skills and digital competences [17,66] can enable students to meet the challenges of tomorrow. Teachers communicate important knowledge to students so that they not only understand the problems but also think about possible solutions as foundations for social change.

Since STEM education helps enhance students' knowledge and skills and change their attitudes towards real-world problems, teaching integrated STEM in secondary schools provides a great opportunity to promote more transformative social change through equal access to STEM education. However, although teachers from 15 provinces across the country were interested in developing STEM teaching projects, the number of teachers from rural and minority ethnic schools in remote provinces that participated is still low. This can be explained by the fact that the existing thinking around STEM education is that it involves a large investment in technology and is thus often only for wealthy private schools, gifted public schools, and students who are naturally oriented towards science. Thus, STEM education has become commercialised and dominated by private high-tech companies and schools in the large cities of Vietnam. The partnership between expensive private education and foreign high-tech companies and organisations has created inequality between the urban and the rural, the rich and the poor, and the ethnic majority and ethnic minorities.

STEM education should be inclusive and available to every student. This appeals more investment and expansion of STEM education in these less developing provinces. By integrating the STEM disciplines, using constructivist pedagogical methods, connecting to a real-world context and making use of technology, teachers and students in these disadvantaged areas are able to catch up with their peers in the big cities.

#### **5. Conclusions**

This study provides meaningful insights into how pedagogical approaches in STEM education can be deployed to teach science in the context of sustainable development. Interviewed teachers were interested in using STEM teaching to address the real-world development issues, such as pollution, sustainable consumption and production, and energy and water conservation. Most STEM teaching projects developed by the teachers integrated from three to four S-T-E-M subjects and applied contemporary constructivist pedagogical methods, such as problem-based learning, design-based learning and cooperative-based learning, which emphasise a student-centered teaching approach. To engage students in the STEM learning environment, this approach focuses on various principles, such as students' independent enquiry, lesson plans, material searching, lesson processes, communitive interactions among students, group working, and student–teacher relationships. This implies that training should be provided to secondary school teachers to enhance their knowledge and application of contemporary constructivist pedagogical approaches to promote students' self-direction, collaboration and problem-solving ability. Teachers must be qualified and skilled, however, so that, through their lessons, students develop their critical and creative thinking and STEM literacy. Importantly, teachers must be capable of driving students to apply and contextualise their learning and to innovate through designing, creating and producing solutions to real-world problems.

In conclusion, inclusive STEM education could contribute to the achievement of SDG4—equitable and quality education for sustainable development and sustainable lifestyles and social equity. It should have received more attention, however, within sustainable development agendas at national and international levels. The implementation of inclusive STEM education requires innovation and change in pedagogical approach, curriculum development, methods of student assessment, school management structure and teacher-support initiatives. It also requires more investment facilities, infrastructure and technology for schools in remote areas, for ethnic minorities, for poor students and for areas of social conflict. It is hoped that, in this way, education in Vietnam can be positively transformed and contribute to the achievement of SDG4. Inclusive, equitable and quality secondary education is for the development of global citizenship and cultural diversity and culture's contribution to sustainable development.

**Supplementary Materials:** The following are available online at http://www.mdpi.com/2071-1050/12/21/8865/s1, Table S1: List of 77 STEM teaching projects.

**Author Contributions:** Conceptualization, T.P.L.N.; Methodology, T.P.L.N.; Investigation, T.P.L.N.; Resources, T.H.N. and T.K.T.; Formal Analysis, T.P.L.N; Data Curation, T.P.L.N.; Writing—original draft preparation, T.P.L.N.; Writing—review and editing, T.P.L.N.; Supervision, T.K.T.; Project Administration, T.H.N.; Funding Acquisition, T.H.N. and T.K.T. All authors have read and agreed to the published version of the manuscript.

**Funding:** Authors acknowledged ADB Loan 3494/3494-VIE, Second Secondary Education Sector Development Program II (SESDPII).

**Acknowledgments:** Our special thanks is given to teacher participants in professional trainings organized by SESDPII in the Highland, Central and Northern regions of Vietnam who participated in this study. We also thank the SESDPII's administrative staffs for their logistic support; Yen Nhi Pham, Thang Trinh and My Ngan Le Hai for their support in data entry.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Active Learning Methodologies in Teacher Training for Cultural Sustainability**

#### **María-Pilar Molina-Torres 1,\* and Raimundo Ortiz-Urbano <sup>2</sup>**


Received: 5 October 2020; Accepted: 29 October 2020; Published: 30 October 2020

**Abstract:** Within the framework of sustainable development, the concept of cultural heritage is linked to the heritage awareness of a specific spatial context, and to the conservation of collective memory. Despite the clear interest in cultural sustainability, the lack of research devoted to interpreting different teaching methods for transmitting patrimonial assets and preserving natural heritage is noteworthy. For this reason, the present study takes various scientific contributions as the background for considering sustainable education as a fundamental instrument to recover and conserve heritage resources, both from an informative and educational point of view. The aim of this study was to assess the training of new teachers in terms of the particular skills needed to develop active learning methods related to the teaching of heritage sustainability. The research sample consisted of 488 students in the Degree in Primary Education program. The methodology combined both qualitative and quantitative techniques, to obtain information from both observation and an opinion survey given to the students. The results show several opinions related to achieving and facilitating the implementation of innovative methodologies, due to a lack of initial university training. Generally speaking, this work provided an opportunity for students to analyze a series of prejudices regarding their working methods, and to overcome excessive theorization in their university studies.

**Keywords:** educational innovation; social sciences; competences; sustainability education; active methodology; heritage; teacher training; university education

#### **1. Introduction**

Currently, curricular proposals do not include sustainability among those teaching skills intended to be achieved by university teachers [1]. Moreover, education for sustainability is not a reality in university classrooms in Spain. Universities have a responsibility to train future teachers and to integrate sustainable culture into training processes as an essential factor in promoting education for sustainability among teachers, as well as in the citizenship they will form [2].

Our proposal relates to the educational methods that future professionals in the teaching of historical heritage and sustainable culture should implement in their primary education sessions. At the time of implementing this research proposal, the didactic models potentially chosen by the university students of the Degree in Primary Education, and the autonomy they acquire during their training, have been both taken into account in order to give priority to educational competencies over theoretical content.

In this regard, Thomas [3] pointed out that, in order to achieve these changes and be able to implement them in the educational field, participation at the community level is necessary to promote sustainability and social responsibility. In fact, two key ideas—global citizenship and volunteerism—are part of education on sustainability. The current trend is to distinguish between the terms "culture" and "sustainability", which are interrelated and subject to political, social, and scientific processes, and which, in any case, are difficult to clearly define [4].

This situation has led to the current state of research over the last two decades, which has resulted in numerous papers in this area [5–12]. According to Barth, Godemann, Rieckmann, and Stoltenberg, little attention has been paid to the development of key competencies related to sustainability so far [5]. For this reason, this study deals with both the conservation of cultural heritage and the development of key competencies in sustainability, through the learning methodologies. In addition, it adopted participatory research approaches and new perspectives on social sustainability. For Tweed and Sutherland, sustainability in urban environments tends to focus on technical issues, as well as on the broad contribution it can make to a sustainable urban development [6]. Additionally, other authors have validated these opinions and contributed to describing a sustainable urban environment both visually and culturally [8,10]. In particular, a review of these studies showed the importance of educating people on the value of urban heritage and its integration into sustainable development—hence the need for the present study.

Currently, more value is given to monumental elements in relation to historical events, without giving much importance to the natural elements that shaped the landscape of the area—which are also a sign of cultural identity. As a discipline, sustainable education blends in with the environment to raise popular awareness about nature conservation. However, identifying the term "environment" only with natural spaces is only a partial view if we do not associate it with a delimited social environment and a particular historical reality. Previous studies showed some disadvantages around cultural sustainability and heritage conservation, as two concepts that come together in a society which is both global and unequal from a local point of view [13].

The opportunities and setbacks attributed to this relationship have been researched in depth. Nowadays, this same society is immersed in a series of challenges involving advances in the economy, the environment, and the culture of sustainable development [14]. However, although the relationship between the individual and the society is progressively changing, sustainable human development acts as an indicator linked not only to economic growth, but also to social, cultural, and environmental growth.

To that end, one of the challenges a university faces is incorporating sustainability into the higher education curriculum as a fundamental component of the training of future professionals [15]. Initial training needs to be improved from its basic levels, so as to direct it towards professional practice and access to the teaching profession [16]. In this regard, the didactic training of teachers in teaching methods is an opportunity for teacher learning and a better implementation of these methods in their career development [17]. Nowadays, cultural sustainability and historical heritage are not associated with each other in the curricula, appearing separately and without relating the concepts and theories that define them.

In the same vein, it must be noted that this methodological approach must be backed up by strategies which favor the search for solutions to problems experienced in the real world. As a result of this research of a social nature, a quest for answers arises, and attitudes that promote citizen competencies are encouraged [18]. From this perspective, sustainability should be understood as the balance and long-term maintenance of both natural and cultural resources and processes in a particular territory. Hence the acquisition of citizenship skills at university allows for the conservation of the environment, as well as a sense of belonging to the environmental culture [19]. Thus, education for sustainability involves a three-pronged approach of environmental education, education in social sciences, and citizenship education [20].

Likewise, the area of civic and cultural education has reached a prominent position. Citizen's training results in education in values, along with sustainable human development. For this purpose, students learn and implement their professional skills in university practice through a real educational context. In this way, the specification of curricular elements in the classroom will inspire new ways of

thinking among future educators [21,22]. In fact, teachers have to face several challenges which reflect the poor sustainability culture they receive at university.

Among these, it is worth mentioning the implementation of traditional methodologies, together with the conceptual strain of the curriculum in undergraduate studies, as shown in Table 1. In this context, the training of these undergraduate students should be directed towards developing skills, abilities and knowledge that will guarantee progress in their practical application of methodological strategies [23]. This is a matter of undertaking a change in the teaching methodology that focuses on the student's learning process, in an educational context which extends throughout their whole study life [24].


**Table 1.** From the traditional model to the active-participatory method.

Source: Drawn up by the authors.

Thus, heritage education does not seek to train experts in a particular subject—on the contrary, the goal is to awaken curiosity and a sense of belonging to a place, based on the knowledge of its closest references, so as to make use of them in a sustainable way [25]. This view allows us to understand and explain the value of heritage elements as references of the community's memory (symbolic-identity dimension), overcoming its material considerations (historical-artistic dimension). It is here where active learning methodologies in primary education promote new methodological strategies based on the implementation of different teaching styles, such as project-based learning, cooperative work, and a flipped classroom, that allow us to work in a cohesive way on heritage, historical, and cultural sustainability.

Therefore, the primary goal of this research was the compilation of methodological strategies used to develop competencies in sustainability and heritage commitment. The following specific objectives are also included to achieve this main goal: (1) promote models of sustainable practices; (2) facilitate the combination of research and teaching; (3) investigate the academic difficulties that students face in recognizing the content of cultural sustainability in teaching; and finally (4) analyze the didactic resources so as to implement an active methodology.

#### **2. Materials and Methods**

#### *2.1. Participants and Sample*

The study population (N = 488) was made up of students in the second cycle of the Degree in Primary Education. Two groups were taking the subject "Didactics of the Environment" in their fourth year (N = 123; 25.2%) at the University of Córdoba, and six groups were taking the subject "Didactics of Social Sciences" in their third year (N = 365; 74.8%)—specifically, 134 students at the University of Cádiz and 231 at the University of Cordoba were included. The average age of the students who participated in this research was 22.

The selection of the sample was not of a probabilistic nature: we selected participants from among groups of participants whose teachers—who were in charge of the abovementioned subjects—had been directly involved in educational research. Another aspect that stands out in this study is the competencies and curricular differences the subjects had, as shown in Table 2. It should also be noted that, although the number of students in Didactics of the Environment was lower, the representativeness of the groups was similar.


**Table 2.** Specific competences of both subjects.

Source: drawn up by the authors.

#### *2.2. Design of the Investigation*

A cross-sectional design was used—it was fundamentally descriptive and interpretative, typical of qualitative research that applies non-numerical data collection and analysis. In addition, it was complemented by a non-experimental, survey-based, quantitative design. To this end, a battery of closed questions was drawn up, in order to analyze the methodological and professional training students acquire in the theoretical and practical sessions of the Degree in Primary Education [26,27]. This type of study is common in research related to the field of education, since it would respond to the amount of learning acquired, together with the evaluation of different professional skills. This methodology has allowed us to collect wide-ranging information on the topics we have worked on.

#### *2.3. Data Collection and Analysis Instrument*

Data were collected through direct observation of the students and through use of a survey, for them to reflect on their initial training in sustainable culture and active teaching methods. The survey was distributed before and after the training, in order to check the degree of knowledge they had gained. This kind of study is very common in educational research, as it can be applied to multiple issues whilst ensuring the validity of the sample [28]. Participants were informed that data collection was part of a study, and their responses were voluntary and anonymous. The research was carried out over three weeks in three practice sessions, lasting one and a half hours each.

The observation system resulted in a class diary which served as a narrative instrument to reflect upon their assessments, hypotheses, and conclusions [29]. We also created Table 3, which consists of several sections grouped in categories, topics and questions, for participants to express their ideas openly. Likewise, the opinions expressed by the students about both subjects—dealing with concepts such as "sustainable", "urban/rural environment", "heritage", and "empathy"—were analyzed through a qualitative evaluation.

We also created Table 4—an overview of the training activities that articulate teaching and learning—using a Likert rating scale of five values ranging from 1 (Strongly Disagree) to 5 (Strongly Agree). Our goal was to respond to the objectives and students' assessments of the different teaching methods in their university studies. In addition, regarding both the design of the survey and data processing, some key questions were defined to identify priority issues. This tool was called "perceptions and beliefs on cultural sustainability and active learning methodologies" and made use of a Likert scale (1–5) (see Appendix A), divided into two sections. Five experts from different Spanish universities took part in its validation. Once both its structure and content were approved, this instrument was delivered before starting the project (pre-test). After finishing it (post-test), the tool was complemented with pre-tests and post-tests to assess the students' opinions on the methodological

teaching at the university, as shown in Table 5. Finally, in the analysis and interpretation of the information, an Excel spreadsheet was used as technical support, so as to make the required graphs and calculations.



Source: Drawn up by the authors.


**Table 4.** Most common learning methods in the study subjects.

Source: Drawn up by the authors.


**Table 5.** Opinions of the students regarding historical and heritage education (N = 488).

Source: Drawn up by the authors.

#### **3. Results**

The most significant results are from information gathered from direct observations and the narratives collected in the journals. The most outstanding themes focused on the methodologies used in the theoretical and practical sessions of the university. The students stated the following: "Many of the methods implemented by the teaching staff are not useful for our training as teachers".

These results also show the impact and implementation of a learning method in a sustainable education subject and heritage culture, where less innovative educational resources are used and which, consequently, are closer to the style of a master class, with taking notes as the learning tool. In line with this, the following was stated: "The sessions of the subjects are monotonous and focus on taking notes"; "It is standard practice to learn the characteristics of the more traditional methodologies".

In fact, when competence-based training was mentioned, they pointed out that "competence-based work is not covered in our practical sessions at the university". They added that "the content is not motivating in terms of the way it is explained by the teacher". They made clear, with their reflections, that the teaching methods used in their university training do not enrich their future as professional teachers. Similarly, when they talk about their experiences with the most active learning methods, they point out: "Active learning methodologies are not explained in these subjects". In this context, they say the following: "We know about some of the most innovative teaching tools and resources because we have dealt with them in other subjects".

In this context, it is now clear that the training of future teachers does not reflect a procedure focused on experimentation with practical issues based on real situations, but rather reflects a particularly theoretical approach. For this reason, the students demand the following: "We would like to learn what project-based learning (PBL) or more active methodologies consist of". It is certain that the opportunity to acquire more innovative knowledge about methodology leads to the development of professional skills from the university period onwards.

To these contributions, we must add the following: "Our teacher does not use historical or natural heritage to teach about sustainable development." This would indicate a lack of interest on the part of the teaching staff around including examples of curricular sustainability. Finally: "The use of ICTs should be directed towards the search for—and analysis of—historical and sustainable resources"; and also: "Digital tools provide new, up-to-date knowledge to discover economic, social, and environmental problems".

In terms of the categories related to terms and expressions such as "sustainable", "urban/rural environment", "heritage", and "empathy", opinions were based upon their personal learning experiences. The answers to each of the questions accurately described the relationship between sustainable culture and development. In practice, they were interested in expressing their doubts about both expressions: "I don't understand the relationship between sustainable culture and sustainable development, as well as the meaning of both concepts"; "I think that sustainable development and sustainable culture are not related, they are different concepts".

However, it is clear that knowledge about environmental education is not completely absent. The students think about the responsible management of the natural areas of our city, such as the Sotos de la Albolafia and the conservation of the Guadalquivir River. In fact, they wondered: "Are these two natural areas protected?"; "Who conserves them legally?"; "Do the current regulations preserve these natural monuments?"

On the other hand, the interpretations about urban and rural locations of the city were specific and partial. They highlighted the following: "To me, the difference between an urban and a rural environment is that the former is inhabited by humans, whereas the latter is not". To which they added discreetly: "I'm not quite sure ... but I think 'urban' would be a city, and 'rural' would be a field or land where no one has to live".

In terms of the issues raised in relation to historical heritage and empathy, the opinions shared a common factor: the approach to heritage assets that they did not associate with cultural sustainability. This is how they pointed this out: "I know the environment of the Mosque-Cathedral better thanks to a didactic trip with my class group at the university". Alternatively, they indicated the following: "I think that a monument like the Mosque was placed here geographically because of its historical importance"; "I don't know the causes and consequences, but it could be because of the importance of Cordoba in history".

The answers in the qualitative evaluation show the students' difficulties in translating these concepts into practice and building them from reflection. Based on all the student contributions, it is remarkable that they were not able to link sustainability to environmental education, as well as to the heritage and cultural assets of their city. These conceptual shortcomings provide an accurate description of their lack of knowledge within their professional development. In their assessments, they emphasized the value of natural and cultural heritage, even if they did not quite understand the relationship between the topics and the categories presented to them for comment.

With regards to student assessments and the different types of teaching methodologies used in their undergraduate studies, their knowledge of more traditional methodologies was where they obtained better marks, thanks to memorizing theoretical concepts. Nonetheless, autonomous work and feedback in working groups also occupy a distinct place, although they recognized that they have made greater efforts to link theory and practice.

As Figure 1 shows, in both subjects, around 4.2 and 4.6 out of 5 of the quantitative evaluations recognized that the traditional and lecture-based methodology plays a leading role in most of their university education. At the same time, between 3.8 and 3.9 of the assessments confirmed that an active and participatory methodology favors meaningful learning. Therefore, it should be noted that the teaching methods implemented today continue to give priority to the teacher's presentation, dictating notes, and the theoretical exam. Even so, they agreed to carry out research and use digital resources to interpret heritage and natural assets.

On the other hand, as Table 5 shows, the questionnaire responses suggested that the acquisition of methodological skills and abilities linked to historical and environmental education lead to higher values in subsequent tests than in preliminary tests. In fact, the results obtained show a moderate interest in the acquisition of educational skills. The vast majority of the items obtained had a value between 3 and 4, both in the pre-test and in the post-test, being higher in the latter. The items best considered in both tests corresponded to item 10 (4.41 and 4.46), showing a significant degree of interest in learning the evaluation instruments of an educational project to adapt them to Primary Education. The lowest values were related to the design of didactic itineraries (3.08) in the pre-test, and the learning of historical content (3.37) in the post-test, possibly because these are topics that the students cover more frequently in their university studies.

**Figure 1.** Comparison of students' methodological assessments according to the subject. Source: Drawn up by the authors.

Based on the coefficient of variation, the values in the post-test are more distributed, less homogeneous, and not centralized with respect to the mean, presenting more disparate values than in the pre-test. From these data, it can be verified that at the beginning of the project, the students were motivated and willing to undertake a didactic experience that they were not familiar with. However, after the implementation of the proposal, their interest decreased; we suppose that this may be due to the historical knowledge that they have of their university degrees, and the lack of practice in the development of this type of didactic proposal.

The results of the qualitative and quantitative evaluation show the difficulties of students in recognizing heritage elements and cultural sustainability. Likewise, the interpretation of environmental and historical concepts was a challenge during the evaluation due to the students' lack of prior knowledge. In the acquisition of this conceptual content, we considered the assessment of the analyzed geographic space and historical time. Thus, the qualitative and quantitative results are similar for the groups of students from both universities, despite them being engaged in different subjects. This indicates the same patrimonial and natural identity for the university students, in addition to similar degrees of interest in learning teaching methods to transmit this knowledge.

#### **4. Discussion**

In this study, the opinions given by the students at the Universities of Cádiz and Córdoba were an essential factor in finding out the methodological shortcomings and lack of knowledge of the didactic models used in their lessons. This has a negative impact through familiarizing students with working in a traditional way, such that they learn little about the issues of cultural sustainability. Meeting the challenge of thinking historically and sustainably lies in the training of future teachers in the university environment, in such educational stages as Primary Education.

Unfortunately, the university training assessed in this study does not delve into the most relevant aspects for the students' future teaching performance, nor does it work on the skills needed to address real classroom problems [30]. Thus, this methodological approach must be accompanied by strategies favoring the search for solutions to problems experienced in the real world, and must not be limited to memorizing conceptual contents in a way that makes little sense in a society where information is available at all times.

This study does have several limitations. One is related to the lack of methodological knowledge of the students' educational practices. It is a fact that the doubts of the students come from their university training and the lack of innovative teaching methods, which are scarce compared to expository methods. These static and not very novel methods have a negative impact not only on their academic knowledge, but also have a demotivating impact and practical implications for curriculum development. Regarding the strengths of the study, the students showed interest in being trained in active learning methodologies, in order to understand their closest environments and implement these methodologies in their professional futures.

Nevertheless, despite the training in citizenship and sustainability skills and their use in schools, teaching of cultural sustainability is still attached to the lecture-based learning mode and use of written tests to evaluate conceptual content. These practices do not allow us to analyze present and past events, nor to establish differences or similarities that would make cultural heritage and sustainable development a discipline which is both evolving and interesting.

Within the quantitative and qualitative evaluation investigating teacher training needs and cultural sustainability, the need to develop active learning methodologies to provide "education for sustainability" in teachers was analyzed. The literature on this subject shows that the development of an evaluation rubric to analyze generic sustainability competencies and determine the level of introduction of sustainability in teaching-learning activities is relatively limited [31].

In this paper, we discuss the different levels of achievement in terms of competencies, taking into account, on the one hand, the interests of the students in their training process [32], and, on the other hand, that not all subjects have the same level of acquisition of competencies in sustainability. This experience was also developed with students and teachers of Primary Education, who recognize the importance of citizenship skills not only in educational practices, but also in the community itself.

#### **5. Conclusions**

Keeping in mind the limitations of the study, we must remember that the personal views of the students are clear on the insufficient training of new teachers [33], and the problems they have in distinguishing between such concepts as sustainability and environmental education. In this regard, Moraes and De la Torre [34] stated that environmental education should be shaped as a permanent meeting space between students, teachers, and knowledge, oriented towards the common purpose of understanding citizen training.

However, there is one positive and common aspect to the views of these future primary education teachers: their affirmative responses on issues related to sustainable education and the practical impact it has on their professional training. Despite the opinions related to the learning of these topics in order to facilitate their knowledge and implementation in a classroom, doubts were raised about the function and definition of these assets. As O'Byrne, Dripps, and Nicholas [35] indicated, notwithstanding the proliferation of academic papers which propose definitions and standards for the field of sustainability and its core concepts, less research has been done to evaluate the state and curricular content of existing degree programs in terms of sustainability.

The results of this study, regarding education for sustainability, raise some questions regarding the educational practices that can be replicated in Primary Education classes [36]. It should be borne in mind that the methodological approaches the students in this study have received in their university training may favor or hinder the acquisition of a series of skills which can be useful for exchanging proposals and reflecting on teaching methods as a teacher. In this context, proposals for improvement in their personal training can support students to get involved and take charge of their own professional practices [37].

Finally, it should be noted that, from our research, in addition to meeting the objectives set, we have concluded that the difficulties students may face when exposed to an innovative and more active methodology depend not only on the training of their university teachers, but also on the priority given to conceptual content as opposed to educational skills [38]. Adequate initial training of new teachers translates into the acquisition of didactic elements present in the teaching–learning process, allowing them to design specific actions addressing both the causes and the consequences of education for sustainability [39,40].

**Author Contributions:** Both authors have contributed in the same way in the realization of this work. The evaluation has been carried out by authors equally. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Spanish Ministry of Science, Innovation and Universities, grant number PGC2018-097481-B-I00.

**Acknowledgments:** The authors gratefully acknowledge the support of the universities of Cádiz and Córdoba (Spain).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**

Scale for evaluating students' opinions: "Perceptions and beliefs on cultural sustainability and active learning methodologies".


Source: drawn up by the authors.

#### **References**


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Effects of Verbal Interactions between Students on Skill Development, Game Performance and Game Involvement in Soccer Learning**

**Wissam Ben Khalifa 1,2,\*, Mehrzia Zouaoui 1,2, Makrem Zghibi <sup>1</sup> and Fairouz Azaiez <sup>3</sup>**


**Abstract:** The purpose of the study was to examine the effects of verbal interaction between students on skill development and soccer game performance within a socio-constructivist perspective and a cooperative learning model in team-sport teaching. In addition, the usefulness of open verbalization was manifested as follows: (1) a social tool for both actors (teachers and students) to collect and manage reports on their thought processes; (2) a tool to stimulate reflection and critical reflection on performance to induce transformation during game action projects. Participants were 18 boys and 12 girls aged (15 ± 0.4 years) from a Tunisian school (ninth grade). They were placed in either the experimental group (with verbal interaction) or the comparison group (without verbal interaction) and then were tested before and after a 12-lesson soccer unit (approximately two hours/week). Skill competence was assessed using three tests: a 15 m ball dribbling test, the Loughborough Soccer Passing Test (LSPT) and a shooting accuracy test. Game performance was measured using the Game Performance Assessment Instrument (GPAI) in which the outcome variables assessed included (a) decision-making (DM), (b) skill execution (SE), (c) support (S), (d) game performance (GP), and (e) game involvement (GI). While both groups showed significant improvements in their short-passing ability, no such improvements were found in dribbling and shooting. In contrast, only the verbal interaction group produced significant improvements in overall game performance. In conclusion, if the objectives of the physical education curriculum are to promote team-sport teaching methods and quality game play, and create a reflexive learner, verbal interaction may be an effective tool for developing tactical understanding through cooperative learning.

**Keywords:** debate of ideas; questioning; teaching games; skill execution; decision-making

#### **1. Introduction**

One of the main strategies for improving school learning in a physical education class has been identified as increasing opportunities for discussion [1]. Discussions and the time required for face-to-face interaction allow students to solve problems and engage in thinking [2,3]. Interactions promote learning outcomes and are more task-oriented than trivial and focused on something other than the task [4]. Darnis and Lafont (2015) [3] recapitulated this main finding when they suggested that cooperative student discussions (dyadic level) about their game strategies resulted in better motor and tactical skills.

The current study centers upon the use of verbal interactions between students as a cognitive strategy during skill development and subsequent game performance. While interactions can take both verbal and non-verbal forms, verbal interaction is the primary medium of instruction in physical education classes and is often utilized by teachers and

**Citation:** Ben Khalifa, W.; Zouaoui, M.; Zghibi, M.; Azaiez, F. Effects of Verbal Interactions between Students on Skill Development, Game Performance and Game Involvement in Soccer Learning. *Sustainability* **2020**, *13*, 160. https://dx.doi.org/

Received: 24 November 2020 Accepted: 10 December 2020 Published: 25 December 2020

10.3390/su13010160

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/ licenses/by/4.0/).

students to discuss tactical problems and resolve potential conflicts [5]. Consequently, verbal interaction is an important part of human interaction. Such interactions can be defined as total relationships that are reached through speaking, conversation, discussion and debate of ideas [6].

Many education stakeholders have argued that students, rather than the teacher, should be the focus of the teaching/learning system [7]. Student-centered approaches reference exploratory, discovery, cooperative, active, participatory or project learning [8,9]. Despite some differences in the implementation of different student-centered frameworks, all are based on the common principle that "the only learning that significantly influences behavior is self-discovered" [10]. In this context, there are a number of common features that characterize student-centered learning strategies: (1) The decision-making responsibilities associated with the planning, implementation and evaluation of activities are transferred, at least in part, from the teacher to the student [11]. (2) The teacher usually assumes a facilitating role [12], and as facilitator, his or her role is to encourage and support the appropriation of the learning process by students. (3) Students are explicitly invited to learn from other students and to educate them through peer or small group teaching [13] and peer-to-peer working groups based on complementary interests or levels of performance [14,15]. (4) The teacher prompts students to reflect thoughtfully and creatively and challenges them to find solutions to problems they encounter [16]; in this context a current recommendation for student-centered design, for example, is that the teacher use the "questioning" tool [17,18] and debate of ideas [19]. (5) The teacher encourages students to learn more about their development and skills and, consequently, adjust their learning strategies [8].

In such a student-centered approach, students are encouraged to make sense of a new contribution in their learning. This contribution is linked to their previous knowledge and, working with their peers through sustained verbalization and debate, to building a shared understanding [6]. The use of verbalization as a cognitive strategy has its roots in the socioconstructivist approach of Russian psychologists such as [20]. The socio-constructivist approach is based on a key premise, namely, the Zone of Proximal Development (ZPD). From a theoretical point of view, Vygotsky's concept of ZPD presents a helpful means to think about the importance of language and verbal interaction in learning [17]. We use Vygotsky's notion of ZPD and the complement of the game-based approach to improve learning [21,22]. Both these theories emphasize the role of language in learning, for example, in the discussion/verbal interaction between students and between students and teachers in the game-based approach [23]. These conditions respond to Vygotsky's postulates and are adequate in our opinion to report the creation of a ZPD. Roth and Radford (2010) [24] proposed that ZPD can be considered as an interactional implementation that allows all participants to become both teachers and learners. They argued that it is useless to think of participants in terms of expert (higher skill level) and novice (low skill level) because this vision masks the fact that experts and novices must demonstrate cultural competence to participate in discussion so as to lead to learning. If knowledge is required in interaction, the learning can be multi-dimensional (take place in any direction) [6]. As a result of these theories, many authors have engaged in the search for teaching techniques aimed at putting students at the center of their own learning. Indeed, during the verbalization sequences, the spontaneous declarations of the learners convey meanings about situational state-action. These revealed meanings attributed to the state-action characterize the development of strategic and procedural knowledge that may be related to more gaming experiences of a particular activity [25].

At the pragmatic level of collective sports intervention, studies on the verbal interactions and debate of ideas have focused on two pedagogical paths, suggesting implementations of the debate of ideas at different moments of the learning process. On the one hand, Gréhaigne and Godbout (1995) [26] presented an operationalization in three stages: a first play time (action time where students are in action); a second stage for co-observation and co-evaluation; a third time for the debate of ideas. In this perspective, the time for debate

and verbal exchanges is based on the definition of social roles (e.g., observing, listening, taking the turn of speaking roles) serving as a support for co-constructions of rules of action. On the other hand, the Teaching Games for Understanding (TGfU) model was originally developed by Bunker and Thorpe (1982) [27] and offers an alternative to the technicist approach of team sports at school. From small-sided game situations, learners are invited to "appreciate the game" to conceptualize the aim, then to become aware of important tactical aspects to solve the problems posed by game competitions. To encourage the co-emergence of pragmatic concepts that are useful for the development of team sports skills, we also build on the work of Chang et al. (2006) [28]. They proposed to set up phases of exchanges, debatesofideas within the teams between game sequences. They highlighted the phenomena of extracting the rules of effective action from a dialogical and shared space of cognition within an 11-year-old team in a basketball unit.

In addition, from the dual perspective of cooperative learning and TGfU formats, Dyson and Casey (2012) [29] developed proposals for cooperative learning in team sports. Their proposals are based on the use of social roles (e.g., observers, coaches) and small group discussions to promote learning in team sports among young and novice students.

Teaching conceptions in this study are largely inspired by the work of Dyson and Casey (2016) [30] and Darnis and Lafont (2015) [3] in an integrative approach of cooperative learning, verbal interactions applied to motor learning and didactics of collective sports [31].

Numerous studies across various educational contexts have shown that when learners are encouraged to verbalize about what they learn, it improves their learning [32–34]. In previous work [31], verbal interaction and debates of ideas between peers provide the construction of action rules and information when participants are faced with a problemsolving setting allowing them to focus on specific tactical rules [3,35]. The construction of tactical skills in a specific game learning situation can be presented as implicit teaching associated with a socio-constructivist approach [3]. From this perspective, Chang et al. (2006) [28] suggested that language production about action strategies produces a positive effect on basketball learning, the construction of effective action modalities and game organization. Another study by Lafont, Proeres, and Vallet (2007) [36] reported the positive effect of verbal exchanges on interpersonal relationships and tactical acquisition and on shooting in a basketball team game among French primary school children. Following this theoretical framework, García-López and Gutiérrez (2015) [37] insisted on the need to study closely the interactions of students during group learning.

It should be noted, however, that in most of the studies conducted to date, the effects of verbal peer interaction were studied where the dyadic learning unit consisted of two students. In this study, the focus was on group learning in a more macro-analytical design (small groups of five students).

The aim of the present study was to examine the impact of including opportunities for verbal interactions between students during a unit of soccer. Specifically, it was hypothesized that students in the experimental group would achieve higher post-test performance scores in both technical skill tests and game performance measures than those in the comparison group who did not engage in verbal interaction between game sequences.

#### **2. Method**

#### *2.1. Participants*

The participants in this study were 18 boys and 12 girls (age, Mean (M) = 15.4 and Standard Deviation (SD) = 0.59; experience in football practice, M = 3.86, SD = 1.81) from one ninth-grade physical education class (60 min of effective learning, once a week) in a Tunisian school. The research was conducted in accordance with the guidelines of the Declaration of Helsinki. The participants and their parents were informed about the study details. Then the parents signed an informed consent form as the studied students were under 18. The research project was approved by the Scientific and Ethics Committee of the High Institute of Sports and Physical Education of Kef (Tunisia). The teacher was an experienced researcher in the didactics of collective sport games and a football (soccer) specialist (football trainer certificate and practice). Before starting the study, he had experience in using cooperative learning and teaching small-sided games with different age groups in Tunisian school.

An introductory lesson was organized for team selection. As a method of team selection suggested by Siedentop (1994) [38] and recently used by Farias, Valério, and Mesquita (2018) [39], six students were elected to form a selection committee that cooperated with the teacher to compose six heterogeneous but balanced teams. Each team included the same number of girls and boys and different skill levels (from lower to higher qualified).

After this procedure, teams were randomly but equally distributed to an experimental and a comparison group learning condition. The experimental group (*n* = 15) was assigned as the "motor learning + instructions + verbal interaction" learning condition, while the comparison group (*n* = 15) was designated the "motor learning + instructions" learning condition. Students were allocated to three equal teams in each group. Each of the teams within both the experimental learning condition and comparison learning condition was comprised of five students (four players on the field and one substitute).

#### *2.2. Pre-Intervention Phase*

Before starting the intervention, both groups were invited to complete thefirst part of the course (eight sessions with two per week, each lasting 60 min). Each session was focused on the acquisition of technical skills (i.e., pass, dribble and shooting) and tactical skills (4 vs. 4 small-sided games at the end of the session). The small-sided game (2 × 6 min) put in opposition a team from the experimental group (with verbalization) and a team from the comparison group (without verbalization). This phase consisted of teacher-centered lessons, which aimed to: (1) teach the necessary declarative and procedural knowledge about appropriate tactical decisions, and (2) remedy social skills (e.g., encouragement, listening, respecting others) during the debate of ideas sequences in the experimental group. These sequences were filmed for use during the meetings (35 to 45 min after each session) with the students of the experimental group to stop in a positive climate of conversation. In this context, Soller's (2001) [40] taxonomy of conversation skills in collaborative learning was used. The taxonomy is designed to facilitate the recognition of an active learning conversation. It introduces each learning conversation skill (Active learning, Conversation, and Creative conflict) into sub-skills (for example, Request, Inform, Acknowledge) and attributes (for example, Suggest, Rephrase, Suppose, Explain, Justify) (for more details see [41]).

#### *2.3. Intervention Phase*

All students participated in the same 12 teaching sessions. Each teaching session focused on one major operational objective to assist the students in developing their tactical understanding of offense, defense, and associated techniques within soccer using a small-sided game format (i.e., 4 vs. 4). These are listed in Table 1.




**Table 1.** *Cont.*

Each session was scheduled for 60 min. Lessons began with a standardized 15 min warm-up consisting of jogging, coordination movements and dynamic stretching, and ending with 4 × 10 m sprints. After the warm-up, each team in the experimental group played a 12 min game against a team from the comparison group (2 × 6 min halves + 3 min half-time). During the half-time period, the students in the experimental group engaged in three minutes of verbal interaction (play–discuss–play) while the students in the comparison group engaged in passive recovery (play–passive recovery–play) where no verbal interaction was permitted. The verbal interactions between students in the experimental group were videorecorded to ensure the participation of all students during the verbalization. The time for verbal interaction was limited to three minutes [3]. Teams rotated through a (a) warm-up, (b) play, (c) observation cycle throughout this game-play lesson segment. Figure 1 provides a graphic of the sequence of activities for each team in the experimental condition.

**Figure 1.** Structure for each team in the experimental condition.

#### *2.4. Verbal Interaction*

The use of the debate of ideas aimed to increase the level of interaction within groups during questioning episodes [19].

#### 2.4.1. Questioning

As in Metzler's (2017) [12] tactical games model, the games concept approach was based on higher-order questioning [42]. These questions helped students reflect on information and commit it to memory; they could be used as a management tool to draw

students into the lesson and keep them focused; and they could develop thinking skills, creativity and encourage discussion/debate of ideas.

#### 2.4.2. Debate of Ideas

By regularly offering confrontation and reflection times during the motor learning lessons, students were able to acquire an approach that would make them more aware of their actions [43]. Over time, provoking student verbalization has become an important approach in physical education, despite the fact that motor tasks are more related to perceptual-motor and intellectual characteristics [44]. In this context, Gréhaigne and Godbout (2014) [45] highlighted the interest of the debate of ideas. They defined this didactic process as a situation in which students explain and exchange ideas about the facts (tactical problems), based on observation or personal experience.

#### 2.4.3. Tactical Problems Discussed in Debate of Ideas

The students' debate of ideas focused on the tactical problem of placement [46], the tactical problem of maintaining possession of the ball [47], identifying the particular strengths of the opposition team, an action plan to deal with these strengths in the previous game and an action plan in order to be effective in the next part of the game [19].

#### *2.5. Intervention Validity*

The teacher's role in each of the teaching sessions was to (a) present the learning objective to the entire class at the beginning of each session, and to explain how this might be realized within the small-sided games; (b) conduct and supervise the standardized student warm-up; (c) provide individual and collective instructions and feedback to each of the teams; (d) provide prompts for the students in the verbalization learning condition and facilitate the debate of ideas between these groups between game 1 and game 2; and (e) provide a recap of the lesson for the entire class.

The principal researcher accompanied the teacher to each lesson in order to validate the pedagogical approaches during the intervention phase of the study. Prior to each lesson, the researcher checked the teacher's lesson plans to ensure that session objectives, instructional cues, deductive questions and prompts for the student verbalization were present. After each lesson, the researcher discussed his observations with the teacher to ensure fidelity to the teaching sequence highlighted above.

Metzler (2017) [12] emphasized the importance of verifying that any model implemented in physical education led to the expected learning outcomes of students. Using word-for-word spreadsheets, we believe that the teacher met the requirements of how to teach games, as identified by Mitchell, Oslin, and Griffin (2013) [47], and thus had a relatively high fidelity retention [48] in the implementation of the tactical act model in games teaching. During the completion of cooperative learning (CL), the teacher ensured that the key characteristics [49] (i.e., positive interdependence: links between group members; individual accountability; face-to-face interaction: head-to-head discussion within the group; interpersonal and small group skills; group processing;small heterogeneous teams; group goals, and teacher-as-facilitator) took place along the unit.

#### *2.6. Data Collection*

Pre-and post-test data were collected during the students' regular physical education lessons.

#### 2.6.1. Technical Skills

*15-m ball dribbling*. A 15-m ball dribbling test was used to assess the student's ability to control a ball while moving [50]. Each participant began 3 m behind an initial photocell gate, and after 3 m of moving in a straight line, entered a 3 m slalom-dribbling section marked by three sticks of 1.6 m height and placed at 1.5 m from each other. The ball was then kicked under a 0.5 m height hurdle placed 2 m from the third stick while the participant crossed it. Finally, the participant freely kicked the ball toward either of two small goals placed diagonally 7 m on the left and right sides of the hurdle and ran 7 m to the finish line, where the second photocell gate was placed to stop the timer.

*Loughborough Soccer Passing Test (LSPT)*. The modified version of the LSPT was used to measure short-passing ability. In this version, the contribution of decision-making was added to the original passing test [51,52]. LSPT total performance time (time necessary to complete the test after adjusting for penalties and/or bonus time) was selected for analysis. The LSPT has been shown to be both reliable and valid, with the detailed protocol and a schematic representation of the test available from [52] or [53]. All participants were familiarized with the LSPT during two practice sessions prior to testing.

*Shooting test*. To assess shooting on goal, the participants completed the Mor and Christian (1979) [54] test, which involved sending a ball through a series of circular hoops attached to each corner of the goal. The shooter started at the lower left target, proceeded to the lower right target, then aimed at the upper left target, and finished with the upper right target. The total successes of 16 trials were scored.

#### 2.6.2. Game Performance

To measure soccer game performance, the Game Performance Assessment Instrument was used [55]. The game played during the pre-and the post-tests was a small-sided game (SSG): 4 vs. 4 players on a 20 m × 40 m pitch size [56] on an outdoor field. The objective of the participants in the assessment game format was to keep possession of the ball in order to score more goals than the opposing team and win the game. The duration of 4 vs. 4 SSG (10 min) during the pre-test and the post-test were strictly controlled. Both assessments were conducted in similar conditions and at the same time of day to limit the potential effects of circadian variation on physiological variables [57]. Moreover, the teacher offered verbal encouragement to the participants (e.g., good work, keep it up, etc.) to maintain a high work rate during the games. The teacher provided a replacement ball in cases where the ball went out of bounds to allow for an improved continuity of play [58]. Goalkeepers were excluded from the investigation, and the offside rule was not applied.

Three elements of game play from the GPAI, decision-making (DM), skill execution (SE), and support (S), were used to assess students' game performance, and these were the elements evaluated on an individual base from videotapes of game play. The DM category consisted of students making appropriate choices about what to do. For example, if a student in possession of ball decided to shoot it through an open goal or pass to an open teammate when the opportunity was suitable, the coder recorded it as appropriate. When the student passed the ball at a bad time or to a marked teammate, an inappropriate decision was recorded by the coder. The SE category included passing, dribbling and shooting skill and was assessed at each student–ball contact as an efficient or inefficient action (e.g., technique used in the game situation, such as proper gesture toward or away from the ball, changing body posture). The S category was assessed when the player was without the ball. It consisted of students calling for the ball (to be in a favorable position to receive a pass).

To assess the impact of the intervention sessions, observers coded each aspect of game performance separately (i.e., with and without the ball actions, in both attack and defense) via GPAI constructs. DM, SE and S of game performance were tallied and summed for all students both as appropriate/efficient and inappropriate/inefficient for each heading. In accordance with the recommendations of Griffin, Mitchell, and Oslin (1997) [59], performance indices for DM and SE were calculated on the basis of appropriate to inappropriate actions ratio. The decision-making index (DMI), skill execution index (SEI) and support index (SI) were calculated using the following formula: number of appropriate DM, SE or S / number of inappropriate DM, SE or S. Then, a global measurement of appropriate/inappropriate game performance (GP) and game involvement (GI) actions were also calculated by the following formulas:

GP = (DMI + SEI + SI)/3 and GI = Σ (number of appropriate and inappropriate SE, DM and S).

#### 2.6.3. Coders' Training and Reliability

Two coders (with football knowledge) belonging to the Tunisian National Observatory of Sport and having two and three years of experience in observation methodology were in charge of the analysis. As a preliminary step for observations, the coders were trained for approximately 24 h spread over two weeks to analyze decision-making and execution of pass, dribble and shot actions. The training of the coders was carried out by an expert in the methodology of observation (Tunisian Football Association). A meeting was established in advance to clarify and finalize the instrument of observation and coding criteria. Intercoder reliability was checked via the observation of two 10-min games not used in the study. Inter-observer reliability was calculated by the formula: Agreements/(agreements + disagreements) ×100, and agreement for the sampled two games was higher than the 80% set prior to data collection (89.7%).Once this value was calculated, Cohen's kappa index was used. Values greater than 0.90 for all dependent variables were obtained, and those exceeding the value of 0.81 from which adequate agreement was considered. The data thus achieved the reliability necessary for subsequent coding.

To ensure the temporal reliability of the measurement, Cohen's kappa index was used. The same coding was done twice, with a time interval of 10 days. Cohen kappa values were between 0.81 and 1.00.

#### *2.7. Data Analysis*

Statistical analysis was performed using SPSS version 24 for Windows (SPSS Inc., Chicago, IL, USA). The normality of data sets was checked using the Kolmogorov–Smirnov test, which led to the use of parametric statistics. A 2 × 2 repeated measures analysis of variance (ANOVA) was used to examine the effect of "teaching method" (with or without verbal interactions), "time" (pre- and post-intervention sessions) and their interaction (teaching method × time) on each of the eight dependent variables (the skill tests of passing, dribbling, shooting, GP, DM, SE, S and GI) with a priori alpha set at 0.05.

When a significant interaction effect was found, posthoc tests were completed where Bonferroni corrections were employed to control for the multiple comparisons to protect against type-1 error. Magnitude of change expressed as Cohen's d coefficient was employed to give a rigorous judgment about the differences between the two teaching methods [60]. Effect sizes (ES) were considered trivial, small, medium and large for values of 0 to 0.20, >0.20 to 0.50, >0.50 to 0.80 and >0.80, respectively [61].

#### **3. Results**

Descriptive data generated from each of the eight dependent variables (the skill tests of passing, dribbling, shooting, GP, DM, SE, S and GI) for each of the two groups at each time point (pre- and post-intervention) are presented in Table 2, while the inferential analyses appear in Table 3.

**Table 2.** Statistics for skill tests, game performance, and game involvement.




Note: M = Mean; SD = Standard Deviation.

**Table 3.** Statistics of the ANOVA with 2 × 2 repeated measures teaching method (with and without verbalization) × time (pre and post).


Note: LSPT = Loughborough Soccer Passing Test; GP = Game Performance; DM = Decision-Making; SE = Skill Execution; S = Support; GI = Game Involvement. Only significant effects are indicated, with \* *p* < 0.05; \*\* *p* < 0.01; \*\*\* *p* < 0.001.

#### *3.1. Skill Tests*

For passing skills, there was a significant main effect for time and a significant teaching method × time interaction. The post hoc tests showed that there was a significant change between pre- and post-intervention in passing skills (LSPT final time) in both groups, but times were significantly lower for the experimental group (Figure 2). No significant effects for dribbling and shooting skills were found (Table 3).

**Figure 2.** Mean skill tests performances both for groups with verbalization (WV) and without verbalization (WOV) teaching methods collected before (pre-intervention) and after (post-intervention). Error bars indicate within participants' standard deviation. \* Denotes a significant difference between values.

#### *3.2. Game Performance*

Significant main effects for time and a significant teaching method × time interaction were observed (Table 3). Posthoc comparisons revealed that there was a significant difference between the pre-intervention and post-intervention of game performance for the experimental group (Figure 3).

For decisions, significant main effects of teaching method was observed, while both groups improved from pre- to post-test, the experimental group had significantly greater gain scores (Table 3). Posthoc comparisons revealed that there was a significant difference between the pre-intervention and post-intervention for the experimental group (Figure 3).

For skill execution, both groups improved from pre- to post-test, the experimental group obtained significantly greater gain scores (Table 3). Posthoc comparisons revealed that there was a significant difference between the pre-intervention and post-intervention for the experimental group (Figure 3).

For support, a significant teaching method × time interaction was observed; while both groups improved from pre- to post-test, the experimental group had significantly great gain scores (Table 3). Posthoc comparisons revealed that there was a significant difference between the pre-intervention and post-intervention for the experimental group (Figure 3).

#### *3.3. Game Involvement*

Significant main effects for teaching method and time were observed, but no teaching method × time interaction. Posthoc comparisons indicated a significant difference was present for both groups from pre- to post-intervention with a slight improvement of the experimental group (Figure 3).

**Figure 3.** Game performances both for groups with verbalization (WV) and without verbalization (WOV) teaching methods collected before (pre-intervention) and after (post-intervention). Error bars indicate within participants' standard deviation. \* Denotes a significant difference between pre-intervention and post-intervention values for each teaching method.

#### **4. Discussion**

The aim of the present study was to examine the impact of including opportunities for verbal interactions between students during a unit of soccer. It was hypothesized that students in the experimental group would achieve higher post-test performance scores in both technical skill tests and game performance measures than those in the comparison group who did not engage in verbal interaction in between game sequences. The discussion will first address the effects of verbal interaction opportunities on technical skills before moving onto highlight differences in game performance and game involvement measures.

#### *4.1. The Effect of Play-Discuss-Play on Skill Tests*

In terms of technical skills, the intervention of between-game discussions was particularly effective in improving the pass technical performance, but not so for dribbling and shooting. A previous study [62] suggested that these results, showing greater improvements in short-passing ability for the experimental (play-discuss-play) group, can be explained by the close association between the action of passing and the decisional compo-

nent leading to improvements in the execution of a pass during the game [63]. Indeed, a recent study [64] suggested that passing complexity resides more in deciding "who to pass to" and "when" than in the technical execution of the skill itself. An additional plausible explanation for these results may be that the content of the 12 lessons focused on passing and movement rather than on the other two technical skills of dribbling and shooting. At the same time, we cannot forget that the factor related to the quality of student practice was identified as the most crucial factor for their learning [65], although it is important to grade the importance of task organization to enable learners to accumulate high-quality repetitions, as this has been found to correlate with improved skill learning. Related to the first point, during learning sessions the frequency of shots on goal was low and unequally distributed among participants. In addition, during initial learning there may be more limitations in some technical aspects of play (i.e., dribbling and shooting) when compared to others. Finally, the dribbling test showed a lower level of reliability compared to the other technical skill tests (0.74).

#### *4.2. The Effect of Play-Discuss-Play on Game Performance and Game Involvement*

Major results of the present study showed the effect of learning in small groups, where play-discuss-play in the experimental group was more efficient, both individually and collectively, than the comparison group on the GPAI-dependent variables (i.e., DMI, SEI, and SI) post-intervention. These differences were also significant at the post-intervention assessment in terms of GP and GI for the experimental group. It is therefore assumed that the changes observed in the current study were due to the verbal interactions that were afforded to students in the experimental group.

The results of this study are one attempt to extend the application of verbal interaction in learning games at a macro-analytical level where decision-making is necessary to enhance students' game performance. The results support the socio-constructivist perspective to team-sport teaching where researchers have suggested that verbal interactions improve the construction of tactical knowledge by students and the development of their decisionmaking abilities [3,66,67]. Effective decision-making strategies are developed through verbal interaction in small cooperative learning groups that enable effective long-term changes in tactical behavior. Blakemore and Robbins (2012) [67] affirmed that decisionmaking in adolescents, like the participants in this current study, is remarkably sensitive to social contexts (verbal interactions), since it occurs in a competition game with peers. In addition, verbalization and reflection on their own performance support the need for participants to be more aware of the main informational constraints they may face in their future competitive performances [68].

Specifically, the character of content development conceptions and explicit teaching strategies ranked in priority order in the current study from the preparation phase to the end of the intervention (direct instruction, discovery-based instruction, debates of ideas), and the quality of engagement of students in problem-solving processes (e.g., identification of tactical issues and team building of tactical solutions) had an effect on game performance development. Furthermore, the evolution achieved by the experimental group through this teaching unit reflected the cognitive and situated learning processes [69]. Prior studies [70,71] argued that to have a more effective game-play, perception, understanding and reflection of learners are key elements. Progress in the ability of students in the experimental group to play team games (e.g., soccer) arose as a result of an interdependent relationship between the pedagogical conceptions used and the level of cognitive and social engagement that stems for students from subject matter [72].

In this study, a solutions-based approach to content development [73] was implemented through pre-established verbal interaction sequences (debate of ideas) at specific moments of the game [74]. This design (play–discuss–play) had a positive effect on students' decision-making, skill execution, support and game involvement but not on technical skill tests (in isolation). In this context, Farias, Mesquita, and Hastie (2019) [13] emphasized the use of guidance-based strategies to create effective instructional interactions (e.g., peer-topeer teaching approach). The verbal interaction sequences were solicited by questioning episodes. The prominence of a tactical questioning provided by the teacher in game-based teaching cannot be ignored or neglected; this proposal was recently evoked by Harvey, Pill, and Almond (2018) [75]. This is consistent with studies revealing that active participation of students in the analysis of tactical problems and the search for appropriate solutions or game plans through tactical questioning produces a higher rate of tactical decision-making than do teacher-centered approaches [76,77].

In our study the incorporation of verbal interaction sequences (debate of ideas) in the structure of the learning unit meant that consequently the teacher ceded his place (responsibility) to the students. Their responsibilities manifested themselves in identifying tactical problems, collectively constructing solutions to these and a more appropriate action plan to address problems and provide rules of action [3]. However, Ward (2006, p. 12) [78] stated that "students follow rules such as 'if this ... do this or that' ... " Rules are particularly useful and are likely to play a vital role in social skills training and teaching tactics (Ward 2006, p. 15) [78]. In summary, in the debate-of-ideas setting, each team sets up a first action project, which is then tested in gameplay. This can in turn lead to the development of a new action plan with the implementation of links between rules of action and rules of organization of the game [26]. In doing so, students gradually gather their tactical knowledge and improve their decision-making skills.

Verbalization as a cooperative behavior can contribute to improving team coordination and effectiveness. On this point, researchers noted that effective teams are more willing to ask for and accept help and give or receive feedback [70,71]. In contrast, a lack of communication in the comparison group teams exposed them to an increasing level of abstraction and ambiguity in their monitoring of their game performances [79]. Indeed, Fiore et al. (2003) [80] affirmed that a decrease in team members' situational awareness occurs in the absence of verbal, paralinguistic and other sensory cues.

Our data are consistent with the results found by Mesquita, Farias, and Hastie (2012) [81]. These authors noted that the teaching of a football unit in a sport education design supported by the structure of learning tasks, namely: (a) having strategic problems to solve, (b) the practice of skills in the game situations, (c) students always performing alongside their teammates, and (d) the time and space needed to think about the game, giving students a chance to improve the skills execution as well as their tactical decision-making.

Another study by Nathan and Haynes (2013) [82] showed that a Teaching Games for Understanding (TGfU) based design led to improved decision-making and skill execution related to hockey game learning, knowing that TGfU often uses questioning and discussion/debate of ideas [83,84]. Harvey et al. [84] confirmed that TGfU unit soccer revealed significant changes in game performance (skill execution and support), and overall measures of game performance (game performance and game involvement) were assessed by the GPAI in a 3 vs. 3 soccer game. A recent study aimed to analyze differences in decision-making and action execution after a program of intervention based on the TGfU model [64]. The study revealed a significant improvement in decision-making and skill execution capabilities in players aged between 10 and 11 years after 22 sessions, but not after the first 11 sessions. Our experimental design intervention included a pre-intervention phase (eight sessions) and an intervention phase (12 sessions) and the study population was 15-year-old mixed students and the majority were novice. On the other hand, the population studied by Pizarro and her colleagues were male players characterized to have expertise level in sport. Hence, it appears that males in the latest study showed high pretest scores in offensive decision-making and skill execution (e.g., dribbling), and increasing the defensive pressure for males was not sufficient to elicit a more sophisticated game performance, which may have limited their margin of progression after 11 sessions.

Our findings are consistent with the results obtained by Práxedes et al. [85] in the Spanish sport context among young male football players (10 to 11 years old). Researchers found that after applying the intervention program based on the TGfU model and including the application of questioning and debate of ideas (a maximum of 2 min) in a context of small-sided games (e.g., 4 vs. 4), the players in the experimental group showed better game performance (e.g., decision-making in the pass and dribbling actions), and better skill execution (e.g., in the pass action), compared with the players from the control group. These results suggest that the application of verbal interaction in a context of small-sided games must be taken into account to foster tactical training/teaching in young footballer/learners and to improve their tactical behavior.

In a comparative study of three teaching groups conducted in Belgium using a volleyball practice course among university students [86], found that the student-centered tactical questioning group had significantly improved in terms of tactical game performance (decision-making process), from pre-test to post-test (after five lessons), compared to the other two teaching groups (i.e., teacher-centered and student-centered without tactical questioning). In the current study, students appeared to benefit from the tactical awareness implemented in the student-centered cooperative learning model and the concepts of questioning and debate of ideas as they have been systematically called upon to react and reflect on their own game problems, which could have given increased meaning to the content learned. Tactical awareness in the student-centered instructional group with questioning and tactical discussion must be attributed to the active role of the students in the teaching–learning process, as the tactical awareness of the comparison group (without interaction) did not reach the student performance level of the experimental group. Therefore, the development of decision-making capacity and skill execution mastery of students is based on the development of tactical awareness [87]. The successful negotiation of the scenarios that confront students in games requires the interaction and simultaneous application of tactical awareness/knowledge, decision-making, and skill execution [88]. In other words, these three elements are inseparable [23].

Questioning and debate of ideas are useful tools to develop decision training, and they are often used with other instructional tools (e.g., video-guided debates) [89]. In any case, decision training based on questioning and debate of ideas has proved to be useful for application in sport and teaching contexts. It leads to improvements in skills execution, decision-making skills and tactical skills [83].

The two instructional groups merely differed from each other in the responsibility of the students for the teaching–learning process and the implementation of goal-oriented observations through evaluation time (e.g., formative evaluation) in each practical session during the intervention period. Therefore, qualitative goal-oriented observations and the evaluation of well-described tactical principles of team members and themselves seemed to provoke the performance advantage of the learner-oriented cooperative learning and verbal interaction (questioning and tactical discussion). As part of this idea, formative evaluation is defined "as the iterative processes of establishing what, how much and how well students are learning in relation to the learning goals and expected outcomes in order to inform tailored formative feedback and support further learning, a pedagogical strategy that is more productive when role is shared among the teacher, peers and the individual learner" [90]. Furthermore, formative evaluation and feedback among students were inherent characteristics of our design to help students take control of their own learning. Formative evaluation as a didactic strategy improves self-regulation learning [91]. Selfregulated students should to be able to actively interpret external feedback, for example, from other students, in relation to their internal and shared goals [92]. Feedback construct was exclusively accentuated in the experimental instructional group with tactical questioning and debate of ideas using qualitative goal-oriented observation forms, in contrast with the comparison group. As a result, it is suggested that verbal interaction (asking tactical questions followed by a debate of ideas) regarding these perceptual observations and evaluation times (e.g., formative evaluation) helps to create a thoughtful learning environment in which decision-making and critical thinking are developed [93]. Students in the experimental group seemed to be able to use a knowledge-based heuristic, while other students' decisions (comparison group) were based on a general heuristic of the field. Heuristics can be described as the simple rules that humans often use to make quick and effective decisions with limited information and shared attention to achieve their goals [86].

In each lesson of this study, students practiced the main game (4 vs. 4) in which their performances and their involvement in the game were evaluated during the pre-test and post-test sessions. It was supported that this increase in playing time was advantageous to improving student game-play performance [39]. In addition, the persistent composition of students in the same teams throughout the learning unit involved their participation in the debate sequences devoted to strategy and problem-solving as a group [11,13]. In this perspective, Gréhaigne, Caty, and Godbout [6] stated that the tactical thinking inherent in regular or formal contribution to debate of ideas assists students to deepen their knowledge of the strengths and weaknesses of every member of the team and the frequency of cooperation activities and the exchange of knowledge among team members increased [94].

Regarding GI, students in both groups showed a high level of involvement from pre- to post-intervention. The GI index is calculated by summing the sums of all the tactical behaviors (numbers of appropriate/effective and inappropriate/ineffective actions) for each participant game component. Thus, using such an index in the analysis of the student's game performance and involvement may give a false reading and interpretation. For that reason, the analysis should be treated with caution unless we use their GP index score alongside that of GI [95]. Therefore, having a high GI score does not mean that the comparison group is better than the experimental group in overall game performance since GP showed high values in the second group. In this context, Memmert, and Harvey [95] stated that the use of GP is more suitable for higher levels (i.e., Grade 9) than the GI, as they seek to provide more effective choices as their understanding of the game improves and vice versa for younger students.

However, in motor learning through verbal interaction, team members treat information and make decisions according to a certain quantity and quality of information [96]. Decisions made can be converted into actions (game plan, skill execution) in later task efforts at a very slow pace. These effects need to be monitored and negotiated during verbal exchanges between students. For example, mistakes that are made in game play can be immediately discussed and negotiated by students through verbal interactions between games, offering to the two teaching actors (teacher and student) opportune circumstances for a better learning next time. If such circumstances increase, the performance will improve as a result of the provided comments. This teaching/learning process is supported by progressive and skillful instruction [97] and would include teaching strategies such as questioning [64], debate-of-ideas settings [97], or team talks [95].

The application of verbal interaction (e.g., questioning and debate of ideas) as a teaching instrument has likely had a crucial influence on the obtained findings, and its utility as a tool to enhance game performance (e.g., decision-making and execution skills) can be confirmed [98,99].

Assigning students to groups and expecting them to know how to cooperate does not ensure that this will happen. Appropriate skills for cooperative learning are important for successful group work and are not owned by everyone. Listening to each other, encouraging everyone to participate, and trying to understand each other's perspectives are examples of skills suggested by Gillies (2003) [100]. More positive views on learning outcomes from cooperative learning of older students may reflect improved skills in group work.

In particular, to use the cooperative discussion group, we took into consideration social skills (e.g., encouragement, listening, respect for others) to guide an active learning conversation (see Section 2.2, Pre-Intervention Phase). Other studies need to be more aware of the social context and the concerns and needs of their students such as preferences for working with peers, attitudes to working in groups, friendships (see [101]). This knowledge can increase the productivity of social interactions during the cooperative discussion group [89]. Moreover, students do not learn while playing or repeating technical gestures. In order to achieve the construction of meaning, it is necessary to analyze after the fact what happened. In this case, the importance of the digital competence of teachers is specified in order to develop innovative methodologies linked to socio-constrictive education and analytically managing information in physical education class [102,103]. Methodologies associated with innovation must be an inexorable part of daily educational practice and must be supported by the use of the educational and technological resources available [104].

#### **5. Limitations**

This study is notable in that it was the first to study the effect of verbal interactions between students on skill development, soccer game performance and involvement within a socio-constructivist perspective in team-sport teaching. There are, however, some limitations, the first of which is the small sample of student participants. Future research is needed with an expanded and varied population (e.g., age, gender) to provide more generalizable results.

Another limiting aspect of this study is that its design was limited to the analysis of quantitative data. In future studies, there would be considerable value in collecting qualitative data to investigate what is specifically going on during each of the verbal sequences. However, we believe that the results of this current study are still clear, and in addition, provide evidence that support our conclusions about decisional/tactical skills learning in school context.

Based on the Cohen (1988) [60] indices, we obtained trivial-to-moderate effects for experimental group (with interaction) progression, although small effects for the learning condition. These observed effects were probably due to various factors such as the pronounced heterogeneity within the teams, the students being novice players, and the limited length of the intervention. These factors that characterize the ecological validity status seem to have mitigated the obtained effects. Furthermore, no more remarkable differences in tactical performance indices could be related to the student expertise [105]. In fact, a preliminary study has proclaimed that inexperienced players use individual actions to solve the contextual problems of the game [106].

#### **6. Conclusions**

Results from the current study provide evidence that including opportunities for verbal interactions during breaks in game play leads to positive changes in game performance, specifically by promoting the abilities to make more appropriate decisions. In turn, these decisions are transformed into effective actions on both the collective and individual levels. Learning may be regarded as a continuous dialogue, which involves students receiving feedback from and providing feedback to, other students about tactical strategy, skill execution, and purported discoveries during games. Further research may focus on generating qualitative data alongside quantitative data and examining the effect of the verbal interaction frequency on students' skills and game performance. As a conclusion, and drawing on socio-constructivist and cooperative learning settings, we suggest that through tactical team-sport teaching, the game can be considered a vital space to examine the interactive physical, social and cognitive factors of student learning.

**Author Contributions:** Four authors contributed to this research: W.B.K. acted as the first author throughout the data analysis and manuscript drafting; M.Z. (Mehrzia Zouaoui) contributed to data collection, measurement development, reviewing, and revising; M.Z. (Makrem Zghibi) contributed to manuscript drafting, reviewing, and revising; and F.A. contributed to data analysis and manuscript completion. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Acknowledgments:** The authors thank the Regional Board of Education for making it possible to collect data in school. Special thanks to Doctor Stephen Harvey for his valuable advice in developing this work.

**Conflicts of Interest:** No potential conflict of interest was reported by the authors.

#### **References**


## *Article* **From Transdisciplinary Research to Transdisciplinary Education—The Role of Schools in Contributing to Community Well-Being and Sustainable Development**

**Susanne Kubisch \*, Sandra Parth, Veronika Deisenrieder, Karin Oberauer, Johann Stötter and Lars Keller**

Education and Communication for Sustainable Development, Department of Geography, University of Innsbruck, 6020 Innsbruck, Austria; sandra.parth@uibk.ac.at (S.P.); veronika.deisenrieder@uibk.ac.at (V.D.); karin.oberauer@uibk.ac.at (K.O.); hans.stoetter@uibk.ac.at (J.S.); lars.keller@uibk.ac.at (L.K.) **\*** Correspondence: susanne.kubisch@uibk.ac.at

**Abstract:** This article looks at transdisciplinary research and discusses the possibilities of translating this concept into a new type of education, which we will call *Transdisciplinary Education.* Following the adoption of the Sustainable Development Goals by the community of states, there has been increased international recognition of education as being a key driver for sustainable development. Considering the global grand challenges of the 21st century, the integration of Education for Sustainable Development at all school levels ought to be prioritized in order to empower young people to contribute to sustainable development. Collaborating with out-of-school partners and doing research on real-world problems within their lifeworld, help students develop the competences necessary for responsible citizenship, while at the same time contributing to community well-being. Both concepts transdisciplinary research and *Transdisciplinary Education* acknowledge the responsibility of addressing social relevant problems and the significant role of those who are and who will be affected by these challenges. The project *Science Education for Action and Engagement Towards Sustainability (SEAS)* aims at analyzing different partnerships between schools and out-of-school institutions in European countries. By comparing the collaborative formats and providing a concept and method pool for educators, *SEAS* targets facilitating the integration of *Transdisciplinary Education* in formal schooling in the future. This article gives insights into the Austrian research-education collaboration *k.i.d.Z.21*. Drawing on experiences of *k.i.d.Z.21* and taking up characteristics of transdisciplinary research, opportunities and challenges of integrating *Transdisciplinary Education* in formal schooling are discussed.

**Keywords:** transdisciplinary education; involvement of young people; (education for) sustainable development; community well-being; empowerment

#### **1. Introduction**

«There is nothing permanent except change» (Heraclitus, 540 BC). This famous quote from the Pre-Socratic philosopher Heraclitus of Ephesus, made long before the beginning of the Anthropocene, can also be applied to the dynamics, world society has to address in the 21st century. Humans are increasingly faced with rapid changes and complex problems, like anthropogenic climate change (CC) or the current Covid-19 pandemic. However, in view of these challenges, which are among the great global grand challenges of the 21st century, humans not only suffer the repercussions of these challenges, but also bear responsibility, holding the power to significantly determine their further magnitude [1–4]. Technological breakthroughs and political regulation alone will not be enough to address these challenges. In this context, innovative forms of research and partnerships will be necessary, which transgress disciplinary and system boundaries and give space for new types of knowledge and competence development [4–6].

**Citation:** Kubisch, S.; Parth, S.; Deisenrieder, V.; Oberauer, K.; Stötter, J.; Keller, L. From Transdisciplinary Research to Transdisciplinary Education—The Role of Schools in Contributing to Community Well-Being and Sustainable Development. *Sustainability* **2021**, *13*, 306. https://doi.org/10.3390/su 13010306

Received: 27 November 2020 Accepted: 28 December 2020 Published: 31 December 2020

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Against this backdrop, a claim in the world of science emerged, namely that science needs to become more socially responsible, recognizing the value to research problems in an inter- and transdisciplinary manner [7–12]. With regards to the social responsibility of science, the consideration of the needs, experiences, and values of those affected by a problem are significant. Thus, shifting away from research on people to research with people, should be the core of transdisciplinary (TD) research. This implies that scientists should regard society as an equal partner and research problems, like CC, which are socially relevant. Hence research becomes a mutual learning process for all partners involved and requires a shift from a unidirectional top-down information flow, to research that transforms all systems involved [8,13–15]. The significance of TD research to cope with current and future challenges and to create more sustainable and socially accepted solutions is also acknowledged by the European Commission and is considered in the concept of Responsible Research and Innovation, which is part of the research funding program Horizon 2020 [16].

The role of education as a key driver dealing with the challenges posed by anthropogenic CC and further sustainability challenges, has been acknowledged by the community of states for almost three decades. Both Agenda 21 and the Decade of Education for Sustainable Development (ESD), began connecting education and sustainable development. Debates on Sustainable Development Goals (SDGs) which have continued over the past five years have also centered on the important role education can and should play in achieving the set targets. The Paris Agreement (2016) affirmed that, SDG4 Quality Education is considered to be a central element for reaching all other SDGs [17–20]. Climate Change Education (CCE) in particular plays an essential role with regards addressing the challenges included in the SDG 13 Climate Action [21].

Considering the fundamental role of education and the value of TD research against the background of anthropogenic CC, the significance of the young generation in contributing to a sustainable present and future becomes apparent. Young people are not only the most affected by future sustainability challenges, but also the decision-makers of today and tomorrow [22,23]. Further, via the Fridays For Future movement, young people impressively demonstrate that they can be the pioneers and the creators of the momentum of change toward social transformation today [23,24]. A transformation of values and lifestyles is considered to be decisive in order to address the challenges posed by anthropogenic CC and further challenges of our time like inequality and poverty [4,25–28].

Although recognizing the value and responsibility of each individual domain—of TD research and of education—in addressing socially relevant issues [9,11,12,16,19], the potential which will be achieved by uniting both domains and integrating them in the form of a *Transdisciplinary (TD) Education* in formal schooling, is only minorly reflected in school reality. Until now, the concept of TD has predominantly been adopted in higher education [29–33], and is rarely found in everyday school education. Where TD does exist in formal schooling, it takes the form of relatively short-term, research or communityeducation partnerships (e.g., [34,35]).

This publication aims to introduce the new concept of *Transdisciplinary Education* and discusses its opportunities and challenges which are derived from the concept of TD research and TD in higher education. Additionally, experiences from the Austrian researcheducation collaboration *k.i.d.Z.21*—*kompetent in die Zukunft* (in English: *competent towards the future*) are presented. This publication also highlights system impedances, conflicts, and questions which might arise by introducing *TD Education* in formal schooling. These questions will be addressed in future research within the EU funded Horizon 2020 project *Science Education for Action and Engagement towards Sustainability (SEAS)* of which *k.i.d.Z.21* is a part of.

A short introduction to the concept of TD and its meaning for formal schooling now follows, along with a presentation of the Austrian research-education collaboration *k.i.d.Z.21*. Subsequently, there is a discussion how to implement *TD Education* in formal schooling against the background of its opportunities and challenges. Finally, a general and for *k.i.d.Z.21* and *SEAS,* a specific conclusion is drawn.

#### **2. From Transdisciplinary Research to Transdisciplinary Education**

In order to achieve a social transformation toward a sustainable future, anthropogenic CC and its consequences require a powerful educational response [17–21,27,36]. Young people have to be empowered to address challenges like CC and to actively shape their lifeworld as responsible citizens. Responsible citizenship is interpreted in this publication as citizens who are able to reflect critically on own and others behavior as well as take responsibility for their own actions, while considering temporal and spatial impact of their actions today. This requires, on the one hand, scientific literacy, the ability to apply scientific knowledge and competences wisely in everyday life, and on the other hand, inter- and intrapersonal competences and values in line with ESD, both supporting transformative learning and allowing the making of informed and sustainable decisions [18,19,37–43]. Against this background, Paulo Freire claimed in 1970 that traditional teaching styles, as still commonly used in schools today, lead merely to short-term recall, with little or no lasting effect in the long run [44,45]. In a dynamically changing world these practices are inadequate, as they hinder adopting learning contents to new and uncertain situations [46]. Moreover, discipline-oriented knowledge is insufficient to deal with and pro-actively tackle current and future sustainability challenges, which require a multi-perspective approach [14,47]. Furthermore, the philosopher Ivan Illich, one of the most visionary political and social thinkers, stated in his radical book *Deschooling Society* (1971) that education has to support knowledge exchange between those who want to share and those who want to learn, as well as give everyone the opportunity to make her/his opinion and arguments known by the public [48].

In order to empower young people to cope with a continuously changing world, teaching approaches need to meet the needs of ESD. ESD promotes inter- and transdisciplinary, learner-centered, participatory, and locally relevant approaches [49,50]. TD partnerships which give students the opportunity to actively conduct research on realworld problems and develop sustainable solutions with out-of-school partners address the claims made by Paulo Freire and Ivan Illich and are in line with ESD [17,18,45,48,51–55]. Hence, TD partnerships allow students and teachers to collaborate with scientists, as well as further out-of-school partners in the fields of, for example, politics, economy, ecology, and civil society within the community that the school is located. Besides that, jointly conducting research on real-world issues fosters scientific literacy, while the exchange of knowledge and perspectives in collaborative processes encourages mutual learning of all partners involved and facilitates the generation of transformative knowledge [15,56–59]. The latter is fundamental for the change of current actions and behavior toward sustainability [51,60,61]. Moreover, TD collaboration allows the development of the competences needed, like the capacity to reflect on one's own and others perspectives and critical thinking, which are in line with the claim of ESD, and beyond that foster a responsible population [17,18,37–39,42,43,51,53].

Initial discourses about transdisciplinarity emerged against the background of economic and technological change in the 1960s, which in turn entailed innovation and transformation of research and educational systems [11]. Years later, environmental problems and arising debates about sustainability resulted in the evolution of the concept of transdisciplinarity, which moved away from a more interdisciplinary scientific perspective [62–64], beyond societal needs as the driving force of the research [65,66], toward partnerships between science and society [11,12,67,68]. Consequently, the perspective and contextual knowledge of those affected by a problem gained importance and led to an increasing involvement of civil society in the research process [11,14,67]. TD research contains three fundamental aspects: (1) The starting points are socially relevant issues, which are jointly identified and which are researched by means of integrative scientific methods, with the aim of developing interdisciplinary solutions or strategies for transformation; (2) during the whole research process there is an exchange between scientific and non-scientific partners, the latter, e.g., politicians, require the generated knowledge for decision-making; (3) integration of non-scientific partners, like citizens (or in this case: students), to consider experiences and context-based knowledge [33,57,69,70]. Different forms of partnerships are possible, depending on the degree of integration of non-scientific partners [71].

Translating this concept into the educational context requires that socially relevant issues of students' lifeworld, which are jointly defined, should be considered as fundamental for formal schooling rather than as pre-defined educational content [43] (see Figure 1). Dependent on the age of the students and the type of school, teachers can draw from different collaborative formats and consequently enhance students' involvement in collaborating with out-of-school partners. Shaw et al. (2011) distinguish between young people as source of research data, young people are consulted about the research and young people as collaborators. The most autonomous form, according to Shaw et al. is that young people are empowered and take ownership of the research. The latter two are considered for *TD Education* as they originate from joint decision-making and consequently are in line with TD research which fosters research with people instead of research on people. The collaboration between students and out-of-school partners involves jointly developing and planning different research phases. If students get the opportunity to take ownership of the research, teachers and out-of-school partners give only support and guidance when needed [72]. Consequently, students design the research process on their own, from the development of the research question, to the selection of the evaluation method, and to the assessment as well as the presentation of results and solution strategies [54,73]. During this form of partnership, young people simultaneously take ownership of both their learning process and their lifeworld [74,75].

**Figure 1.** From transdisciplinary (TD) research to transdisciplinary education. Source: [13], modified.

Beyond that, scientists should be involved to monitor and support the collaboration process, teachers can additionally benefit of the partnerships with scientists and further out-of-school partners, to jointly work out with students, actual research findings and practical context-based knowledge.

Inspired by the approach of TD research and Responsible Research and Innovation [11,16,57], this paper argues that mandatory anchoring this new form of education

*Transdisciplinary Education* in formal schooling, offers schools the opportunity to play a part in having a social impact while educating a responsible student body.

The opportunity that TD Education offers community well-being and sustainable development, raises a few questions, which will partly be addressed in this paper and in future research within the project *SEAS*. These questions are as follows:


#### **3. A Long-Term Research-Education Collaboration on Climate Change: The Example of** *k.i.d.Z.21*

The research-education collaboration *k.i.d.Z.21—kompetent in die Zukunft* (translated as: competent toward future) was developed against the backdrop of today's youth being increasingly confronted with the global grand challenges of the 21st century, specifically with the consequences of anthropogenic CC [5,76]. Consequently, they have to adapt and go through a transformation process, for which young people, as today's and tomorrow's decision-makers, have to be prepared [77]. Besides raising awareness about CC and its consequences, *k.i.d.Z.21* aims at generating acceptance and the need for action, linked with a necessary social transformation, while at the same time strengthening young peoples' capacities to act and adapt to the challenges posed by societal, economic, and ecologic changes [78]. Since the foundation of the collaboration in 2012 between the Department of Geography of the University of Innsbruck, Austria, and the Karl-von-Closen Gymnasium, a southern German high school, the network has continuously grown with the addition of many more Austrian, German, and Italian schools. To date, almost 2,500 high-school students have been involved in the one full school-year project. The network includes more than 100 voluntary (scientific) experts in the fields of CC and environmental ethics, tourism, bio-, pedo-, and cryosphere. These are alongside established (scientific) CC experts as well as young scientists.

*k.i.d.Z.21* is based on the idea of a transdisciplinary dialogue between students and (scientific) experts of different disciplines and on moderate constructivist approaches to learning [77,78]. Hence, the setting of *k.i.d.Z.21* fosters exchanges between students and (scientific) experts and gives students the opportunity to actively complete research on real-world problems within school and out-of-school settings [55,79].

The project consists of different modules (Figure 2), which, among others, are based on concepts of TD mentioned above [11,57,80–82]. During a school year, the students (aged between 12 and 18 years) are actively involved in these modules.

At the beginning of the project (and the school year), during the *kick-off event*, students have interactive *workshops, lectures + debates on CC* with CC experts and further out-ofschool partners like politicians and peers, getting the opportunity to discuss CC and its consequences. During the school year, the students receive *classical school lessons on CC*. All teachers in the participating schools are asked to include the topic of CC in their school lessons. The teachers are autonomous with regards how far and by which methods they integrate this topic. Furthermore, for at least six months the students develop and work on their *individual CC research project*, which can be related to the human or social sciences (e.g., arts, history etc.) or related to the natural sciences (e.g., mathematics, biology etc.) or, even better, be interdisciplinary. The *individual CC research projects* are presented at the end of the project. According to the moderate constructivist approach [52,80,82], the whole concept is up to the students. This resulted in students creating their own creative learning projects, e.g., some students established an initiative at their school to encourage

other students to go to school by bike. The project year finishes at the end of the school year with an *Alpine Research Week*, where the students once again come in contact with experts in the fields of tourism, environmental ethics, alpine vegetation, and glaciology. The high Alpine environment offers an authentic setting, where students become researchers, doing research on the consequences of CC on the above mentioned fields [55]. Based on inquiry-based learning students are asked to develop their own research questions, find adequate methods for data collection and collect, analyze, as well as interpret the data. The (scientific) experts offer the students support during the whole research process. On the last day of the *Alpine Research Week*, the synthesis day, local experts are invited to discuss the questions remaining unanswered.

**Figure 2.** *k.i.d.Z.21* learning setting during a project year (one full school year) (source: [55], modified).

The continuous scientific monitoring and evaluation of the participating students via online questionnaires at the beginning (pre-test) and the end of the project year (posttest), helps modify the setting and already showed interesting results regarding students learning outcomes against the background of CC [55,79,83–85].

#### **4. Beyond Requirements of TD Education to Opportunities and Challenges**

Referring to the initial questions (see Section 2), the following section deals with the opportunities and challenges of *TD Education* for schools as well as requirements in the fields of educational, staff, and institutional development [86]. Moreover, it highlights system external implementation requirements in the context of teacher training. This discussion is based on the experience of the research-education collaboration *k.i.d.Z.21* and on a literature review of research in the field of TD in higher education and TD research.

#### *4.1. Educational Development—From Theory-Based Teaching to Socially Relevant Issues*

*TD Education* means a shift from theoretical predefined and discipline focused content as a starting point of learning to the joint definition of socially relevant issues within students' lifeworld [31,87,88]. This requires innovative forms of teaching and learning which are in line with ESD [18]. Moderate constructivist and inquiry-based learning approaches seem to be most adequate for *TD Education*, as they are, on the one hand, learner-centered and grant students the opportunity to use their analytic and creative skills and on the other hand allow exchange and reflection of different perspectives, which is essential in collaborative research processes [33,54,73,80,82]. These concepts are also approved by the TD modules within the *k.i.d.Z.21* project [55]. Nevertheless, TD settings require a high degree of self-reliance and active engagement of students as they evolve from a mere recipient to a co-designer of their school lessons [31,74,89–91]. The role of teachers in turn shifts from the classical role of knowledge mediator to coach and process manager, supporting students in their learning processes, enabling reflection among students and only intervening if necessary [33,92,93]. During the *k.i.d.Z.21* school year, students develop and do research on their *individual CC research project.* The project evaluation shows that even if the students invest a lot of effort in the development of their project, actively dealing with an issue within their fields of interest encourages effective learning in comparison to the classical school lessons as demonstrated by Keller et al. (2019). The same is true for the *Alpine Research Week*, in which students do research and are supported by experts and their teachers whenever needed. The TD dialogue and active involvement in an authentic learning setting during this week, raises students understanding of CC and its consequences and lead to an increasing learning effectiveness [55]. Moreover, this TD project raises awareness and self-efficacy of students which are important predictors for climate-friendly action [83,85].

Moreover, *TD Education* also requires the rethinking and the redefinition of learning objectives and evaluation modes, as the learning content is jointly agreed with the students at the beginning of the TD partnership. The determination of learning objectives and evaluation modes therefore requires some flexibility. Learning objectives should focus on competences generated during the process or should be defined together with students at the beginning of the collaboration. The same can be possible for the evaluation mode.

#### *4.2. Staff Development—Consideration of TD in Teacher Education*

The demands that *TD Education* makes on teachers' competences not only with regard to alternative pedagogical and didactical methods, but also in using different forms of student's involvement within partnerships, not only affect schools but also universities, since these need to be considered in teacher training programmes [94,95]. The same is true for the organizational competences which are required to plan and maintain the partnerships [31,94–97]. Consequently, according to Germ (2018) it is necessary that universities build academic capacities doing research on TD and necessary (didactical) competences, and consider these in the curricula for teacher training [98]. Following a learning-by-doing approach, it would be highly recommended that university students get in contact and do research with out-of-school partners [30,32]. The implementation of TD Education in schools further raises the question of how to re-train teachers already working within the classical education system. In the *k.i.d.Z.21* project, for example, the project team invests a lot of time in training the teachers how to apply innovative didactical methods like moderate constructivism and inquiry-based learning in the sense of CCE and ESD [19,54,55,81]. The teachers themselves take part at the *Alpine Research Week* and do research in this authentic TD learning within the teacher training programme [55]. As the respective working group is also responsible for teacher education of the subject Geography and Economies at the University of Innsbruck, TD, moderate constructivist approaches of learning and ESD as well as CCE are part of the teacher training. Moreover, students get the opportunity to support the working group in different modules of the project.

#### *4.3. Institutional Development—Structural Anchoring of TD Education*

Since TD partnerships require a continuous exchange between students, teachers and out-of-school partners and are very time-consuming [37,57]; one possibility to integrate *TD Education* in formal schooling would be a TD project seminar [96,99]. This in turn needs to be considered in school curricula. One of the *k.i.d.Z.21* schools for example, incorporates two hours each week for the students to work on their *individual CC research projects*, which would also be recommendable for a TD project seminar, in order to foster continuous exchange with out-of-school partners. Another possibility would be to allocate different periods within the school year for the TD exchange with out-of-school partners, as it is commonly handled in the *k.i.d.Z.21* project. The findings of the study by Keller et al. (2019) and of further studies within the project [83–85] support the fact that the dialogue with out-of-school partners within the TD modules have already a high learning effect.

#### *4.4. Personal Development—Beyond Knowledge to the Generation of Competences for a Responsible Citizenship*

*TD Education* leads to a democratization of learning as students get the opportunity to actively shape their own learning process, while contributing to the community well-being and sustainable development. Moreover, both by means of the process of collaboration and by researching, students may develop competences needed for responsible citizenship [31,39,42,43,47,54,100]. The findings of the study by Deisenrieder et al. (2020) also support an increased self-efficacy and climate change awareness among students taking part at the *k.i.d.Z.21* project. Furthermore, students' climate-friendly behavior is raised by doing their own research and being in dialogue with out-of-school partners in the project [85]. The latter is also confirmed by further studies beyond this project [101–103]. Moreover, actively researching an issue of interest raises scientific literacy and encourages students to identify with, take ownership of, and consequently responsibility for the issue [31,33,43,103–106]. Furthermore, as demonstrated by Parth et al. (2020), *k.i.d.Z.21* students also act as multipliers on their families in relation to climate-related knowledge attitudes and actions [84].

As mutual learning takes place within TD partnerships by continuous dialogue and exchange of different perspectives, not only students but all partners involved make progress and develop competences necessary for a responsible citizenship [42,103,107].

#### **5. Conclusions**

This publication aims to demonstrate that solving today's and the future's challenges are no longer a matter for scientists and politicians alone, but requires a responsible citizenry [38,39,41–43]. Therefore, new forms of research and education are necessary which allow mutual learning and foster new types of knowledge and competences generation for transformative learning [11,13,42,51,60,67]. However, traditional teaching styles as commonly practiced in formal schooling are inadequate to address the challenges in a dynamically changing world [45,47]. Approaches to teaching and learning need to be in line with the claims of ESD, which promote inter- and transdisciplinary, learner-centered, participative and locally relevant approaches [18,49,50].

Inspired by the concept of TD research [11,67] and Responsible Research and Innovation [16] and drawing on the experience within the research-education collaboration *k.i.d.Z.21*, this publication aims to introduce the new concept *Transdisciplinary Education* which should become an integral part of formal schooling. *TD Education* transgresses system boundaries and empowers young people, the ones who will be the most affected by future sustainability challenges [22,23], to actively shape their lifeworld by participating in the development of sustainable and societal accepted solutions within their community. Actively dealing with socially relevant issues within their lifeworld, collaboratively doing research and being in dialogue with out-of-school partners from science, politics, and further sectors within their community, students develop competences necessary for a responsible citizenship, as demonstrated by studies within the Austrian research-education

collaboration on climate change *k.i.d.Z.21* [83–85]. Within the international EU Horizon 2020 funded project *Science Education for Action and Engagement Towards Sustainability (SEAS)*, which was submitted in November 2018 under the call *Science with and for Society* different formats of TD partnerships both in regard to temporal, structural, and operative aspects will be analyzed in a cross-cultural setting from projects partners in Austria, Belgium Estonia, Italy, Norway, Sweden, and the United Kingdom. *K.i.d.Z.21* is representative of the Austrian TD collaboration. Comparing different methodological approaches and implementing different tools within the partnerships, *SEAS* aims at giving answers to the initial raised questions (Section 2) and providing a concept and method pool of good practice examples and lessons to be learned for educators to establish and facilitate the integration of *TD Education* in formal schooling.

In addition to having a great potential to contribute to community well-being and thus to sustainable development, *TD Education* also may challenge schools as it requires developments on educational, staff, and on an organizational level. These necessary developments lead to the need to rethink aspects of the traditional school system, especially in regard to the objectives of teaching and learning which are defined in school curricula and how to consider *TD Education* in school reality. Furthermore, anchoring *TD Education* in formal schooling involves a necessary development in higher education in the field of teacher training and also challenges out-of-school partners [98,107–109]. Consequently, different systems involved come into conflict which differ in terms of interests, time structures, and administrative aspects [107]. Analyzing and comparing the opportunities and challenges for schools, but also for out-of-school partners, of different formats of TD partnerships and finding solutions for the just named challenges, will be a demanding task for future research within *SEAS* and the country-specific partnerships. However, it will inform how to best integrate the concept of *TD Education* in formal schooling.

Yet, as TD research and the education of (young) people play an increasingly important role in dealing with the complex global grand challenges of the 21st century [11,16,19], this publication also aims at encouraging research beyond these projects to find innovative answers in order to translate the concept of Transdisciplinary Research into *Transdisciplinary Education*. Acknowledging the important role of education [18], educating a 'powerful force for social change' (Bentz & O'Brian 2019, p.1) [23], this publication concludes that schools not only have great potential, but also a responsibility to contribute to community well-being and beyond that to sustainable development.

**Author Contributions:** Conceptualization, S.K. and L.K.; methodology, S.K.; investigation, S.K.; resources, S.K.; writing—original draft preparation, S.K.; writing—review and editing, L.K., V.D., K.O. and S.P.; visualization, S.K.; supervision, L.K. and J.S.; project administration, L.K. and J.S. funding acquisition, L.K. and J.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This project receives funding from the European Union's Horizon 2020 research and innovation program under grant agreement No. 824522.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) of the Universtiy of Innsbruck (protocol code 49/2019 and 1 October 2019).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** No new data were created or analyzed in this study. Data sharing is not applicable to this article.

**Acknowledgments:** We would like to thank Karen O'Brien, Department of Sociology and Human Geography of the University of Oslo, Norway, for her valuable contributions to the conceptualization of this publication. Furthermore, we would like to thank Thomas Allmer, Institute of Geography, University of Innsbruck, and Lisa Thompson for the help with style and language.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Is It a Good Idea for Chemistry and Sustainability Classes to Include Industry Visits as Learning Outside the Classroom? An Initial Perspective**

**Omar Israel González-Peña 1,\*, María Olivia Peña-Ortiz <sup>2</sup> and Gustavo Morán-Soto 3,\***


**Abstract:** Learning outside the classroom (LOtC) activities are part of pedagogical methodologies that are currently applied in the development of student skills. The objective of this study is to determine the perceptions of faculty and undergraduate students concerning industrial visits and define the advantages and disadvantages of these activities. A survey was designed with 17 questions, from a sample of 296 students and 32 professors from various chemistry and sustainability courses. The statistical samples correspond to a population of 2275 students and 246 professors. Descriptive statistics were used to analyze and compare participant perceptions on industrial visits, as LOtC activities. Results indicated a positive perception for making industrial visits, generating more interest in the class material and helping students acquire knowledge. Despite this positive perception, it was found that professors are unlikely to organize industrial visits frequently due to the work required to plan, perform, and evaluate these activities. This issue suggests that approximately 40% of the students may lose the advantages that LOtC activities could offer. Professors must be motivated and supported by administrators to include industrial visits in their courses as a teaching strategy to provide a beneficial experience to the majority of students enrolled in chemistry and sustainability undergraduate programs.

**Keywords:** learning outside the classroom (LOtC); industrial visits; STEM education; education for sustainable development; educational innovation; higher education

#### **1. Introduction**

It is common for instructors, nowadays, to use different methodologies and pedagogical strategies to encourage their students to develop skills that will serve them in their academic and professional performances. One of the most utilized methods to promote better learning outcomes is known as learning outside the classroom (LOtC). This method has become popular at different educational levels. It can be used through teaching strategies, such as the inverted classroom [1], problem-based or project-based learning [2], professional practices [3], or industrial visits [4], among others. Educators use these pedagogical practices related to LOtC to inform, teach, and motivate students to develop a greater interest in course content [5]. As a result, using these pedagogical tools contributes to the development of the cognitive skills necessary to train professionals to face the most pressing challenges around the world [6].

Likewise, in the decade of 90s, the American Association for the Advancement of Science [7], the National Research Council [8], and the National Science Foundation (NSF) began using Sciences, Technology, Engineering, and Mathematics (STEM), with the intention of promoting and categorizing different activities, programs, and public policies,

**Citation:** González-Peña, O.I.; Peña-Ortiz, M.O.; Morán-Soto, G. Is It a Good Idea for Chemistry and Sustainability Classes to Include Industry Visits as Learning Outside the Classroom? An Initial Perspective. *Sustainability* **2021**, *13*, 752. https://doi.org/10.3390/su13020752

Received: 17 December 2020 Accepted: 12 January 2021 Published: 14 January 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

where the disciplines of Sciences, Technology, Engineering, and Mathematics (STEM) intervene. Therefore, with the term STEM, it was sought, among other things, to encourage innovation and problem solving in the educational system [9]. However, later in the next decade (2000–2010), there was still little use (or confusion) for the term STEM. Thus, in 2009, the National Assessment of Educational Progress (NAEP) in engineering began evaluating technology literacy in the U.S. [9–11]. In 2010 and 2014, initiatives to promote technology and engineering education were launched by the National Assessment Governing Board (NAGB) to include social problems related to STEM issues, such as: energy efficiency, climate change, sustainability, etc. [9,12]. In parallel, the framework for The Programme for International Student Assessment (PISA) 2006 [13] was structured. As a result, they included, in curricular subjects, the relevance of studying in local, regional, and global context issues, such as: health, energy efficiency, natural resources, environmental quality, hazard mitigation, etc. [9,13]. Furthermore, among the different educational activities, the LOtC method has been listed as a recommended tool for students and instructors in the United Nations Educational, Scientific, and Cultural Organization (UNESCO) program: Teaching and Learning for a Sustainable Future [14,15]. The United Nations is actively contributing to the design and development of educational processes that could help students understand the importance of creating technological solutions in a sustainable way. One of the most relevant projects of the United Nations is the program: Teaching and Learning for a Sustainable Future, which is a multimedia program focused on training instructors in education for sustainable development. The United Nations Educational, Scientific, and Cultural Organization (UNESCO) has shown particular interest in sustainable development, presenting a report in relation to the final monitoring and evaluation of the decade of education for sustainable development [16,17]. This report shows that many education systems around the world are making efforts to address sustainability issues, aiming to confront social, environmental, and economic challenges in a more sustainable way. These efforts to address sustainability issues could be seen at the secondary level [18], basic vocational education [19], or universities that need to assess whether their programs are strongly involved in linking student curricula with the general objectives of sustainable development [20]. This growing interest, in teaching students and professionals about sustainable development goals, could be addressed in a better way if professors included LOtC to facilitate their students' active learning. In this sense, there have been examples of courses that have included hands-on activities with the goal of developing sustainable projects, such as building prototypes of powered cars without internal combustion in one-week projects [21], or full semester projects [22]. These LOtC methods could promote student interest in sustainable development, especially if these methods could show them the most relevant environmental issues in their locality, and how industries are looking for solutions to improve the sustainability of their processes.

Although learning experiences outside the classrooms have resulted in better development of student skills [6,23], complications have arisen while planning implementation of these strategies in schools. Among the primary complications: schools must have trained staff as well as an infrastructure that allows students to explore outside the classroom. These basic requirements of learning methodologies outside the classroom are often missing, making it difficult to apply and adapt these LOtC activities to classes in public educational institutions and developing countries [24]. Our research was intended to complement the existing literature on LOtC by determining the perception of undergraduate chemistry students, and their professors, on visits to companies or industries. Likewise, this study is relevant due to its aim to find a more structured way to plan and evaluate the impact of industrial visits on students and professors. In the end, the results of this research can be the basis for programming future industrial visits by university students and, possibly, generate more studies on this learning strategy.

#### **2. Purpose of This Study**

The analysis of existing literature on LOtC techniques found that most studies obtained their data from a single part of the teaching process, based on information collected from students, without considering the instructors. Or vice versa—they took into account the opinions of professors without collecting information on students. This study seeks to supplement the current literature on LOtC by establishing reflection and integration of learning experiences that take place outside of the classroom. Moreover, this research focus on industrial visits, seeking to complement the existing LOtC literature by interpreting the opinions on these visits by both students and university professors in chemistry and sustainability areas not previously reported, according to the bibliographic search carried out in this study. Therefore, this work is relevant because there is currently no information that can help college educators organize and evaluate these activities in a standardized way, and these missing parts may negatively affect the outcome of these industrial visits. To achieve the purpose of this study, we posed the following research questions to guide our work:


#### **3. Theoretical Framework**

This research is based on the out-of-classroom learning methodologies reported in the "Learning Outside the Classroom" manifesto published by the Department for Education and Skills in United Kingdom [25]. This manifest indicates that young people should experience the world beyond the classroom as an essential part of their learning and personal development. Therefore, studies that motivate professors can be found to encourage children and adolescents to learn outside of the classroom [5]. Literature recognizes that learning strategies outside of the classroom provide students with opportunities to develop skills—much better than just experiencing a classic classroom education; in this way, outside activities complement in-classroom instructions [2].

It is important to mention that, for LOtC strategies to be effective, and motivate students to take an interest in the class, these strategies must have well-thought-out goals from the outset, as well as clear objectives and tasks that can be accomplished in a set time frame [26]. If outside learning strategies are planned considering these indications, students will likely be able to use what they have learned outside of the classroom in all kinds of school activities, as well as in daily life. In the end, students will fortify their interests in learning and studying new topics by performing activities, such as field trips, laboratories, inverted classrooms, and industrial visits, among others [27].

As the goal of the United Nations Education for Sustainable Development is to improve access to education in order to transform society in a sustainable way, they support activities that help instructors incorporate education for sustainable development into teaching and learning activities [16]. This support for more education (for sustainable development) could positively influence school administrators and stakeholders to promote public policies at global, national, and regional levels; creating better possibilities of an alignment of visions and goals, at all educational levels, with a long-term plan for sustainable development.

#### **4. Previous Studies**

Undergraduate STEM programs are usually missing a strong link to educate students in a cooperative effort between academia and industry to excel in their new jobs. In this sense, [28] found that stimulation, in the innovative behavior, can be enhanced by LOtC activities that promotes student learning, self-efficacy, and career planning. A wide variety of studies can be found that use LOtC strategies with all types of students around the world. Studies on adolescents reported better development of their emotional and social skills and attributes when using activities related to LOtC [29]. Elementary and secondary school students reported better development of their social interactions and showed better communication and collaboration with classmates when their instructors used LOtC strategies [24,30]. In addition to the benefits of social interaction, the activities that take place outside of the classroom, in outdoor environments, can improve the concept of personal values, such as freedom, fun, autonomy, and a sense of satisfaction, which promotes the integral development of children [5].

Pedagogical outings are different LOtC strategies. These pedagogical outings allow students the opportunity to deal with different contexts, areas, or fields of study, where there are not enough educational resources to develop a subject extensively as in a classroom setting [4]. In these outings, learning opportunities come from having direct experiences with the phenomenon under study, motivating students to ask questions and get involved in research projects that could provide students with better educational and social abilities [31].

Other studies have used the project-based learning strategy for teaching outside of the classroom in high school courses. These studies suggest that collaborative projectbased-learning could result in better academic performance, facilitating the exchange of ideas among all the team members during the planning and elaboration of the project [32]. The project-based learning strategy could be considered as a good LOtC activity after an industrial visit, as professors and students learn more about solving real-life problems. In addition, options, such as lab practices, do not require a student to leave the school to apply learning strategies, similar to those outside of the classroom. These types of practices are widely used in chemistry courses with positive effects on students' motivation [33,34].

In general, activities outside of the classroom involve connecting knowledge acquired in the classroom to information and applications used in different environments, allowing students to develop important abilities that facilitate their critical thinking development. This way of creating and developing knowledge can be related to the constructivist theory of meaningful learning. This theory describes that prior knowledge constitutes the basis for new knowledge, as long as there is a logical relationship between them [35]. The knowledge obtained through learning outside of the classroom uses previously acquired knowledge to experiment and draw conclusions, which help develop and expand the body of knowledge [36].

Concerning college students, examples of various LOtC methods that are used for teaching STEM can be found, where the objective is that students develop autonomous and critical thinking. It has been found that project evaluation can help students develop interdisciplinary and specific competencies, allowing them to know what their roles are or what might be their participation within the industrial sector [37]. Other activities, such as fieldwork, allow students the opportunity to analyze and assess issues related to the quality of the products and services in the current market [38]. In addition, these LOtC strategies have proven to have a positive impact on the development of students' study, communication, and scientific writing skills [33], as well as the interests they take in their courses [1,39]. Moreover, LOtC strategies can facilitate the application of previously learned knowledge, helping students manage a higher level of cognitive processing [40,41]. Other studies have associated college-level course development with the industry, including talks, and research projects with sponsorships [42]. Other options include offering one-year positions in different industries [43], and promoting regular industrial visits to different companies, where students can disassemble and assemble industrial machines [44,45].

Another approach that has been sought to achieve LOtC is to use computational tools, such as virtual reality [46] or gamification [47]. In one virtual reality environment, students participated in emulating an iron-making industrial visit; therefore, this study presents a method of avoiding the possible disadvantages of costs, transportation, and high standards of industrial safety that are typically involved in student industrial visits [46]. Moreover, using gamification as a LOtC activity showed that this method works well at promoting enjoyment with learning—with better performance—when students are evaluated using games as a learning tool [47].

Despite the advantages that LOtC methods have shown concerning the development of student knowledge and abilities, few studies have registered evidence of college-level science courses that incorporate these strategies as a teaching method. Previous studies that analyzed industrial visits in college courses have found that both students and professors consider these industrial visits as positive experiences that facilitate student learning. The issue with these studies is that their main goal is to analyze student learning activities in general, and they are not focused on industrial visits [38,48,49].

Literature suggests that most college students have not experienced industrial visits or any other LOtC activity [49]. This could stem from the lack of well-structured plans from universities to implement these didactic strategies [50]. The lack of opportunities to learn using outside of the classroom strategies in university courses could ultimately affect student development [51]. In addition to losing opportunities to improve academically, students without LOtC experiences also miss opportunities to develop as professionals with social commitments [3].

Regarding chemistry, it has been pointed out that cognitive skills that can be developed with LOtC strategies are essential for performing scientific processes and laboratory work. The skills developed with these strategies allow students to reflect on the topics learned and how they learn those things, giving them the tools needed to transfer knowledge beyond the classroom [32,52]. Students in chemistry-related careers need to develop skills to apply theoretical learning, to practice in the laboratory, and solve problems in their working lives [53]. For this reason, professional programs in universities should try to incorporate more activities connected with industry and research in the areas of chemistry.

The incorporation of teaching outside of the classroom in college chemistry courses can also serve as a networking bridge for universities with businesses. For example, industrial visits could facilitate students' professional training so they can graduate from college equipped with better skills to solve problems presented in the industrial sector [54,55]. This academic–industry link not only includes the possible resolution of industrial problems, but also the development of better oral and written communication skills due to the direct contact with potential future employers [56]. Another advantage of incorporating LOtC strategies into college chemistry courses is that these experiences could have a positive impact on students' interest in continuing with graduate studies [57]. This is because students obtain access to all types of resources and professionals that might help them identify topics of their interest in the study of chemistry, which may ultimately motivate them to consider the possibility of becoming researchers or scientists in these subjects [57].

#### **5. Methods**

This research was based on a multi-method approach (see Figure 1), conducting a qualitative phase followed by a quantitative phase [58]. For the qualitative phase, a survey was developed with the goal of collecting data about visits to the chemistry industry. This survey was developed after the authors could not find a validated instrument on a survey that collected data on this specific topic in the current literature. Hence, a new instrument was designed to facilitate information gathering about industrial visits, related to the teaching of sustainability and chemistry related courses. The design of the new instrument began with a search to find the key questions that should be included in a survey, to understand how students and professors from sustainability and chemistry related majors perceived such visits. Semi-structured interviews were conducted with two chemistry professors who had several years of experience utilizing industrial visits in their courses. During the interviews, open-ended questions were asked that allowed the professors to discuss their positive and negative experiences during their visits to industries connected with chemical processes [59]. The two interviews were audio-recorded and transcribed for

analysis. Each interview was coded line-by-line through an open coding strategy similar to that used in grounded theory methodology [60].

**Figure 1.** Multi-methods research design.

This qualitative analysis strategy allowed researchers to find themes and meaning units that served as the basis for our survey questions [61,62]. Seventeen multiple-choice questions were drafted in Spanish to determine the frequency, possible impediments, and reasons for making industrial visits. Two focus groups were put together to test the content and wording of the new instrument questions [63]. The first focus group consisted of four professors from different chemistry courses, and the second of ten students in various chemistry programs in the university where this research was conducted. During the focus groups, participants were asked to read the questions and possible answers, to give their feedback about the clarity of the statements, and identify possible misunderstandings in the content of the survey [25]. In the end, some questions were modified for clarity according to the feedback received by chemistry students and professors. In particular, questions 1–7 and 16 were designed to be similar for students and professors. Additionally, questions 8, 13, 14, and 15 were labeled as professors' questions because it did not make sense to ask those types of questions to students. Questions 9, 10, 11, 12, and 17 were designed to be answered for students and professors, but syntax was required to be different in the structure of the questions, to be separated; this means that those questions had subtagging: (I) for student and (II) for professors. In other words, the information gathered was the same for both groups (please refer to the Appendix A to see the questionnaire).

After the content and face validation process [25], the survey with 17 questions was distributed on paper. This survey was translated to English by one of the researchers for publication purposes only (see Appendix A). The information was collected from a sample of 296 undergraduate students and 32 professors in chemistry-related bachelor's programs during the fall semester of 2016 at a public university in the state of Jalisco, Mexico. For context, Jalisco is one of the most important industrial states in the country, with a value of sales of manufactured products of \$400,641,994 (thousands of Mexican pesos, MXN) [64]. The main productive activities in the state of Jalisco include industry sectors, such as steel, mechanical, textile, computers manufacturing, communication, measurement equipment and other electronic equipment, components, and accessories [65]. However, there are also relevant and considerable industrial activities in the region, such as food, beverages and tobacco, tanning and leather or substitute materials, paper, plastic, and rubber industry, chemical industry, transport equipment manufacturing, metal products, and basic metal industries, among others [64]. Due to this high and diverse industrial activity, the state of

Jalisco is also considered one of the main polluting areas in Mexico. This region generates high polluting emissions in the air, water, and soil. Mexican environmental regulations are currently aiming to facilitate the efficient classification and management of all waste and emissions. This is where industrial visits could help universities develop curricula that consider the current needs and regulations that all chemical processes need to fulfill for appropriate waste management, at a laboratory scale and beyond [66].

Both professors and students were instructed to answer the questions labeled with their descriptions, or not labeled at all, avoiding the questions labeled with a role that did not match their own role in the class (professors avoided questions designed for students, and vice versa, students avoided questions designed for professors). These special instructions resulted in several questions that were not answered because they were designed for a different context for that particular participant (see Appendix A). The size of the sample obtained was larger than that determined by the formula shown below [67]:

$$m = \frac{pqNz\_a^2}{(N-1)d^2 + pqz\_a^2} \tag{1}$$

*z*2 *<sup>α</sup>* is a statistical parameter that depends on *N*; *q* = (1 − *p*) is the probability that the event will not occur; *d* is the accepted estimation error; *p* is the fraction or proportion of the elements in the sample that possesses the characteristic of interest. *n* and *N* represent the sample size and the target population, respectively.

Students in the sample were enrolled from first through fourth semesters of the bachelor's degree programs in chemistry, chemical engineering, and chemical pharmacobiology, and were between 19 and 21 years old. The sample of professors normally taught subjects in the area of sustainability and chemistry, such as general chemistry, physicalchemistry, chemistry of the cell (biochemistry), environmental chemistry (environmental pollution), and analytic chemistry. They had 15 to 25 years of experience as university professors. Students and professors, in this sample, were purposely selected from chemistry courses that were required to be taken by students during the first four semesters of their major in the university where the study was conducted. This sampling strategy facilitated data collection since students from fifth or higher semesters usually have flexibility of their curricula in chemistry majors that no longer require mandatory chemistry courses. The surveys were distributed and collected during student class time. Subsequently, survey responses were digitalized to conduct the statistical analysis.

For the quantitative analysis, the responses of professors and students were analyzed separately. Additionally, data from students and professors were separated into two more groups for better interpretation of the information collected. The first two groups contained students and professors who had experienced industrial visits in the past, while the additional two groups contained students and professors who had never experienced industrial visits (see Table 1). As a result, four groups were analyzed in separate ways; the first group clustered students with industrial visit experiences; the second group clustered professors with industrial visit experiences; the third group clustered students without industrial visit experiences; and the fourth group clustered professors without industrial visit experiences (see Table 1). These groups facilitated the analysis of the data and allowed a better interpretation of the results. For the data analysis, tables and graphs were assembled with the data from each of the questions separately. Responses of students and professors were compared in all four groups to contrast the answers according to their experience with industrial visits. Finally, these tables and graphs of the four groups were compared side-by-side to determine possible differences or similarities between the professors' and students' opinions on industrial visits, to answer the research questions adequately.


**Table 1.** Industrial visits made by students and professors.

#### **6. Results and Discussion**

Table 1 presents the groups made up of professors and chemistry students who made none, or at least one, industrial visit in their college curricula.

Of the 181 students who reported having made an industrial visit during their course work, 150 (83%) reported that their greatest motivation to accept the invitation for an industrial visit was to develop knowledge, and be up-to-date with the methods and procedures currently used in the chemistry industry. The analysis of the same question to the 18 professors who had organized one or more industrial visits revealed that 17 (94%) of them responded that their greatest motivation to organize the visit was to reinforce the learning of their students, and ensure that they were up-to-date with the processes currently used in the chemistry industry.

On the other hand, the professors and students who had never made an industrial visit gave very different answers to the question about why they had not participated in industrial visits during their university studies. Of the 115 students who had not made an industrial visit, 77 (67%) reported not receiving an invitation from the professors or university. The professor responses showed that the majority of the 14 professors who had not organized industrial visits had not been involved in this learning activity outside the classroom due to a lack of time (12 of the 14 professors (86%) gave this response).

The results displayed in Table 2 show the semester that the students and professors consider most appropriate to visit a chemistry-related industrial visit.


**Table 2.** Semesters in which chemistry professors and students prefer to make industrial visits.

In addition to the semester preference indicated for making industrial visits shown in Table 2, 91% of all students reported that they would like to make an industrial visit every semester. In comparison, 91% of the professors prefer these learning activities outside of the classroom to be programmed more sporadically, with only one industrial visit programmed throughout their students' university careers, or one every one or two years.

Table 3 shows the number of students and professors who made an industrial visit, and whether they had a clear objective of what they sought to achieve visiting a specific industry.

**Table 3.** Number of students and professors with a clear objective in their industrial visits.


As a complement to the results shown in Table 3, 92% of students who had made an industrial visit reported that they already had an idea of what they would see in the industry prior to their visit. On the professors' side, 100% of them who had already made an industrial visit indicated that they already knew what was in the chemistry industry before visiting it.

The majority of students who had made industrial visits reported that this learning strategy outside of the classroom helped them improve their knowledge of chemistryrelated topics (see Figure 2).

**Figure 2.** Students who believed that industrial visits improved their learning.

The results shown in Table 4 indicate that it is common for students not to be evaluated for their industrial visits. Moreover, 50% (9) of the professors who visited chemical industries reported that they did not feel that this activity should be included in the final grade of their students.

**Table 4.** Experience evaluating visits to chemical industries.


Another interesting result (regarding the opinion of professors about making industrial visits) was that 100% of the professors (18) considered that the industrial visit encourages student participation and learning. All of the professors who made industrial visits in the past felt that these should be included as teaching resources in their classes. However, 22% (4) of these professors reported that this could generate extra work, which could create problems when planning their classes in the future.

Most of the chemistry students and professors made at least one industrial visit (see Table 1). The percentage of students (61%) who had experiences with industrial visits was slightly larger than that of the professors (56%). Although these results suggest that industrial visits are a known and used teaching strategy in chemistry-related careers, it is worth noting that a considerable number of students and professors (approximately 40%) do not use this outside-the-classroom learning strategy. This may reduce the learning opportunities for students who have not had the chance to be involved in industrial visits, which may have had a positive influence in student interests in science applications in real life environments [23,49].

Most chemistry students (83%) and professors (94%) who experienced industrial visits in the past reported having sought this type of learning outside the classroom activity with the motivation to update themselves, and learn more about the subjects taught in class. This suggests that both professors and students seek to engage in industrial visits to develop their knowledge within an environment where they can interact with people highly-trained in current chemical techniques and processes. Figure 1 shows that 89% of students who experienced an industrial visit felt that this activity improved their learning experience. These results were somehow expected based on previous literature suggesting that students usually prefer industrial visits as a learning activity instead of regular lectures in the classroom, or other types of LOtC activities [49,68].

Students who have had experiences with industrial visits learned more about applying the topics taught in their classrooms, and these experiences helped them develop more critical and participatory attitudes [4,27]. It should be noted that, not just students are motivated to learn from industrial visits, professors also update and enrich their knowledge, from an industrial point of view [2,69,70]. In addition to the academic benefits that students and professors could obtain from industrial visits, the industry could use these industrial visits as a first step for recruiting future chemistry professionals that could show interest and potential for working in the visited industries [71].

Although industrial visits, for chemistry teaching at the college level, turned out to be an activity that piqued the interest of professors and students, it is clear that this learning outside of the classroom activity has not yet been adopted by certain professors who prefer other teaching methods (see Table 1). These professors reported that the main reason for not considering industrial visits as a teaching strategy in their classes was the lack of time, with 86% of them reporting that they did not have the time to plan and make a visit to an industry related to chemistry. This lack of time for organizing an industrial visit on the part of the professor goes hand-in-hand, with the fact that 67% of students reported that they had not made any industrial visits because their chemistry professors had never invited them.

These results suggest that increasing the number of industrial visits by chemistry students requires convincing professors that the time spent on this learning outside of the classroom activity can result in better academic development for their students. Although existing literature mentions that there are various impediments for programming industrial visits [15], these results indicate that lack of time is the biggest obstacle that the professors encounter when thinking about including an industry visit in their chemistry courses. This lack of time should be considered by university administrators in charge of the workload of chemistry professors. If professors do not motivate their students to engage in activities connected with the chemical industry, then students are unlikely to show interest in making industrial visits on their own [72]. Among possible issues that may complicate the planning and organization of industrial visits could be the arranging of transportation from the university to the selected industry. This transportation issue could be a major challenge if there is no sufficient industry around the university, creating possible conflicts for students and professors, who must invest the whole day around an industrial visit—that may negatively affect their academic activities, research projects, or other personal activities [73]. Another challenge for planning industrial visits could be related to safety and confidential regulations that industries may have. The accommodations that any industry must arrange before a visit, to avoid possible safety issues when they allow outside people to enter their building, and perhaps, the will of some other industries to keep their processes and materials a secret, to maintain an edge over their competitors, are important factors that need to be discussed in advance of any possible industrial visit [74]. In the end, planning and conducting industrial visits requires time and work, and most of this workload is normally handled by the professor alone. This workload could be the reason why some professors choose not to include industrial visits in their courses, and universities and chemistry departments could help ameliorate this issue by having administration take charge of planning and coordinating industrial visits.

The results in Table 2 show that the students do not have a clear preference over which semester might be best suited for their first industrial visit. Although the largest group of students (37%) selected the third semester as the best option for industrial visits, the other semesters (first, second, and fourth) also had considerable preferences (no less than 18% for each semester). This indicates that chemistry students may be interested and prepared to take advantage of making industrial visits in any semester from the beginning of their curriculum onwards.

Concerning the professors: 50% of all professors selected the first semester as the best option for inviting their students to industrial visits (see Table 2). This suggests that most professors think about putting their students in touch with the chemistry industry, since the first semester can give them better opportunities to develop their skills and interest in chemistry-related topics. Although the professors believe that industrial visits could have a positive impact on student training, 91% of them showed a preference to conduct

learning outside the classroom activities only once a year, or once every two years. This result must be related to the extra work involved in organizing and attending an industrial visit, which leads professors to think that it is better to organize these visits sporadically instead of once each semester. Concerning the students: 91% of students reported that they would like to have an industrial visit scheduled every semester so that they would have more opportunities to observe how their in-class studies are applied, as well as improve their learning experiences.

All chemistry professors who had made an industrial visit (18) reported having begun their industrial visit with a clear and well-planned objective. Although all professors had a clear goal at the time of the industrial visit, only 78% of students reported that they knew the purpose of their industrial visit before it started. This suggests a lack of clear communication between the professors and students when planning and making an industrial visit, given that 22% of the students attended without understanding the purpose of the activity beforehand. This problem could be mitigated if professors and administrators try to include students in the industrial visit organization and in the decision of which companies they might visit. Then, there would be a space to discuss what may be the most relevant aspects in the visit, and what kind of processes might be observed, in detail, in each industry. Students may feel that arranging visits to certain industries could be a waste of time because they have other interests that are not related to some industries, and visiting an industry in which students are not interested may have a negative effect on their motivation and behavior during the visit [73]. In this sense, the work of planning and performing an industrial visit must be shared between professors, administrators, and students, so that the objectives of the industrial visit are clear, and students can take advantage of the benefits from this learning outside of the classroom activity [3,75]. An additional advantage of including students during the industrial visit organization process is that possible schedule complications with students could be avoided, since students may have other classes or extracurricular activities that may need to be rescheduled with anticipation [73].

Another feature that could improve industrial visits is the evaluation at the end of this activity. It is noteworthy that 58% of the students and 72% of chemistry professors reported never having assessed the process and results of their industrial visits (see Table 4). This lack of an evaluation system can complicate the establishment of an objective, making it difficult to establish whether the goal was met.

If professors could include an evaluation rubric that, from the outset, establishes the learning objective for the industrial visit, then this learning outside of the classroom activity would have a better chance of success, and it might foster student interest in the most relevant activities and processes of the chemical industry they attended. This evaluation would also help professors set realistic objectives oriented to the learning of their students. It would also help professors consider the teaching tools and resources that are available at the location selected for the industrial visit [50]. The lack of assessment of industrial visits may be related to the little interest some professors have shown in including industrial visits as part of the final grade of their chemistry courses. Half of the professors reported not considering the performance and learning of their students during industrial visits as part of the final grade, which may send the wrong message to students that industrial visits are not important, and do not have a well-structured objective.

It is possible that the time and extra work of coming up with objectives and methods of assessment for industrial visits are why 22% of the professors think that including these visits in their course schedules every semester would create long-term problems. This is supported by the observation that, although 100% of these professors recommend including industrial visits as a teaching resource to stimulate meaningful student participation and learning, not all professors are willing to include this activity in their evaluation methods because this requires more work and time than they normally put into planning their chemistry course. An alternative LOtC activity that could have a similar impact than industrial visits could be industrial talks [49]. These industrial talks need to be wellplanned with clear goals and objectives to help professors and students take advantage of the expertise that different professionals could provide during their talks [76]. Similar to the industrial visits, the industrial talks would need extra work and new evaluations, but the workload is less times consuming than planning an industrial visit; with less difficulties to solve by professors and administrators.

At the end, students who complete a chemistry-related major, having some knowledge about the current industry needs, could develop abilities that may help them experience faster incorporation into the industry. This knowledge about industry needs could be facilitated to college students if their universities invite them to get involved with industrial visits, or invite industry professionals that could give them a talk about the current industry processes and technology [74]. Having these types of LOtC activities would help strengthen academic plans and create better learning environments. Students would have better opportunities to develop their skills, helping them feel more motivated to engage in professional activities related to their majors. The positive influence that industrial visits may have on chemistry students' development and motivation could also be a good starting point for solving sustainability problems; as having more prepared chemistry professionals can contribute to regional development, which ultimately would collaborate toward fulfilling the 2030 Agenda for Sustainable Development goals.

#### **7. Conclusions**

Results of this study suggest that industrial visits are well accepted by college chemistry students and professors. Students believe that this LOtC activity allows them to improve their knowledge, develop their critical analysis skills, and become more interested in sustainability and chemistry-related topics (see Figure 3). Professors who teach courses in sustainability and areas related to chemistry believe that the industry visits create the right environment for their students to participate more actively in class and engage in professional activities related to their careers (see Figure 3). Despite knowing the aforementioned advantages, these professors feel that scheduling industrial visits in their courses is time-consuming. They prefer that these visits only take place sporadically, with one visit every year or two. If these professors feel that they do not have enough time and support from their university departments to organize and conduct industrial visits (see Figure 3), then these industrial visits are unlikely to take place. Another point that complicates industrial visits is that the professors rarely evaluate this activity, and they usually decide not to include the performance and learning of their students during the industrial visit in the final grade of their courses (see Figure 3). This lack of evaluation may jeopardize the consideration of the industrial visits as teaching methods in sustainability and chemistry courses. Lack of an evaluation system could also complicate the planning and organizing of this activity, since it is not known whether the objective of the visit was well-understood and achieved by the students.

Although chemistry students and their instructors have a positive outlook on industrial visits, there are still a considerable number of students who have never had the opportunity to make one. This situation may detract from the development opportunities of students who are not included in industrial visits. This responsibility relies on chemistry professors and administrators. Administrators in charge of sustainability and chemistryrelated programs in universities must strive to create environments where industrial visits are common practice among their professors. For this, administrators must consider the number of courses and the resources they assign to each of their professors each semester, providing professors with enough time to plan and complete all of their academic activities, including visits to chemical industries. If the professors are motivated to make industrial visits, and they are given space in the class calendar to organize and carry these out, then more professors would likely include these LOtC activities, which may ultimately help their students' professional development.



**Figure 3.** The advantages and disadvantages of organizing and conducting visits to chemical industries.

#### **8. Limitations and Future Work**

This research was conducted in a city university campus in one of the most important industrial sectors of the western region of Mexico. This city is Guadalajara in the state of Jalisco, in which it provides to this particular university the possibility of having many different options to select industries from a large variety of productive activities. Having different industries close to the university may facilitate planning and conducting industrial visits, and could also help professors choose industries with processes and activities that may fit the syllabus of classes and student interests in a better way. If any university location is different from the context presented in this research, then the university's administrators and professors should consider different issues that may complicate planning and conducting industrial visits, such as transportation and finding the right industry to visit. This issue could be more evident for different cultures and contexts [48].

This survey was a good starting point for documenting how professors and students feel about industrial visits as a LOtC activity, but future versions of this survey can be expanded to collect further and more specific information about the needs for each class. Future data collections could also help chemistry educators understand more about sustainability issues that might be relevant for undergraduate students in sustainability and chemistry majors. To this end, professors may consider visiting industries related to green chemistry or green energy solutions in current processes, which could ultimately help promote student interest in designing sustainable solutions to current regional and global issues.

**Author Contributions:** All authors contributed equally to this work. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) in in the office of the Research Coordination of the Academic Secretary of the University Center for Exact Sciences and Engineering of the University of Gudalajara. and with the principles established in the Code of Ethics of this institution, protocol number: IV/2018/117, session of March 2, 2018 of the General University Council.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Data is contained within the article.

**Acknowledgments:** The authors would like to acknowledge the Writing Lab, Institute for the Future of Education, Tecnologico de Monterrey, Mexico, for the financial and logistical support received to be this article an open access source.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A Questionnaire**

Survey translated to English (original questions were in Spanish).

	- (a) Getting out of the classroom/university
	- (b) Learning new topics and actualize my knowledge
	- (c) Relaxing
	- (a) Lack of time
	- (b) These visits are expensive
	- (c) No one has ever mentioned the possibility of participating in/organizing an industrial visit
	- (a) First semester
	- (b) Second semester
	- (c) Third semester
	- (d) Fourth semester or later
	- (a) One each semester
	- (b) One every year
	- (c) Only one during the entire major

(a) Yes (b) No

9. (**I**) If you have been involved in at least one industrial visit in college, do you consider that participating in industrial visits could facilitate your learning about chemistry topics? **Just for students**

(a) Yes (b) No

(**II**) Do you consider that the students participating in industrial visits could facilitate their learning about chemistry topics? **Just for professors**

(a) Yes (b) No

10. (**I**) If you have been involved in at least one industrial visit in college, have you ever been evaluated about the industrial visit? **Just for students**

(a) Yes (b) No

(**II**) Have you ever evaluated your students on the industrial visit? **Just for professors**

(a) Yes (b) No

	- (**II**) If you have previous experience organizing/realizing at least one industrial visit for your college students, do you consider that these industrial visits should be evaluated as a part of your students' final grades? **Just for professors**

#### (a) Yes (b) No

12. (**I**) By doing at least one industrial visit; do you think that these experiences of industrial visits encourage your participation and learning in chemical subjects? **Just for students**

(a) Yes (b) No

(**II**) If you have previous experience organizing/realizing at least one industrial visit for your college students, have you ever evaluated your students' performances on any of these industrial visits? **Just for professors**

#### (a) Yes (b) No


(a) Always (b) Almost always (c) Rarely (d) Never

17. (**I**) When carrying out an industrial visit, do you deliver a report of the industrial visit? **Just for students**

(a) Always (b) Almost always (c) Rarely (d) Never

	- (a) Always (b) Almost always (c) Rarely (d) Never.

#### **References**


### *Article* **Effects of a Thermal Inversion Experiment on STEM Students Learning and Application of Damped Harmonic Motion**

**Omar Israel González-Peña 1,\*,†, Gustavo Morán-Soto 2,†, Rodolfo Rodríguez-Masegosa 1,† and Blas Manuel Rodríguez-Lara 1,†**


**Abstract:** There are diverse teaching methodologies to promote both collaborative and individual work in undergraduate physics courses. However, few educational studies seek to understand how students learn and apply new knowledge through open-ended activities that require mathematical modeling and experimentation focused on environmental problems. Here, we propose a novel home experiment to simulate the dynamics of a flue gas under temperature inversion and model it as damped harmonic motion. After designing and conducting the experiment, twenty six first year students enrolled in STEM majors answered six qualitative questions to inform us about their epistemological beliefs regarding their learning process. Their answers imply that this type of open-ended experiments may facilitate students' understanding of physical phenomena and point to the significance of physics instructors as promoters of epistemological development. In general, students described this activity as a positive experience that helped them connect an environmental phenomenon with a fundamental physics concept.

**Keywords:** temperature inversion; damped harmonic motion; stem education; teaching methodologies; sustainable sevelopment goals; sustainable education; epistemology; mathematical modeling; educational innovation; higher education

#### **1. Introduction**

Finding strategies to promote student engagement in introductory physics courses is a challenge of our times. For instance, something as simple as identifying the preferences of students for learning physics by demonstrative problem solving on a blackboard or supervised independent collaborative work may improve their learning process [1].

In a traditional learning environment, introductory physics curricula is usually designed with the laboratory at the core of the learning process. It becomes a place to learn theoretical concepts as well as to conduct experiments [2,3]. This methodology increases student engagement in physics courses and improves conceptual understanding through manipulation of instruments and materials [3,4] to generate and process experimental data [5]. A well designed experiment may help STEM students develop self-regulated learning strategies [4,6,7], giving them the opportunity to build their own conclusions and boost their knowledge about physical phenomena and its interpretation [8]. Self-regulation and motivation is usually driven by epistemic beliefs [9,10] that describe the way students think about the nature of knowledge and knowing [11].

On the other hand, it is of utmost importance to develop Modeling Instruction (MI) plans to involve students beyond the four walls of the laboratory as engagement and cooperation play an essential role in the learning process of physics concepts [12]. These plans must improve the academic success of our students by boosting their interest

**Citation:** González-Peña, O.I.; Morán-Soto, G.; Rodríguez-Masegosa, R.; Rodríguez-Lara, B.M. Effects of a Thermal Inversion Experiment on STEM Students Learning and Application of Damped Harmonic Motion. *Sustainability* **2021**, *13*, 919. https://doi.org/10.3390/su13020919

Received: 11 December 2020 Accepted: 13 January 2021 Published: 18 January 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

and involvement; for example, implementing active learning [13–15] using passive [16] or interactive [17,18] content as well as social media platforms [19] to promote students interaction with their whole learning environment within the framework of distance learning methodologies.

We believe that, in general, a well-designed instruction strategy facilitates STEM students understanding of physical phenomena that, in time, motivates the design and development of open-ended challenges aiming to provide solutions to urgent global issues. Here, we focus on the issue of air quality control and pollution to elaborate on the concept of simple and damped harmonic motion applied to particulate dynamics in the atmosphere. This issue is specially relevant for us as five Mexican cities (Mexico City, Monterrey, Guadalajara, Toluca and Leon) rank among the 13 cities with worst air quality according to a recent report from the Organization for Economic Co-operation and Development (OECD) [20], and it is common to observe pollutants trapped by temperature inversion in our daily life. Studying simple and damped harmonic motion applied to the dynamics of a particulate in the atmosphere could provide fertile ground to boost students cognitive process and to create awareness within the framework of the 2030 Agenda of Sustainable Development Goals. In Section 2, we state our purpose in detail and follow it with a literature review on teaching the damped harmonic oscillator in Section 3. Then, we present the details of our theoretical model and experimental setup in Section 4. Sections 5 and 6 show the methods and results of our qualitative research in detail, in that order. They are followed by a brief discussion in Section 7. Finally, we close with our conclusion in Section 8.

#### **2. Purpose**

Students learning new topics usually undergo an epistemological process where they reason about specific information obtained from different sources, then they claim knowledge of these new topics [21]. Kitchener's work suggests that open-ended problems or activities are more likely to engage students on an epistemological process than solving problems in class [22]. Our research aims to help physics educators by analyzing the effect of constructing and experimenting with a simulator of thermal inversion on students understanding of the damped harmonic oscillator.

Here, we propose a methodology to construct an isobaric troposphere simulator using air confined by a glass vessel where it is possible to introduce a foreign gas and make it oscillate by controlling the temperatures at the bottom and top ends of the container to simulate temperature inversion. We presented this activity to STEM students enrolled in first year introductory physics courses and asked them to collect experimental data of the dynamics to compare it with a numerical simulation of the damped harmonic oscillator. A goal for our students was to find values for the various parameters of the system that provide a good fit between experiment and theory. This was a collaborative activity for teams of four students that submitted a single project report. Individuals underwent an argumentative test that served as evidence to evaluate and accredit the understanding and mastering of a technical competence. In addition to this academic evaluation process, a cohort of students answered a questionnaire looking for their point of view on the effect of this experiment in their understanding of the thermal inversion phenomenon and its relation to damped harmonic motion.

#### **3. Literature Review**

The harmonic oscillator is at the core of our modeling of real-world devices involving integrated circuits, fluid mechanics, optical systems, and quantum technologies among others. Thus, engaging STEM students in an cognitive process that allows them to claim knowledge of this concept and extend its use beyond particular examples becomes a fundamental objective of physics education.

Pendulum and spring-mass systems are the standard textbook example of harmonic motion in introductory physics lectures. A spring-mass experiment is simple enough to introduce the idea of damped oscillations by measuring the position of the mass [23–27]. Conducting experiments with pendulum may improve student satisfaction under selfevaluation of their learning experience and knowledge of the subject [28]. Of course, real world devices are not completely harmonic; both spring-mass systems [29] and pendulum [30] beyond the small displacement or oscillation angle limit, in that order, serve as examples of basic non-linear models that undergraduate students may build using simple materials.

In addition to physics concepts, the oscillation of complex systems, like membranes or strings, allows the introduction of differential equations [31] and Fourier methods to physics problems [32]. In this direction, coupling a pair of simple harmonic oscillators may ease introducing the idea of coupled differential equations to students with some experience in classical mechanics [33]. From the simple to the complex, the harmonic oscillator offers an opportunity to understand the significance of mathematical modeling and visualize the effect of variable manipulation to engage STEM students into learning by physical interpretation [34].

Furthermore, analogies to the concept of harmonic oscillators may help students understand the workings of real-world devices and phenomena. For example, the infrared spectrophotometer [35] may be modeled as a spring-mass oscillator and the membrane vibration happening inside a microphone [36] is an analogy to a driven harmonic oscillator [37,38]. For more advanced courses, the idea of a classical harmonic oscillator may be extended to the quantum realm, for example, using basic calculus and algebra [39] or studying fluorescence in diatomic sulfide [40]. The motion of an electron in the presence of a two-dimensional potential is another simple example of harmonic motion [41] and analogies using spring-systems with coupled masses may help to introduce the formation of quantum bands to STEM students [42].

On the education side, the idea of interactive conceptual instruction using collaborative problem solving [43], computational simulations [44] and virtual laboratories [45], or alternative learning methods [46] show improvement clarifying misconceptions related to harmonic motion. Furthermore, flipped learning using software simulation of electrical circuits shows improvement in the students knowledge of the damped harmonic oscillator [47].

Our experimental proposal focuses on the importance of exploring systems beyond the spring-mass and pendulum in order to boost the epistemic cognitive process. In this direction, electronics present an opportunity to introduce highly controllable damping and non-linearities to harmonic oscillators beyond mechanical systems [48,49] and, for advance courses, the ability to produce, for example, time-dependent control to introduce continuous symmetries and its invariants following Noether theorem [50]. We may look into space and introduce the damped harmonic oscillator using the dynamics of particular celestial bodies [51] or to more complex setups, for example, lasers and optical resonators [52], classical gases confined by harmonic potentials [53], or the oscillation of a superconductor ring levitated by a magnetic field [54]. In particular, we are interested in fluids as they are exceptionally helpful to elucidate the effect of non-constant friction forces [55,56].

In the following, we present a simple experiment that allows STEM students to simulate an isobaric troposphere to explore, for example, the relation between damped harmonic motion and the dynamics of a particulate under temperature inversion at home.

#### **4. Experimental Methods**

The core of our proposal is a toy model of the troposphere to visualize how a gas cloud under temperature inversion behaves like a damped harmonic oscillator. Our experimental set-up, Figure 1, builds upon the idea that a transparent container whose ends are covered by a good thermal conductor may serve as a simulation of an isobaric troposphere at atmospheric pressure *p* with control of bottom and top temperatures, *Tb* and *Tt*. An inlet in the bottom, a flue or exhaust, allows us to introduce a foreign gas, in our case the result of paper combustion, that we model as a non-interacting sphere with constant density *ρ<sup>g</sup>* in order to follow its center of mass motion. This approximation allows us to account the dynamics for the collection of combustion gases and particulates moving along with them as a whole.

**Figure 1.** Sketch of the experimental setup used to simulate an isobaric troposphere. A clear container with two temperature sensors and an inlet to inject smoke.

Let us start with the model. Temperature control at the bottom and top ends of our simulation allows us to assume an air density that depends on the height. For the sake of simplicity, we suppose constant pressure and linear temperature gradient that allows us to approximate the air density,

$$
\rho\_{\text{air}}(y) = \rho\_b (1 - \varepsilon y), \qquad \text{with} \qquad \varepsilon = \frac{1}{h} \left( 1 - \frac{T\_b}{T\_t} \right), \tag{1}
$$

up to first order on the height *y*, that we take as zero at the container bottom and *h* at the top such that *y* ∈ [0, *h*]. We define the air density at the bottom and top ends,

$$
\rho\_b \equiv \rho\_{\rm air}(0) = \frac{pM}{RT\_b} \qquad \text{and} \qquad \rho\_t \equiv \rho\_{\rm air}(h) = \frac{pM}{RT\_t} \tag{2}
$$

in that order, where our local atmospheric pressure at 540 m above the sea level is *<sup>p</sup>* <sup>=</sup> 101,388 Pa [57], the molar mass of air is *<sup>M</sup>* <sup>=</sup> 28,965.4 <sup>×</sup> <sup>10</sup>−<sup>6</sup> kg mol−<sup>1</sup> [58,59], the ideal gas constant is *R* = 8.31447 J mol−<sup>1</sup> K<sup>−</sup>1, and the temperatures are given in Kelvin.

Assuming the collection of combustion gases and particulates moving along them as a non-interacting microscopic sphere allows us to model its center of mass dynamics using Newton second law,

$$m\frac{d^2y}{dt^2} = -w + F\_B - b\frac{dy}{dt},\tag{3}$$

where the forces in the right-hand-side of the equation are the weight *w* = *mg* pointing downwards, the buoyant force *FB* = *ρ*air*V*gas*g* pointing upwards, and the Stokes drag for a sphere moving through a viscous fluid proportional to the drag coefficient, *b* = 6*πr*gas*η*air in terms of the radius of the spherical particle *r*gas and the viscosity of the air *η*air, and the velocity of the particle *dy*/*dt*.

The mass of the effective particle is given by its density and volume, *m* = *ρ*gas*V*gas, such that its weight becomes *w* = *ρ*gas*V*gas*g* and the dynamics reduce to the following second order differential equation,

$$\frac{d^2y}{dt^2} = \left[\frac{\rho\_{\text{air}}(y)}{\rho\_{\text{gas}}} - 1\right]y - \frac{b}{\rho\_{\text{gas}}V\_{\text{gas}}}\frac{dy}{dt},\tag{4}$$

where we assumed that the gas density change induced by the temperature gradient is negligible at the time scale of the experiment and that the height of the container is small enough to produce no significant changes in the value of standard gravity. For the sake of simplicity, we assume that the viscosity of air has a negligible change with the temperature gradient. We use our linear approximation to the air density inside the container to unfold the model,

$$\frac{d^2y}{dt^2} = -\frac{\varepsilon g}{\rho\_{\text{gas}}}y + \left(\frac{\rho\_b}{\rho\_{\text{gas}}} - 1\right)y - \frac{9\eta\_{\text{air}}}{2\rho\_{\text{gas}}r\_{\text{gas}}^2}\frac{dy}{dt},\tag{5}$$

into that of a damped oscillator,

$$\frac{d^2y}{dt^2} = -\omega\_0^2 y + a\_0 - \gamma \frac{dy}{dt},\tag{6}$$

where the temperature difference controls the sign of the frequency,

$$
\omega\_0^2 = \frac{\mathcal{G}}{h\rho\_{\text{gas}}} \left( 1 - \frac{T\_b}{T\_t} \right). \tag{7}
$$

Without considering the rest of the terms in the right-hand-side of the oscillator equation, if the temperature at the top is lower than that at the bottom, *Tt* < *Tb*, we have a positive squared frequency *ω*<sup>2</sup> <sup>0</sup> > 0 that yields a harmonic oscillator and we will see our gas sample oscillate. In the opposite case, *Tt* > *Tb*, we have a negative squared frequency *ω*2 <sup>0</sup> > 0 that yields an inverted oscillator and our gas sample will rise and remain at the top of the container. For an isothermal simulation, *Tt* = *Tb* the squared frequency is null, *ω*2 <sup>0</sup> = 0 and only the external effective acceleration,

$$a\_0 = \left(\frac{\rho\_b}{\rho\_{\text{gas}}} - 1\right)g\tag{8}$$

has an effect on the dynamics. Without considering the rest of the terms in the right-handside of the equation, if the gas density is larger than that of the air at the bottom of our simulator, *ρ*gas > *ρb*, the effective external acceleration is negative, *a*<sup>0</sup> < 0, and the gas sinks to the bottom of the container and stays there. If it is smaller, *ρ*gas < *ρb*, the effective external acceleration is positive, *a*<sup>0</sup> < 0, and the gas rises to the top of the container and stays there. If they are equal, *ρ*gas = *ρb*, the effective external acceleration is null and the gas does not move upwards nor downwards. Finally, the approximate drag frequency for the spherical particles of gas,

$$\gamma = \frac{9\eta\_{\text{air}}}{2\rho\_{\text{gas}}r\_{\text{gas}}^2},\tag{9}$$

where we assume a constant air viscosity as it changes from *η*air(*T* = 213.5 K) = 0.0171 × <sup>10</sup>−<sup>3</sup> Pa s to *<sup>η</sup>*air(*<sup>T</sup>* = 313.5 K) = 0.0218 × <sup>10</sup>−<sup>3</sup> Pa s for a temperature gradient of 100 K [60]. We take the average of these values as our constant viscosity, *<sup>η</sup>*air = 0.01945 × <sup>10</sup>−<sup>3</sup> Pa s [60].

Our toy isobaric troposphere model allows the simulation of diverse dynamical phenomena. Temperature control at the ends of the simulator, for example, allows to switch the driven and damped oscillator between inverted or harmonic behaviour. Changing the atmospheric or foreign gases gives even more options to control and explore the parameters of the model. In the following, we focus our observations on temperature inversion.

Under standard conditions, the temperature gradually falls with the increase of altitude,

$$
\Gamma = -\frac{dT}{dy}.\tag{10}
$$

This is known as the thermal lapse rate; for example, the dry adiabatic lapse rate is around <sup>Γ</sup> <sup>≈</sup> 9.8 <sup>×</sup> <sup>10</sup>−<sup>3</sup> K m<sup>−</sup>1. Temperature inversion is the phenomenon that occurs when the thermal lapse rate Γ changes sign from positive to negative; that is, a hot layer of air with low density hovers above a colder one with high density. In these situations, it is possible to observe smoke, or other pollutant gases, form a ceiling as the top low density layer of air stops their ascend.

In order to have a reference for the behavior, we present a numerical experiment in a container that is 0.15 m tall, take the standard value of gravity *g* = 9.81 m s<sup>−</sup>2, the bottom of the container at room temperature *Tb* = 298 K leading to the density of air *ρ<sup>b</sup>* = 1.225 kg m−<sup>3</sup> [61], the top of the container heated to *Tt* = 373.15 K and a constant viscosity of air *<sup>η</sup>*air <sup>=</sup> 1.945 <sup>×</sup> <sup>10</sup>−<sup>5</sup> Pa s. We assign the gas a density of *<sup>ρ</sup><sup>g</sup>* <sup>=</sup> 1.140 kg m−<sup>3</sup> with a radius of *rg* = <sup>6</sup> × <sup>10</sup>−<sup>3</sup> m that represents about 0.06% of the total volume of the container. These assumptions provide constants for the differential equation,

$$
\omega\_0 = 3.400 \text{ rad s}^{-1} \text{ \textdegree} \tag{11}
$$

*a*<sup>0</sup> = 0.731 m s<sup>−</sup>2, (12)

$$
\gamma = 1.567 \text{ rad s}^{-1},
\tag{13}
$$

leading to the damped behaviour shown in Figure 2a. Figure 2b shows the effect of random variations on the temperatures, densities, effective particulate radius, air viscosity and initial velocity, *Tb*, *Tt*, *ρb*, *ρ*gas,*r*gas, *η*air, *v*<sup>0</sup> , following a normal distribution with mean value provided by the parameters above and standard deviation equal to one percent of the mean. We want to stress how such a small change in parameters produces a strong change in the dynamics.

**Figure 2.** Damped oscillation for (**a**) a single particle of gas in the experiment with effective parameters provided by Equation (11) to Equation (13) and (**b**) mean value and region delimited by one standard deviation above and below the mean for a thousand random realizations with parameters following independent normal distributions.

We ask our students to reproduce the experimental setup, Figure 1, at home using a transparent tempered glass container to avoid fractures from temperature gradients; for instance, we use a coffee jar. In order to record the temperature at the bottom and top, we use two Vernier Stainless Steel Temperature Probes placed inside the glass container and a Vernier LabQuest Mini controller. These may be substituted by simple atmospheric

thermometers in contact with the external facet of the container at home. The sensors and a pewter straw to let smoke in are secured in place on the container lid using modeling clay to guarantee a good seal. We recommend securing a disposable plate to the container in order to hold ice cubes and secure access to the inlet straw that connects to a smoke container using plastic tubes; for instance, we used a candy jar to contain smoke from paper combustion but a party balloon or plastic bag may play the role. A light bulb lamp or an iron covered in aluminium foil may be used to change the temperature at the top of the container. Finally, we followed the effective center of mass dynamics of the flue gas using a logitech HD Pro Webcam C920 and Vernier Logger Pro but any given video capturing device and open source software like Tracker Video Analysis and Modeling Tool should do.

Figure 3 shows our experimental setup at home. As the container is not hermetically sealed, the pressure inside should be constant and equal to the atmospheric one. Our experiment allows the control of the input speed of the smoke as well as the bottom and top temperatures. We ask the students to experiment with these three parameters to explore the different dynamical regimes available. In particular, we ask for a detailed analysis of a case whose dynamics are an analogy to the damped harmonic oscillator. Their experimental data should allow them to fit for the dampening frequency *γ* and the smoke density after figuring out the input velocity of the smoke without any temperature gradient. Figure 4 shows a sequence tracking of the approximated smoke cloud center of mass in an experiment with bottom and top temperatures in the order of 292.35 K and 306.35 K, respectively.

**Figure 3.** Experimental setup used to simulate temperature inversion in an isobaric troposphere.

**Figure 4.** Example from an experimental measurement sequence. The yellow dot indicates the approximated center of mass of the smoke cloud.

Figure 5a shows experimental data points compared to the analytic model using an atmospheric pressure of *p* = 102,300 Pa and temperatures of *Tb* = 292.75 K and *Tt* = 303.95 K at the bottom and top of the container, respectively, leading to approximate air densities of *ρ<sup>b</sup>* = 1.218 kg m−<sup>3</sup> and *ρ<sup>t</sup>* = 1.173 kg m−<sup>3</sup> accounting to a height difference of *h* = 0.08 m between the temperature sensors. We assume the smoke density of about *<sup>ρ</sup>*gas <sup>=</sup> 1.210 kg m−<sup>3</sup> with radius *<sup>r</sup>*gas <sup>=</sup> 5.97 <sup>×</sup> <sup>10</sup>−<sup>3</sup> m and initial velocity of the order of *v*<sup>0</sup> = 0.145 m s−<sup>1</sup> leading to an effective acceleration, *a*<sup>0</sup> = 0.066 m s<sup>−</sup>2, as well as effective oscillator and damping frequencies *ω*<sup>0</sup> = 1.93 rad s−<sup>1</sup> and *γ* = 2.027 rad s−<sup>1</sup> , in that order. The difference between experimental data and the dynamics under our educated guess may be due to variations in any of the assumptions, as we discussed before, and provides an opportunity for discussion. Our ansatz provides a good starting point for a better fitting using, for example, Newton least squares method or Levenberg-Marquardt method for nonlinear least squares yield an effective acceleration, *a*<sup>0</sup> = 0.053 m s<sup>−</sup>2, effective oscillator and damping frequencies *ω*<sup>0</sup> = 2.025 rad s−<sup>1</sup> and *γ* = 2.137 rad s−<sup>1</sup> , in that order, and initial velocity *v*<sup>0</sup> = 0.156 m s−<sup>1</sup> that provides a better fit shown in Figure 5b.

**Figure 5.** Example from an experimental measurement sequence (data points) and its corresponding fit (solid line) from (**a**) theoretical parameters and (**b**) a numerical fit using these parameters as starting point.

#### **5. Qualitative Research Methods**

In the following, we address the qualitative methods used to analyze our students point of view regarding the effect of this learning activity on their understanding of the damped harmonic oscillator.

#### *5.1. Participants*

We selected a cohort of 26 students from a total population of 185 individuals that conducted the experiment while enrolled in different groups of the same five-week long introductory physics module during the Fall 2020 semester. The modules were lead by instructors that voluntarily accepted to distribute our questionnaire with six open-ended questions at the end of the experiment. Students were offered extra credit if they decided to answer the questionnaire; all of these 26 students answered the six questions and we collected their answers before the module ended. Our students were in the first semester of diverse STEM majors at Tecnologico de Monterrey where they are required to participate in challenges related to real-world issues as part of their professional competencies development [3,62–64].

#### *5.2. Data Collection*

Our thermal inversion experiment aims for our students to explore damped harmonic motion in a unique way to try and help them understand this concept. We distributed a questionnaire after they finalized the experiment in order to analyze the effect of our openended activity on their understanding of the damped harmonic oscillator. The six questions from the questionnaire were adapted from the Engineering Related Beliefs Questionnaire (ERBQ) [65]. The ERBQ aims to measure the beliefs of students regarding the nature of STEM-related knowledge and knowing and it has proved a successful tool for the analysis of STEM students epistemic process after solving open-ended problems [66]. These six selected questions are related to open-ended problem solving aspects of the cognitive process of STEM students. They were translated to Spanish language and then adapted to our thermal inversion experiment context to ask students about the certainty and sources of their physics knowledge. A researcher with experience teaching and conducting education research in both Spanish and English at Mexican and U.S. universities made the translation. The experience of this researcher facilitated the interpretation and accurate translation of the meaning of every question. The aim was to adapt those questions in order to reflect the context of Mexican students in their physics courses and avoid possible cultural misunderstandings [67]. Additionally, four physics instructors provided feedback about the clarity and the content of the questions aiming to reconcile possible misinterpretations. Their comments helped to establish content and face validity for the questionnaire [68]. The questionnaire was back translated to English language by the same researcher for the purpose of this paper, see Appendix A.

#### *5.3. Data Analysis*

The four researchers involved in this study conducted the qualitative analysis. We analyzed the data collected with the questionnaire using open coding to let emerging codes to stay as close as possible to the own words and ideas of the students [69]. The final codes for each student were compared side by side with codes from other students aiming to find similarities that could be coded together into meaning units [70]. This coding philosophy is similar to the methodology proposed in the constructivist grounded theory [71]. This qualitative analysis approach helped us draw conclusions on how our thermal inversion experiment influenced our students knowledge and understanding of the damped harmonic oscillator. It also helped us ensure that these meaning units appropriately reflected the responses and feelings about the experiment of our students. We used the epistemical belief system framework proposed by Schommer-Aikins [72] as a lens to analyze and compare these meaning units. This framework helped us determine how the epistemological beliefs of our students evolved during the learning process and how they developed and acquired new knowledge. At the end, we selected some of the most relevant comments from the students to support our findings. These comments were translated from Spanish to English language by the same researcher that translated the questionnaire and are included in the following section.

#### **6. Qualitative Results**

Half of our students (13) reported using a single methodology or theory for designing and developing their thermal inversion experiment, while the other half (13) used a combination of two or more different methodologies or theories for their design. Regardless of whether they used only one methodology or different methodologies, our students reported that the most likely starting point for their thermal inversion experiment design was the methodology previously explained by their instructor. Student A4 noted: "I thought of some less orthodox methods but I mostly focused on the one taught in the lessons."

Most students (20) reported asking for help to their instructor during the design and development of their thermal inversion experiment. Seventeen students reported looking for more than one source of information; for example, asking other instructors, classmates, or family members to confirm and complement the information about the damped harmonic oscillation they had. Student A19 noted: "We asked the challenge's instructor for help. However, we needed further help and requested it from friends in a senior class." Only one student reported completing the inversion experiment without asking for help.

When students were asked if they searched for additional sources of information to complement what they knew about the damped harmonic oscillator, almost all of them (25) decided to look for additional information in different sources outside of what they learned in their course. Most of these students (17) looked for more than one source of external information. The most recurrent sources of external information were videos (15) and websites (13). Other less common sources were textbooks (5) and scientific papers (3). Eight students reported looking for only one external source of information. Only one student decided not to look for external sources of information.

Almost all students (24) reported that the thermal inversion experiment helped them to better understand the damped harmonic oscillator. These students stated that this experiment helped them clarify some doubts about damped harmonic motion, while they learned different ways to apply this topic in real life problems. Student A23 noted: "It was really helpful. Before, I thought that harmonic movement only applied to springs, but now I understand that it also applies to more complex systems such as fluids." On the other hand, only two students mentioned that this experiment was confusing and it did not help them understand the damped harmonic oscillator. Most of our students (23) stated that the thermal inversion experiment could have different final results that could be valid depending of the methodology used during the design, or some differences in the obtained measurements; seven of these students argued that their responses need to be supported by theory to be considered a valid response.

Half of our students (13) stated that the thermal inversion experiment might have better results in their understanding of the damped harmonic oscillator if the instructor could spend extra time explaining the theory and giving them more details about how to apply this concept to solve different problems in different contexts. This issue was more evident for seven students that mentioned having struggles to answer the argumentative test due to difficulties adapting their knowledge and experiences from our thermal inversion experiment to the context of the text. Student A11 noted: "I felt prepared to answer questions related to the challenge, but the argumentative exam had nothing to do with thermal inversion."

#### **7. Discussion**

Students in our research showed that they expected instructions from their professors in order to design the experiment following what they have previously learned and practiced in classes. They were open to follow different methodologies to design and develop their experiment, but they stated that the guidance from an expert is the key to success in this type of open-ended activities. This may be related to a low level of epistemological maturity for college level students [73], where they need to learn new knowledge and ways to apply it from an epistemological authority that is considered the only source of reliable information [11]. Most of our students asked for help during the experiment design and development, showing they are likely to search for an epistemological authority that could help them to develop their knowledge during the experiment process as well. Students that asked for help reported going directly to their instructors with specific doubts and questions. This behavior pinpoints the importance of physics instructors and the information they provide to their students before and during the development of the experiment.

Students reported that they are likely to seek additional information sources to analyze what they know and solve some doubts; videos and websites are the most common places where they look for more information. This interest in seeking additional sources of knowledge may be seen as a self-regulated learning action [10]. Promoting these types of actions may ultimately help students understand class material and perform better in solving open-ended problems. Although most students searched for additional sources of information, very few consulted scientific publications and textbooks to expand their knowledge and answer questions. This lack of interest in scientific publications is an area of opportunity to improve the research skills of our students and should be considered by physics instructors when advising their students on the advantages of seeking information supported by scientific evidence [74]. Physics instructors should be aware that their students are likely to search for the most accessible source of information, and then build their knowledge from that source rather than looking for more reliable sources which may be harder to find and analyze to construct their knowledge [75].

At the end of the thermal inversion experiment and the argumentative test, almost all students described this activity as a positive experience that helped them to better understand the damped harmonic oscillator and how this physical concept may be applied in different contexts. This type of open-ended activities may help students develop their knowledge and make them think about different sources where they can look for information to solve their doubts. Giving the opportunity to experience these type of experiments to students may help physics instructors facilitate the epistemological development of their students; different studies have shown similar results for different courses and contexts [66,76]. Reaching higher levels of epistemological maturity may benefit the development of students as critical thinkers, making them more likely to think that the development of their knowledge is their responsibility and that they need to search and confirm their own knowledge and beliefs [73]. This is relevant for physics instructors because some students stated that they felt the knowledge about thermal inversion that they learned and experimented was not transferable to different instances involving the damped harmonic oscillator. This lack of abilities to apply the same physics principle to solve different problems is common in low levels of epistemological maturity [73]. This position on how to learn and apply new concepts could hinder the possibilities of our students to understand complex physical phenomena [77].

#### **8. Conclusions**

It is important that physics instructors strive to provide enough information to their students so that they develop sufficient confidence to successfully complete open-ended activities like our thermal inversion experiment. The process of preparing students with the basic knowledge to perform experiments such as the one presented here needs certain precautions. Physics instructors must provide enough information without leading the entire experience so that their students have the opportunity to solve emerging issues on their own. They must take the role of facilitators, holding their students responsible for solving their own doubts by searching for reliable sources of information that help them develop their knowledge and solve their doubts. Our thermal inversion experiment may be used by instructors to reinforce students knowledge of the damped harmonic oscillator and to motivate them to evolve from low to mature levels of epistemological maturity where they are more likely to seek and develop their own knowledge. It may also help students develop research interest and skills leading to a deeper understanding of scientific concepts in advanced semesters. In addition, the possibility of controlling the various parameters in our isobaric troposphere simulator allows for the real-time visualization of different dynamics that may be useful in courses in Earth Science or Environmental Engineering. This type of experiment opens a window of opportunities for faculty to propose more challenging activities that might facilitate facing the great issues that society has related to the 2030 Agenda of Sustainable Development Goals.

**Author Contributions:** All authors contributed equally to this work. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Tecnologico de Monterrey grant number NOVUS-2020-308.

**Institutional Review Board Statement:** Ethical review and approval were waived for this study by Tecnologico de Monterrey.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** O.I.G.-P. acknowledges to Arath Marín-Ramírez for the literature collecting process on the previous studies in harmonic oscillator in the education context. R.R.-M. is grateful to his students for agreeing to share their evaluations anonymously for the academic analysis of this work. B.M.R.-L. acknowledges fruitful discussion and support from Benjamin Raziel Jaramillo Ávila. O.I.G.-P., R.R.-M. and B.M.R.-L. acknowledge NOVUS, the Writing Lab, Institute for the Future of Education, Tecnologico de Monterrey, Mexico, for the financial and logistical support.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyzes, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **Abbreviations**

The following abbreviations are used in this manuscript:


#### **Appendix A. Questionnaire**

1. When preparing your answers on the thermal inversion experiment, did you consider several methods to answer the questions or did you just consider one method to find the solution? Please explain the reasoning that led to the methods that you used.

2. Did you ask for help to answer the questions? If so, who did you ask (for example, instructors, classmates, friends, family, or tutors)?

3. When answering the exercises or preparing the report, did you check any additional source of information (such as videos, books, or tutorials) besides the data given to you by the instructors? If so, which sources did you check?

4. Do you think that experimental activities based on the application of a physical concept (such as the activity on damped movement) helps improve your understanding on thermal inversion? Please explain the reasoning behind your answer.

5. Do you think that the questions you ask yourself during the experiment only have one correct answer, or that there could be more than one correct answer depending on your interpretation of certain variables and data? Please explain the reasoning behind your answer.

6. Did you feel that you had sufficient background knowledge to understand the thermal inversion experiment and to correctly answer the argumentative exam? If not, please explain which subjects would have helped you better understand the experiment.

#### **References**


## *Article* **An Online-Based Edu-Escape Room: A Comparison Study of a Multidimensional Domain of PSTs with Flipped Sustainability-STEM Contents**

**Félix Yllana-Prieto, Jin Su Jeong \* and David González-Gómez**

Departamento de Didáctica de las Ciencias Experimentales y Matemáticas, Universidad de Extremadura, 10003 Cáceres, Spain; feyllanap@unex.es (F.Y.-P.); dggomez@unex.es (D.G.-G.) **\*** Correspondence: jin@unex.es

**Abstract:** The use of active and flipped methodologies has increased in recent years. Here, gamification uses typical elements of a game in different contexts, including that of education. Specifically, Escape Room games used as educational tools have potential for teaching–learning, and they can be beneficial because they can improve students' motivation and emotions toward learning. This is particularly valuable in science, technology, engineering and mathematics (STEM) courses, where the cognitive factor and multidimensional domain are closely connected. This research presents an online-based Edu-Escape Room with science and sustainability contents as an educative tool in a STEM course. With the intervention proposed, we analyze how this tool influences the multidimensional domain (attitudes, self-efficacy and emotions) of pre-service teachers (PSTs). According to attitude and self-efficacy analysis, it is observed that most of the items analyzed show an increase in self-efficacy and more positive attitudes after the intervention. In particular, Question 11 (Q11) indicates a significant difference. Concerning the results for emotion, the positive emotions "joy", "satisfaction" and "fun" are significantly increased after the intervention. However, the negative emotions "nervousness", "frustration" and "concern" also increase, partly due to the game characteristics. The proposed activity had a medium effect on items with significant differences except for the emotion "frustration", where the intervention had a large effect according to effect size (ES) analysis. According to the principal component analysis (PCA), the attitudes, self-efficacy and emotions of the PSTs are positively correlated, and the influence of the proposed activity shows a significant improvement in these variables. Finally, the structural equation modeling partial least squares (SEM-PLS) analysis showed the effects that the instruction has on the PSTs' emotions and also that they had a significant effect on the positive attitudes towards and self-efficacy in science. Therefore, there are multiple benefits in the multidimensional domain of PSTs of having implemented the proposed online-based Edu-Escape Room.

**Keywords:** Escape Room; gamification; attitude/auto-efficiency; emotion; PST; STEM education; flipped classroom

#### **1. Introduction**

With the popular increase in active and flipped learning methodologies in the classroom, the use of games can help to support students' satisfaction, grades, collaboration and motivation for learning [1–3]. Thus, game-based education has provided the opportunity for an interdisciplinary approach that takes into consideration how to promote a realistic environment with the aim of achieving a better student experience [2,3]. In this context, Deterding [4] indicated that gamification normally uses the typical elements and techniques of games in different contexts, such as those of business or education. In particular, in university education, various studies show successful practices through the use of gamification [5–7]. They even reveal that the students who participate tend to achieve better academic performance, motivation and attitudes towards their courses [5–7]. In a

**Citation:** Yllana-Prieto, F.; Jeong, J.S.; González-Gómez, D. An Online-Based Edu-Escape Room: A Comparison Study of a Multidimensional Domain of PSTs with Flipped Sustainability-STEM Contents. *Sustainability* **2021**, *13*, 1032. https://doi.org/10.3390/ su13031032

Academic Editor: Marc A. Rosen Received: 26 December 2020 Accepted: 16 January 2021 Published: 20 January 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

context of gamification, an Escape Room, a didactic application, is a live-action game in which participants can discover clues, solve puzzles and perform tasks in one or more rooms in order to reach a specific goal that will allow them to leave the room in a limited amount of time [2,8,9]. Thus, Nicholson [10] points out the importance of connecting the teaching-learning activities into the history and context of the Escape Room so that the students can be involved in the narrative process and can then be inspired to connect with real problems and develop an intrinsic motivation to learn and explore [11,12]. However, the use of gamification and Escape Rooms in higher education is still in a premature phase; they are scarce tools, although they have potentials for teaching–learning. In these situations, with higher demands and challenges, the Escape Room is a gamification tool that can be used in science, technology, engineering and mathematics (STEM) courses along with the students' multidimensional domains [13,14].

Along with gamification in active and flipped methodologies, STEM has received increasing attention in recent years for its ability to provide proper preparation for students [14,15]. The UN decade of education for sustainability development (DESD) has described current STEM education situations [16,17] that can promote communal awareness of values and increase life-long training [16,18–20]. STEM education proposes the integration of various scientific disciplines as a cohesive entity, the teaching of which is integrated and coordinated so it can be used for problem-solving in everyday situations [21]. In fact, Wiswall et al. [22] suggested that students who participate in programs and activities focused on STEM courses obtain better grades in STEM subjects than those who do not participate. Moreover, it was observed that students participating in these STEM programs choose STEM degrees more frequently than students who did not participate. However, despite the implementation of STEM methodologies being useful in promoting scientific literacy for students, disinterest is one of the main causes of a negative attitude towards science [23]. According to Vázquez and Manassero [24], students begin to show disinterest in scientific disciplines at an early age, initiating a negative image of science in general and a subsequent abandonment of the idea of choosing a STEM career. In addressing this challenge, it is important to take into account the preferences of students in order to generate interest in scientific disciplines from an early age since these preferences and interests persist into adulthood and can facilitate the choice of STEM careers [25]. Therefore, it is important to increase the positive affective domain of students in STEM courses, which can be done through gamification and Escape Rooms in active and flipped methodologies.

In the context of gamification and Escape Rooms and with the use of active and flipped methodologies, student satisfaction, grades, collaboration and motivation have been increased in the classroom [2,3]. This indicates that different types of positive emotions are promoted and highlights the importance of motivation, interest, academic self-concept, cognitive activity, learning achievement and fun [26]. Here, students experience the proposed enigmas, as well as their usefulness in promoting collaboration and teamwork in an educational environment [26,27]. In this context, emotions in science teaching–learning play a fundamental role because they are closely connected with the cognitive factor. Some authors [28,29] have described the dependence between the rational and the emotional domain and argued that there is an increasingly clear relationship between the two domains [30]. Encouraging the presence of positive emotions in students during the study of a subject favors learning, while the generation of negative emotions limits it [28,30]. However, some studies [31,32] reveal that these types of activities also generate emotions such as stress and frustration, especially in items or tests of greater difficulty. The emotional factor is also related with self-efficacy in the teaching–learning processes [27]. According to Bandura [33], teachers' self-efficacy refers to a teachers' individual awareness of their effectiveness, that is, the degree of their capacity to motivate students to higher levels of performance. Some studies [34,35] found a positive correlation between self-efficacy and academic success, such that a strong sense of self-efficacy predicts increased performance and success, which then become grounds for greater self-efficacy. In particular, Kazempour [36], in a study of future teachers, points out that understanding the negative

experiences and difficulties with science for pre-service teachers (PSTs)allows one to better understand their low self-efficacy and negative attitude towards science teaching, as well as the interrelation between beliefs, self-efficacy and attitude towards science if lasting changes that lead to an improvement in their teaching practices are wanted [36–38]. In addition, PSTs' attitudes are important, influencing students' motivation to choose STEM careers and influencing cognitive-affective variables such as self-efficacy, attitudes, motivation and beliefs [37,38]. It is important to generate positive emotions in the teachers in training so that they feel competent and trained to teach STEM, since it depends on them to generate interest in STEM for future generations [39]. Therefore, it is necessary to have a specific study to have a balance the aforementioned as gamification application to improve PSTs' multidimensional domain in a flipped STEM course.

This research aims to analyze the effects of the multidimensional domain of PSTs produced by the online-based Edu-Escape Room used as an educative tool to teach content related to the Universe in a course at the Primary Education degree of the University of Extremadura (Spain). This study focuses on the design, development and subsequent analysis of the online-based Edu-Escape Room. Specifically, we analyzed the intervention affects the attitudes, self-efficacy and emotions of the PSTs. Here, an effect size (ES) analysis was used to measure the magnitude of the activity effect, and it was checked whether they followed any distribution pattern. To complete the study, a principal component analysis (PCA) and the structural equation modeling, partial least squares (SEM-PLS), were conducted with the objective to interpret the variables with significant differences and determine how these variables correlated.

#### **2. Materials and Methods**

In this research, an online-based Edu-Escape Room of contents related to the Universe applied into a flipped class as a teaching tool is analyzed. A comparative study was carried out to analyze attitude, self-efficacy and emotion of PSTs as a multidimensional domain through a questionnaire given before and after the proposed intervention (preand post-test).

#### *2.1. Sample and Instruments*

The study was conducted with 42 PSTs enrolled in a science course called "Knowledge of the natural environment in primary education", which is taught using a flippedclassroom teaching methodology. The course was taught in the senior course of the Primary Education degree (University of Extremadura, Cáceres, Spain) during the first semester of the 2020/2021 course. No constraints were imposed, and the PSTs freely chose whether they wanted to participate in this research.

The PSTs who participated in the study had a mean age of 23 years old, with 90% in the 15–20 age range and the remaining 10% equally distributed between ages 26–30 and over 40. The PSTs had a mean university entrance grade of 7.46 (scale 1 to 10). It is important to note that the PSTs enrolled in this program had mostly studied Humanities or Social Sciences during their pre-university education (69.05% of the participants). Figure 1 provides more complete demographic information about the sample.

Regarding the instruments used in the study, the collection of data was carried out using a questionnaire with two sections in order to measure the different variables studied: attitude and self-efficacy, and emotions towards science and the proposed intervention. This questionnaire was based on quantitative items that could be measured by a fivepoint Likert scale from 1 to 5. There were a total of 34 items in the questionnaire, i.e., 24 items regarding attitude (Q1 to Q7 correspond to AF\_1 to AF\_7) and self-efficacy (Q8 to Q24 correspond to ATT\_1 to ATT\_17) and 10 items (5 positive as E\_1 to E\_5 and 5 negative emotions as E\_6 to E\_10) regarding emotions towards science and the intervention proposed (see Table 1).

**Figure 1.** Demographic information about gender, pre-university background and pre-university access of the sample.

First, a 24-item questionnaire was used to measure the attitude and self-efficacy towards science and the proposed intervention, online-based Edu-Escape Room, of the participants. Attitudes towards science and intervention proposed by PSTs were analyzed and measured using the instrument designed by Schruba [40]. Here, the PSTs' attitudes were assessed by considering their favorable or unfavorable feelings towards science and the intervention proposed. Self-efficacy about science and the intervention proposed were measured by various items adapted from the science teaching efficacy belief instrument– preservice (STEBI-B) [41], developed to analyze the self-efficacy of instructors with regard to science and science teaching. The validity and reliability of both questionnaires (attitudes and self-efficacy) were tested and confirmed in various studies [42–44]. For both attitudes and self-efficacy towards science and the intervention proposed, a five-point Likert scale was applied, in which the lowest value was "Strongly Agree" and the highest value was "Strongly Disagreed" before and after intervention implementation (pre- and post-test).

Then, a 10-item questionnaire was used to measure the emotions of the PSTs regarding the expectation of making an online-based Edu-Escape Room about the Universe, along with other science topics, before and after its development (pre- and post-test). This questionnaire has been validated in various research published [39,45,46]. Here, the PSTs were questioned about the intensity of 10 emotions: 5 positive emotions were joy, confidence, satisfaction, enthusiasm and fun, and 5 negative emotions were concern, frustration, uncertainty, nervousness and boredom. They were selected from the emotions that the PSTs could experience in the face of a teaching-learning process according to certain classifications as positive and negative emotion groups of Bisquerra [47]. Specifically, these emotions were quantified from the value of 1 to 5 (a five-point Likert scale), corresponding to the value 1 to never have felt that emotion and the value 5 to have felt it very intensely. The value 3 is the intermediate value, and values 2 and 4 determine "a little" and "sufficient", respectively.

**Table 1.** Questionnaire description about attitude, self-efficacy and emotions. The nomenclature between parentheses refers to the type of item (ATT: attitude, AF: self-efficacy and E: emotion item).



#### *2.2. Course Context and Procedure*

The participants, PSTs, were in the same group, studying the subject in which the intervention was implemented. The course is called "Knowledge of the natural environment in primary education" and is organized into three different blocks of contents. The course contents comprised current challenges in science education and learning to teach science in primary school with different strategies and contents at the primary education stage—such as the importance of science for sustainable development and environmental issues and the role that science has to prevent it—through projects on the curriculum of the knowledge of the natural environment in primary education and resources and didactic materials. This course has a total of 150 h (6 credits). Table 2 provides specific information in the context of the course teaching plan.

Before applying the intervention proposed, the online-based Edu-Escape Room on content related to the Universe, the 42 PSTs answered the questionnaire about attitudes, self-efficacy and emotions towards science and the intervention (pre-test). Due to the COVID-19 outbreak, the activity was made in an online session through Zoom, in which the PSTs connected to a virtual meeting organized by the instructors. In this meeting by videoconference, a synchronous follow-up of the activities completed by the PSTs was developed, clues were provided and doubts were resolved. Therefore, the student– instructor interaction was closely interchanged during the activity despite not being in a

face-to-face environment. In the context of the online-based intervention proposed, the PSTs completed the Escape Room individually through their computer, tablet or mobile from their respective homes. Finally, as soon as they finished this activity session, they were dedicated to answering the questionnaire that analyzed attitudes, self-efficacy and emotions towards science and intervention proposed after completing the online-based Edu-Escape Room (post-test). All interactive educational materials were provided to the PSTs through the online University platform, Moodle, for a proper teaching and learning process.

**Table 2.** Course description about teaching plan proposed.


To motivate and involve the PSTs to actively participate in this session, the rating of this activity was based on the speed with which the participants solved the onlinebased Edu-Escape Room. Furthermore, the instructors encouraged the interaction and involvement of participants through the online session and had an active role and constant attention to the PSTs' doubts and concerns.

#### *2.3. Online-Based Edu-Escape Room Design*

The online-based Edu-Escape Room designed deals with the contents related to the Universe; specifically, the theoretical contents of the activity are based on the solar system, the Sun and its role as a source of renewable energy for sustainable development. Other general concepts such as definitions of planets, asteroids, satellites, movements of planets, order in the solar system, density, perimeter, radius and diameter were also studied.

First, the concepts of "gamification" and "Escape Room" used as a teaching tool in the class were explained to the participants. Then, when this theoretical basis was explained and understood, the online-based Edu-Escape Room started as a class activity. It consisted of five assignments that the PSTs should solve during the online session using their computers and other devices individually (see Figure 2). The online-based Edu-Escape Room was designed in such a way that solving a challenge leading to the next task, and it was not possible to advance to the next test without having solved the previous ones.

**Figure 2.** Challenges that the participants must solve to complete the online-based Edu-Escape Room.

One of the best-known scientists in the study of the Universe is Carl Sagan. As shown in Figure 2, in the context of the activity, he sent a letter to the participants through the university's online platform with all the instructions they must follow to complete the activity (1). This letter guides students to two links; the first to a digital lock (which they could not open because they did not know the password) and the second to a digital puzzle (2) that could be completed by dragging the pieces with the computer mouse. After solving this puzzle, the participants could read the question "What positions do the Sun, Earth and Moon have during the eclipse of the Moon?", and giving a correct solution to this question would unlock the next challenge on the platform. The next task was a crossword puzzle (3) about the general contents of the solar system. Some words in the crossword were marked and, by ordering them in a logical way, the participants already knew that they had to ask for a Quick Response (QR) code from the instructors in order to download the next challenge. The following task (4) consisted of the data about the planets of the solar system (composition, order, satellites and perimeter) that the PSTs should find out by searching in notes, books or on the Internet. Here, some of the words and numbers were marked, and if they were properly arranged, a mathematical operation with a three-digit number is obtained. This numerical combination should be input into the digital lock (5) to reach the final goal and complete the online-based Edu-Escape Room.

#### *2.4. Data Analysis*

A descriptive analysis was firstly carried out due to its common use as the most appropriate way to characterize, describe and draw conclusions from the data of a sample [48,49]. To measure the reliability and validity of the questionnaires used in the study, Cronbach Alpha Coefficient was applied, demonstrating that the questionnaires used were reliable, as shown in Table 3.

**Table 3.** Cronbach Alpha test results.


\* Values above 0.7 confirm that the questionnaire used is adequate.

To reach plausible conclusions about the effects of the evaluated teaching tool, the homogeneity of the sample was tested. The Shapiro-Wilk normality test was performed to establish whether the data were normally distributed or not. As data were not normally distributed, non-parametric statistical tests were performed. Here, the Mann–Whitney test was used to establish the existence of significant differences between the values of the variables under study before and after the intervention. The ES was also calculated to measure the strength of the relationships between variables. Then, the PCA was calculated to reduce the dimensionality by projecting each data point onto only the first few principal components. In these cases, the statistical software SPSS (SPSS statistics 22.0) was used. Finally, the SEM-PLS analysis was conducted in order to assess the relationship between the variables studied in this research. For this analysis, the SmartPLS software was employed.

#### **3. Results and Discussion**

The data collected through the questionnaire were statistically analyzed to obtain plausible conclusions about the multidimensional domain of PSTs. Once the reliability of the questionnaires using the Cronbach Alpha Coefficient and the non-normality of the data using the Shapiro-Wilk test had been analyzed, the results were classified according to the different statistical tests (comparative analysis of means, ES test, PCA test and SEM-PLS analysis) carried out on each of the variables studied (attitudes, self-efficacy and emotions).

#### *3.1. Comparative Analysis of Survey Questionnaires*

In order to measure the effects of the online-based Edu-Escape Room proposed on the PSTs multidimensional domain, the answers given by the participants in the 24-item questionnaire about attitude and self-efficacy (7 and 17 items, respectively) were analyzed. Figure 3 shows the comparison of means (scale 1 to 5) of the 24 items about attitude and self-efficacy before and after implementing the online-based Edu-Escape Room (pre- and post-test). It should be noted that most of the items corresponded to issues related to positive attitude and high self-efficacy towards science; a score of 5 on the scale indicated a positive attitude or high self-efficacy. Only items Q15, Q16, Q17, Q18 and Q19 were associated with issues related to a negative attitude and low self-efficacy towards science, so a value of 5 on the scale corresponded to negative attitude or low self-efficacy towards science. Although there was a slight improvement in most items, statistically significant differences were observed only in Question 11 (Q11) "I like the challenges of scientific activities" (*p*-value = 0.031), where the mean changed from 3.74 to 4.16 before and after the intervention (difference of 0.42 points).

**Figure 3.** Comparison of means before and after the intervention of items related to attitudes and self-efficacy. Highlighted items in red color with significant differences (Q11) and highlighted items in green color with negative attitudes and low self-efficacy towards science.

Concerning the emotions felt by the PSTs before and after the intervention, there was an increase in all emotions, except for confidence after the intervention. It was observed that there were significant differences in 6 of 10 emotions analyzed: 3 positive emotions (joy, satisfaction, fun) and 3 negative emotions (nervousness, frustration and concern). Among the positive and negative emotions, enthusiasm and confidence, and uncertainty and boredom, respectively, the differences that were detected were not significant. Figure 4 shows a result on the comparison of means (scale 1 to 5) of these 6 emotions (3 positives and 3 negatives) in which significant differences had been detected before and after implementing the online-based Edu-Escape Room. Particularly, there were differences of 0.43, 0.47, 0.48, 0.94, 1.73 and 0.67 points in happiness, satisfaction, fun, nervousness, frustration and concern, respectively.

**Figure 4.** Comparison of means with significant differences of emotions before and after the intervention (pre- and post-test).

#### *3.2. Effect-Size (ES) and Principal Component Analysis (PCA) Analysis*

An ES analysis measures the strength of the relationship between variables in a statistical population or an estimate of that amount based on a sample. It can refer to the value of a statistic calculated from a sample of data, the value of a parameter in a hypothetical statistical population or the equation that operationalizes how statistics or parameters lead to the value of the ES [50]. Some studies [51,52] indicate that the ES estimation is useful for determining the practical or theoretical importance of an effect, the relative contributions of factors or the power of analysis. It also provides a way to evaluate the data more thoroughly and precisely. In this research, the estimate of the ES was used to measure the magnitude of the treatment effect [53].

Cohen's *d* is defined as the difference between two means divided by a standard deviation for the data as shown in Equation (1) [53].

$$d = \frac{\overline{x}\_1 - \overline{x}\_2}{\text{s}} = \frac{\mu\_1 - \mu\_2}{\text{s}} \tag{1}$$

*S* is the pooled standard deviation and is calculated using the following Equation (2) [53]:

$$s = \sqrt{\frac{(n\_1 - 1)S\_1^2 + (n\_2 - 1)S\_2^2}{n\_1 + n\_2 - 2}}\tag{2}$$

Depending on the values obtained for the calculation of Cohen's *d*, the ES of between 0.2–0.3 could be considered "small", around 0.5–0.8 as a "medium" effect, and 0.8–1.0 as a "large" effect [53]. Later, Sawilowsky [54] added to these magnitudes of the effects "very small" when Cohen's *d* is less than 0.1, "very large" when Cohen's *d* is around 1.2 and "huge" when Cohen's *d* is around 2 points.

In this study, we calculated the ES (see Table 4) of the variables that showed significant differences between the means before and after the intervention to measure the magnitude of the online-based Edu-Escape Room. These variables consisted of item 11 (Q11) "I like the challenges of scientific activities", the positive emotions "joy", "satisfaction", "fun" and the negative emotions "nervousness", "frustration", "concern".


**Table 4.** The Effect Size (ES) value (Cohen's *d*) of the intervention of variables with significant differences (Mann-Whitney test).

As shown in Table 4, the online-based Edu-Escape Room had a medium effect on all variables with significant differences except for the variable "frustration", where it had a large effect.

The PCA was conducted with the aim to interpret all data collected. The PCA is a method used to describe a data set in terms of new correlated variables (components). The components were ordered by the amount of original variance, so the PCA is useful for reducing the dimensionality of a data set. This tool was useful to summarize large numbers of data and to determine how samples differed from each other (pre- and post-test data), which variables contribute more importantly to this difference and how variables correlated [55,56]. The PCA was done for the variables with statistically significant differences, i.e., Q11 item of attitudes and self-efficacy questionnaire, "I like the challenges of scientific activities" and the emotions "joy", "satisfaction", "fun" (positive emotions), "nervousness", "frustration", "concern". Other emotions analyzed "confidence", "enthusiasm", "uncertainty" and "boredom", which were aggregated as well but did not have much influence. Figures 5 and 6 show a PCA diagrams obtained from this analysis.

Figure 5 shows principal component 1 (PC1) and principal component 2 (PC2), which explain 62.9% and 13% of the total variance, respectively. PC1 (62.9% of the variance) can group samples in two groups. One group belongs to positive emotions and is in the positive part of PC1, while negative emotions are assembled in a second group in the negative part of PC1. No sample grouping is observed along PC2.

Figure 6 shows principal component 1 (PC1) and principal component 2 (PC2), which explain 34.4% and 30.5% of the total variance, respectively. PC1 (34.4% of the variance) can arrange the sample in two groups (pre- and post-test answers). Pre-test answers are located mainly in the negative part of PC1, while post-test answers are grouped in the positive part of PC1. No sample grouping is observed along PC2. Thus, PC1, which represents the effect of the instruction methodology in the emotions toward activity implemented of PSTs, can distinguish between the PSTs before and after the intervention, presenting more positive emotions after completing the activity.

**Figure 5.** The principal component analysis (PCA) diagram, X and Y axis show principal component 1 and principal component 2 that explain 62.9% and 13% of the total variance, respectively. Green points are positive emotions, light green point is Q11 and orange points are negative emotions.

**Figure 6.** PCA diagram, X and Y axis show principal component 1 and principal component 2 that explain 36.1% and 31.5% of the total variance, respectively. Blue points are answers to pre-test and orange points are answers to post-test. The numbers associated with each point belong to each student's answer in pre-test (1) and post-test (2).

#### *3.3. SEM-PLS Data Analysis*

In order to get a better understanding of the influence of the instruction designed on the studied variables, an SEM-PLS analysis was carried out. The model was built grouping the variables into "positive emotions" (E\_1 to E\_5), negative emotions (E\_6 to

E\_10), "self-efficacy believes" (AF\_1 to AF\_7) and "attitudes towards science" (ATT\_1 to ATT\_17). Based on the results, all positive emotions have a positive significant effect of self-efficacy beliefs (t-value: 2.105, *p* = 0.035), while negative emotions showed a negative significant effect with self-efficacy (t-value: 2.697, *p* = 0.007). According to the designed model, we also observed that there was a strong significant effect between attitudes and self-efficacy towardss science (t-value: 18.017, *p* = 0.000). Finally, the loads of each item were close to or greater than the recommended value (>0.700). Considering such values, the reliability, and validity of the constructs can be granted [57]. Results are summarized in Figure 7.

**Figure 7.** Results of the relation between the studied variables after applying the structural equation modeling partial least squares (SEM-PLS) analysis. t-values are shown for the inner and outer model. Paths have been highlighted based on relative values.

#### *3.4. Discussions*

Comparison of the results revealed significant differences about the multidimensional domain of the PSTs. Statistically significant differences were observed in the Q11 "I like the challenges of scientific activities" of the attitude and self-efficacy questionnaire, which showed an increase of more than 0.4 points compared to the pre-test. Maintaining

positive attitudes and fostering high self-efficacy towards the online-based Edu-Escape Room of science was associated with higher performance and motivation in the contents covered [58,59]. Sequentially, an active teaching methodology such as a virtual Escape Room could lead to increased attitudes and self-efficacy of participants [14,30], which was especially useful in STEM courses.

The emotions analyzed showed statistically significant differences in 6 of 10 emotions analyzed: 3 positive emotions—joy, satisfaction and fun—and 3 negative emotions nervousness, frustration and concern. All these emotions had an increase after the proposed activity. Concerning positive emotions, positive emotional states were encouraged while applying active instructional methodology such as an Escape Room game and flipped classroom [2,27,29,39]. This was especially important in STEM disciplines as students began to show a deficiency of interest in science at an early age, which might impact the adulthood and decreased the choice of STEM careers later in life [30,60]. The increase in negative emotions could be due to a multitude of factors. One of these was the pre-university background of PSTs; most of the sample (69%) had studied Humanities or Social Sciences in High School. This component was associated with lower motivation and the presence of an unfavorable image and negative emotions towards science in general [29,61]. The difficulty of developing the proposed intervention in an online environment might be another factor. Online teaching had various disadvantages such as the feeling of isolation, excessive self-management or the absence of face-to-face interaction between instructors and students. Moreover, it could affect factors such as the student's capacity for independence, the difficulty of follow-up, coordination and communication, and the difficulty of the process in comparison with traditional teaching. This was essential in practical scientific activities that required synchronous follow-up like the proposed activity in this study [59,62].

According to the ES analysis results, it was observed that the online-based Edu-Escape Room has a medium effect on all variables with significant differences except for the variable "frustration". This type of effect was within the acceptable range of ES, and there were some factors such as the heterogeneity of the sample that could decrease the effects of the intervention [63,64]. The online-based Edu-Escape Room had a large effect on the negative emotion "frustration". According to another research study [9,31], when doing an Escape Room, it was important to be aware that it was normal that some negative emotions might appear such as frustration or nervousness due to game components. Frustration might arise for a variety of reasons, such as the difficulty of a challenge, the time taken to solve it or the participant's lack of knowledge. According to some studies, the learning processes that generated knowledge were likely to produce emotions such as frustration, uncertainty, anxiety and boredom [65–67]. However, given the complexity of interactions between emotions and variables, emotions such as frustration, anxiety and shame were potential activating elements of the learning process [67].

Regarding the PCA, the variables were arranged into two groups: positive variables (Q11 and positive emotions) and negative variables (negative emotions). Concerning questionnaire answers, there were two groups: pre-test answers and post-test answers. It was observed that pre-test answers were located mainly in the negative part of the x-axis as well as negative variables, and post-test answers were grouped in the positive part of x-axis of the diagram as well as positive variables. Thus, it could students before and after the intervention could be distinguished. There was a positive correlation between the Q11 and positive emotions with the post-test, and there was a correlation between negative emotions and post-test. The results were consistent with other studies [44,56,59]. This information completed the results of the means comparison analysis and the ES analysis. The PCA results confirmed that there was a dependency correlation between the results of the Q11, positive emotions and post-test and between the results of the pre-test and negative emotions. Thus, the proposed online-based Edu-Escape Room could be effective for improving the multidimensional domain of PSTs.

Finally, the SEM-PLS analysis allowed us to determine the effects of the studied variables after applying the online-based Edu-Escape Room developed as an educative tool to teach STEM contents in the University course of the bachelor's degree in Primary Education. According to the results obtained, there PSTs' emotions played a significant effect on the PSTs' self-efficacy beliefs, and these beliefs also have a positive effect on their attitudes towards science.

#### **4. Conclusions**

The study analyses the effects of multidimensional domain of PSTs produced by an online-based Edu-Escape Room developed as an educative tool in the University course within the bachelor's degree in Primary Education at a Teacher Training College (Spain).

According to the results, there is an increase in positive attitudes and high self-efficacy items after the intervention. In particular, Q11, "I like the challenges of scientific activities", had a significant difference and its value increased considerably. Regarding the emotions analyzed, there were significant differences in three positive emotions, "joy", "satisfaction" and "fun" and three negative emotions, "nervousness", "frustration" and "concern". Specifically, there was an increase in these emotions after implementing the online-based Edu-Escape Room. Here, positive emotions were generated due to Escape Room games used as an educative tool, which promoted motivation, interest and fun. Negative emotions were mostly generated because of the difficulties of an online environment. The online-based Edu-Escape Room had a medium effect for items with significant differences except for the emotion "frustration", where the activity had a large effect. Here, negative emotions such as frustration were frequently viewed as detrimental to motivation and learning, but they were also interpreted under some circumstances as beneficial in a learning process. It was also observed that there was a correlation between some items of attitudes, self-efficacy and positive emotions and the post-test, which indicated that there were benefits in the multidimensional domain of the PSTs after having implemented the online-based Edu-Escape Room. Thus, this finding was supported by the SEM-PLS analysis, which showed the tool's effect that the PSTs' emotions influenced to a significant effect of the positive self-efficacy and attitudes towards science. Hence, this tool could be utilized in future works by adding improvements to achieve better results and greater benefits in the multidimensional domain of PSTs.

**Author Contributions:** Conceptualization, F.Y.-P., J.S.J. and D.G.-G.; methodology F.Y.-P., J.S.J. and D.G.-G.; software, F.Y.-P., J.S.J. and D.G.-G.; validation, F.Y.-P., J.S.J. and D.G.-G.; formal analysis, F.Y.-P., J.S.J. and D.G.-G.; investigation, F.Y.-P., J.S.J. and D.G.-G.; resources, F.Y.-P., J.S.J. and D.G.-G.; data curation, F.Y.-P., J.S.J. and D.G.-G.; writing—original draft preparation, F.Y.-P., J.S.J. and D.G.-G.; writing—review and editing, F.Y.-P., J.S.J. and D.G.-G.; visualization, F.Y.-P., J.S.J. and D.G.-G.; supervision, J.S.J. and D.G.-G.; project administration, J.S.J. and D.G.-G. All authors have read and agreed to the published version of the manuscript.

**Funding:** The authors gratefully acknowledge the Consejerería de Economía, Ciencia y Agenda Digital de la Junta de Extremadura y Fondo Europeo de Desarrollo Regional (FEDER)—Project IB18004, which support this research possible.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by Ethics Committee of University of Extremadura (ref. 94/2018).

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data available on request.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Are Cross-Border Classes Feasible for Students to Collaborate in the Analysis of Energy Efficiency Strategies for Socioeconomic Development While Keeping CO2 Concentration Controlled?**

**Roberto Araya 1,\* and Pedro Collanqui 2,3**


**Abstract:** Education is critical for improving energy efficiency and reducing CO2 concentration, but collaboration between countries is also critical. It is a global problem in which we cannot isolate ourselves. Our students must learn to collaborate in seeking solutions together with others from other countries. Thus, the research question of this study is whether interactive cross-border science classes with energy experiments are feasible and can increase awareness of energy efficiency among middle school students. We designed and tested an interactive cross-border class between Chilean and Peruvian eighth-grade classes. The classes were synchronously connected and all students did experiments and answered open-ended questions on an online platform. Some of the questions were designed to check conceptual understanding whereas others asked for suggestions of how to develop their economies while keeping CO2 air concentration at acceptable levels. In real time, the teacher reviewed the students' written answers and the concept maps that were automatically generated based on their responses. Students peer-reviewed their classmates' suggestions. This is part of an Asia-Pacific Economic Cooperation (APEC) Science Technology Engineering Mathematics (STEM) education project on energy efficiency using APEC databases. We found high levels of student engagement, where students discussed not only the cross-cutting nature of energy, but also its relation to socioeconomic development and CO2 emissions, and the need to work together to improve energy efficiency. In conclusion, interactive cross-border science classes are a feasible educational alternative, with potential as a scalable public policy strategy for improving awareness of energy efficiency among the population.

**Keywords:** STEM education; energy efficiency; CO2 emissions; APEC databases; cross-border classes; sustainable development

#### **1. Introduction**

Education is critical for improving efficiency in the use of energy resources [1–3]. However, there is a perception among the population that reducing energy consumption has very little impact [4]. Early education could therefore be one option to help overcome the barriers to energy-saving behaviors. Indeed, it is widely acknowledged as being one of the key social tipping elements for achieving rapid global decarbonization and climate change mitigation [5]. Moreover, several multinational organizations have been working on different educational interventions. For example, the APEC Energy Working Group (EWG) has been discussing energy efficiency since 1996. This group considers that enlightening younger generations on the importance of energy throughout elementary and secondary education is vital [6]. The Organisation for Economic Co-operation and Development (OECD) Learning Framework 2030 provides a vision and certain underpinning principles for the future of education. The first challenge proposed by this framework is an

Cross-Border Classes Feasible for Students to Collaborate in the Analysis of Energy Efficiency Strategies for Socioeconomic Development While Keeping CO2 Concentration Controlled? *Sustainability* **2021**, *13*, 1584. https://doi.org/ 10.3390/su13031584

**Citation:** Araya, R.; Collanqui, P. Are

Academic Editor: Pedro Guilherme Rocha dos Reis Received: 28 December 2020 Accepted: 29 January 2021 Published: 2 February 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

environmental one: Climate change and the depletion of natural resources require urgent action and adaptation [7]. There are, however, several challenges with including energy efficiency in early education. Effective energy efficiency education requires the integration of different school subjects. However, truly integrating science, technology, engineering, and mathematics (STEM) is a significant challenge [8]. Furthermore, if we add socioeconomic development into the mix then the level of challenge is increased. Teachers are not used to such integration [9]. They typically teach isolated subject matters, perhaps because they themselves were trained compartmentally, with large walls dividing different university departments. In addition to this barrier, the goal of including real-world problems is even more challenging. Although teachers receive training from the mathematics, science, and education departments, they never receive any training from the engineering department. Therefore, their training is more heavily focused on understanding nature, and not on building solutions. A further challenge that is posed is the idea of cooperative learning and working in teams in order to build energy-efficient solutions. In most classes, students either listen to the teacher or work on their own, even when cooperative learning has been empirically proven to produce positive learning effects. "Despite the very robust evidence base of positive outcomes, co-operative learning 'remains at the edge of school policy'" [10]. Isolation is a deeply rooted classroom practice that is very difficult to change since teachers themselves work alone with almost no interaction with their peers. Cooperation is also difficult as schools are composed of completely independent units, i.e., isolated classrooms [11]. Moreover, an additional challenge is having students from different countries learn together and share their research, as well as explaining their understanding and their solutions. This is more demanding than simply having cross-border curricula [12]. It also requires the development of collaborative skills across countries. In this sense, students need to learn to work with teams from around the world; particularly when it comes to decarbonization strategies, where regions and countries cannot work in isolation [6].

In this article, we propose and test the feasibility of the cross-border approach for dealing with these challenges. The research question of this study is whether interactive cross-border science classes with energy experiments are feasible and can increase awareness of energy efficiency among middle school students.

#### **2. Theoretical Framework**

The proposed approach is based on previous experiences with cross-border public classes [13,14]. After receiving extensive feedback from the APEC Tsukuba community [15], an improved version was implemented. This version included content more geared towards the socioeconomic development of developing countries, as well as better use of technology to reach sustainable development. Given that [16–18] showed that technology can have a significant impact on learning, we include some promising online features. However, online courses can be controversial. In higher education, the completion rate for MOOCs is very low and has not improved, despite six years of investment in course development and learning research [19]. According to [20] (pp. 14,900) "online education provides unprecedented access to learning opportunities, as evidenced by its role during the 2020 coronavirus pandemic, but adequately supporting diverse students will require more than a light-touch intervention." For example, online forums can build a sense of classroom community and aid learning [21], but students do not always use these tools. On the other hand, hands-on activities for early education have proven to produce slight gains in conceptual and procedural knowledge in early energy education [1]. This is despite the fact that some misconceptions regarding energy saving and carbon-emission reductions are not easily overcome, such as foliage around the house not being simply for the sake of beautification, and grasses planted on the roof of the building not being meant only for decoration [1]. The aim of this paper is to find an adequate blend of online education with face-to-face and hands-on teaching in order to achieve good levels of student understanding. Our aim is to help them find solutions that will allow for socioeconomic development

while improving energy efficiency and controlling CO2 concentrations—that is, to achieve balanced economic growth that ensures environmental protection.

The approach reported in this paper has several novel contributions. The first contribution is the introduction of an innovative new type of lesson that integrates two classes simultaneously. These are known as cross-border public classes, where two classes from different countries connect online, attend a lesson synchronously, and share their learning experience and points of view. This interconnection between classes is very important for facilitating imitation and sharing ideas, therefore helping scale the solution to national and regional levels. However, this new type of lesson requires a different didactic design and a more demanding technical infrastructure. From the designers' and teachers' point of view, the lesson has to consider that there are two classes, and that at least one of them is unfamiliar to the teacher. Furthermore, they also have to consider that the two classes do not know each other. Additionally, the didactic strategy has to help establish a fruitful communication between students from two classes that have never met each other before. These are students from different cultures who live thousands of miles apart. In this particular setting, obtaining a dialogic teaching structure, where all students participate actively, is even more challenging. The goal is not to have the students passively watch a lecturer, as is the case with most online courses [19]. Instead, the goal is to have the students receive guided inquiry instructions, where they can explore possible answers, conduct experiments, make predictions, come up with explanations, get to know their classmates from a different culture, and share their results and points of view with them, as well as peer review their classmates' solutions and explanations. One way of achieving this goal is to use ConectaIdeas, a special cloud-based platform that includes a support system for teachers [16]. This is an online platform that we have adapted for cross-border public classes. All of the students connect to the platform synchronously and answer openended questions on computers, tablets, or smartphones. The ConectaIdeas teacher support system helps the teacher analyze the students' responses in real time. The system also summarizes the answers and generates world clouds and concept maps in real time. These are directed graphs that link the keywords used most frequently by the students. These written responses and maps complement the verbal responses given by a small sample of students. A major challenge for the teacher is to maintain engagement in both classrooms at the same time, as well as being able to listen to the different students' reactions in order to adjust the lesson in real time.

A second contribution of this study is the work that is done to address a real-world problem using APEC databases to learn about energy consumption and energy sources, as well as their trends over the last 40 years. Furthermore, the lesson promotes analysis of the relationship between energy consumption and economic development, as well as its environmental implications. APEC and other institutions have gathered a complete set of statistics on this issue. This is a very powerful source of information, which is constantly being updated. This provides lesson designers and teachers with a unique opportunity to design lessons that are connected to real-world problems and situations that are relevant to students from different countries. In this sense, comparisons between economies are facilitated enormously. It is an invaluable source of material and a powerful mechanism for engaging students.

However, given the age and interests of middle school students, it is important to connect the issue to their everyday lives and provide concrete examples that are familiar to them [2,22]. A third contribution of this study is therefore the use of simple physical models to understand the connection between energy, socioeconomic development, and CO2 concentration, as well as strategies for controlling the environmental implications of increased levels of energy consumption. CO2 is a complex compound that cannot be seen directly. It is invisible; it cannot be heard nor touched. In order to be able to measure CO2 concentration in the air, certain instruments are required. Although the instruments themselves are not expensive, they are not readily available in schools. However, there are some simple experiments that can help visualize CO2. Nevertheless, these experiments take

up precious time and involve complex coordination if all of the students are to participate actively. This coordination is made all the more difficult when there are two or more classes that are not located in the same room. In this lesson, we use a very simple physical model. It is quick and easy and requires no materials at all. Students just have to breathe in a very small space. In this case, we suggest the students cover their mouths and noses with their hands and breathe in through the small gap between their hands. The teacher just has to make sure the hands insulate the air coming from the nose and do not leave any other gaps between their fingers. After a few seconds, students can feel an increase in the heat and humidity. They can also smell the odor of their breath and experience difficulties breathing with less O2 and more CO2. Furthermore, with some physical activity, such as jumping, CO2 concentration increases more rapidly and is immediately detected by the students in a small, closed space. This includes more heat and humidity, higher levels of CO2 and therefore increased difficulty when breathing. This difficulty to breathe creates a very direct connection to the environmental problem that must be solved urgently. Moreover, the direct experience with CO2 ensures that the activity is emotionally engaging and interesting for the students. They can easily relate it to everyday experiences, such as doing physical activity in a crowded classroom or a gym full of people [23,24]. All of the students can participate and give explanations as to what is going on. The physical activity is a good model of the economic activity of a country, where more CO2 emissions are produced as more energy is consumed. The lesson design also includes thinking about what happens in a bedroom, which is used as a model for the earth. In this sense, students are invited to think about a bedroom after a full night's sleep in order to model the increase of CO2 concentration in the earth's air. The teacher can then dramatize the need to find a solution to the increase in CO2 concentrations, highlighting the fact that, unlike the bedroom, there are no windows we can open on earth to solve the problem. By doing so, the teacher can show that another kind of solution is required.

A fourth contribution of the study is the classroom observation strategy that is adopted. In standard public classes, teachers and didactic experts observe the lesson. Then a panel analyzes and comments on the lesson. In this project, observers use SmartSpeech [25], a smartphone app that allows them to record events based on a rubric and make comments. After the class, observers upload their observations and can view them on a web-based graphical interface. Using the same app, the teacher records their speech and then uploads it to the cloud. After selecting some keywords, an algorithm counts the words and the connections between them and generates a concept map. This is a directed graph, where the main concepts are located on the nodes and where arrows connect the nodes based on the frequency with which pairs of concepts appear in the same paragraph. Additionally, the direction of the arrows reflects the order in which the concepts appeared. Efficient classroom observation is critical for measuring, sharing, and giving feedback, therefore allowing efficient pedagogical practices to be scaled [26–28].

This experience builds on previous cross-border educational experiences using Information and Communications Technology (ICT) support. Cross-border public classes and lesson study are an innovative form of teacher education and teacher professional development. For example, [29,30] illustrated a cross-border lesson study using an internet-based bulletin board system (BBS) between pairs of Japanese and Australian schools. He reported a significant cultural impact on students, awakening their cultural interest in the content and developing their hermeneutic or subjective attitude towards collaboration. There are several new ICT-based features used in the APEC project reported in this paper. The lesson features a live video transmission and augmented synchronicity, with the students jumping and answering questions live and online. Cross-border public classes and lesson study are also an additional development of massive team games across classrooms [31–34]. In these activities, dozens of schools play synchronized games with one or two teams per classroom, with each team comprising approximately 10 students. Another new addition with the APEC project is the cross-border design and the lesson study cycle that is collectively developed over 20 months by a multinational team of researchers and teachers. The cross-border lesson reported here is a further development of a previous cross-border lesson on energy efficiency [13,14]. Having received feedback from several teachers and researchers, this latest version of the lesson now includes different activities for teaching socioeconomic development and its environmental implications.

#### **3. Materials and Methods**

The lesson was designed with several educational goals on energy efficiency and sustainability in mind. It was also designed to establish connections with the goals of the math and science curricula. The lesson plan prescribed a duration of one hour as shown in Table A1.

Firstly, students should realize that there is a need to compare countries using per capita estimates and not only total amounts. This means that students should realize that the size of the population for each country should be considered when looking at their CO2 emissions. A larger population means more emissions. Therefore, the size of the population is critical. During the lesson, the teacher should help students discover this critical factor. During the implementation of this lesson, the students managed to detect this variable. For example, they suggested agreeing on a sample size and then taking a sample of the population of that fixed size in each country. The CO2 emissions for those samples could then be calculated and compared.

Secondly, students should also consider population growth. This is also a critical factor when studying trends in CO2 emissions over the last 50 years. The teacher should conduct the lesson so that students discover that a proportion of the growth in CO2 emissions may be due to population growth. They should therefore look for ways to discount this.

Thirdly, students should sense CO2 concentration by themselves and connect it to their everyday lives. The teacher should help them take very basic measurements by having them breathe in a very small, closed space. After a while, students should sense a change in the temperature and humidity, as well as experience greater difficulties when breathing.

A fourth goal was to learn to use modeling [8,35]. This is a critical skill. It is one of the main goals mentioned in the US Next Generation Science Standards, as well as in the Common Core Standards for Mathematics. It is a component of core science and engineering practices. In this lesson, the teacher helps the students model the level of economic activity of a country using the rhythm of their jumping, while also modeling the effects of socioeconomic development on CO2 concentration by having them consider the quality of the air in their hands while jumping. The students should realize that jumping increases the temperature, humidity, and CO2 concentration. They should notice the rise in CO2 concentration based on the increased difficulty of breathing. Another modeling activity was to consider their bedroom as a model of the earth. They have to imagine the air quality and CO2 concentration in their bedroom after a full night's sleep with closed doors and windows. The teacher then asks for possible strategies for cleaning the air, before asking whether these strategies could also be applied for cleaning the air of the whole earth.

A fifth goal was to have students think about the environmental consequences of developing countries reaching a quality of life similar to that of a developed country. Students should give a rough estimate of energy consumption per capita, total energy, and total CO2 emissions if all countries were to reach the levels of socioeconomic development of a developed economy. Students should then realize the need to produce and consume energy more efficiently. They have to realize that the entire world wants to live like they do in developed countries. An increase in efficiency is therefore a must in order to reach sustainable development.

A sixth goal was that students learn to develop critical thinking [36] and be able to collaborate, share explanations and predictions, inquire, understand other students' points of view, take turns, summarize, write explanations, and do peer review with classmates from other countries. This is a significant challenge. According to the outstanding educational researcher L. Cuban [37], "writing about inquiry is far easier than practicing it in

classroom lessons." In a recent study, educational expert D. Willingham [38] concludes, "it is no surprise that programs in school meant to teach general critical thinking skills have had limited success."

Following traditional Japanese lesson studies [39], the cross-border team came to an agreement and produced a detailed lesson plan. The observers in both countries also received the lesson plan in order to track and record the class events. The lesson plan is shown in Table A1, which is located in the Appendix A. It has three columns: one with the time (from 0 to 60 min), another with the teacher's actions, and a third column with a prediction of the students' reactions.

Two sessions were held (Figure 1). The first session was taught in Lima, Peru, with a class from Santiago, Chile, also connected. The second session was taught in Santiago, with a class from Lima also connected. The first session was a first pilot whose objective was to test the idea of a cross-border class and to test the planning of the lesson. In the second session the students went from other schools. In this work we report the second session. A total of 33 eighth grade students participated, 20 from Chile and 13 from Peru. 21 were female and 12 were male. Median age was 14 years old.

**Figure 1.** (**a**) Cross-border public class taught in Lima. In the classroom, Peruvian students are paying attention to the teacher. The Chilean class is connected by Skype and projected on the screen, along with the answers submitted on the ConectaIdeas platform by the Peruvian and Chilean students. (**b**) Cross-border public class taught in Santiago. The Chilean students answer on tablets, while the Peruvian students answer on computers (not shown).

After the teacher had the students present themselves, he then introduced the goal of learning about energy production and consumption, as well as the environmental implications. He introduced the notion of air quality and CO2 concentration. At this point, the teacher introduced a first model. He made the students breathe in with their nose between their two hands, producing a small, closed volume of air surrounding the nose. The teacher asked each student to take 10 breaths and then asked what they sensed (Figure 2).

**Figure 2.** Peruvian students breathing through their nose into the small space between their hands.

The students answered that there was an increase in temperature and humidity, as well as a decrease in the air quality. The teacher then told them that this was due to an

increase in the concentration of CO2 in the small volume of air surrounding their noses. He then showed Figure 3a and read question 1: *In which country have CO2 emissions grown faster, and what might the cause have been? Explain your answer.* The teacher mentioned that the graphs corresponded to real data from Chile and Peru, but that he would not reveal which country was which. This is a didactic strategy to capture interest, but also a strategy to avoid comparisons of superiority between the countries. The teacher also mentioned that energy is measured in a precise and standard way, but that the graphs did not include the name of the units of energy. These students most likely did not know the exact units for measuring energy. The didactic strategy was therefore to concentrate on the socioeconomic and environmental trends rather than on concepts from physics related to energy. The teacher made sure the students understood the graph by asking for the meaning of the axes and asking for the CO2 emissions for both countries at different dates. The students submitted their answers on their computers and the teacher received them in real time. The teacher then asked how the two countries should be compared. Following a discussion, the students realized that the size of the population was an important factor for explaining the amount of CO2 emissions. They then suggested using equal sized samples of the population for both countries.

The teacher then showed a second slide with Figure 3b. The graph plots trends in per capita CO2 emissions for both countries. On the platform, the teacher then asked question 2: *In which country did per capita CO2 emissions grow faster, and what might the cause have been? Explain how you came to your conclusion*.

**Figure 3.** (**a**) CO2 emissions and (**b**) CO2 emissions per million people from 1980 to 2014 in both countries, and the corresponding question 1 and question 2 that students had to answer on their devices.

The students answered on their devices. After reviewing and commenting on the answers they received, the teacher then suggested that the two graphs could be used to calculate the population of both countries. This was a great opportunity to test the notion of division where the denominator is a rate. The teacher therefore asked question 3 on the platform: *Based on the previous graphs, in which country has the population grown faster? Explain in your own words*. The results can be easily interpreted by students since at that age they know the size of the population of their country. With this information they can work out which country is which. Some of the students realized this and mentioned it in their written responses.

The teacher then introduced the relationship between energy consumption and CO2 emissions. First, he showed the graph on the left in Figure 4a and asked question 4: *In which country has energy consumption per person grown faster, and what might the cause have been? Explain your answer.* The teacher and students also commented on the trends in total CO2 emissions. The teacher then suggested that the students jump 10 times while holding their hands around their noses and sensing what happened. The students realized that with more energy (jumps) the temperature, humidity, and CO2 concentration increased more quickly. By doing so they could relate energy consumption and socioeconomic development with CO2 concentration.

The teacher then showed the graph from Figure 4b and asked question 5: *Based on the previous graphs, compare the CO2 emissions per unit of energy consumed. Explain your reasoning.* The plan was for the students to realize that it was an almost constant rate, similar for both countries, and that they have not changed much in the last 50 years. However, given time constraints this important topic was not analyzed in depth.

**Figure 4.** (**a**) Total energy consumed/million people and the corresponding question 3. (**b**) CO2 emission/energy consumed and question 4 that the students had to respond to in writing on their computers or notebooks.

Next, the teacher asked the students to analyze Figure 5a [40]. This is a trend from the last 16,000 years. The students realized that there was a significant change. The teacher asked for the source of this change: What are the sources of energy and energy consumed by hunter gatherers, and also during the Roman Empire, why did it then decrease during the middle ages, and why has it skyrocketed in recent years? The students referred to electrical devices, transportation, and heating. The teacher then asked them to analyze buildings, streets, highways, and bridges, as monuments of captured energy. After that, the teacher showed Figure 5b [41]. The students realized it was another time scale, and then commented on the peak in CO2 concentrations. They analyzed the source of this peak and its relation with the peak in energy consumed per capita.

**Figure 5.** Graphs and the corresponding questions: (**a**) question 3, (**b**) question 4. Students had to respond in writing on their computers or notebooks. (**a**) is adapted from [39]. (**b**) is adapted from [40].

The teacher then introduced the bedroom model. He asked the students to imagine what happens to the air quality after a night of people sleeping with the door and windows closed. The students answered that there would be an increase in temperature and CO2 levels. The teacher asked for the solution. The students suggested opening the windows. The teacher then asked what would happen if there were more people in the bedroom, doing physical activity like a workout in a gym. Students answered that CO2 concentrations would be very high. The solution again was to open the windows. The teacher then asked if this solution could be applied to Earth. The students realized that there are no windows in this case and that another type of solution would be needed. The teacher then asked question 6 on the platform: *If every person in the world consumed as much energy as people*

*from developed countries, then how much would CO2 emissions increase, what environmental implications would this have, and how do you suggest we solve it?*

#### **4. Results**

How do we measure the impact on 21st-century skills? Some of the data we obtained through the ConectaIdeas platform can help us measure the students' reasoning, writing, communication, and collaboration skills. For example, the platform can tell us the percentage of students who answered open-ended questions, the average length of their responses, the meaning of the their responses based on the frequency with which they mention core concepts and the connections they establish between said concepts, the percentage of students who wrote a review of another student's response, the percentage of students who received feedback, and the percentage of students who included words related to the learning goals for the lesson (i.e., normalize populations, use of models, making predictions, making suggestions, etc.).

The cloud-based platform received and summarized the written responses to the open-ended questions that were asked during the cross-border public class. For each open-ended question that was asked on the platform, the teacher received the answers from all of the students and the corresponding directed graphs with a network of the main words used by the students. A total of 29 students answered the first question, 27 students answered the second question, 28 answered the third question, 29 students answered the fourth question, and 30 students did peer review.

For the first question—*In which country have CO2 emissions grown faster, and what might the cause have been?*—all of the students recognized that the blue country had grown faster. They correctly interpreted the graph. This is illustrated by the concept map in Figure 6. Some of the answers submitted by the students included:


As can be seen from the answers above, the students gave explanations related to industrialization, transportation, vegetation, and population. These answers and explanations were in line with the predictions made in the lesson plan. These predictions can be found in the right-hand column of the lesson plan, titled "Predicted reaction of the students." The prediction was: From the graph, students will identify in which country CO2 emissions have grown faster. They also proposed different causes, such as population growth and industry. However, it is interesting to note that only one student identified that the blue country was Chile.

**Figure 6.** Auto-generated directed graph that summarizes students' answers to the first open-ended question. The main nodes are "blue," "country," "more," and "CO2" The map basically suggests that the blue country has more CO2.

However, the written explanations given by the students did not include any mention of development or economic growth. They also failed to mention the growth in transportation, industry, and population in the 34 years between 1980 and 2014. All of their explanations used only static variables. The lesson plan did not highlight this potential issue. During the session, the teacher did not pay any attention to this issue either. This important issue was only detected once the lessons had been taught, and following an in-depth review of the written responses.

The teacher asked the students how they could compare the two countries, especially given the difference in the size of their populations. After some discussion, a couple of students suggested taking the same size sample for both countries. This then led the teacher to agree on a sample of one million people. However, it was not clear whether all of the students clearly understood the meaning of this step. Following this, the teacher then asked the second question: *In which country did per capita CO2 emissions grow faster, and what might the cause have been? Explain how you came to your conclusion*. Some of the written responses included:


The lesson plan made two predictions regarding the students' reactions. One prediction was that students would look for ways to compare, for example, using a sample, and then understand what CO2 per capita is and how it grows. The second prediction was that students would realize that in one of the two countries the per capita emission was fixed, so the total increase in CO2 could be due to population growth. However, per capita CO2 emissions were increasing in the other country. Most of the answers seemed to agree with these predictions. However, the second answer revealed that this student still did not understand that the information on the graph was per capita or per million people. On the other hand, it is interesting to see how one student gave a more dynamic explanation (the last of the answers shown above). This response mentions increased

electricity consumption over the "past few years." It was also this student who successfully identified the country as being Chile.

The plan was then to ask question 3 on the platform: *Based on the previous graphs, in which country has the population grown faster? Explain in your own words*. However, this question was only asked verbally. Instead, the teacher asked question 4 using the platform: *In which country has energy consumption per person grown faster, and what might the cause have been? Explain your answer.* Some of the answers included:


All of these responses used static variables, not variations, except for the fourth one. The third answer shows that this student still did not understand that the population sizes had been normalized.

The next question from the lesson plan was only asked verbally. The next question asked on the platform was therefore, *If every person in the world consumed as much energy as people from developed countries, then how much would CO2 emissions increase, what environmental implications would this have, and how do you suggest we solve it? Explain your reasoning and peer review your classmates' responses.* Some of the students' responses included:


All of the students realized that CO2 emissions would increase. However, they produced different estimations. Some of them just said that there would be more CO2 emissions. Other students estimated an increase by a factor of three or four, but they did not give any justification for this estimation. In fact, it would appear that these estimations were based on what the teacher said in class. He showed the left-hand graph from Figure 6 and commented that energy consumption per capita in Chile was about a third of the USA, whereas in Peru it was about a quarter. This suggests that there may have been a process of proportional reasoning. Other students suggested an increase by a factor of five, but again gave no justification. Other students proposed a factor of 100, while several others predicted that CO2 emissions would be a million times higher. Some students even suggested that CO2 emissions would "explode." None of the students gave any justification for their estimation. The teacher identified this diversity in the students' responses and tried to get them to think about their estimations. With more time, the students would probably have engaged in some form of proportional reasoning, explained their assumptions, and justified their estimations. This is something that should be considered for future versions of this lesson.

In addition to the ConectaIdeas platform, the SmartSpeech observation app [25] was also used by some of the teachers who observed the cross-border public class. Observers followed an observation protocol to select different options based on what they saw in the class, as well as writing their comments. Most observers selected the Classroom Observation Protocol for Undergraduate STEM (COPUS) rubric [42], proposed for STEM lessons by Physics Nobel Prize winner Carl Wieman and his team. The distribution of events observed is shown in the pie chart in Figure 7. Most of the time was spent lecturing (45%) and asking open-ended questions (43.2%). There are also observations relating to other events, such as the teacher moving among groups of students, conducting experiments, and waiting.

**Figure 7.** Auto-generated directed graph of the teachers' actions during the lesson. The events were marked by observers using the SmartSpeech app on their smartphone. The directed graph is a concept map summarizing the observers' selection of events according to a specific rubric, in this case the COPUS rubric.

SmartSpeech also provides a timeline of events, with the duration and occurrence of each event (Table A2).

SmartSpeech displays the time schedule graphically (Figure 8). In this case, we can see that experiments were conducted at two different points during the lesson. These events correspond to the breathing activities.

**Figure 8.** Auto-generated time graph summarizing the events recorded by the observers. On the horizontal line is the time. The graph displays the events labeled in the first column as horizontal bars according to the time of their occurrence.

During the class, the teacher recorded his speech on his smartphone using the Smart-Speech app. Once the lesson was finished, the app uploaded the recording to the cloud and used Google services to produce a transcript of the recording based on automatic speech recognition. The teacher then selected some of the main concepts that were used most frequently, with SmartSpeech generating a corresponding concept map (Figure 9). In this case, we can see how the concept of energy was related to more heat, growth, and odor. Additionally, electricity and transportation were also linked to the concept of energy. All of these issues were connected to the idea of "more energy," which is the main arrow on the map. This was the concept that summarized the whole lesson.

**Figure 9.** Auto-generated concept map summarizing the main concepts from the teacher's discourse and the relationships between them. It basically suggests that a country with more energy impacts contamination and CO2 emissions, but also that countries and people generate more heat, odor, and growth.

The use of these electronic devices and software helped with the implementation of the cross-border public class. The ConectaIdeas platform and the SmartSpeech App were combined to create a performance support system [43] that helped the teacher deliver a cross-border public class [16]. It allowed the teacher to contact all of the students, receive their written responses to open-ended questions, and receive a concept map as a summary while they were teaching.

#### **5. Discussion**

Lesson study and public classes are a very powerful strategy for delivering professional development to teachers. It is a Japanese teacher education and training strategy that has been around for more than 150 years. For several years, APEC economies have been developing STEM lessons based on critical, real-world problems, such as emergency preparedness for events such as tsunamis, hurricanes, earthquakes, landslides, and forest fires [44]. In this paper we report on the design and implementation of a lesson on a new problem: energy efficiency and decarbonization. There was also the additional challenge of designing a cross-border public class in order to help students develop interpersonal skills and learn to cooperate with peers from other countries, as well as developing cross-cutting concepts related to energy, social development, and the environment. The goal was to tear down the walls that separate and isolate classrooms, and bring them closer to what is happening in the real world. Using computers and mobile devices, we can enable the shift from isolated technology labs to technology-rich classroom contexts [45].

Did this cross-border public class meet its goals? From the written responses that were submitted we obtained some evidence of student achievement. We saw how the lesson helped students understand the cross-cutting nature of energy. Several of them connected it to concepts from physics, such as heat and movement (kinetic energy). They also wrote about the transformation of energy, such as going from heat or fuel to movement. In addition to this, they also made connections with the concept of energy from biology, such as metabolism, calories, diets, and food. Finally, they also made connections to socioeconomic development, for example, comparing transportation, electricity consumption, and fuel use in the two countries, as well as looking at trends from the last 40 years. Students also related energy to CO2 emissions.

Another goal was to help students think critically. According to [36], this is a central goal of science education: Students should learn to think critically about scientific data and models. In this sense, the authors claim that this ability is developed through repeated practice by making decisions based on data and receiving feedback on those decisions. In this cross-border public class, the students reviewed APEC databases on energy production and consumption, as well as CO2 emissions. They had to understand the information, its graphical representations, and the different temporal trends. They had to make meaningful comparisons between countries, understand the implications and share their views. They also had to write down their observations, explanations, and suggestions, and they received feedback from their peers, some of whom were from another country. Therefore they had to learn to listen, read, and respond in writing, commenting on the suggestions and opinions of students from another country. Moreover, they had to use a couple of models to understand the experimental data in more depth and make predictions. As suggested by the Next Generation Science Standards (NGSS) and several authors [35], models provide scientists and engineers with tools for thinking. Such models can help them visualize and make sense of different phenomena and experiences, before developing possible solutions to design problems. The framework for K–12 science education [8] states that by the end of 12th grade students should be able to represent and explain phenomena using multiple types of models and move flexibly between model types depending on their purpose and use. In this cross-border public class on energy efficiency students used several models and discussed their respective strengths and weaknesses.

However, students were still missing some critical concepts. Firstly, not all students realized that there was a need to normalize the data. Even when the teacher displayed information on emissions and electricity consumption per million people, some students did not appreciate its significance and importance for cross-country comparisons, as well as for understanding causal mechanisms that could explain temporal trends. Secondly, most of the students only gave argumentations using static variables in response to the temporal patterns they identified. For the increase in emissions or electricity consumption over the 35-year period under study they only mentioned the current state of important factors in the respective countries, but not the changes in those factors over the 35 years.

More specifically, they did not explicitly consider the impact of socioeconomic growth. Moreover, given that the global emission intensity (fossil fuel CO2 emissions per GDP) rose in the first part of the 21st century, socioeconomic growth plays a more significant role than could be predicted by just proportional reasoning. Previous emission intensity projections failed since they did not consider unanticipated GDP growth in Asia and Eastern Europe [46]. Thirdly, students did not provide any justification for their predictions when asked how much CO2 emissions would increase if all countries lived like developed economies. Some students gave reasonable estimates, probably based on proportional reasoning. However, they did not justify their arguments. They did not explain their assumptions or the proportional reasoning underpinning their estimations. Finally, the majority of students did not consider or explicitly argue for the need to improve energy efficiency. These are issues that should be considered for future versions of the lesson.

There is also another important topic that teachers may wish to include in future versions of the lesson: the Jevons paradox [47]. This paradox is critical for understanding the environmental impact of technological transformations that generate more efficiency. According to this paradox, an increase in efficiency leads to an increase in consumption, not the other way around. This is completely counterintuitive. In 1865, William Jevons observed that a more efficient use of coal led to an increase in its consumption. Watt's innovations with steam engines made coal a more cost-effective power source and therefore it started to be used in more industries. This is perhaps the most widely known paradox in environmental economics. It would certainly engage students in a deep discussion that should be highly relevant to citizens of developing countries. Doing so may provide them with the necessary awareness and knowledge they need to face the challenges of energy consumption and the environment in the 21st century.

#### **6. Conclusions**

In summary, we can conclude that the main objective of the lesson was met. We found that cross-border classes are a viable option. We also found that even though the two classes were far apart from one another, all of the students in both classes participated actively. Students were able to collaborate in seeking solutions together with others from the other country. In addition, we found that it is possible to run an online crossborder class that is not merely expository, but instead active and focused on inquiry and experimentation. We also found that it is possible to develop verbal and written argumentation in all students. Furthermore, we also discovered that synchronous peer review is entirely possible, with students reviewing and commenting on each other's written responses in real time. Additionally, it turned out to be a very attractive class for the students, with students from two countries working together to explore solutions to the issue of promoting socioeconomic development while also keeping CO2 emissions in check. That is, students from two countries worked to imagine and discuss strategies that achieve economic growth that is compatible with sustainable development. These preliminary findings are encouraging signs that with middle school education it is possible to change the current perception among the population that reducing energy consumption has very little impact.

**Author Contributions:** Conceptualization, P.C. and R.A.; data curation, R.A.; formal analysis, R.A.; funding acquisition, R.A.; investigation, P.C. and R.A.; methodology, R.A.; project administration, R.A.; software, R.A.; supervision, P.C. and R.A.; validation, P.C.; visualization, P.C. and R.A.; writing original draft, R.A.; writing—review and editing, R.A. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the Chilean National Agency for Research and Development (ANID), grant number ANID/ PIA/ Basal Funds for Centers of Excellence FB0003.

**Institutional Review Board Statement:** Ethical review and approval were waived for this study, due to it being a class session during school time. The activity was revised and authorized by the respective principals.

**Informed Consent Statement:** Student consent was waived due to authorization from principals. Given that there are no patients but only students in a normal session in their schools, within school hours, and using a platform that records their responses anonymously, the principals authorized the use of anonymized information and the pictures for publication.

**Data Availability Statement:** The minimum data set that supports findings is shared in Appendix A, figures, and anonymized student responses.

**Acknowledgments:** Support from ANID/PIA/Basal Funds for Centers of Excellence FB0003 is gratefully acknowledged.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **Appendix A**

**Teacher Goals:** Give students the opportunity to


**Goals for students:** to recognize the cross-cutting concept of energy, its relationship to CO2 emissions and socioeconomic development, and recognize the need to work together to improve energy efficiency to reach sustainable development.

**Mission:** to help achieve social development across the world with greater energy efficiency.

**Materials:** graphics, ConectaIdeas platform.

**Table A1.** Lesson plan for the cross-border public class on energy efficiency.


**Table A1.** *Cont.*



**Table A2.** Time schedule of events using the COPUS rubric.

#### **References**


### *Article* **COVID-19 Inspired a STEM-Based Virtual Learning Model for Middle Schools—A Case Study of Qatar**

**Azza Abouhashem, Rana Magdy Abdou, Jolly Bhadra, Nitha Siby, Zubair Ahmad and Noora Jabor Al-Thani \***

Qatar University Young Scientists Center (YSC), Qatar University, Doha 2713, Qatar; a.saad@qu.edu.qa (A.A.); rana.abdou@qu.edu.qa (R.M.A.); jollybhadra@qu.edu.qa (J.B.); nitha.siby@qu.edu.qa (N.S.); zubairtarar@qu.edu.qa (Z.A.)

**\*** Correspondence: n.al-thani@qu.edu.qa; Tel.: +974-4403-3979

**Abstract:** An unprecedented turn in educational pedagogies due to the COVID-19 pandemic has significantly affected the students' learning process worldwide. This article describes developing a STEM-based online course during the schools' closure in the COVID-19 epidemic to combat the virtual science classroom's limitations that could promise an active STEM learning environment. This learning model of the online STEM-based course successfully developed and exercised on 38 primary–preparatory students helped them to overcome the decline in their learning productivity. Various digital learning resources, including PowerPoint presentations, videos, online simulations, interactive quizzes, and innovative games, were implemented as instructional tools to achieve the respective content objectives. A feedback mechanism methodology was executed to improve online instructional delivery and project learners' role in a student-centered approach, thereby aiding in the course content's qualitative assessment. The students' learning behavior provided concrete insights into the program's positive outcomes, witnessing minimal student withdrawals and maximum completed assignments. Conclusions had been drawn from the course assessment (by incorporating both synchronous and asynchronous means), student feedback, and SWOT analysis to evaluate the course's effectiveness.

**Keywords:** digital transformation; digital learning resources; STEM; virtual science classroom

#### **1. Introduction**

The COVID-19 pandemic stirred educators to develop contingency plans to prevent the disruption of learning caused by schools shutting down [1–4]. The measures taken by the governments to maintain education were devoid of solid countenance, as the situation was new and unfamiliar to the decision-makers and responsible authorities. Promptly, home-based education ecosystems using flexible learning approaches were adopted to foster students' independent learning abilities. This proceeded to entrusting a considerable responsibility of extending to the students on the teachers' virtual tutoring skills and experience [5,6]. Despite the massive efforts in bridging the gaps among the different teaching strategies to foster flexible learning, the efficiency of online teaching was nevertheless lagging the gauge. Even though the teachers had undergone the necessary training within a short period, there was a need to convert the existing resources to online teaching materials.

Previous studies on the development of online courses suggest that cultivating interaction among the students is essential to ensuring their development [7–10]. Three types of interaction are vital to fostering an active learning community during online courses, which include interaction with content, interaction with instructors, and interaction with peers [11–14]. Content interaction includes observing videos, interactive media, and the use of search engines [15]. As far as the student–instructor interaction is concerned, Dixson [9], Kehrwald [16], and Yu et al. [5] emphasized the interactive and active presence of a facilitator, who should connect with the students through activities by giving a notion of

**Citation:** Abouhashem, A.; Abdou, R.M.; Bhadra, J.; Siby, N.; Ahmad, Z.; Al-Thani, N.J. COVID-19 Inspired a STEM-Based Virtual Learning Model for Middle Schools—A Case Study of Qatar. *Sustainability* **2021**, *13*, 2799. https://doi.org/10.3390/su13052799

Academic Editor: David González-Gómez

Received: 11 January 2021 Accepted: 19 February 2021 Published: 5 March 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

a "real instructor". The facilitators may incorporate both asynchronous (emails) as well as synchronous (chats) means to communicate with the students [15,17,18]. In the case of student–student interaction or peer collaboration, the facilitators can accommodate both asynchronous (discussion forums) as well as synchronous activities, like video conferencing or real-time chatting [15,17,18]. Such connections aid the students with course engagement, despite the deficit of a physical presence of a facilitator or a school classroom [19–21].

To create an optimistically impactful online course, the focus should also be imparted to the learning environment's ambiance, because it provides an essential support to students in accomplishing their learning goals [10,18,22,23] and leaves a positive impact on their constructive learning behavior [23,24]. Though most of the research on creating an active learning environment in virtual classrooms has aided the educators, barely any assistance was provided to science teachers in incorporating hands-on scientific activities in the online course material. Even though the facilitators worldwide have provided customized solutions depending on their target audience and lesson plans, an authoritative evidence-based study has not been previously performed in this context.

As the "new normal" of blending online and offline has emerged, learning settings have gradually been formed to gratify the shortcomings under the COVID-19 regime. However, with a specific focus on the virtual science classroom, we have designed and implemented a STEM-based research course for middle school, and assessed the student engagement with cautiously incorporated hands-on science activities. The course was created to combat the shortcomings in students' interaction on a virtual science platform and, in the due process, effectively transfer a peer collaboration space into the learning arena. The course developers ensured that it offers a good scope for researchers to create science lab spaces at students' homes through the developed science-based course material. The online sessions were carried out employing the synchronous and asynchronous activities like real-time course sessions, online meetings and chats, real-time educational games, live presentations, poster seminars, etc., and exploring learning videos, assignments like hands on scientific experiments, and group discussions.

#### **2. Objective of the Course**

The school shutdown has barred the students from accessing a physical classroom that had previously offered them ample opportunity to experience peer-collaboration and engaged them for productive outcomes. However, currently, the students have been driven to choose limited compensatory learning resources through digital platforms and resources. This unprecedented situation also prevented them from performing group-oriented handson activities conducted in their school laboratories and restricted their active learning to passive virtual lectures at home premises under the guidance of parents. Hence, an interactive online STEM course is designed for primary–preparatory students with a transparent and concise objective of overcoming some of the existing shortcomings of a virtual classroom (as described in the discussion section) and enhancing their engagement.

#### **3. Methodology**

#### *3.1. Research Questions*

The study employs a research- and evaluation-based research framework to design the interactive online course based on the course developers experience with adequate research backings. The paper is primarily focused on evaluating the effectiveness of the course design while applied on the participants under voluntary terms, and henceforth carefully portrays the impact of each learning resource on the student response. While addressing the impact of course design, the following research questions (RQs) were potentially considered.

RQ1. Did this course design provide adequate action through diverse digital resources for active virtual learning, resultantly improving the student attitudes towards online learning environment?

RQ2. Was the designed course successfully implemented in attaining its target objectives like improved student interaction and creative peer collaboration?

RQ3. Did the course design provide scope for improvement and replication?

RQ4. Did the design offer course flexibility and applicability, especially under limited learning opportunities?

#### *3.2. Research Methods*

The pilot study reported in this paper was a design experiment carried out for research and evaluation. The course design and implementation was an experiment that was assessed by the feedback mechanism approach, performed daily after the course session on the participants. This approach was relied upon to directly rectify the shortcomings in the multiple session course activities, teaching approach, and implementation flaws before heading towards each succeeding session.

Course design is a crucial parameter in assessing scientific lab-driven courses' efficient execution, especially online platforms. The online STEM course has been implemented using different assessing tools to analyze the respective student outcomes. The content for this one-week course was based on a STEM topic, forces and motion, as per the standards outlined by Qatar Science Curriculum (QSC) for primary and preparatory stages (grades 5, 6, and 7). The course participants include 38 students (24 girls and 14 boys) from seven different public and private schools in Qatar. Though the course was voluntary, free of cost, and open to all based on a first come first serve basis, the preference was offered to the nationals. The maximum number of students to be accommodated within a session was restricted to a maximum of 15, and thus the program facilitated two batches for girls and two batches for boys. The two course batches for girls were organized during the first week, followed by the two batches of boys in the second week. Since the course accommodated diverse team activities, the participants were further randomly grouped into eight groups of females (four groups in each batch) and seven groups of males (four groups in each batch with one group withdrawal). The facilitator, mentor, and student distribution is as illustrated in Figure 1. The course witnessed the mentorship of eight female undergraduate (UG) national students, each guiding one team of participants and studying their learning behavior as a part of their summer internship program. Each UG student acts as mentor to one group each of female and male students, as illustrated in Figure 1 as well. In the case of female students, the ratio of the UG mentors to the number of students was 1:3, and in the case of males, it was 1:2, as boys from middle school need additional supervising due to their behavioral patterns [25–27]. The UG students were motivated to join the study voluntarily to fulfill their summer internship curriculum (as a part of their degree requirement) by performing research on innovative online STEM learning approaches. They recorded daily observations on the student learning outcomes that mostly illustrated the participant response to the diverse activities and assignments. The UG mentors were also assigned to solve any technical glitches for the smooth functioning of the course. The daily assessment made on each participant group also influenced the facilitators to amend the teaching approach for each succeeding day, considering the students' setbacks on a respective day. The UG mentors held a key responsibility in assembling the students for the respective sessions, encouraging them to actively participate within the session through continuous follow up, motivating them to answer the facilitator's questions and compete with each other, and involving them in discussion with peers. They prompted the students to engage in real time discussions and inspired the students to respond timely to the facilitator. Overall, they assessed daily student attendance and participant response to the synchronous activities and monitored their assignments that were required to be accomplished through asynchronous sessions.

**Figure 1.** Graphical representation of facilitator, mentor, and student distribution in each female and male batch. During the first week, there were two female batches; each batch consisted of four groups of students and was administered by three facilitators. Each group of students comprised of one university undergraduate student and three middle school students. Similarly, the second week carried out two male batches of students. One group of students withdrew from the course in between, leaving three male groups (second batch) at the end of the week. The three facilitators covered both the male and female batch of students.

The course was designed by a team of STEM professionals (Ph.D./Masters/bachelor's degree holders) who have an extensive experience in designing workshops and educational activities for more than five years. Previously, they have been working in close collaboration with faculty members at different research centers and colleges at Qatar University under the Al Bairaq Program [28]. The course developing team were also responsible for delivering the course sessions without any fail and thereafter addressed as facilitators/course developers.

#### *3.3. Course Design*

The course is constructed under an organized framework of STEM programs, as shown in Figure 2, facilitating both synchronous and asynchronous activities that cultivate curiosity, execute research, and draw out creative innovation through engineering a product. The framework accommodates diverse assignments daily to evaluate and understand their improvement in engagement and collaboration. This was duly considered to address the research questions RQ1, RQ3, and RQ4, later evaluated from the student daily feedbacks. Synchronous learning accommodates activities commencing from introduction of force to transferring knowledge by creating a "product car" as illustrated in Figure 2b. Meanwhile asynchronous activities included daily assignments, product, and documentation of videos of completed offline challenges. The students' groups performed the following STEM-based activities for the first four days of the class during both online and offline sessions, culminating with an engineering design-based project on the final day as described in Figure 2a.

**Figure 2.** Graphical representation of course methodology is displayed along with learning standards and digital resources implemented. (**a**) A set of asynchronous and synchronous activities implemented during the course from day 1 to the end of the course. (**b**) Learning standards that are focused on each synchronous activity commencing from introduction of force to transferring knowledge by creating a "product car". (**c**) Digital resources implemented to provide course, with MS Teams (File:Microsoft Office Teams (2018–present).svg.) as learning platform and PowerPoint presentation (File:Microsoft Office PowerPoint (2018–present).svg.) [29,30] to present the content through games, videos, and simulations. (**d**) Evaluation criteria of final product car made from recyclable materials, by testing it on 2–3 different materials with different surface friction, capable of covering a minimum distance of 50 cm.


which has to follow a set of requisites during the assessment as given in Figure 2d. The product development should be carried out within a week as an obligatory condition from the course's last day. After successfully testing the prototypes, the teams present their working products through web conference to a judging panel, comprised of researchers from multiple backgrounds in science and engineering. The judges included engineers (civil, industrial, and chemical majors) from the University and an outreach specialist from the oil and gas industry. The detailed explanation of the product's functioning is further presented as a poster to evaluate the outcomes based on the pre-requisites. While all the activities were executed during synchronous sessions, the completion of assignments and product development were carried out under asynchronous activities. The different criteria (refer to Figure 2d) that were expected to be satisfied while designing a product include the following.


#### *3.4. Course Breakdown*

Each day facilitates a ~60-min learning session as per student preferences to promote flexible learning opportunities on a real time learning platform. During the synchronous session as illustrated in Figure 3 and Table 1, the session was categorized into sub sessions —introduction session, scientific activities session, and concluding session. The scientific contents and the learning objectives were addressed during the scientific activities sub session, whereas the other two sub sessions mainly focused on the smooth functioning of the course session. The three sub sessions are distributed accordingly in Table 1.

**Figure 3.** Distribution of each course session of 45 min into different sections of each session daily plan.


**Table 1.** Time allocation for different synchronous activities that included scientific learning and course organizing.

\* Asynchronous session and the time taken to complete were not documented, as it was performed at home under parent supervision.

During the introduction sub session, a distinct time was allotted for all attendees to log on time, followed by discussing the submitted offline assignments and concluding with the recap time, the last two sessions being exempted for the first day as seen in the Table 1. However, this time was equalized on the first day by allotting time to introduce course participants and their different teams to the whole batch.

The scientific activities session accommodated the STEM course activities on and off the session, depending on the feedback of the students (will be addressed later in Section 4.3). The table clearly demonstrates the decrease in the time allotted for stimulating curiosity activity, as the students showed self-motivation without any prompt. As most of the scientific session time was allocated to acquaint the students with scientific knowledge, the more inquiry based activities were provided to the students. Moreover, engineering design challenge activity was practiced off-session through assignments as the students can take more time on completion.

The concluding session was basically carried out to summarize the period, launch assignments, address students on the next session requirements, and obtain feedback. The time for feedback session was not limited in order to give room for student expression. The launching assignments sub session did utilize a major time period on the final day, as the students were briefed on making a car activity as a part of their design challenge.

The course design also ensures that a curriculum standard is delivered on each day to obtain the required learning outcomes, as in Table 2. Each session's core content is developed and modified, based on the curriculum standards and learning outcomes, as portrayed in Table 2 below. The facilitators also ensure that each day's teaching approach is modified prior to the delivery, according to the feedback, student performance, and learning behavior during the preceding course session. They modify the web-based learning resources that were implemented to engage the students, hence ensuring that no resources are repeated over time.


**Table 2.** Distribution of adapted curriculum standards and their respective learning outcomes.

#### *3.5. Digital Learning Resources*

To implement the course effectively, content delivery is smoothly organized with different digital resources and assignments to guarantee an innovative learning experience. Since STEM courses integrate multiple hands-on activities to bolster creativity and cognitive development, the course developers modified the science experiments to implement digital learning. The initial outline for activities included procedures that involved elaborate experimental setting, which needed parental or teacher supervision. The initial activities were organized appropriately for the physical classroom and hence required the usage of scissors and the guidance and safety protocols. However, those activities were replaced by simple online simulations (will be introduced in Section 4.1) that could be implemented to obtain the same expected learning outcomes. The Microsoft application MS teams was chosen as the online educational platform to deliver the course, with the students familiar with it from their regular school sessions. In fact, the facilitators also did not want to allot time to familiarizing the primary–preparatory students with a new forum. It may consume time and overshadow the course's main objective, to deliver STEM learning on force motion. The facilitators highly recommended MS teams to engage with the students, as it aided in carrying out effective group discussions by enabling students to communicate through private channels offering limited visibility to group members only. However, the developers considered the maximum possible ways to construct the course implementing the different interactive resources offered by MS teams, for example, meeting chat box, the whiteboard, raise your hand icon, the benefits of setting private channels, icon, etc.

The digital resources are efficient moderators in ensuring student engagement on online platforms for voluntary sessions with a prolonged duration. Table 3 below enlists different online teaching resources including PowerPoint, videos, online games, puzzles, simulations, etc., (refer Figure A1a–d from Appendix A) employed during the course to establish the respective course objectives in escalating a positive attitude from the students. All these resources will be specifically introduced in Section 4.1. The digital resources were chosen to smoothly facilitate the topic presentation to fascinate the students and educate them in the respective context. The interactive games, which are useful resources for quick assessment with instantaneous outcomes, are also included from diverse websites to avoid repetition and build student interest. Student groups are also pursued by the UG

mentors and facilitators for their assignments, as well as offering assistance with the aid of WhatsApp applications to hold group discussions and idea sharing.

**Table 3.** Distribution of diverse learning resources implemented to achieve the respective content objectives.


#### *3.6. Course Assessment*

The developers included assignments as a primary assessment tool for analyzing the students' behavior to measure their interactivity during the course. The assignments are carried out asynchronously during the offline period. The students perform the assignments at home under parent supervision if necessary. The assignment activities were carefully designed to fit tasks that prompt students to draw out their creativity, test their reasoning skills, build resilience, showcase artistic talents, implement technical knowledge, and solve diverse science, math, and engineering problems. The multi-competent assignments that were designated to the students during the entire course are tabulated as below (see Table 4).


**Table 4.** Brief details of multiple assignments designed to achieve the respective learning experiences.

#### **4. Results**

The study findings that include the course features and the counter student behavior were laid out to analyze the effectiveness of the course design and its implementation, thereby addressing all four research questions. The following data collection tools were used in drawing out the findings of the study.


The analysis of the course features was performed by examining the student behavior and feedback towards accomplishing each course activity as well the asynchronous assignments. The following sections detail the findings based on different factors that contributed to the course's successful design and implementation.

#### *4.1. Assessment of Course Content and Design*

The topic, 'force and motion', was carefully considered for the course, as per the QSC requirements. As the course design implemented different hands-on activities, it was quintessential for the developers to overlook the safety guidelines, ensuring that the materials and activity procedures were safe to be implemented at home. Moreover, the materials' sourcing was restricted, as the course was provided to the students when the entire country was under lockdown due to the pandemic. The situational crisis prevented the students from leaving their houses or accessing school labs or material stores, thereby forcing the developers to create content that was fully functional from their home premises.

The content was designed in such a way to make sure to retain the students interested during the sessions in a cordial manner, by providing a wide variety of activities that offered dynamic and interactive content to engage and learn. The course successfully applied various digital resources (refer Table 5 and Figure A1a–d from Appendix A) like different learning videos, real-time simulations, hands-on activities, and multiple games to understand the concepts of force and motion. As games are an effective strategy to engage students on online platforms, numerous interactive games were introduced to grab their attention. All the activities except for the assignments were executed live in the facilitators' presence during the virtual sessions, thereby visualizing their commitment throughout the session. The course content was successful to a significant extent in integrating STEM subjects, thereby correlating forces to real-life applications. In the due process, it was also observed that students exhibited enhanced teamwork through online communication on social platforms like WhatsApp and MS teams, thereby marching forward to social learning.

Some of the online games (for example, https://www.gameflare.com/online-game/ seesaw-ramp-car-balance/, (accessed on 23 February 2021)) included are concise and function as a single level task of asking the participant to predict the title of the lesson or activity. Such short games are highly effective in creating short-term excitement in students. The virtual simulation (for example, https://phet.colorado.edu/sims/html/forces-andmotion-basics/latest/forces-and-motionbasics\_en.html, (accessed on 23 February 2021)) was integrated into a part of four-day lessons to adhere the students to the program through invoking student interest and engagement. The developers also ensured that the simulation resources are not repeated in the consecutive sessions, to avoid students getting bored quickly. The respective simulations were chosen, as they easily assisted the students to attempt multiple times, varying the inputs and observing the corresponding outputs. The 3D simulations also gave them a holistic view of the concept, offering them situations they can easily imprint on, thereby improving students' engagement. Zacharia and other researchers [32–35] found that when the teachers interacted with the computer-based simulations, the explanations they constructed were more fluent, more detailed, scientifically more accurate, and involved more formal reasoning. Similarly, games involving puzzles and crosswords were complimentary online engaging resources that were utilized to address the students' reasoning skills and problem-solving competences.

The facilitators had ensured that the implemented videos were finely audible with attractive sound modulation and understandable for students. The videos were created either by the facilitators or from other existing educational resources. The course also made room for animation videos [36] to convey the lesson objectives, as the children exhibited more interest in animation for educational purposes. The videos were further implemented as an exceptional input to assessing the students, as they were requested to observe and analyze a situational problem from the provided video as in problem-solving activity. The respective activity helped the facilitators interpret their observational skills and their application of critical thinking to the corresponding situation. Importantly, the videos were short-timed so that the participants could not divert their attention. The course developers ensured that the inclusion of the digital resources was balanced and rightly timed according to the background study conducted on the behavior of students on online learning.


**Table 5.** Distribution of diverse teaching resources for the course.

During the sessions, the students zealously engaged in the course, actively participating in all the online and offline activities. Amid the online session activities, extroverted students with excellent leadership qualities and social learning skills were often observed to provide oral answers quickly. Meanwhile, most of the participants were introverts, who acknowledged considerable activity through meeting chats to address the facilitators' questions. However, the introverted students did seem to perform relatively better along with the extrovert students on hands-on activities, as they were accomplished individually from their homes. They recorded videos using mobile phone cameras and thrived on WhatsApp group conversations with their team members, wherein most of the course driven group discussions were facilitated. Few students had difficulty approaching group-oriented activities and discussions and exhibited poor socializing skills either due to their behavior or due to lack of associating with the digital transformation. As the participants were from middle school, most of them also had to access their parents' guidance for gadgets or course-related follow-up activities.

The hands-on scientific experiments were crucially important in the course content for attracting the students to participate in the courses without any compulsion. The students were requested to perform activities as a part of live sessions and assignments, attributing to reflections of the corresponding live session (refer Figure 4a,b and Figure A2a–c from Appendix A). They consistently performed these activities and recorded their act as a part of creating evidence and documenting the result of their work as videos (student video links are attached in the Appendix B). The students' documentation was performed with their parents' assistance in holding the camera or measuring the distance traveled by their toy car and/or arranging the setup. The videos were also promoted to be published on social platforms such as YouTube [37], thereby using their personal inclination towards social media platforms. Resultantly, the students were self-motivated and encouraged to improvise their work with each attempt, thereby learning and self-assessing themselves in the meantime. Moreover, the course developers also ensured that the experiments were conducted with readily available materials like cloth hangers, plastic cups, rope, paper, plastic bottles, and bottle caps, etc., guaranteeing material access to every student at home. Though the materials were commonly found in homes, the facilitators also tested the student's capability to replace these materials in case of a deficit. However, to their surprise, the students exhibited critical thinking and problem-solving competencies by successfully accomplishing their target with limited materials, guidance, and workspace.

**Figure 4.** Examples of random hands-on assignments and activities accomplished by the students in the form of images or videos. The videos can be found in the links provided in Appendix B. (**a**) An instance from the video, an assignment turned in by the student on day 1. (**a**) Origami frog making, an integration of science and simple engineering, was introduced to the students in order to acquaint them with force and its effects. (**b**) A sample balloon rocket, made by the students as a part of one of their assignments. (**c**) One of the product cars as made by a student participant of a random group using recyclable materials and balloon. (**d**) A similar product was also made by the same group member using different materials and design. (**e**) A battery powered car made by a third team member of the same group, which could also move on water due to its foam base that keeps it afloat. This car was chosen to be the best product from the respective group.

The design project activity, included in the course, was the course's key highlight in deriving student creativity, problem-solving, and reasoning potential, as the students successfully transferred the knowledge for creating the product. The students built a toy car (refer Figure 4c–e) by satisfying a set of criteria mentioned in the methodology section on page 7.

The facilitators were keen to include opportunities for the students to make predictions and interpretations. The exercises on predictions and interpretations are critical factors in activating and refining the student's prior knowledge and enhancing their engagement. Each activity satisfied the lesson plan's core objective, as most of the students responded positively to the UG mentors, who kept tabs on their assignments and technical glitches, and guided them during group discussions. The facilitators also notified that all the course certificates were provided to those who completed their assignments without any fail.

The students exhibited enhanced participation for hands-on activities despite the restrictions of accessing the materials on their own in contrast to having been accustomed to settings with readily available materials at their school classrooms or labs. They even showcased their problem solving and critical thinking dispositions while accessing the required materials by replacing them with better alternatives in case of deficit. During the design project assignment, which was to be accomplished as teamwork, each member designed and developed a product independently, as they did not have the opportunity to gather around due to social distancing norms. Once they succeeded, they tested their products in front of their teammates via a web conference call. They displayed the product to each other and compared the results to choose the best design according to the criteria, in order to compete with other groups. Some groups even resorted to improving their final product even after meeting the requisite conditions. For example, in the case of one girls' group, student 1 built a balloon car (as in Figure 4c) that was able to satisfy all the preset criteria. However, student 2 made a toy car, which also exhibited similar performance concerning speed and distance (as in Figure 4d). However, student 3 changed the design of the vehicle (car) to function itself as a boat as well, powered by a battery source, which was tested in a tub filled with water (as in Figure 4e) and chosen to be best in performance and design (videos of testing is available on request).

#### *4.2. Assessment of Student Learning Behavior*

Student attendance was a major criterion to understand that they were interested in participating in the course, irrespective of the fact that the sessions were entirely voluntary. The motivational factors that drove the students to attend the courses out of their own free will could be attributed to intrinsic parameters like student interest, cultivated from the course. Moreover, since the course was organized during the summer vacation, it compensated for their free time. The students who attended the courses received merit certificates at the end of the course to participate actively without any fail, depending on their attendance. Table 6 and Figure 5 below provide the distribution of students who attended the course entirely and those who withdrew from the course. The withdrawal of the students was attributed to either medical unfitness in the case of one student or lack of interest in the case of other three as per the facilitators' reports. However, we observed that the number of female students were retained until the end of the course in contrast to that of males. The drop-out males did attend most of the sessions; however, lack of interest in submitting assignments discouraged them from completing the voluntary program. However, the students were not forced to leave the course due to the incompletion of assignments.

**Table 6.** Number of total enrolled students and their gender-based distribution.


**Figure 5.** Graphical representation of number of student participants who have completed the course along with the number of withdrawals.

#### *4.3. Upgradation of Course Content*

The effectiveness of the course was assessed by the results from the feedback mechanism implemented in the daily session. During the five days of the course, the students were required to provide feedback for the following three open-ended questions.


The analysis of the three open ended feedback questions was carried out daily by the UG mentors, who notified the facilitators to improve the teaching approach, learnercentered instruction, hence implementing the changes according to the student choice. This method led to providing the students with a voice and equally participating in the pedagogical change that improves the teaching approach with time.

The course was improved in the instructional approach owing to the recommendations of the students. Some of the students requested more activities, whereas others asked for time to complete the sessions' online challenges. Some students even asked to include more days of activities to the course schedule; meanwhile, others did not have any suggestions at all. One of the students did suggest that the mentors did not give fair chance to all. This feedback was rebounded by introducing an online interactive tool, "wheel of names", which chooses students randomly from a list of entries, thereby solving their problem. Though the facilitators considered providing more time for the challenges, they could not accommodate more activities within the designed course due to the time allotted for the course. The recommendations put forward by some of the students, on giving equal chances to all participants to answer queries, for which they were awarded points on a daily score-chart, were also considered on the following days. The graphical representation of student feedback that portrays satisfaction with respect to each course feature is also displayed hereafter in Figure 6. The feedback from the students was analyzed, and the frequently recurring course feature was chosen as a keyword. The keywords derived from their course include activities, challenges, teamwork, interaction, etc. While the students

stated the teaching approach of the mentors, for example, "the mentors should give a fair chance", "the mentors should mute participants", or "the mentors should give more time to finish challenges", the keyword represented is "mentor approach". In case of stating "nothing" for their dislikes or recommendations, we represent the keyword as fully satisfied. The Figure 6 presents the graphical representation of participant preferences for each of the mentioned keywords or course features.

**Figure 6.** Graphical representation of participant feedback to the three questions, "What did you like the most from the session?", "What did you like the least from the session?", and "What would you like to change in the session?". (**a**) Male participant response to the most liked feature in the course session. (**b**) Female participant response to the most liked feature in the course session. (**c**) Male participant response to the least liked feature in the course session. (**d**) Female participant response to the most liked feature in the course session. (**e**) Male participant response to the feature likely to be changed in the course session. (**f**) Female participant response to the feature likely to be changed in the course session.

As the course program was based in Arabic language, the data representation was based on feedback stated in the Appendix C attached, and contains transcripts translated by the "Google translate" application (a sample of original WhatsApp chats is provided in Appendix D).

All digital resources were included in the course by cautiously considering the preferences of the students and balancing time management. They demanded activities based on simulations and games in their daily feedback, thereby inciting the developers and facilitators to include more weightage for the same. Videos were provided either to deliver content knowledge or to introduce a game and hence were mostly short timed to play as 1-min videos, limiting their maximum daily playtime to 3 min. Since PowerPoint presentations were used to deliver the whole session course content in the form of videos, games, simulations, etc., they were accounted to be implemented for the maximum duration. The online engaging resources include interactive applications like whiteboard for illustration and the wheel of names that was applied to select students on a fair basis for each game, thereby avoiding conflicts in the midst of session. The graphical representation of digital tool distribution that displays the daily usage of each digital tool during the respective session is provided below in Figure 7a–f.

**Figure 7.** (**a**) Graphical representation of the duration of the multiple digital resources implemented during each session per day. The time duration for each resource was chosen based on the student feedback that stated their interest towards them as well considering the time limits of the session. The resources were applied in each session considering the student feedback received on the preceding

day. (**b**) Graphical illustration depicting the implementation of PowerPoint presentation throughout the entire program. The graph looks constant, as PowerPoint (ppt) was the main tool to impart the course, whereas the other resources were implemented from the PowerPoint. (**c**) Simulations are provided more weight on alternate days to balance the timing as well the curriculum demands. (**d**) The graph depicting the usage of online games decreases exponentially with each day, corresponding to the increase in content weight as well as the demands to attract the students, as more required during the initial days. (**e**) Online engaging resources are also following an exponentially decreasing trend similar to that of games, depending on the lesser need for engaging students with the progress in days. (**f**) The implementation of videos is constant throughout, as they are provided in the form of short videos to warm up students.

#### **5. Discussion**

The emergence of virtual classrooms with the closure of schools due to COVID-19 had exhibited various setbacks while its impact on student development in terms of interactivity, communication, and engagement was assessed. The students were mostly confined to online lectures serving their curriculum's core topics, providing few opportunities for activities requiring an ambiance of laboratories or courses. Despite providing flexible learning opportunities with customized course timing and duration for the students, the virtual learning platforms could not promise scope for active student participation and innovation with enhanced peer collaboration. Moreover, in contrast to the past summer vacations and camps, owing to the social distancing restrictions imposed all around the country, the students were confined to their homes without any opportunities to engage in extracurricular activities or even be involved in any recreational activity with/without peers. They were bored out of inactivity, and the timing was appropriate to impart them with a learning-for-fun kind of course to study their learning behavior without any obligation. As the study's objective was focused on developing a course to counter the existing drawbacks of the online learning outcomes, we accomplished the methods using principles of learnercentered instructional design considering the voluntary nature of the course execution. The study evaluated the effectiveness of the developed course content by implementing a weeklong course on the students from primary–preparatory schools. Overall, the program appeared to have served as a competent learning resource for our sample, as the students shared their experiences, being acknowledgeable in terms of interest and reflections.

#### *5.1. SWOT (Strengths, Weaknesses, Opportunities, and Threats) Analysis*

The course content development as well as execution were well examined in terms of their strengths and weakness as well as opportunities and threats faced by the facilitators as well as students. Hence, a SWOT analysis matrix was developed from the overall analysis of the course program as described in Table 7.

#### 5.1.1. Strengths and Weaknesses

The course is designed and implemented according to students' convenience with session duration and timing, thereby offering flexible learning opportunities. The students also partake in the course teaching approach by providing daily feedback that plays an important role in determining the facilitation of different types of activities. The course also encourages the setting up of simple science activity labs within the limited home premises. Setting up labs requires room for parent involvement, as they assist in arranging materials, supervising, and assisting in performing experiments, which in turn can positively upgrade the student learning attitudes. Moreover, the course also engages UG students to experience learning by mentoring the participant students, thereby reinforcing their knowledge and understanding. Overall, the course has displayed promising aspects in terms of enhancing tech savviness of students as they progress through different virtual activities and online classes.

However, the course implementation lacked standard assessments that could be used as a tool to measure student-learning outcomes quantitatively. In addition, the number of participants involved in the course was limited due to the constraints in executing online lab activities effectively for many groups at a single time. The course also was constrained to accommodate activities that require safe materials that can be used at home under limited supervision. However, learning was majorly technology driven, thereby enforcing its dependence on materialistic resources rather than intellectual presence.

**Table 7.** SWOT analysis matrix that details the strengths, weaknesses, opportunities, and threats faced by the stakeholders and program developers.


#### 5.1.2. Opportunities and Threats

The course provides oversight for the educators to implement a considerable number of collaborative activities in the same manner. The outcomes of the course reinforce the research backings that highlight the positive impact of out of school programs on student cognitive development. Similar programs do pave a concrete path for self-directed learning for the students, especially through challenging times for global education. On the other hand, learning via similar programs more or less depends on availability of material resources rather than intellectual capital, specifically pointing to the fact that the availability of uninterrupted internet access and gadgets that aid smooth functioning of the course is far more critical than the availability of skilled facilitators and their intellectual contributions. The course can offer internship opportunities for university undergraduate students by guiding them to perform research studies on the learning behavior of students when subjected to similar courses in STEM.

As technology can enable a wide range of students from near and far to access education, student presence during an online session is still a major concern encircling the teaching community with the emergence of virtual learning platforms. Students may also lose their interest and motivation due to the technical glitches in a prolonged period. Hence, the teaching strategies need to be refined from time to time to include all students of different learning dimensions, to retain their self-interest or motivation. Digital divide is also another major factor that can threaten the credit of the course, like all other virtual learning opportunities. However, the course can cover an extensive reach both locally and internationally due to the beneficial medium of the course.

#### **6. Conclusions and Outlook**

Tackling the shortcomings in the online teaching methods, educators came forward with diverse innovative solutions to engage students. Their main challenge was to create online learning content that is relatively engaging and interactive for the students to remain in learning sessions without any compulsion. This paper elaborates on creating a STEM-based virtual course to be delivered during summer vacation, thereby utilizing the idle time of the students for productive learning outcomes. The course successfully explores the effects of different elements like various teaching resources, both digital and traditional, hands-on experiments, and assignments in implementing effective course content. The evaluation of the course implementation was carried out through daily post-session feedback from the 38 participants from middle school that highlighted the importance of including activities and challenges for an active session. This resulted in constructing the course design with maximum time allocation for simulations, games, and hands on activities, thereby enhancing the student participation in the voluntary course. SWOT analysis (qualitative) was also performed to provide clarity on the study, as the paper had limitations in quantitative analysis. Even though the course provides opportunity to students to engage in the course instructional development through daily feedback sessions, their individual learning outcomes need to be assessed quantitatively. Additionally, this model of summer course displayed constraints in delivering scientific experiments with elaborate settings and a higher number of students per session. Additionally, this study implicates a learning model that requires an uninterrupted internet connection to provide a smooth functioning of similar online courses. However, the following highlights of the study could be summarized from the successful development of the course based on student feedback.


**Author Contributions:** Conceptualization: A.A. and R.M.A.; methodology: A.A., R.M.A., and N.J.A.- T.; formal analysis: J.B., N.S., and Z.A.; investigation: J.B., N.S., and Z.A.; resources: A.A., R.M.A., and N.S.; writing—original draft preparation: N.S.; writing—review and editing: J.B. and Z.A.; supervision: N.J.A.-T.; project administration: N.J.A.-T. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Data Availability Statement:** Video recordings and other data of the sessions are available on the reasonable request, respecting the data privacy regulations.

**Acknowledgments:** We are deeply grateful to the office of Vice President for Research and Graduate Studies, Qatar University, for supporting the study. Moreover, we also thank Ras Laffan Industrial City-Community Outreach in supporting us for the effective implementation of the study.

**Conflicts of Interest:** The authors declare no conflict of interest.

**Appendix A. Supplementary Figures that Are Referred to within the Text**

**Figure A1.** Different online activities that can be used to assess student competences. (**a**) Simulation to describe balance of forces. (**b**) A 3D simulation implemented in the course. (**c**) Multiple choice challenge generating application. (**d**) 1d crossword challenge.

**Figure A2.** Examples of random hands-on assignments and activities accomplished by the students in the form of images or videos. (**a**) A rice–pencil friction activity performed during the online session to introduce the students to the concept of friction. (**b**) An image depicting the student assignment of making a free balance out of random materials like hangers, plastic cups, etc. (**c**) An instance from a video illustrating the testing of the sample balloon rocket that works on the Newton's third law of motion, thereby familiarizing scientific laws through applications.

#### **Appendix B. A Few Video Links of Students Work that Have Been Uploaded on Youtube as Referred to within the Text**



#### **Appendix C. Sample of Feedbacks from 20 Students, both Male and Female as Referred in the Text**

#### **Appendix D. Example of Feedback in the Original Form**

The below represented detail is a screenshot of a WhatsApp conversation sent by the UG mentor of team "The Clouds" to the facilitator, providing the student feedback obtained from her team. Student names are masked for privacy.


Note: The Clouds ( ) 14 June 2020.

#### **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Sustainability* Editorial Office E-mail: sustainability@mdpi.com www.mdpi.com/journal/sustainability

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18

www.mdpi.com

ISBN 978-3-0365-3422-0