# **Ensuring Quality Education and Good Learning Environments for Students**

**Improving the quality of higher education through good teaching practices** 

> Edited by Ana B. Bernardo, Adrián Castro-López, Leandro Silva Almeida and Sílvia F. Rivas Printed Edition of the Special Issue Published in *Sustainability*

www.mdpi.com/journal/sustainability

**Ensuring Quality Education and Good Learning Environments for Students: Improving the Quality of Higher Education through Good Teaching Practices**

## **Ensuring Quality Education and Good Learning Environments for Students: Improving the Quality of Higher Education through Good Teaching Practices**

Editors

**Ana B. Bernardo Adri ´an Castro-L ´opez Leandro Silva Almeida S´ılvia F. Rivas**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editors* Ana B. Bernardo University of Oviedo Oviedo Spain

S´ılvia F. Rivas University of Salamanca Salamanca Spain

Adrian Castro-L ´ opez ´ University of Oviedo Oviedo Spain

Leandro Silva Almeida University of Minho Braga Portugal

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Topical Collection published online in the open access journal *Sustainability* (ISSN 2071-1050) (available at: https://www.mdpi.com/journal/sustainability/ topical collections/qual educ good learn environ).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-6736-5 (Hbk) ISBN 978-3-0365-6737-2 (PDF)**

© 2023 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**



## **About the Editors**

#### **Ana B. Bernardo Guti´errez**

Dr. Ana B. Bernardo Gutierrez is an associate professor of psychology of education and ´ development at the University of Oviedo. Since 2000, she has belonged to the university-run research group named ADIR (School Learning, Difficulties, and Academic Achievement). She has participated in several research projects and has funded competitive calls on the challenges that the European higher education field brings with it for students, teachers, and also for several educational innovation projects. As a result of these studies and collaborations with research teams from other universities (national and international), she has published numerous articles in both scientific journals and book chapters, and she has participated in numerous scientific meetings over the last few years.

#### **Adri ´an Castro-L ´opez**

Dr. Adrian Castro-L ´ opez is an assistant professor of marketing at the University of Oviedo. His ´ current research is focused on the circular economy, marketing, artificial intelligence, and education. He has published his work in several journals, for example, the Sustainability, International Journal of Contemporary Hospitality Management, the European Journal of Psychology of Education, Applied Soft Computing, the International Journal of Information Technology and Decision Making, PLOSONE, Frontiers in Psychology, Higher Education Research & Development, Education and Training, Springer Editions, etc. He has served as a referee for the Journal of Retailing and Consumer Services, Consumer Studies, and Electronic Commerce Research, as well as a member on the editorial team for Applied Soft Computing.

#### **Leandro S. Almeida**

Dr. Leandro S. Almeida is a full-time professor at the University of Minho, and he is a member of their Psychology Research Center (CIPsi-UMinho). His current research is focused on higher education, transitions, and students' adaptation to academic success and incidences of dropping out. He is both the director and member of the research projects, which include participants from Angola, Brazil, Mozambique, Portugal, and Spain. Furthermore, he has authored and co-authored psychoeducational tests (intelligence) and scales (academic adaptation, learning strategies, and motives of abandonment). For his dedication to his field, he has been awarded the "University of Minho Scientific Award" (2019).

#### **S´ılvia F. Rivas**

Dr. S´ılvia F. Rivas is professor of critical thinking at the University of Salamanca. He has presented courses and lectures at other universities all over the world on the subject of critical thinking. Furthermore, he has published several articles that focus on critical thinking instruction and assessment. He has contributed to the development of the ARDESOS program in terms of the refinement and improvement of critical thinking skills, and since 2008, they have jointly developed the PENCRISAL critical thinking test. Currently, he continues to develop critical thinking instruction and evaluation tools at the University of Salamanca, within the university-led research group, namely, the "Group of Critical Thinking and Positive Psychology".

## *Editorial*  **Challenges of Higher Education in Turbulent Environments**

**Ana B. Bernardo 1, Adrian Castro-Lopez 2,\*, Leandro S. Almeida 3 and Silvia Fernandez-Rivas 4**

	- 3 School of Psychology, University of Minho, 4710-057 Braga, Portugal
	- 4 Department of Basic Psychology, Psychobiology and Methodology of Behavioral Sciences, University of Salamanca, 37005 Salamanca, Spain
	- **\*** Correspondence: adriancastrolopez@gmail.com

New technologies will change personal and social life as we know it today. In the current society of information in which we live, we gather information and make decisions based on the virtual resources available to us. However, despite the increased quantity and quality of these resources, problems persist. One of them is that such resources are not generalized between countries and between social and ethnic groups within each country. The use and exploitation of new technologies and virtual environments are implemented in all areas of social life, including in the educational context, and such differences call into question the sustainability and scope of the Sustainable Development Goals (SDGs) defined by the UN for 2030.

Thus, in the last few years, higher education systems and learning environments have developed in line with society and the changes that have arisen in the environment, thus influencing their long-term sustainability. For example, the COVID-19 pandemic highlighted the vulnerability of these systems worldwide, mainly due to the difficulty of adapting to new technological teaching resources (focused on new online teaching platforms), their integration into the teaching and research tasks of the academic workforce and the impact that the virtual environment may have on students' learning and assessment. In this sense, initiatives to strengthen the training of teachers and students in the appropriate use of new technologies and virtual teaching–learning environments must be reinforced. Indeed, immersion in the new teaching methodologies demanded by the new technological environments can lead to a paradigm shift in the teaching–learning process, affecting both the development policies of educational systems and the quality of the learning content taught, as well as students' satisfaction.

This book focuses on key factors regarding the sustainability of traditional education systems complemented by new virtual learning environments. Under this perspective, it raises important research questions in different scientific fields. For instance, concerning higher education institutions and their quality systems, is it possible to ensure the planned SDGs, are the opinions of all stakeholders involved in the teaching–learning process being taken into account, and can the educational dimension be strengthened in disadvantaged social environments to promote citizenship, employability and reduce poverty? As far as the new teaching technologies to be incorporated into educational systems and their impact on learning are concerned, are there clear guidelines to identify teaching technologies capable of promoting quality virtual learning, in which absenteeism and dropouts are minimized, are these new tools capable of adequately motivating and satisfying students, providing them with a constant updating of knowledge, and are they able to avoid undesirable aspects such as lack of teacher–student interaction, poor technical and instructional support, or insufficient collaboration for problem solving/group projects? Do they allow inclusive practices for students with some kind of disability? Can mobile devices act as a catalyst for improvement in specific learning processes such as languages? All these questions, and others, show the diversity and complexity of potentialities and problems that may occur in the ever-changing and unpredictable near future.

**Citation:** Bernardo, A.B.;

Castro-Lopez, A.; Almeida, L.S.; Fernandez-Rivas, S. Challenges of Higher Education in Turbulent Environments. *Sustainability* **2023**, *15*, 2965. https://doi.org/10.3390/ su15042965

Received: 31 January 2023 Accepted: 2 February 2023 Published: 6 February 2023

**Copyright:** © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/license s/by/4.0/).

In brief, this Special Issue of *Sustainability* entitled "Ensuring Quality Education and Good Learning Environments for Students" consists of a compendium of 14 articles that provide interesting reflections on the above issues and raise the need to monitor the positive and negative factors observed in the different empirical studies carried out on sustainability in the educational environment.

This volume is an attempt to enhance the research contribution arising from the challenges faced by the current higher education system.

**Author Contributions:** All the authors have contributed equally to this paper. All authors have read and agreed to the published version of the manuscript

**Funding:** This research received no external funding

**Conflicts of Interest:** The authors declare no conflict of interest.

**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

## *Article* **Effect of Positive Parenting Styles as Perceived by Middle School Students on Academic Achievement and the Mediation Effect of Self-Esteem and Academic Engagement**

**Jhong Yun (Joy) Kim <sup>1</sup> and Eunbee Kim 2,\***

	- Hwaseong 18330, Korea

**Abstract:** This study aimed to identify how positive parenting styles as perceived by middle school students affect their academic achievement and to assess the mediation effect of self-esteem and academic engagement. Data concerning 2590 middle school first graders were obtained from the South Korea Children and Youth Panel Survey 2018, and data analysis was undertaken using the SPSS21.0, AMOS22.0, and PROCESS macro programs. This study identified significantly positive effects of positive parenting styles and academic engagement on self-esteem, and of academic engagement on academic achievement. Furthermore, self-esteem and academic engagement had a mediating effect on the relation between positive parenting styles and academic achievement. Therefore, educators should develop programs that improve positive parenting styles, and specific programs for middle school students to enhance their academic engagement and self-esteem.

**Keywords:** middle school students; parenting style; self-esteem; academic achievement; academic engagement; social emotions; performance

#### **1. Introduction**

Academic achievement is highly related to not only personal factors such as selfidentity development but also environmental factors. Various studies have dealt with relevant family aspects in this respect, especially the role of parents. Kim [1], Park [2], and Yoon [3] examined personal factors related to parents' involvement and showed that parenting styles or personal tendencies, such as parental support, can affect academic performance. Further, Park and Hyun [4] examined parents' relationship with academic achievement using self-encouragement as a variable. The results of these studies showed that parents' support for their children's academic careers improved the children's academic performance and achievement.

However, close relationships with children may not always or necessarily have a positive effect on their academic performance and achievement. It has been claimed that excessive intervention could negatively affect academic performance [5]. Additional research is needed to clarify this matter. While one study [6] examined the correlation between academic achievement and self-identity development, that study was limited as it was conducted on middle school students in only one area in South Korea (Korea hereafter). Therefore, the current study sought to enhance the generalizability of its results by using country-level data on Korean children and youth panel data for analysis.

Given that prior studies have shown that individuals' sustainability orientation, sustainable effort, economic environment, and parental support affect children's academic performance, this study considered the aspects of both their learning environments and their personalities. In particular, this study focused on the students' self-esteem and parents' parenting style. In other words, academic achievement can vary depending on the relationship between parenting style and students' self-esteem [7,8].

**Citation:** Kim, J.Y.; Kim, E. Effect of Positive Parenting Styles as Perceived by Middle School Students on Academic Achievement and the Mediation Effect of Self-Esteem and Academic Engagement. *Sustainability* **2021**, *13*, 13233. https://doi.org/ 10.3390/su132313233

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Javier Puente and Leandro Almeida

Received: 1 November 2021 Accepted: 22 November 2021 Published: 29 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

This study thus aimed to determine the effects of parenting style, self-esteem, and academic engagement on academic achievement as perceived by middle school students, using data obtained from the Korean Children and Youth Panel Survey (KCYPS). It also attempted to identify more precisely how certain variables affect academic achievement.

Therefore, the authors propose positive parenting attitudes and high self-esteem as prerequisites for improving academic achievement, which will reduce the tendency of managers to leave. Based on the above study, the following hypothesis was proposed to clarify the impact of the environment and propensity according to academic achievement.

**Hypothesis 1 (H1).** *The positive parenting method perceived by middle school students will related to academic achievement, self-esteem, and academic participation.*

**Hypothesis 2 (H2).** *Self-esteem and academic participation among middle school students will have a mediation effect on the relationship between perceived positive parenting styles and academic achievement.*

#### **2. Theoretical Background**

#### *2.1. Parenting Styles and Academic Achievement*

As family households are generally where humans begin their lives and develop their first human relationships, such places provide a foundation for character and personality formation. In particular, the relationship between parents and children is the most important factor in the latter's growth and development as adolescents. However, definitions of parental attitudes have varied in different studies. Parental attitude is defined as general attitudes and behavior a parent adopts toward a child (Kim and Jang, 2016) [9] Parental attitudes are very important, which is because parents' standards for morals and behaviors are expressed in communication with their children and the children apply the standards to their development [2]. In particular, parents or teachers who guide students at school are interested in the relationship between parentings styles and academic achievement as a major concern for students and children. Parenting styles have been proven to be a strong predictor of academic achievement in previous studies [2,7,10], but the association between academic achievement and positive parenting styles in Korean educational environment and culture remains controversial [5,6]. Therefore, it is necessary to analyze the relation between parentings styles and academic achievement using data from a survey targeting all middle school students rather than specific cases through qualitative research. Although how specific factors relate to each other has been investigated, how academic achievement, self-esteem, learning enthusiasm, and academic achievement are linked to each other is not comprehensively clear yet.

Park (1996) first theorized parental attitudes and classify them into two dimensions: (a) acceptance or rejection and (b) domination or obedience. Reference [11] also classified parental attitudes into two dimensions: (a) affection or hostility and (b) autonomy or control. According to Schaefer, affection refers to parents' positive evaluation of children, emotional expression, and attitudes; hostility refers to attitudes of ignoration or rejection; autonomy refers to acceptability; and control refers to anxiety, protection, coercion, desire to achieve, and interest in health [2]. References [12,13] considered that the sub-factors of parental attitudes involved promise-making, authoritarian, democratic, equal-treatment, and neglectful aspects, while references [14,15] categorized parental attitudes into three dimensions: (a) warmth and hostility, (b) restriction and tolerance, and (c) overprotection and neglect.

According Heo (2012) [16], parenting not only determines the quality of the parentchild relationship, but also has a significant effect on a child's psychological characteristics, which in turn affects the child's intellectual development. These findings are supported by reference [17]. Reference [18] reported a significant correlation between academic performance and school adaptation in a long-term study on the effects of family environment and parent-child relations on children's academic performance. Specifically, they found

that the more children experienced rejection from their parents, the lower were their levels of academic performance and school adaptation. Epstein and Radin claimed that children from families where the father's parenting style was democratic and intimate were motivated positively in terms of academic achievement and school adaptation, whereas children whose fathers frequently punished or restricted them responded negatively to school life and had reduced motivation for achievement. These studies indicated that positive parenting attitudes are closely related to and influence all aspects of children's learning activities, cognitive and affective behavior, and school life adaptation. Accordingly, an empirical study to further clarify the precise effects involved is likely to be useful.

#### *2.2. Mediating Role of Self-Esteem and Academic Engagement in the Relation between Positive Parenting Styles and Academic Achievement*

No matter how motivated learners are to participate in learning, they tend to continue their participation selectively depending on their academic engagement [3]. Therefore, academic engagement is an important element in examining successful school life [19] and continuity in learning participation [20]. Self-esteem as a concept has also been used to evaluate individuals' perception of their own value, and reference [21] defined self-esteem as either a positive or negative attitude toward oneself. Conceptualizing one's value may not only relate to academic achievement but also affect other types of behavior among middle school students. According to previous studies, self-esteem has been reported to be a variable affecting academic achievement or adjustment at school [14,22,23]. Further, students with high self-esteem were shown to have high level of adjustment at school and academic achievement, whereas students with low self-esteem had high levels of anxiety and abnormal behavior. Students who had a positive view of themselves and a firm belief in their abilities realized more outstanding academic achievements [2].

Families' functional characteristics such as parenting attitudes and parenting beliefs have a stronger effect on adolescents than their structural characteristics, such as socioeconomic status and family composition [24]. Parent-child relationships are important [25–27] because children learn and imitate their parents' values, lifestyles, and behaviors through interaction with them. During this process, adolescents develop their values and personalities, self-concepts such as self-respect, and social skills such as social-norm learning [28,29].

Parents' parenting attitudes affect their children's school life adaptation, such as their relationships with peers and teachers as well as attitudes toward schoolwork, in varying degrees [28,30]. Family-related variables, including parents, were generally identified and discussed in studies on school life adaptation in Korea [15,17,28]; among these variables, parental attitudes were consistently reported to affect school life adaptation [31]. Studies found that the more affectionate parental attitudes are [27,32,33], the more receptive [33], democratic [34], and reasonable [35] are the parents, and the more likely it is that there will be a positive effect on adolescents' adaptation and academic achievement.

Negative parenting attitudes, such as parental abuse and neglect, negatively affect adolescents' school life adaptation and learning enthusiasm [36]. Parenting is also an important factor influencing adolescent self-esteem [36–38]. Self-esteem is formed through social interactions involving recognition, respect, acceptance, and evaluation of values imparted by important others, as well as successful experiences as individuals [38,39]. The family setting is generally the primary environment for individual development, with parents most likely to have interacted with adolescents for the longest time, that is, since infancy. Therefore, the effect of parents on the development of self-esteem in adolescents is likely to be significant [40] and can be mutually reinforcing, because positive adolescent self-esteem will affect parental self-esteem positively, creating a positive feedback cycle [34].

#### **3. Methods**

#### *3.1. Research Model*

This study aimed to determine whether self-esteem and academic engagement have a mediating effect on the relationship between positive parenting styles and academic achievement among middle school students. For this purpose, a research model was constructed, as shown in Figure 1.

**Figure 1.** Research Model 1.

#### *3.2. Participants*

This study was undertaken using data concerning 2590 middle school first graders in Korea that was obtained from the KCYPS 2018 conducted by the National Youth Policy Institute. KCYPS 2018 was designed to establish data that can show complex changes regarding the growth and development of children and youth from a systemic and multidimensional perspective with support from the Ministry of Education, Korea. This panel survey was reviewed by the institutional review board of the National Youth Policy Institute and executed using the tablet-assisted personal interview method for multi-stage stratified sampled middle school first graders.

Among the 2590 participants in the survey, 1409 (54.2%) were male and 1185 were female (45.8%). Of these, 1169 (45.1%) students attended middle school in metropolises, 1054 (40.7%) attended middle school in small or medium-sized cities, and 367 (14.2%) attended middle school in the rural areas of Korea. %). In addition, the average age of the participants was 13 years old (Mage = 13.052).

#### *3.3. Measures*

Within the panel data of KCYPS 2018, the questionnaire responses on positive parenting styles, self-esteem, academic engagement, and academic achievement of middle school first-grade students were used for this study. To measure positive parenting styles, the Korean version of the Parents as Social Context Questionnaire for Adolescents developed by Kim and Lee [39] was used. This scale consisted of six dimensions: warmth, autonomy support, structure, rejection, coercion, and chaos. Data from a total of 12 items within the warmth, autonomy support, and structure dimensions were used to analyze positive parenting styles in this study. In Kim and Lee's [39] research, the Cronbach's α were as follows: warmth, 0.882; autonomy support, 0.836; and structure, 0.766, all of which were over 0.6. The sample items include "My parents show me that they love me" and "My parents like to be with me."

Self-esteem was measured using Rogenberg's [23] self-esteem scale translated by the National Youth Policy Institute. Participants were asked to estimate their self-esteem through items such as "I feel that I have a number of good qualities," "I am able to do things as well as most other people," and "I take a positive attitude toward myself."

To measure academic engagement, the Korean Academic Engagement Inventory developed by the Lee and Lee [41] was used. This scale consisted of 16 items: four items for each of the four dimensions of vigor, dedication, esteem, and immersion. Lee and Lee's [41] results showed that the eigen values of all items were over 0.5. The sample questions to measure academic engagement include "I'm good at studying," "I am confident in my studies," and "I become energetic while I'm studying."

The questionnaire for academic achievement consisted of two items—the subjective academic achievement level and academic achievement satisfaction level—as developed by the National Youth Policy Institute [42–44].

#### *3.4. Data Analysis and Research Procedures*

Data analysis was performed using the SPSS21.0, AMOS22.0, and PROCESS macro programs. The analysis proceeded as follows. First, a frequency analysis was conducted on the independent variables, dependent variables, and mediator variables. Also, measuring Cronbach's α was conducted to confirm the variables' reliability. Second, a confirmatory factor analysis was conducted to confirm the research model's validity. Third, a correlation analysis was conducted to identify the direction of relevance and relationships among the variables. Fourth, a research model and an alternative model were constructed and compared to identify which model was appropriate to use for analysis. Fifth, a structural equation was used to conduct covariance structural analysis, confirm whether there are any influencing effects between the study variables; and confirm the research model's goodness of fit. Sixth, mediator analysis was undertaken for verification using the PROCESS macro. According to Lee [41–44], the PROCESS macro is a useful analytic method for verifying both the mediation and control effects as it does not require separate procedures using regression analysis such as those used by Baron and Kenny [12], or the Sobel test. The PROCESS macro uses bootstrapping to verify mediation effects where the basic number of samples has been set to at least 2000, and it determines the resulting values in a single analysis.

#### **4. Results**

To enhance the measuring model's simplicity and accuracy, three sub-categories were determined concerning self-esteem through data parceling, while three and two sub-categories were determined for each of them, respectively, based on the theoretical background. As shown in Figure 2, confirmatory factor analysis was conducted on the research model. The analysis confirmed the model's goodness of fit as well as that the measuring and potential variables were well-organized, as shown in Table 1. Specifically, the fit indexes are acceptable when the comparative fit index (CFI) ≥ 0.9, normed fit index (NFI) ≥ 0.9, Tucker-Lewis Index (TLI) ≥ 0.9, and root mean square error of approximation (RMSEA) ≤ 0.1; as the measuring model had fit index values of CFI = 0.930, NFI = 0.927, TLI = 0.902, and RMSEA = 0.088, it found to be fit for this study [12,45]. The regression weight of variables in Confirmatory factor analysis model was found in Table 2.

**Table 1.** Goodness-of-Fit Indexes for Structural Equation Model.



**Table 2.** Regression Weight of Variables in CFA.

**Figure 2.** Confirmative Factor Analysis.

The means, standard deviations, Cronbach's α values, and correlations of the study variables are shown in Table 3. All Cronbach's α values were over 0.6, and all the variables in the model had a direct relationship with one another. Correlation analysis confirmed that there was a significantly positive relationship between positive parenting styles, self-esteem, academic engagement, and academic achievement.

**Table 3.** Number, Means, Standard Deviations, Cronbach's α, Correlations, and Reliabilities of Study Variables.


Each model's goodness of fit was compared through the comparison between a fullmediation model as the alternative model and a partial-mediation model as the research

model, in accordance with Kelloway [11]. The partial-mediation model was constructed in relation to Hypothesis 2 (H2): Self-esteem and academic participation among middle school students will have a mediation effect on the relationship between perceived positive parenting styles and academic achievement (see Figure 3). The fit indexes of the partial-mediation model showed significance: χ<sup>2</sup> = 1079.401 (degrees of freedom [df] = 45, *p* < 0.001), with CFI = 0.920, NFI = 0.917, TLI = 0.900, and RMSEA = 0.094, as shown in Table 4.

#### **Figure 3.** Partial-Mediation Model.

**Table 4.** Goodness-of-Fit Indexes for Structural Equation Model.


In the alternative full-mediation model, the arrow indicating a direct relationship between positive parenting styles and academic achievement was deleted (see Figure 4). The fit indexes of the full-mediation model were significant: χ<sup>2</sup> = 1080.407 (df = 46, *p* < 0.001) with CFI = 0.920, NFI = 0.917, TLI = 0.902, and RMSEA = 0.093, as shown in Table 4. Compared with the partial-mediation model, the full-mediation model was found to have an equivalent goodness of fit.

Next, a model comparison was conducted based on the difference in values between the alternative and research models. The result showed that the difference between the χ<sup>2</sup> was 1.00, which was smaller than the threshold of 3.84 with the difference in df of 1. Therefore, the difference in χ<sup>2</sup> was not significant. When comparing the values and df of the two models, model verification did not reveal significant results. In this case, the model with higher df tends to be selected [11]. In the comparison between the full-mediation and partial-mediation models, the full-mediation model with a higher df was selected. In other words, the full-mediation model was determined to be more appropriate than the partial-mediation model through comparison of the goodness of fit and difference in χ2. The partial-mediation model (research model) is presented in Figure 3, and the full-mediation model (alternative model) is presented in Figure 4.

**Figure 4.** Full-Mediation Model.

For answering Hypothesis 1 (H1), a structural equation was made and covariance structural analysis was conducted between the study variables. To analyze the relationships among the major variables in this study, the standardization route (β), standard error, and t- and *p*-values were reviewed in the selected research model. The analysis results are presented in Table 5. Positive parenting styles had a significant positive effect on selfesteem (β = 0.581, *p* < 0.001). The relationship between positive parenting styles and academic engagement was significant (β = 0.156, *p* < 0.001), and the effect of self-esteem on academic engagement was significantly positive (β = 0.446, *p* < 0.001). In addition, the effect of academic engagement on academic achievement was significantly positive (β = 0.483, *p* < 0.001). On the other hand, the relationship between self-esteem and academic achievement was not significant.



\*\*\* *p* < 0.001.

The total effect, direct effect, and indirect effect values among the variables for the partial-mediation model are shown in Table 6. Since the full-mediation model was determined to be more appropriate than the partial-mediation model through comparison of the goodness of fit and difference in χ2, the indirect effect of positive parenting styles on academic achievement represents 100% of the total effect. The direct effect of the positive parenting style on academic achievement was found to be insignificant, and the indirect effect showed a larger value than the total effect.

To answer research Hypothesis 2 (H2) more clearly, model number 6 among the models in the PROCESS macro [46] was applied for analysis. The results showed that the indirect effects of positive parenting style as an independent variable, self-esteem and academic engagement as mediating variables, and academic achievement as a dependent variable had the value of 0.3469, as shown in Table 7. Since this value at the 95% confidence level was not 0, the mediation effect was significant.


**Table 6.** The effect decompositions of partial mediation model.

**Table 7.** PROCESS macro Analysis Results.


#### **5. Discussion**

This study aimed to identify the effects of parenting style, self-esteem, and academic engagement on academic achievement among 2590 middle school first graders in Korea, using data obtained from the KCYPS 2018, and to verify the mediation effect of self-esteem and academic engagement in the relation between positive parenting styles and academic achievement.

First, positive parenting styles as perceived by middle school students were found to have a statistically significant and positive relationship with academic achievement, and self-esteem and academic engagement mediated this relationship rather than exercising a direct effect. Specifically, the effect of positive parenting styles as perceived by middle school students on self-esteem was statistically significantly positive, as was the effect of self-esteem on academic engagement. Academic engagement had a significant positive effect on academic achievement, but no significant differences were found in the relation between self-esteem and academic achievement.

These results differed from the results obtained by references [46–48], which showed that self-esteem was directly related to academic achievement and also correlated with academic self-efficacy among first graders in middle school. In this study, self-esteem did not have a directly relation with academic achievement; however, the higher the level of self-esteem among middle school students, the more likely it was for them to positively evaluate themselves, which reinforced high levels of self-esteem.

Second, self-esteem and academic engagement were found to mediate the effect of positive parenting styles on academic achievement. According to references [49,50], the higher the level of self-esteem, the more likely it is for students to have improved learning motivation and achievement, although they could not identify any mediating variable. Further research was thus required to clarify the effects of various variables in relation to positive parenting styles. This study's results support the need to develop specific programs to improve self-esteem and academic engagement, as these variables are closely related to academic achievement among middle school students.

Third, the indirect effect of positive parenting styles on academic engagement involved 62.4% of the total effect (β = 0.415), whereas the indirect effect of positive parenting styles on academic achievement was 100% of the total effect (β = 0.220). The indirect effect of self-esteem on academic achievement involved 86.7% of the total effect (β = 0.248). These results help explain why the variables of positive parenting styles and academic achievement may not be directly related, although positive parenting styles improve self-esteem levels, and self-esteem enhances the academic engagement of middle school students. From this perspective, there is a need to determine how positive parenting styles can be fostered, and to help middle school students improve their self-esteem and academic engagement through consultation with suitably qualified career advice consultants. This will ensure that the students advance successfully to college; are able

to confront experiences such as failure, adversity, and psychological pain; and remain focused on academic achievement. Furthermore, appropriate institutional support, such as administrative and financial assistance from the Korean Department of Education and the Office of Education on Classroom Teachers, may be necessary for those offering career advice in middle school [51,52].

Based on this study's findings, the following recommendations are proposed to improve the academic performance of middle school students. First, it is necessary to supplement and expand parental education in a realistic way to aid the development of adults as parents. Second, it is necessary to strengthen family programs and family friendly systems. If parents and children do not have enough time to spend together, they will have fewer opportunities to interact positively. Third, it is necessary to initiate and diversify programs to increase self-esteem among teenagers. Self-esteem has been shown to have a positive effect on academic performance and a significant effect on the subsequent lives of children. Therefore, institutional support is needed, as noted, to develop and revitalize various activities that can increase teenagers' self-esteem. Thus, self-esteem improvement programs are needed that allow participation by parents and adolescents and that are available for implementation by adolescent activity-related institutions; this is especially true given that prior studies clearly show that parents' positive parenting styles affect adolescents' self-esteem [53,54].

This study contributes to the literature by showing that parents' positive parenting styles sustainably affect middle school students' academic performance, and that academic engagement and self-esteem are mediating factors in this relationship.

#### **6. Limitations**

Nevertheless, this study had the following limitations: First, the questionnaire measuring the network variables was not sophisticated, which made it difficult to measure multiple questions grouped together. In other words, there was a limitation in terms of accurately measuring the responses to the questionnaires asking about positive parenting styles, Self Esteem, Academic engagement etc. Specifically, it did not reflect the respondents' propensity to pursue both functions because they had to choose one of the two questionnaires measuring their perceived parenting style.

Second, it focused on only middle school students. Further studies are needed that consider high school or elementary school students to determine whether this study's findings are applicable at these levels. Third, this study recognized that schools are important environmental systems that provide services to teenagers for their growth and development. However, it failed to consider the extent to which the schools' capacity to identify and address the needs of adolescents may be affected by their level and quality of service organization. Subsequent studies will need to consider schools as structural and environmental variables and assess the extent to which schools as service providers for all adolescents may positively affect the psychological and physical development of adolescents.

**Author Contributions:** J.Y.K. (formal analysis, methodology, software, investigation, resources, data curation, writing—original draft preparation, visualization, project administration); E.K. (Conceptualization, validation, writing—original draft preparation, writing—review & editing, visualization, project administration). All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the Hyupsung University Research Grant of 2021.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Brief Report* **Enhancing Final-Year Medical Students' Clinical Examinations Performance via a Transdiagnostic ACT-Based Intervention**

**Nicholas Tze Ping Pang 1, Eugene Koh 2,\*, Mohd Amiruddin Mohd Kassim 1, Wendy Diana Shoesmith 1, Assis Kamu 1, Chong Mun Ho 1, Jun Rong Ng <sup>3</sup> and Walton Wider <sup>4</sup>**


**Abstract: Introduction**: Performance anxiety is a transdiagnostic construct similar in both sports and clinical medical examinations. Given that acceptance and commitment therapy (ACT) and other similar therapies have transdiagnostic evidence in improving the performance of athletes, the same approach can be adopted for medical students in improving their performance during clinical examinations. This pilot study aimed to assess the efficacy of a brief ACT-based intervention in improving transdiagnostic performance anxiety in clinical medical students through assessing changes in psychopathology scores (depression, anxiety, and stress) and psychological process variables (psychological flexibility, cognitive fusion, and mindfulness). **Methods**: Final-year medical students were randomized into intervention and control groups. A one-day ACT-based intervention was delivered to an intervention group, with the control group crossing over one month post intervention. Both groups filled in sociodemographic questionnaires and questionnaires measuring psychological flexibility, cognitive fusion, mindfulness, depression, anxiety, and stress at three time points: T1 (before intervention), T2 (immediately after intervention), and T3 (one month post intervention). Repeated measures ANOVA was employed to assess the change between the intervention and control groups over time. **Results and Conclusions**: There was a significant change in anxiety scores from T1 to T3. In addition, there were significant improvements in mindfulness, cognitive fusion, and psychological flexibility scores over time in the intervention group compared with the control group. This pilot study builds on small single-sample evidence bases for the efficacy of an ACT-based intervention in non-sports performance enhancement, suggesting that larger-scale randomized trials of similar interventions in clinical medical students may prove equally efficacious.

**Keywords:** performance enhancement; performance anxiety; psychological flexibility; ACT; transdiagnostic approach; transdiagnostic assessment; transdiagnostic interventions; transdiagnostic dimensions; psychopathology; mental health; health conditions; emotional disorders; implementation; dissemination; cost-effectiveness; prevention; evidence-based psychological interventions

#### **1. Introduction**

Mindfulness-based therapies have increasingly gained currency as a panacea for treating multiple disorders across the mood and neurotic spectrum. In the specific group of university student populations, which overlaps with the age of presentation of both clusters of disorders, there is evidence both for individual [1] and group mindfulness [2] for both psychological distress and academic results [3,4]. However, mindfulness-based therapies can be employed as experiential avoidance tools if not coupled with effective

**Citation:** Pang, N.T.P.; Koh, E.; Mohd Kassim, M.A.; Shoesmith, W.D.; Kamu, A.; Ho, C.M.; Ng, J.R.; Wider, W. Enhancing Final-Year Medical Students' Clinical Examinations Performance via a Transdiagnostic ACT-Based Intervention. *Sustainability* **2022**, *14*, 7522. https://doi.org/10.3390/su14137522

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Leandro Almeida, Silvia Fernández Rivas and Jesús-Nicasio García-Sánchez

Received: 3 April 2022 Accepted: 1 June 2022 Published: 21 June 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

action and behavior change. ACT (Acceptance and Commitment Therapy) is an offshoot of pure mindfulness focusing on two aspects: acceptance (mindfulness techniques) coupled with committed action (identification of values and goals and plans in line with them) [5]. It is transdiagnostic and has been used for various conditions, including depression, anxiety, and substance use [6–8].

An exciting new field of interventions is in the field of performance enhancement. There are existing interventions that share similar concepts with ACT per se, including the Mindfulness Acceptance and Commitment (MAC) protocol [9] for increasing "flow" [10] and the Mindfulness Sports Performance Enhancement (MSPE) protocol [11,12]. There is substantive evidence that such transdiagnostic mindfulness-based therapies are efficacious, even for one intervention, in terms of boosting performance by increasing psychological flexibility [13]. Some of the fields that have benefited from this include springboard diving [14], tennis [15], basketball [16], and weightlifting [17]. Apart from sports, military life has been associated with benefits from mindfulness in the sense that perceived stress on hopelessness among soldiers is inversely proportional to the level of mindfulness [18]. A recent study also demonstrated that subjective well-being and self-compassion are positively associated with mindfulness [19].

Nonetheless, there is little research on using ACT-based methods to tackle the specific issue of performance anxiety in a clinical performance setting. This research project proposes using a transdiagnostic ACT-based approach to tackle performance anxiety in clinical examinations for medical students. Clinical medical examinations generally do not merely assess discrete knowledge or skills per se, but rather are very cross-sectional assessments that reward individuals who have the ability to remain calm under pressure [20]. Hence, there is a very large component of performance anxiety that can impair the performance of a previously reasonable candidate. A key part of this is intervening in a way that can increase psychological flexibility [21].

Psychological flexibility is defined as the "ability to contact the present moment more fully as a conscious human being, and to change or persist in behavior when doing so serves valued ends" [22]. This would and should be able to improve performance in medical students, as they are able to do a few things better after attending. Firstly, they focus on the goals and values that drove them to study medicine and identify "away moves" that lead them away from their desired values [23]. Then, they learn the necessary skills of acceptance and being present to help them manage their responses to anxiety rather than pushing it away or internalizing it unproductively. An athlete receiving ACT learns the skills of mindfulness, cognitive defusion, acceptance of difficult internal experiences, and values clarification [9]. These skills will then further enhance behavior changes that facilitate performance enhancement.

The research question is hence whether ACT can improve psychological flexibility and reduce markers of psychopathology in medical students undergoing clinical-based performances. The objective of this study was to assess the efficacy of the ACT-based intervention in improving performance in clinical medical examinations. Thus, the hypothesis was that an ACT-based intervention would result in significant improvements in psychological flexibility, state mindfulness, and cognitive failures, as well as significant decreases in psychopathology, namely depression, anxiety, and stress.

#### **2. Materials and Methods**

#### *2.1. Intervention and Instructional Program Followed*

A one-day ACT-based intervention tailored to clinical medical students was created by adopting methods described in the Mindfulness Acceptance Commitment model and condensing the methods into the triflex model described by Russ Harris. Author EK also adjusted the model on the basis of his clinical experience as an ACT practitioner to ensure that there was a continual coherence between each session within the module. There was a total of 5 sessions during the one-day intervention, comprising of an introduction, being present, being open, being committed, and a closure session added toward the end to encourage students to exchange experiences and skills for performance. This session was added to ensure that the other sessions prior to that were absorbed and utilized. Because of pandemic travel restrictions, the intervention was delivered by EK online while the students were grouped together in the same room. Authors NP and MAMK, who were both experienced in ACT, remained physically in the group to assist and provide feedback to students, which was consistent with ACT principles. Details of the program are explained as per Table 1.

**Table 1.** Contents of the Program.


#### *2.2. Participants*

The study was performed in the medical faculty of a public university on Borneo island, which accepts students on the basis of merit in a nationwide examination and a battery of interviews and psychometric testing. To meet the inclusion criteria, participants in both groups were required to be final-year medical students who were experiencing anxiety about performing clinical examinations and volunteered to participate in the oneday intervention. The exclusion criteria were acute medical or psychiatric illness and non-consent. All medical students were in the same phase of their program and were three months from sitting for the major exit examination (Professional II examination); hence, each participant had equal representativeness. Convenience sampling was employed.

#### *2.3. Procedure and Intervention Design*

The number of experimental groups was two—an intervention and a control group which students were randomized into. The control group crossed over to the intervention after one month. Pre–post monitoring measures were as follows: the intervention group (Group I) completed the questionnaires at three time points: T1 (prior to the intervention as a baseline), T2 (immediately after the intervention), and T3 (one month post intervention). The control group (Group C) completed the questionnaires at three time points too, similar to the intervention group: T1 and T2 (at the same time as the intervention group) and T3 (one month later, prior to their cross-over intervention).

#### *2.4. Variables and Instruments Used*

Both groups were given a set of questionnaires to complete at certain predetermined time points, which measured psychological process variables, namely psychological flexibility, state mindfulness, self-reported failures in perception, memory, and motor function, as well as psychopathology, focusing on depression, anxiety, and stress scores. For psychological flexibility, the AAQ-II was used. For state mindfulness, the Mindfulness Attention and Awareness Scale was used. For self-reported failures in perception, memory, and motor function, the Cognitive Failures Questionnaire (CFQ) was used. The DASS-21 was used to measure depression, anxiety, and stress. All measurements utilized English scales and all instruments used are included in Supplementary Materials in this study.

The DASS-21 questionnaire is a set of three self-report scales measuring the emotional states of depression, anxiety, and stress [24]. Respondents were requested to complete the DASS-21 on the basis of the presence of a symptom over the previous week. They rated each item on a four-point Likert scale ranging from 0 (did not apply to me at all over the last week) to 3 (applied to me very much or most of the time over the past week)*,* with higher scores indicating greater severity. It is a very widely used composite measure of the three psychopathologies measured and is routinely used in clinical practice as well as research. The internal consistency was between 0.85 and 0.89 in this study, while the average variance extracted was between 0.46 and 0.55. Furthermore, discriminant validity was achieved with the square root of AVE ranging from 0.82 to 0.86.

The Cognitive Failures Questionnaire (CFQ) is a 25-question instrument designed to represent cognitive errors in daily life [25]. These questions reflect different aspects of cognitive functioning, including attention ("Do you fail to notice signposts on the road?"), perception ("Do you fail to see what you want in a supermarket (although it's there)?"), memory ("Do you find you forget appointments?"), impulsivity ("Do you say something and accomplish afterward that it might be taken as insulting?"), and language ("Do you find yourself suddenly wondering whether you've used a word correctly?"). Each question is answered on a frequency scale, ranging from never (0 points) to very often (4 points). The total scale ranges from 0 to 100 points. The CFQ is one of the unique scales illustrating this dimension and is used primarily in research settings with robust psychometric properties, including high internal consistency and test–retest stability [26,27]. The internal consistency was 0.96 in this study, while the average variance extracted was 0.77. Furthermore, discriminant validity was achieved with a square root of AVE of 0.94.

The Acceptance and Action Questionnaire (AAQ-II) is one of the most widely accepted instruments in ACT studies and ascertains experiential avoidance and psychological inflexibility [23]. Experiential avoidance refers to the attempt of the individual to ignore unpleasant thoughts, feelings, and physical sensations, which leads to measures against one's values, causing continual harm [5]. Psychological inflexibility is defined as firm psychological action against one's value to avoid distress and uncomfortable feelings or thoughts, ignoring the present moment [22]. AAQ-II is a unidimensional scale with 7 items and is rated using a 7-point Likert scale. AAQ-II has good internal consistency (α = 0.88) and good test–retest reliability over 3 and 12 months at 0.81 and 0.79, respectively [28]. Higher AAQ-II scores reflected significant psychological inflexibility and have also been found to be associated with greater levels of depressive symptoms, anxiety and stress, thought suppression, and psychological distress. The internal consistency was 0.92 in this study, while the average variance extracted was 0.64. Furthermore, discriminant validity was achieved with a square root of AVE of 0.89.

The Mindful Attention Awareness Scale (MAAS) is a popularly used research instrument that measures the level of awareness and attention to the present-moment experience [29]. It consists of 15 items, and participants respond to each item using a six-point Likert scale ranging from 1 (almost always) to 6 (almost never) [30]. Higher scores indicate higher levels of mindfulness. The internal consistency was 0.94 in this study, while the average variance extracted was 0.53. Furthermore, discriminant validity was achieved with a square root of AVE of 0.85.

#### *2.5. Data Analysis*

Data were analyzed according to the intention-to-treat principle. All participants were analyzed according to the condition (Group I or Group C) to which they were initially randomized. Multiple imputation by the SPSS 26.0 software package was adopted for missing data. Data were analyzed with descriptive statistics, and analyses of normality were performed for all continuous data.

Repeated measures ANOVA was employed to analyze the repeated measurement data, as the normality analysis suggested a normally distributed data set. Mauchly's test of sphericity was calculated for all variables to establish whether there were significant differences between the variances of the differences, and if violated, Greenhouse–Geisser corrections were used if ε < 0.75, and Huyhn Feld corrections were used if ε > 0.75. Subsequently, the estimated marginal means for each measured parameter were plotted on a graph for three time points—pre-intervention (T1), post intervention (T2), and one month post intervention (T3).

This study was conducted according to the Helsinki Declaration and was approved by the Institutional Review Board of Universiti Malaysia Sabah (IRB serial number: 110606). Written informed consent was obtained from all the participants involved in this study. Data protection was ensured by utilizing paper forms, which were not stored electronically on any devices, and the paper forms were shredded at the end of the study after a period of two years.

#### **3. Results**

A total of 22 participants volunteered for the one-day intervention out of 83 students enrolled in the final year of the medical program. They were randomly divided into an intervention group and a control group of 11 students each. All continuous data of depression, anxiety, stress, CFQ, MAAS, and AAQ-II scores were normally distributed with skewness and kurtosis less than or equal to ±2, as per normality requirements [31].

For depression scores, Mauchly's test of sphericity indicated that the assumption of sphericity was violated, χ2(2) = 7.970, *p* = 0.019, epsilon > 0.75, and therefore, a Huynh– Feldt correction was used. There was, however, no significant effect of time\*group on the subjects, *F*(1.662, 33.248) = 9.898, *p* = 0.062. As Figure 1 suggests, the raw score trend was that there was a sustained decrease in scores for the intervention group compared with the control group, whereby the scores decreased marginally at T2 but increased at T3.

For anxiety scores, Mauchly's test of sphericity indicated that the assumption of sphericity was not violated, χ2(2) = 1.792, *p* = 0.408, epsilon > 0.75. There was a significant effect of time\*group on subjects, *F*(2, 2) = 5.109, *p* = 0.011. As Figure 1 suggests, the raw score trend was that there was a sustained decrease in scores for the intervention group compared with the control group, whereby the scores decreased marginally at T2 but increased at T3.

**Figure 1.** Estimated marginal means for psychopathology variables.

For stress, Mauchly's test of sphericity indicated that the assumption of sphericity was violated, χ2(2) = 6.243, *p* = 0.044, epsilon > 0.75, and therefore, a Huynh–Feldt correction was used. There was, however, no significant effect of time\*group on the subjects, *F*(1.756, 35.117) = 10.838, *p* = 0.104. As Figure 1 suggests, the raw scores showed that the intervention group had a drop in scores at T2 and then had a small increase at T3, which was still beneath the baseline scores at T1. This differed from the control group, in which there was an increase over time from T1 to T3.

For levels of mindfulness, Mauchly's test of sphericity indicated that the assumption of sphericity had not been violated, χ2(2) = 5.073, *p* = 0.079, epsilon > 0.75. There was a significant effect of time\*group on the subjects, *F*(2, 40) = 2.208, *p* = 0.003. As Figure 2 suggests, the mean scores increased from T1 to T3 in the intervention group, whereas in the control group, they decreased over the course of T1 to T3.

**Figure 2.** *Cont.*

**Figure 2.** Estimated marginal means for psychological process variables.

For psychological flexibility, Mauchly's test of sphericity indicated that the assumption of sphericity was violated, χ2(2) = 10.684, *p* = 0.005, epsilon < 0.75, and therefore, a Greenhouse–Geisser correction was used. There was a significant effect of time\*group on the subjects, *F*(1.398, 27.970) = 7.620, *p* = 0.005. As Figure 2 suggests, the scores decreased for the intervention group (indicating lower levels of psychological inflexibility) from T1 to T3. This differed from the control group, in which the scores increased from T1 to T3 (indicating higher levels of psychological inflexibility).

On measures of cognitive fusion, Mauchly's test of sphericity indicated that the assumption of sphericity was violated, χ2(2) = 7.118, *p* = 0.28, epsilon > 0.75, and therefore, a Huynh–Feldt correction was used. There was a significant effect of time\*group on the subjects, *F*(1.171, 34.125) = 10.327, *p* = 0.001. As Figure 2 suggests, cognitive fusion decreased for the intervention group from T1 to T3. In the control group, however, after an initial decrease from T1 to T2, it rose significantly after T3.

#### **4. Discussion**

This study answered the research question in that it showcased that ACT could improve psychological flexibility and reduce markers of psychopathology in medical students undergoing clinical-based performances. Specifically, the study demonstrated that the ACT-based intervention had beneficial effects on all three psychological process variables that were expected to respond—psychological flexibility, mindfulness, and cognitive fusion. There were also reductions in depression and stress scores; however, the effect of time\*group on both was not significant. However, for the main construct that the intervention was intended to deal with, namely anxiety, there were significant score reductions for the intervention group compared with the control group, with significant time\*group effects. These data thus add value to the literature, as the objective of the study was achieved in that we were able to ascertain the efficacy of an ACT-based intervention in improving performance on clinical medical examinations. The hypothesis was also confirmed in that there were three psychological process variables that demonstrated improvement; on the other hand, depression and stress demonstrated non-significant improvements, whereas anxiety demonstrated significant improvements.

From a theoretical standpoint, this study significantly boosts the existing evidence for ACT-based intervention in performance enhancement, as its sample size was larger than those of the previous single-sample studies focusing on performance enhancement in non-medical populations [10–13]. This is hence the first documented case in the literature looking at the efficacy of ACT-based intervention for medical student-related performance anxiety. In comparison with cognitive-based psychotherapies, which focus on Socratic questioning to reduce negative or irrational thoughts and hence curtail negative emotional responses, ACT approaches may be useful because they promote a more acceptance-based approach to what is essentially a natural response to a stressful situation.

In terms of its application for practice, as an online intervention, it is highly transferable and can hence be conducted with exponentially larger groups. The cohort of final-year medical students in this research study had their final examinations postponed two times, and this created a high level of anxiety due to the uncertainties of delayed employment, possible forestalled scholarships for university studies, and high financial implications for flights to hometowns because of examination cancellations. Hence, it is hoped the lessons obtained from the ACT-based intervention performance anxiety course can be translated to other anxiety-inducing situations in real life.

In comparison with previous studies, mindfulness approaches reduce anxiety amongst health profession students in general [1], with comparable efficacy for group mindfulness [2]. There have also been comparable studies that have found efficacy for mindfulness practices for academic results [3,4]. Consequently, it has been adopted widely for use in medical schools to improve psychological health and well-being, with the intended effect of reducing the student dropout rate [32]. This, however, was a unique result in that it showcased the efficacy of mindfulness specifically for the purpose of performance enhancement. Moreover, this is the first study that specifically used the ACT modality, which emphasizes committed action with value- and goal-setting on top of merely using conventional mindfulness or meditation practices.

This study has several limitations that limit the generalizability of the results. Firstly, the sample size of this study is rather small because of the low uptake rate by volunteers; a higher-powered study would ideally incorporate all 83 medical students in the same batch. However, as this was the first ACT-based performance enhancement intervention in a non-athlete group, it was felt that a pilot study would be best to redress any larger issues with the ACT-based intervention before proceeding to a larger-scale intervention. The other limitation was that we did not use students' actual grades on clinical examinations pre and post intervention. However, this would be inherently highly subject to bias, as all final-year students were undergoing one of four different clinical postings, which have inherently differing difficulty levels and high degrees of variability in end-of-posting assessment formats. For instance, in the surgery posting, one-third of students were reliably expected to fail, whereas for the pediatrics posting, all students were predicted to receive an A or A minus grade. Hence, falsely unobjective results would be obtained if the intervention group had gone from a difficult clinical posting to a comparatively easier one, which would have artificially boosted their post-intervention grade. As a consequence, the researchers decided to use objective markers of performance anxiety, e.g., DASS scores and psychological process variables, as these were unaffected by the wide variances in posting examinations.

#### **5. Conclusions**

In conclusion, this pilot study provides limited but preliminary evidence for the use of the ACT-based intervention as part of an online intervention that may be able to alleviate performance anxiety in clinical medical students. Such ACT skills are highly transferable across the career of a medical student, as doctoring, whether as a resident, a medical officer, or a specialist trainee, inherently implies participation in multiple assessments of one's clinical abilities, such as formalized examinations; workplace-based assessments; or by a jury of one's own peers, subordinates, and patients. It is hoped that a larger-scale fully randomized study involving a statistically significant population of clinical medical students can be employed to further assess the veracity of the significant findings obtained so far in this smaller-scale pilot study.

**Supplementary Materials:** The supporting information can be downloaded at: https://www.mdpi. com/article/10.3390/su14137522/s1. Refs. [23,25,33,34] are cited in Supplementary Materials.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** This study was approved by Institutional Review Board of Universiti Malaysia Sabah (IRB serial number: 110606).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Data available on request due to restrictions eg privacy or ethical.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Students' Perceptions of Active Learning Classrooms from an Informal Learning Perspective: Building a Full-Time Sustainable Learning Environment in Higher Education**

**Lei Peng 1,2, Shitao Jin 1,2,\*, Yuan Deng 1,2 and Yichen Gong <sup>3</sup>**


**Abstract:** Under the influence of education for sustainable development, active learning classrooms, as the new learning environment in higher education, have become increasingly diversified and flexible, with a greater emphasis on student experience and engagement. Most research on these learning environments is centered on formal learning analysis and discussion, whereas empirical research on the perception of informal learning in active learning classrooms outside of class time is lacking. Based on informal learning perceptions, this study explored the strengths and weaknesses of active learning classroom spatial environments using a mixed quantitative and qualitative research approach. Through a questionnaire survey of 154 students and one-on-one semi-structured interviews with 15 students, this study found that most students were satisfied with the informal learning experience in active learning classrooms, comfortable and flexible space perception and humanized learning support facilities in active learning classrooms were critical spatial factors influencing students' informal learning, the private environment and positive learning atmosphere in active learning classrooms could promote students' informal learning behaviors, and for active learning classrooms, better resource management could help them develop a better quality full-time learning environment. Based on these findings, this study makes recommendations for optimizing the environment design and management of active learning classrooms.

**Keywords:** sustainable education; higher education; learning environments; active learning classrooms; informal learning; student perceptions

#### **1. Introduction**

The concept and practice of education for sustainable development and learning environments are inextricably linked. As the physical entity of sustainable education implementation, learning environments carry the development of educational methods, learning concepts, resources and facilities, and cultural cognition [1]. As a dominant learning environment, the development of the classroom is inevitably influenced by numerous driving variables in sustainable education, such as personalized and adaptable learning capacity, learner perception and engagement, and the active learning pedagogical practices involved [2]. Along with the development of sustainable education, the innovation of learning concepts, and the emergence of new technologies, an increasing number of researchers have begun to promote the innovation and practice of classroom space, particularly in the United States, the United Kingdom, Australia, and some education developed countries [3–6], various more innovative and flexible types with an emphasis on active learning attaching great importance to the students' experience and participation in active learning classrooms have emerged.

**Citation:** Peng, L.; Jin, S.; Deng, Y.; Gong, Y. Students' Perceptions of Active Learning Classrooms from an Informal Learning Perspective: Building a Full-Time Sustainable Learning Environment in Higher Education. *Sustainability* **2022**, *14*, 8578. https://doi.org/10.3390/ su14148578

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Leandro Silva Almeida and Silvia Fernández Rivas

Received: 16 June 2022 Accepted: 8 July 2022 Published: 13 July 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Active learning classrooms are student-centered learning environments supported by digital technology and interactive information integration [7] which not only meet the dual needs of constructivism teaching implementation and innovative practice application [8] but also meet the needs of personalized and adaptive multi-oriented training [9]. There are three driving factors behind this. First, both the goals of sustainable education and the development of learning science theory emphasize the importance of active learning skills (e.g., concentration, teamwork, self-regulation, and social adaptation) in the cultivation of future higher education talent [10–12], while the spatial environments that support studentcentered and interdisciplinary collaboration are more conducive to the development of students' active learning skills [13]. Second, the deep integration of information technology and educational concepts enables generating and disseminating knowledge in novel ways. Active learning classrooms that mix technological devices and classroom activities blur the barriers between physical and virtual spaces, promoting a learning environment that is flexible in time and sustainable in space [14]. Third, the promotion of lifelong learning [15] and the establishment of the informal learning paradigm [16] have provided learners with a more diversified and complex personalized learning environment, requiring a richer continuum of learning spaces to meet learners' requirements.

As a new field of development, active learning classrooms are attracting many researchers and practitioners, and existing research focuses on the following two aspects: (1) the description and construction of educational concepts and technical facilities and the spatial design of active learning classrooms and (2) teaching effectiveness and classroom evaluation of active learning classrooms. However, active learning classrooms, as an important carrier of sustainable education development and an important part of the spatial continuum constructed by learning theory, still have gaps in the perception research and effect evaluation of extracurricular students' active knowledge construction, informal learning participation, and personalized learning activities. At present, the construction of active learning classrooms in various universities has increased significantly. For example, Auburn University has built more than 50 new active learning classrooms in 6 years [17], Sichuan University has renovated and built more than 400 active learning classrooms since 2012 [18], and Huazhong University of Science and Technology has also built and put into use 101 active learning classrooms from 2018 to now [19], and a large number of active learning classrooms have become one of the main spatial places for students' informal learning activities. Research on the perceptual experience of unstructured learning activities for such structured environments will contribute to the sustainable development of active learning classrooms by providing better quality opportunities for knowledge construction, collaborative exchange, and practical application to maximize support for high-quality and full-time learning. This study will concentrate on students' spatial perceptions of active learning classrooms from the perspective of informal learning and will utilize empirical research methods to investigate the impact of active learning classrooms on college students' experiences and perceptions of informal learning activities in order to improve the sustainability of active learning classrooms and provide a reference basis for optimizing the design and management of learning spaces. Based on this, this study proposes the following three research questions:


#### **2. Literature Review and Research Framework**

#### *2.1. Theoretical Research and Practical Exploration of Active Learning Classrooms*

Active learning classrooms are not isolated physical environments but rather a carrier and expression of the constructivist learning concept and learning space continuum, implying a shift from the traditional teacher-led paradigm of knowledge transfer to a student-led paradigm of construction and problem solving [6,20]. Furthermore, active learning classrooms should not be viewed simply as a new teaching environment, as teaching is simply one of its purposes. The learning environment also implies a shift toward sustainable learning activities, where learning does not only take place in a specific classroom but is a continuous and open process [21]. Active learning classroom environments allow for richer learning expressions, such as self-study, group activities, and workshops.

Theoretical research on active learning classrooms includes exploring design principles and the analysis of empirical evaluations. In terms of design principles, the pedagogy-spacetechnology framework developed by Radcliffe et al. [22] explains the leading theoretical principles of active learning classrooms. Pearshouse et al. [23] illustrated the relationship between learning environments and education through the why-what-what framework. In terms of empirical evaluation, it primarily investigates the practical impact of different new teaching techniques [24,25], advanced communication and information technologies [26,27], and redesigned innovative space forms used in active learning classrooms [6,28]. Simultaneously, organizations and national teams such as the UK Higher Education Funding Council [29], Australia's Next Generation Learning Environment Project [30], and the United States' National Learning Infrastructure Initiative (NLII) have systematically sorted out and established corresponding learning space evaluation guidelines.

The practical exploration of active learning classrooms began in the early 21st century with the Student-Centered Active Learning Environment for Undergraduate Program (SCALE-UP) [31] proposed by North Carolina State University, which formed the foundation for active learning classrooms with a student-centered teaching and learning environment [32]. MIT's Technology Enables Active Learning (TEAL) space [7] improves student cooperation and learning by utilizing more complex visual media simulations and personal response systems. The University of Minnesota introduced the Pedagogy-rich, Assess learning impact, Integrate innovations, Revisit emerging technologies (PAIR-up) active learning classroom [33] in 2006, increasing the flexibility of the space and allowing teachers and students to experience a new classroom design and variety of teaching strategies. The University of Iowa's Transform, Interact, Learn, Engage (TILE) classroom spaces [34] combine faculty instructional strategies with the design of classroom spaces and offer a greater diversity of technology equipment and classroom sizes. In 2015, Thomas Jefferson University built several active learning classrooms of various shapes and sizes, collectively known as "Nexus Learning Hubs", which offer a variety of furniture configurations and maximize each student's workspace, creating many collaborative group environments [35]. Auburn University designed and built more than 50 active learning classrooms, named Engaged, Active Student Learning (EASL), from 2011 to 2017. The university has specially set up management and maintenance departments for these classrooms and cultivated a number of diversified teaching staff suitable for these classrooms to encourage students' active participation [17]. Huazhong University of Science and Technology (HUST) commissioned 101 active learning classrooms in 2018, known as "smart classrooms", which incorporate technologies such as the Internet of Things (IoT), multi-screen interaction, and smart interactions, as well as more comfortable furniture and interior decoration to meet innovative teaching and learning developments [19].

In general, prior research has made many important attempts from many perspectives toward active learning classrooms, but there is still potential for exploration in two areas. First, most studies begin with the perspectives and expectations of designers and educators, with relatively little research on students' experiences and perceptions of actually using active learning classrooms, while research on students' perceptions of this new learning space based on their motivation and learning experiences is a critical element for the further development of active learning classrooms. Second, existing studies mainly focus on the research and evaluation of formal learning in active learning classrooms, whereas there is a relative lack of research on students' extracurricular learning activities in active learning classrooms. As for active learning classrooms, most of the time, they are open for

informal learning activities. How to make it a truly full-time sustainable education learning environment needs a more comprehensive perspective.

#### *2.2. Informal Learning and Its Environmental Impact*

Informal learning is an active learning behavior that is self-initiated, self-regulated, and self-responsible by learners outside of formal school education. Its knowledge comes from learning diversity, which is a kind of integrated learning with a social nature [36]. American adult educator Victora J. Marsick defines informal learning as a de-structured form of learning with a sense of active learning closely related to the surrounding environment and social conditions [16]. At present, research on informal learning mainly focuses on learners' experiences and perceptions, and its findings can visually reflect learners' preferences and effects on informal learning. Among this research, qualitative research methods include one-on-one structured interviews [37], focus group interviews [38], and behavioral logs [39], while quantitative research methods include questionnaires [40], behavioral measures [41], and environmental preference surveys [42], and mixed research methods include a combination of questionnaires and interviews [43,44], as well as a combination of delayed photography and focus group interviews [45]. In general, there are many examples in the literature in the study of informal learning that use qualitative or quantitative research methods, but there are few examples in the literature that use mixed research methods.

Unlike formal learning, informal learning emphasizes the initiative, diversity, and randomness of students' learning, so the design of its learning environment should also focus more on students' self-direction and multi-dimensional interactive experiences [46] in order to promote communication and cooperation, achievement demonstration, and learning participation in informal learning activities. Research on the design of informal learning environments can currently be summarized by the following four points: first is the comfort of the indoor environment, such as indoor lighting [38], ventilation, noise [37,47], temperature [48], materials, and color [49], second is a flexible spatial layout, such as flexibility of furniture [49], diversity of spatial layout, and the openness and privacy of the space [50], third is a positive spatial atmosphere, such as a good learning atmosphere [36], better spatial accessibility, rich spatial level, and humanized infrastructure [51], and fourth is modern electronic facilities, such as interactive whiteboards [52], wireless microphones, and interactive visual software. Because of the social character of informal learning in higher education, research at this stage has focused on public spaces on campus, such as libraries, student cafeterias, dormitory buildings, and other potential outdoor public places. However, with the emergence of active learning classrooms, their high-quality learning environment has attracted many students to use them for informal learning outside of the curriculum. Therefore, there is an urgent need to investigate the impact of the environmental design of this learning space on students' experiences and perceptions of informal learning.

To summarize, active learning classrooms, as a new generation of learning environment, can not only meet formal learning needs but also promote students' free and flexible informal learning, encourage students to think and solve problems independently, and cultivate students' sense of cooperation and social awareness. However, how to optimize and design "full-time learning catalysts" such as active learning classrooms with an informal learning perspective has become an important research issue in current and future sustainable learning environments in higher education.

#### *2.3. Research Framework*

Based on the above research, we can learn that the learning environment in higher education is one of the main factors affecting students' informal learning experiences and perceptions and that a better-quality learning environment can promote more active informal learning. This study focused on the learning environment of active learning classrooms to investigate the influential relationship between students' informal learning experiences and the perception of space in active learning classrooms. As shown in Figure 1, a blended research approach was used to obtain students' satisfaction with the informal learning experience in the existing learning environment and the critical spatial elements that can enhance and improve students' informal learning in order to propose corresponding recommendations and strategies for the subsequent optimization and design of active learning classrooms as sustainable educational environments.

**Figure 1.** Research framework.

#### **3. Research Methodology**

#### *3.1. Research Design*

This study used a student-oriented questionnaire and semi-structured interviews to obtain empirical data on the effects of active learning classroom environments on college students' perceptions of informal learning. The combination of quantitative and qualitative research methods has the following advantages: first, it can reveal differences between individual and group responses, which helps to improve the study's explanatory and persuasive power [53], and second, it can construct a more comprehensive understanding and perception of the learning environment by the respondents, which helps to improve the findings' generalizability and practicability.

The research subjects selected for this study were active learning classrooms at HUST. These research subjects were selected for the following reasons: the active learning classrooms at HUST are arranged following the student-centered design principle advocated by SCALE-UP and TEAL, and the environment design and technology updates have been carried out to a certain extent according to the development of higher education and information technology. This can reflect a learning environment concerned with sustainable education development. In addition, HUST built and used more than 110 active learning classrooms in 2018, all of which are designed with different types of spatial features based on the demands of professional courses and disciplines, and the rich sample size and diverse spatial environments can be sufficiently representative. The active learning classrooms in this school can be divided into four types according to their building size, space layout, and furniture design (Table 1): the table and chair integrated type, individual splicing combination type, multi-person splicing combination type, and multi-person fixed combination type. Finally, this university's active learning classrooms are used on a daily basis to take a large number of general and specialized subject courses, and a large number of students study informally in the classrooms outside of class time, so their layout and usage patterns are abundant and varied to meet the individual needs of different learning activities. As shown in Table 2, in HUST, the table and chair integrated type of classroom is small in size, and their centripetal and parallel layouts are mostly used for small-class teaching, while the free layout is mostly used for students' extracurricular group activities. The centripetal and parallel layouts of the individual splicing combination type of classrooms can be used for general and specialized courses, while the free layout is mostly used for individual and independent learning of students. All three usage layout modes of

the multi-person splicing combination type of classroom can meet the needs of the course, and their free and centripetal layout can be used for informal learning discussions and extracurricular communication among students. The multi-person fixed combination type of classroom is more versatile in its use due to the fixed layout, and the electronic monitors that combine with each multi-person desk make them the primary learning venue for student debriefing and discussion outside of class.


**Table 1.** Four types of active learning classrooms on the HUST campus.

The investigation of this study was conducted in the autumn semester of 2021, and the questionnaires and semi-structured interviews were conducted at the same time and continued for 3 months. The specific research content of this study can be divided into two phases. The first phase was a quantitative study in which the active learning classrooms at HUST were visited during non-class periods and questionnaires were distributed to the students in the classrooms who were engaged in informal learning activities such as extracurricular independent learning or extracurricular group learning, after which the collection and related analyses were conducted. The second phase was a qualitative study, which involved validation and extension of the results of the quantitative analysis. To ensure uniformity of the findings, the interviewees were recruited from the students who participated in the questionnaire and engaged in one-on-one semi-structured interviews in active learning classrooms. During the interview, the interviewees' thoughts were not influenced. They could evaluate the learning environment according to their own

experiences and perceptions to obtain more accurate and reliable research data. The consent of the participating students was obtained before the questionnaire survey and the semi-structured interviews began.

**Table 2.** Usage of 4 active learning classroom types at HUST.

#### *3.2. Questionnaire and Interview Design*

The questionnaire used in this study was divided into two parts. The first part asked students to provide their current active learning classroom numbers and personal information, including gender, academic stage, and subject major. The second part was a Likert scale to assess the impact of the active learning classroom environments on students' perceptions of informal learning activities. The dimensions of this scale were based on Lei et al. [54] 's questionnaire "Exploring Active Learning Classroom Space Factors Affecting Students' Learning Experiences", which classified classroom environments into four dimensions: "Instructional Interaction", "Furniture Perception", "Learning Support", and

team report

"Physical Environment", and it has been shown to have a significant impact on students' learning experiences in each of these four dimensions. The questions of the scale were based on the "Classroom Environment Quality (IEQ) Survey Scale" by Choi et al. [55] and the "Influence of Student Performance in Active Learning Classroom Environment Scale" by Yang et al. [56]. While extracting the mature scale, this study also combined the spatial attributes and characteristics of the current active learning classrooms at HUST, as well as the behavioral characteristics of students' informal learning activities to compile each specific dimension and item of the questionnaire. The final redesigned scale contained 22 items, including 8 questions on instructional interaction, 4 questions on furniture perception, 3 questions on learning support, and 7 questions on the physical environment, all analyzed using a 5-point Likert scale (Appendix A). Statistical software SPSS 24 was used as the main tool to analyze the questionnaire, the specific data analysis process can be found in the Supplementary Materials. A total of 160 questionnaires were distributed in this study. After excluding incomplete, repeated, or invalid questionnaires, 154 valid questionnaires were finally used for analysis, and the questionnaire return rate was 96.2%. Table 3 shows the sample distribution of students in this survey. Because HUST is an institution of higher education with a focus on science and engineering, most students in this university study science and engineering disciplines, and fewer students study natural and social sciences, so the sample distribution of students in this study is basically consistent with the overall distribution of students' attributes such as gender, major, and grade level at this university. In addition, the sample size of the students participating in this survey was statistically significant compared with the number of specific items on the scale [57]. The internal consistency test was conducted on the questionnaire data, and Cronbach's alpha value was 0.939, indicating that the reliability of this questionnaire was good.


**Table 3.** Sample distribution of students participating in the questionnaire (N = 154).

The interviews used in this study included the following topics (Appendix B). First, how often do students come to active learning classrooms for informal learning? Second, in what ways do students engage in informal learning activities? Third, what specific factors in active learning classrooms influence students' learning activities? Fourth, what is the students' satisfaction with the active learning classroom environments? Fifth and finally, what needs and preferences do students have for informal learning spaces? Before the interviews began, the students interested in participating were recorded and screened, aiming for an even distribution of students interviewed in terms of major, grade level, and the type of classroom they were in. Fifteen participants were finally identified for the interview in this study. Table 4 presents the main background characteristics of the students interviewed, and these interviewees were able to cover as many grades, majors, and disciplines as possible at this university with a high degree of statistical significance. The interview time for each participant was controlled to be between 10 and 20 min. After the full transcript of the interview, the texts were categorized and analyzed in depth using the programmed grounded theory proposed by Strauss and Corbin [58], and the interview materials were computer coded using Nvivo qualitative analysis software. The interview content was processed through three levels of coding. Open coding (level 1 coding) focused on the material itself, assigning various themes to the meaning expressed in the interview content and categorizing it according to its attributes [57]. According to the relationship between each code (causal, time, semantic, situational, etc.) to connect and cluster, the axial coding (level 2 coding) was formed. The axial coding could be summarized and integrated using the core theme and finally form the selective coding (level 3 coding). The coding process was conducted separately by the corresponding author of this paper and another independent coder and then corrected by multiple rounds of discussion to improve the internal confidence.

**Table 4.** Background characteristics of the students who participated in the interviews.


Note: Due to the limitation of the interview sample size, the participants' educational levels in this study were only distinguished between undergraduate and graduate students, and they were not subdivided in terms of grade level.

#### **4. Results**

#### *4.1. The Current State of Student Perceptions of Active Learning Classrooms Based on Informal Learning Experiences*

Based on the informal learning experience, this study conducted a descriptive statistical analysis of the questionnaire scale of students' perceptions of active learning classrooms, and the results are shown in Table 5. In active learning classrooms, the overall satisfaction of students' informal learning experiences was 4.15, which was between "relatively satisfied" and "very satisfied", indicating that the overall satisfaction level of the students' informal learning activities in active learning classrooms was good. Among the environmental dimensions of active learning classrooms, the students were most satisfied with the informal learning experience of "furniture perception", followed by the dimensions of "instructional interaction" and "physical environment", and were least satisfied with the informal learning experience of "learning support". Regarding the standard deviation, the "physical environment" dimension of active learning classrooms had the smallest value, indicating less dispersion among individuals in the group and that students' perceptions of it were relatively consistent. The "learning support" dimension of active learning classrooms had the largest value, indicating that the degree of dispersion between individuals in the group was large and the students' perceptions of it were relatively different.


**Table 5.** Student satisfaction statistics in active learning classrooms from the informal learning perspective.

Note: The larger the value of the mean, the higher the student satisfaction is, and the smaller the value of the mean, the lower the student satisfaction is.

#### *4.2. The Critical Environmental Elements Affecting Students' Perceptions of Informal Learning in Active Learning Classrooms*

In this study, exploratory factor analysis was conducted on the questionnaire of students' perceptions in active learning classrooms based on the informal learning experience to reduce the scale data, simplify the data analysis, and verify the validity of the scale structure. The KMO test and Bartlett's spherical test analyzed the scale items, and the KMO test result was 0.91, which was bigger than 0.9, and the Sig of Bartlett's spherical test took a value of 0.000, both indicating that the scale data were suitable for factor analysis. After reclustering, Table 6 shows the spatially influential factor division of the active learning classrooms after reclustering. After several explorations, one question item with a factor component coefficient less than 0.5 was finally excluded, and four factor components with eigenvalues greater than 1 were obtained, while the cumulative contribution of the reclustered factor principal components was 70.837%. According to the initial dimension design of the scale, Factor 1 is related to the attributes of tables and chairs and spatial attributes of the active learning classroom, so factor 1 can be named the "spatial perception" dimension. Factor 2 is related to the active learning classroom's sound, light, thermal, and decorative environments, so factor 2 can be named the "physical environment" dimension. Factor 3 is related to the display and interactive devices in the active learning classroom, so factor 3 can be named the "interactive learning" dimension. Factor 4 is related to the user-friendly facilities and storage space in the active learning classroom, so factor 4 can be named the "learning support" dimension. The name of each factor was replaced by "ALC\_F1", "ALC\_F2", "ALC\_F3," and "ALC\_F4," respectively.

In order to control the effects of demographic variables such as student gender, academic stage, major discipline, and active learning classroom type on the regression analysis, independent sample *t*-tests were conducted for student gender, and one-way ANOVAs was conducted for the student academic stage, major discipline, and active learning classroom type, respectively, and it was found that none of these demographic data had a significant effect on this linear regression model.

In order to explore the critical spatial elements that influence students' perceptions of informal learning in active learning classrooms and establish regression equations, this study conducted a linear regression analysis between the reclustered spatial perception factors of active learning classrooms and students' overall satisfaction with informal learning. This linear regression yielded a Durbin–Watson value of 1.497, indicating the good independence of the students who participated in the study. Second, the VIFs between the independent variables involved in this linear regression model were all less than three, indicating that there was no multicollinearity in the data. As shown in Table 7, the linear regression model fit well, with R2 = 0.658, indicating that the four active learning classroom spatial perception factors collectively explained 65.8% of the variance in their overall satisfaction with informal learning. Among them, the "spatial perception" and "learning support" factors could significantly affect the students' overall satisfaction with informal learning. (All *p* values were less than 0.001, and the Beta values were 0.324 and 0.286, respectively.) The regression equation is as follows: The overall informal learning satisfaction = 0.253 + 0.342 × ALC\_F1 + 0.231 × ALC\_F4.


**Table 6.** Spatial factorization of active learning classrooms after reclustering.

Extraction method: principal component analysis; rotation method: Kaiser normalized maximum variance method. The rotation converged after 7 iterations.

**Table 7.** Linear regression between active learning classroom spatial factors and overall satisfaction.


\*\*\* *p* < 0.001. Variables: (constant), ALC\_F1 = "spatial perception", ALC\_F2 = "physical environment", ALC\_F3 = "interactive learning", and ALC\_F4 = "learning support". The dependent variable is the overall informal learning satisfaction with active learning classrooms.

*4.3. Strengths and Weaknesses of Active Learning Classroom Environments from the Perspective of Students' Informal Learning*

This study used the three-level coding of grounded theory to analyze the interview data, and 146 key points were obtained through the first-level coding. The second-level coding was conducted according to the relationship between the first-level codes, and 10 second-level codes were summarized, including technical support, supporting facilities, physical environment, space perception, furniture design, learning purpose, learning atmosphere, learning activities, environmental status, and improvement measures. Then, the third-level coding was conducted according to the internal relationship of the 10 secondlevel codes, and 3 core influencing factors were obtained—the space dimension, student dimension, and management dimension—as shown in Table 8.

The frequencies and percentages of the 10 second-level coding categories and 3 thirdlevel coding core factors were obtained in the coding process, as shown in Table 9. Among them, "spatial perception" (19.80%), "furniture design" (12.94%), "improvement measures" (12.43%), and "learning atmosphere" (11.85%) were important secondary coding categories affecting the informal learning activities in active learning classrooms, while the "spatial

dimension" was the important core element in the evaluation process of active learning classrooms from the perspective of students' informal learning.

**Table 8.** Three-level coding of students' perceived evaluation of informal learning in active learning classrooms.


By analyzing the categories and core factors of the coding, as well as the frequency and percentage of second-level and third-level coding, this study could yield the following three findings from the perspective of the students' experiences with informal learning use:

(1) Suitable spatial perception

The spatial dimension (43.26%) was the largest core factor in the interviews, and it reflected the most intuitive perception of students' experiences in active learning classrooms. First, the spatial perception of the classroom was an important influencing factor (19.80%) in students' choice of active learning classrooms as a place for informal learning. For example, most students believed that active learning classrooms were more spatially private or that active learning classrooms were more spatially self-controlled, and so on. Second, many students were satisfied with the furniture design of the active learning classrooms (12.94%), namely with larger desks, comfortable seating, and movable furniture attributes that are more adapted to informal learning activities. Third, the supporting facilities (5.33%) and physical environment (4.98%) of the active learning classroom also impacted students' informal learning experiences, and the students expressed relatively positive perceptual feedback concerning them.

(2) Positive learning atmosphere

The student dimension (24.87%) was the most relevant result for exploring students' perceptions of informal learning. First, the students considered the learning atmosphere to be the main factor influencing informal learning activities in active learning classrooms (11.85%), and many considered active learning classrooms suitable for independent informal learning activities. Second, the students' learning activities in active learning classrooms (7.37%) were more diversified and abundant than in traditional classrooms, such as online course learning, essay writing, or course assignments. Third, the students' learning purpose (5.65%) also affected informal learning activities, such as reviewing exams, waiting for the curriculum, and finishing papers.

(3) Relative lack of resource management

The management dimension of active learning classrooms (18.98%) is also a critical factor influencing students' informal learning. First, the improvements proposed for active learning classrooms (12.43%) could indicate poor spatial perception in students' informal learning experiences, such as timelier logistical management, increasing the number of active learning classrooms built, and increasing the number of electrical outlets in classrooms. In addition, the current state of the active learning classroom environment (6.55%) is also a factor that influences whether students choose active learning classrooms for informal learning activities, such as the availability of space in active learning classrooms and the proximity of active learning classrooms to student dormitories.


**Table 9.** Frequency and percentage of perceived evaluations in active learning classrooms from the informal learning perspective.

#### **5. Discussion**

As a new learning environment in the higher education learning space continuum, active learning classrooms increasingly influence students' learning experiences. Improving students' informal learning experiences and perceptions in active learning classrooms is critical to building a full-time sustainable education environment. By distributing questionnaires to students and conducting semi-structured interviews, as well as using descriptive statistics, linear regression, and grounded theory methods to process the obtained scale data and interview transcripts, this study concludes the status of students' satisfaction with the current active learning classroom environment as a place for informal learning, as well as the critical active learning classroom environment factors that can influence students' informal learning experiences and perceptions, with the aim of suggesting certain design strategies for subsequent active learning classroom optimization and enhancement.

#### *5.1. Students Were Most Satisfied with the Furniture Design Dimension of the Active Learning Classroom and Least Satisfied with the Learning Support Dimension*

As a result of sustainable development education [59], next-generation learning spaces [30], intelligent teaching environments [60], and other educational concepts, active learning classrooms have changed and innovated to a certain extent compared with traditional lecture-based classrooms in terms of learning concepts, design principles, form features, and usage methods [61]. In terms of furniture design, the active learning classrooms have completely changed from the regular and fixed furniture form of the previous classrooms to a more autonomous, inclusive, cooperative, and flexible principle, giving students more diverse and creative furniture use scenarios. This abundant and diverse furniture design fits well with students' self-initiated, self-regulated, and self-responsible informal learning activities. Various types of furniture designs, such as the integrated type of tables and chairs, the single-person spliced combination type, the multi-person spliced combination type, and the multi-person fixed combination type, can meet the diverse learning forms and learning purposes of informal learning, such as electronic collaborative communication, interactive sharing of ideas, individual independent reading and learning, and the interaction of teamwork. In this study, the students' satisfaction with the movable, diverse, and comfortable furniture design of active learning classrooms reached a value of 4.29, which was the most satisfying spatial element of the students' informal learning process. Parsons (2016) found in a study that spliceable combinations of semicircular desks in active learning classrooms had a more positive impact on students' usual learning communication and interaction and was able to form different patterns of furniture combinations, depending on the students' activities [62]. Yeoman et al. (2019) argued that by changing the furniture design in the room, different categories of informal learning activities could be supported, including student-centered learning, individual learning, and collaborative learning, so more flexible furniture could build a more active informal learning environment [49].

The design principles of active learning classrooms are based on new learning theories such as active learning, collaborative learning, and deep learning, and their main emphasis is on the diversification of learning spaces and the enrichment of technological equipment. Although the more flexible and diverse environment of active learning classrooms has already had a more positive impact on students' learning experiences, the lack of humane facilities in classroom spaces still exists, such as insufficient storage space, uneven distribution of power outlets, and unstable network signals. However, this overemphasis on learning styles and activities often neglects students' most basic personal use needs. In this study, the students' satisfaction with the learning support facilities in active learning classrooms was only 3.81, which was the least satisfactory element of the space for the students' informal learning process. Porterfield et al. (2020) found that adding personal storage space to active learning classrooms or including an adequate number of electrical outlets could better facilitate student learning activities [63], and Robert et al. (2015) found that although students rated active learning classrooms positively, the lack of user-friendly amenities in classrooms could also lead to negative student learning experiences [64]. This finding was also reflected in the student interviews, where 8 of 15 respondents (53.3%) felt that the storage space in the active learning classroom needed to be increased. Twelve students (80%) felt that the number of electrical outlets in the active learning classroom was low and wanted more to be added. It is worth noting that the need for active learning classroom learning support elements is generally higher in the upper grades (students in their senior year and above) than in the lower grades (students in their junior year and below) due to more e-learning or team-based learning in the upper grades, such as online classes, writing papers, and organizing the contents of electronic reports, as well as a more homogeneous learning style in the lower grades. However, according to the interviews, the students in the early grades also preferred the active learning classroom as the place for their extracurricular learning due to its more comfortable indoor environment, more convenient learning support facilities, and the ability to work with multiple students to meet their personalized learning needs.

#### *5.2. The Spatial Perception and Learning Support Dimensions of Active Learning Classrooms Are the Critical Factors Influencing Students' Informal Learning Experiences*

Informal learning, as an unstructured act of active learning by students [6], not only enriches the way students learn but also contributes, to a great extent, to their social and cultural engagement [16]. With the development of sustainable education, more informal learning has been generated on campus, and students have gradually become the dominant learners [8]. A large amount of learning outside of class time has become a new way for students to acquire knowledge. In addition to serving as a place where formal learning occurs for students, active learning classrooms can also meet the contextual, collaborative, self-directed, and flexible characteristics needed for students' informal learning. In this study, the linear regression between the environmental dimensions of the active learning classroom and students' overall satisfaction with the informal learning experience showed that flexible spatial perception is the critical factor affecting the overall satisfaction with the informal learning experience in the active learning classroom (*p* < 0.001, Beta = 0.324). In addition, compared with other learning environments, the comfortable and flexible spatial perception of active learning classrooms can also influence students' informal learning more effectively for the following reasons. First, there is the more comfortable spatial environment. For example, one respondent said, "It is comfortable to study in an active learning classroom, the environment is great, and a good spatial environment can motivate greater motivation to learn". Second, there is the flexible and variable layout of the individual study space. One student explained this by saying, "The classroom space can be changed at the will, enabling the formation of multiple independent individual study spaces". Third, the layout of the space supports multi-person interaction and group practice. Some respondents said, "The classroom space is particularly suitable for group collaboration, and the form of space that can be put together and combined helps us broaden our ideas and share everyone's knowledge and ideas". The findings of some researchers can also confirm this point. Oliveira et al. (2016) found that when the overall spatial environment of the learning space was not comfortable or attractive, it would reduce the time for students to study there [65]. Granito et al. (2016) showed that students prefer flexible classroom space layouts because they can flexibly switch between different learning styles [66].

Education in the 21st century is inseparable from the support of information technology. Convenient and portable computer equipment and electronic learning spaces covered by a wireless network are important factors in improving students' learning satisfaction. The learning styles of contemporary college students are more diversified and diverse, such as learning through the interactive use of multiple electronic devices, interactive team learning with network connectivity, and visualized experiential learning, all of which require rapid development and responsiveness from the learning environment. As a space for formal and informal learning, the active learning classroom should pay more attention to the combination of physical and virtual as well as humanized and informational to give students a more comprehensive and holistic environment to support their learning. Although student satisfaction with the learning support elements of current active learning classrooms is low, a linear regression based on the relationship between environmental factors in active learning classrooms and the overall satisfaction with students' informal learning experiences can reveal that the learning support dimension of active learning classrooms remains a critical spatial element influencing students' informal learning experiences (*p* < 0.001, Beta = 0.286). Its main performance is: First, the available storage space. Active learning classrooms have increased the amount of usable space per person, and there are more areas in the classroom where personal belongings can be placed, with some

respondents stating, "There is more space in the classroom to place personal belongings properly". Second, there is the wireless coverage of WiFi devices. Active learning classrooms are designed with the addition of ICT in mind, and their network environment is relatively more complete. Some interviewees mentioned, "The network signal of the active learning classroom is stable, and they usually choose this place to complete their online courses". Third, there are evenly distributed power outlets. Active learning classrooms are designed with evenly distributed power outlets on all four walls of the classroom to facilitate the power needs of students' information-based learning. Some interviewees said, "More adequate power outlets make it easier to use computers and other electronic-based devices, and enable longer learning time".

#### *5.3. The Spatial Privacy and Learning Atmosphere of Active Learning Classrooms Can Promote Informal Learning for Students*

Informal learning emphasizes the social, diverse, collaborative, and purposeful nature of learning compared with formal learning [36]. Therefore, a learning environment that focuses more on spatial privacy is more effective for students' informal learning [44]. Students welcome the relatively more private learning spaces of active learning classrooms for informal learning, and their quiet and comfortable spatial environment and free and abundant spatial layout can facilitate informal learning styles such as individual study, collaborative discussions, gatherings and meetings, retreats, and readings [40] while also supporting students' choice of learning activities according to their needs and preferences [67] and increasing the frequency of use [41]. Of the 15 respondents who participated in the one-onone interviews, 11 students (73.3%) believed that the spatial privacy of the active learning classroom had a more positive impact on informal learning activities than other informal learning spaces, as evidenced by the following. First, a high-quality physical environment creates better privacy. For example, some respondents said, "The classroom is a quiet and private learning environment with good sound insulation in the indoor environment as well as walls for shielding". Second, the flexible space layout allows for a separate learning environment. One student stated, "When other people are studying in the classroom, I can move the furniture to an unoccupied corner to study and avoid the interference of others". A study by Deng et al. (2017) showed that most students prefer to study in a quiet and solitary environment so that the quiet learning environment allows them to concentrate on their reading and study fully [68]. Beckers et al. (2016) found that students prefer to choose seats in remote corners for informal learning to avoid the distractions brought by other people [50].

Active learning classrooms have a stronger learning atmosphere than other learning spaces. In formal learning spaces, students experience them through their senses and give them meaning for learning, and the students become more familiar with the classroom space. In addition, active learning classrooms often have adequate learning furniture and equipment, and they also give users a certain degree of self-control while allowing students to develop a positive sense of spatial belonging, which are the spatial elements that bind students together and build a stronger learning atmosphere. In this study's interviews, seven students (46.6%) mentioned the learning atmosphere of the active learning classroom and believed that its spatial atmosphere was closely related to learning activities. For example, some students mentioned that they "prefer to come to the active learning classroom with their classmates for extracurricular learning to monitor and promote each other". "The active learning classroom has a high learning atmosphere, and seeing everyone's learning behavior can improve one's motivation". In studies such as those by Harrop et al. (2013) and Waldock et al. (2016), it was found that students, both academically and socially, learn near friends and peers, creating an atmosphere of a learning community where everyone can work together, and students can develop a sense of enrichment that "I came here to learn and my friends are already here, so I joined them", meaning that students are more motivated to work in a shared learning environment [67,69].

#### *5.4. Better Resource Management Helps Students Have a Higher-Quality Informal Learning Environment in Active Learning Classrooms*

The active learning classroom uses many new technology devices, and the layout of the space is also more abundant and flexible to meet the diverse learning needs of students. More accessible resources and available space also require more time management and support, including the proximity of classroom locations, the length of open hours, the number of locations, and the management and maintenance of learning resources. These dimensions of active learning classroom management have a substantial role in enhancing the effectiveness of learning environments, but the current theoretical and practical research on active learning classrooms and informal learning has not received the corresponding attention. Among the 15 interviewees, 13 students (86.6%) mentioned the management status and subsequent improvement of active learning classrooms, which are mainly reflected in three aspects. First, there is the more timely classroom management. Some respondents mentioned, "The classroom has some leftover items and trash after class, which can disturb other students studying". Other respondents also mentioned, "The furniture and technology in active learning classrooms should be managed more frequently; some furniture storage spaces are dirty, and some equipment is damaged and needs to be disposed of on time to ensure an efficient learning environment". Second, there are the longer open hours. Ten of the students who participated in the interviews (66.7%) mentioned their willingness to engage in longer learning activities in the active learning classroom. Some students responded, "While active learning classrooms are preferred for extracurricular learning, their opening hours are somewhat limited and hinder extracurricular learning". Another student mentioned, "Active learning classrooms are comfortable and the equipment is smarter, but they are also more strictly managed and open for shorter periods of time than traditional classrooms, while I would like to study in the active learning classroom for longer periods of time, the reality is often the opposite". Third, there is the more accessible classroom space. One student explained this by saying, "Active learning classrooms are great for studying, but I would also choose other traditional classrooms closer to the dorms". "We would choose a study space closer to us, even if its space use is not as good as an active learning classroom". Overall, the interviews with students revealed that students are willing to spend more time in the active learning classroom for informal learning activities due to its better environment, comfortable layout, variety of spatial features, and abundant configurations. Therefore, the logistic management, equipment maintenance, classroom open hours, and classroom accessibility should be reorganized accordingly.

#### **6. Conclusions and Limitations**

With the flourishing research and practice of active learning classrooms in recent years, its changes and innovations in learning styles, teaching models, spatial environments, and educational technologies have profoundly impacted sustainable education development. The Education 2030 Framework for Action [15], released by OECD in 2015, defines the importance of lifelong education and full-time education and emphasizes that students' informal learning is "valuable learning", being as valuable as formal learning. As a new learning environment for sustainable education, active learning classrooms have emerged in relevant theoretical and practical studies [54,59]. However, there is no academic consensus on the use and theory building of informal learning in active learning classrooms. Based on students' perspectives, this study explores the impact of the sustainable educational environment in the active learning classroom on students' informal learning perception. Using a combination of quantitative and qualitative empirical analyses, this study found that first, students were more satisfied with their informal learning experiences in active learning classrooms, and most students preferred active learning classrooms as their informal learning environment. Second, flexible and comfortable space perception and humanized learning support facilities in active learning classrooms are critical spatial factors affecting students' informal learning. Third, a private environment and sufficient

learning atmosphere in active learning classrooms can promote students' informal learning ability. Fourth, for active learning classrooms, better resource management helps them form a better-quality and full-time learning environment.

The findings mentioned above have the following implications for the optimal design and sustainable development of active learning classrooms in the future.

First, active learning classrooms should actively experiment with more diverse and flexible space layouts as well as emphasize more comfortable and student-centered environmental perception designs, such as by using more acoustic and soft-colored interior materials to improve students' spatial comfort, increasing the usable area of classroom chairs and tables to improve students' usage comfort, designing a variety of flexible tables and chairs to meet different learning activities such as individual and multi-person learning, expanding more areas for leisure and learning in classrooms, and designing electronic interactive devices that can meet both the needs of the curriculum and the needs of students' extracurricular learning activities.

Second, the active learning classroom should improve and increase more diverse and convenient humanized facilities, such as through multiple lockers for personal use in active learning classrooms, floor outlets evenly spaced on the floor that can be hidden for storage, WiFi signal boosters that can be used by dense numbers of people, retail vending machines for students who use the classroom for long periods of time, and information interaction software that students can use independently.

Third, the environmental design of active learning classrooms should focus on the privacy of the space and the establishment of a learning atmosphere, such as by designing some mobile partitions that can enrich the space level so as to form a diversified personal learning space from a private one semi-open to the informal learning use of the classroom, designing a more integrated and wrapped furniture design to meet the personal privacy needs of students when they study independently after class, and increasing learning service facilities such as mobile whiteboards or fixed interactive whiteboards to encourage students' extracurricular team communication and cooperation and improve students' control over the sound, light, and thermal environment of the classroom to meet students' needs for a learning space atmosphere.

Fourth, the timeliness and accessibility of logistical and resource management in active learning classrooms can be improved by, for example, extending the available time of the classroom, increasing the frequency of equipment inspections and indoor cleaning of the classroom, increasing the space supply of the active learning classroom in extracurricular time, repairing and maintaining various technical equipment and furniture facilities in the classroom in a more timely manner, and making the site of the active learning classroom closer to the dormitory and canteen for students.

Student-centered active learning classrooms are designed primarily to promote active and deep learning, which requires a higher level of student engagement. By improving the environmental design and experiential perception of the learning space, it can effectively promote students' learning involvement [70]. In addition, similar to formal learning, learning engagement in students' informal learning behavior is also an intermediary variable to improve students' abilities of active learning and deep learning. The more frequent students' informal learning behavior is, the more they will participate in learning [71], and this kind of participation behavior will also increase students' active learning ability. In summary, the construction and development of learning environments are closely related to students' learning perceptions. By rethinking students' informal learning experiences and perceptions, the high-quality development of active learning classrooms can be more comprehensively enhanced, thus enhancing students' full-time learning experiences, which in turn promotes student learning engagement and active learning in active learning classrooms. In addition, as a new type of learning environment for sustainable education development, active learning classrooms should not only meet the learning theories and educational strategies for formal learning but should also focus on and enhance the scenarios used by students for informal learning. This holistic classroom environment

that combines formal and informal learning creates a full-time learning environment that promotes student active learning to achieve the goal of quality and sustainable higher education development.

This study also has certain limitations due to research capacity and time constraints. First, the experimental data come from students' self-reporting, which may affect the authenticity of the data due to discrepancies in students' self-perceptions, and research methods such as objective measurement and performance evaluation can be integrated in the future. Second, the time of the study was relatively limited, and this study chose data from one semester. In the future, a more in-depth and reasonable exploration can be conducted by tracking research on students' long-term learning space use processes. Third, the object of the study was relatively singular, and only one university was selected for the collection of student data and classroom models in this study. In the future, more diverse studies can be conducted by investigating several universities with different disciplinary and professional characteristics at the same time, which makes the research findings more convincing and generalizable.

**Supplementary Materials:** The following supporting information can be downloaded at: https://www. mdpi.com/article/10.3390/su14148578/s1, Spatial satisfaction in two types of classrooms\_Raw Data.

**Author Contributions:** Conceptualization, S.J. and Y.G.; methodology, L.P., Y.D. and S.J.; software, S.J.; validation, L.P., Y.D. and S.J.; formal analysis, S.J.; investigation, Y.D. and S.J.; resources, L.P., Y.D. and S.J.; data curation, S.J.; writing—original draft preparation, S.J.; writing—review and editing, L.P. and S.J.; visualization, Y.D. and S.J.; supervision, L.P. and S.J.; project administration, L.P.; funding acquisition, L.P. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the National Natural Science Foundation of China, grant number 51978294.

**Institutional Review Board Statement:** This study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Medical Ethics Committee, Tongji Medical College, Huazhong University of Science and Technology, authorization number: 2022-S046.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data used to support the findings of this study are included within the article.

**Acknowledgments:** The authors would like to thank the teachers and students at Huazhong University of Science and Technology for their support and encouragement.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A. Questionnaire Template**

*Questionnaire Survey on Satisfaction with the Informal Learning Experience in Active Learning Classrooms at Huazhong University of Science and Technology*

Hello! Thank you very much for taking the time to fill out this questionnaire. This questionnaire aims to ascertain students' informal learning satisfaction with the classroom space and serve as a resource for optimizing classroom spaces in universities.


A. Male B. Female

	- A. Undergraduate B. Master's degree C. PhD D. Other
	- A. Philosophy, economics, and law
	- B. Education, literature, and history
	- C. Science, engineering, agriculture, and medicine
	- D. Military science, management, and art


	- A. Very dissatisfied
	- B. Dissatisfied
	- C. Neutral
	- D. Satisfied
	- E. Very satisfied

#### **Appendix B. Interview Outline**

*Semi-Structured Interview Outline of the Informal Learning of Students in Active Learning Classrooms at Huazhong University of Science and Technology*

Semi-structured interviews will explore students' experiences and perceptions of using active learning classrooms for informal learning outside of class time and attempt to draw out the impact of the learning theories in active learning classrooms on students' learning activities and technology use.

The main questions and possible follow-up questions are as follows:

	- Your frequency of extracurricular time in the active learning classroom is?
	- When do you usually come to the active learning classroom?
	- Do you use the active learning classroom with your classmates for informal collaborative learning outside of class time? If so, how satisfied are you with collaborative learning in the active learning classroom? What design and environmental factors affect teamwork?
	- Do you choose active learning classrooms for breaks or chats outside of class time?
	- How did you feel the first time you came to an active learning classroom outside of class time?
	- What do you think about the physical environment, furniture design, and technical equipment of the active learning classroom that attracts you to informal learning in the active learning classroom?
	- What do you think is particularly attractive about the active learning classroom? How does it help you during your informal learning sessions?
	- Which aspect of the active learning room do you think you are not satisfied with, or what do you think could be improved?
	- What are some differences between active learning classrooms and traditional lecture-based classrooms?
	- How have these differences you noticed impacted your learning?
	- Did the active learning classroom and traditional lecture classroom environments make a difference in your attention or engagement during informal learning? If so, what do you think might cause these differences?
	- What differences do you notice between active learning classrooms and public spaces and libraries?
	- How have these differences you noticed impacted your learning?
	- Does the active learning classroom and public learning environment make a difference in your attention or engagement during informal learning? If so, what do you think may have contributed to these differences?

#### **References**


## *Article* **Comparative Analysis of Dropout and Student Permanence in Rural Higher Education**

**Alfredo Guzmán 1,\*, Sandra Barragán <sup>2</sup> and Favio Cala-Vitery <sup>2</sup>**


**\*** Correspondence: alfredo.guzman@asturias.edu.co; Tel.: +57-321-254-0363

**Abstract:** The growing dropout and low permanence of rural students in higher education has become a central problem in the education system, both affecting the quality conditions of training programmes and preventing the materialisation of the benefits that achieving this educational level entails for society. However, the study of these events in rural populations is scarce, resulting in an inadequate treatment of dropout and, consequently, the impossibility of consolidating student permanence. Thus, the aim of this article is to identify which individual, academic, socio-economic, and institutional variables influence the dropout and the retention of the rural student population in higher education. To achieve this purpose, a cross-sectional study was defined. The sample used was a non-probabilistic sample with an n of 269 rural Colombian students who were administered a self-report questionnaire that assessed 59 variables. Data analysis was based on means comparison and cluster modelling. The results show that dropout and permanence in rural students is related to the educational level of the father, family and work obligations, the need to move from their place of residence, the academic average in higher education, satisfaction with the choice of programme, communication with the institution, and the attention of teachers, among other things.

**Keywords:** educational quality; higher education; dropout; permanence; public policies; institutions; modelling

#### **1. Introduction**

The term quality is widely used in higher education systems worldwide to ensure excellence at both the institutional and training programme levels [1–3]. In this sense, it is necessary to recognise that, before the 1980s, quality in higher education was an internal matter for Higher Education Institutions (HEIs); however, after the 1980s, quality at the level of training became a matter of public policy, making quality assessment an internal activity of HEIs, as well as an external activity of interest to states [4,5]. Thus, in the quality assessment exercise, various standards which allow us to understand the current state of substantive functions (teaching, research, and relations with the external sector), as well as those complementary to these functions, have been generated [6].

In this scenario, there are multiple indicators that evaluate the quality of the educational system, HEIs and training programmes; however, the student dropout rate and its counterpart, the permanence rate, have become one of the main indicators [7–9], since they allow us to identify whether training programmes manage to provide society with professionals who meet the diverse demands that society generates on a continuous basis [10]. Hence, if an HEI, a training programme or an education system does not rank below the average dropout rate at national or global level, it is considered to be of low quality, leading to intervention through the development of institutional policies and public policies to avoid this scenario.

**Citation:** Guzmán, A.; Barragán, S.; Cala-Vitery, F. Comparative Analysis of Dropout and Student Permanence in Rural Higher Education. *Sustainability* **2022**, *14*, 8871. https://doi.org/10.3390/su14148871

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Leandro Silva Almeida and Silvia Fernández Rivas

Received: 30 May 2022 Accepted: 18 July 2022 Published: 20 July 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

The intervention generated by both HEIs and the State is not only related to the outcome of the dropout or permanence rate, but also to the effects that these educational events bring to society, by limiting or achieving the materialisation of the benefits of higher education (e.g., higher income, increased productivity and better security rates, etc.) [11]. This makes both dropout and permanence at the educational level a matter of interest for the academic community, as well as for policy makers.

In accordance with the above, many studies have sought to establish the variables that explain the materialisation of these events both in HEIs [11–18] and in the education system [19–24]. However, their study still lacks multiple perspectives, generating indications that HEI and state policies have not been effective, thus high dropout rates and low permanence in education systems persist. An illustration of this is often the situation in OECD nations, where in 2018 the dropout rate was near to 64.5%, or, in the case of Latin America, the dropout rate was close to 54% [25]. In addition, the dropout rate since the beginning of the COVID-19 health crisis has increased, especially among vulnerable student populations (e.g., those displaced by conflict, Afro-descendants and rural populations, among others) [26].

The study of dropout and permanence in rural populations is placed within the framework of perspectives that have been little addressed by both academics and education policy decision makers [11,25]. As a result, both HEIs and states have dealt with dropouts in this student population with generic strategies that are applied equally to all types of students, without considering the individual, academic and socio-economic aspects of the students and the institutions in which they study. This has led to an increase in the dropout rate and a low permanence rate, thus affecting the quality of the training programmes offered in these areas [27]. Hence, it is necessary to evaluate the institutional and public policies that should be implemented in the rural student population to prevent and mitigate the event of dropout, in order to achieve the permanence of this type of student in the education system. In the analysis of this problem for rural students, the need arises to know what variables influence the decision to abandon or remain in the educational process.

Thus, the aim of this article is to identify which individual, academic, socio-economic and institutional variables influence the dropout and retention of rural students in higher education. The Colombian education system was selected for this study because most of the previous research on dropout or permanence of rural students has been carried out in developed countries [28–32], and not in contexts of social disparity as marked as the Colombian case, where rural areas have been characterised by violence and conflict by various armed actors, which has led to marginalisation, inequality in the income of the population, regional differences and various social tensions [11]. Hence, this analysis in the Colombian rural population, as an added value, allows us to understand what other variables influence dropout or permanence, providing new perspectives for the academic community, as well as for public policy and HEI decision makers.

This article is structured in four sections. The first presents the theoretical framework and contextualisation of dropout and retention in Colombia and the studies developed; the second contains the methodology that allowed the fulfilment of the objective; the third shows the results; the fourth discusses the main findings and offers the conclusions of this study.

#### **2. Theoretical Background**

#### *2.1. Dropout and Permanence in Higher Education*

Dropout as an event that affects education systems does not have a unique meaning, being the result of the different actors involved in its study, such as researchers, HEIs, states and social organisations, among others [25]. That said, the literature tends to conglomerate definitions of dropout into two main groups. The first group is a compilation of conceptualisations derived from the academic study of dropout; the second group is operational, established by states in the framework of education systems to facilitate the quantification of the event [11,33].

In this sense, the present article is framed within the first group, which allows for the analysis of multiple variables that can lead to the early termination of a student's academic studies. Thus, dropout is defined as "the cessation of the relationship between the student and the training programme leading to a higher education qualification before the qualification is recognised. An event of a complex, multidimensional and systemic nature, which can be understood as cause or effect, failure or reorientation of a training process, compulsory choice, or response, or as an indicator of the quality of the education system" [34] p. 6. The use of this meaning of dropout permits the integration of perspectives on the study of this event. In the case of permanence, there is a greater consensus regarding its conceptualisation, which is understood as "the permanent initiative of HEIs to generate strategies to strengthen institutional capacity, which contribute to reducing drop-out rates". It is also an important element in the elaboration of "the institutional educational plan" [35] p. 194.

In view of the various actors involved in the research of dropout and permanence in higher education, the multidisciplinary orientation in its study stands out [35]. This has led to the examination of illustrative factors, both innate and external to the student, which can be categorised as: individual, socio-economic, academic and institutional. This categorisation has been used in previous studies, such as those of Fonseca and García [36], Barragán and González [16,24], Donoso and Schiefelbein [37] and Guzmán et al. [11,25], among others. In addition, states have made use of this categorisation to define public policies to prevent and mitigate dropout at the educational level, as is the case in Colombia [38]. Figure 1 presents the dropout model based on the categorisation of variables; each cluster of variables is hereafter referred to as a determinant. It should be noted that the variables of one determinant have the capacity to relate to and influence one another. These same determinants can, in fact, also explain permanence in higher education.

**Figure 1.** Conceptual model of determinants of dropout. Note: Each determinant groups n variables v1, v2, . . . , vn. Reprinted with permission of Guzmám et al. [21]. Copyright 2021 Frontiers.

The individual determinant explains the characteristics related to the student and his or her individual environment that specifically affect the choice of whether to leave higher education without completing it or to remain in it [38]. In higher education, the individual determinant variables have been widely debated, as several research studies have largely attributed them to the materialisation of the dropout event [39]. An example of this was evidenced in the study by Georg [40], who found that 95% of dropouts from German HEIs were explained by the characteristics of the individual at the time of entry into the institution. The socio-economic determinant refers to the variables of the social and economic environment that affect the student and his or her family and that directly or indirectly affect dropout or permanence [38]. Previous studies have been divergent, since some research has indicated that this type of variable does not influence dropout or permanence in higher education [41,42] and others have highlighted the influence of these variables on student completion of the educational process [43–45].

The academic determinant relates to the achievement of learning outcomes, the advancement of proficiency, student performance and other components that impact the management of instruction and learning at all levels of instruction [38]. In general, the findings of previous studies identify that the variables of this determinant have a great impact on student dropout and permanence in higher education, especially because of the demands of the educational level, as identified by Heidrich et al. [15], Choi and Kim [46], as well as Stewart et al. [47,48]. Transition to higher education [14] and student perspectives (e.g., self-efficacy and self-management) [48] are closely related to dropout at the educational level.

Finally, the institutional determinant explains those characteristics of HEIs that allow for the correct development of the educational process [38]. Previous research has found that the high levels of attrition and retention in HEIs are related to their size in terms of number of students, the quality of the training programmes, programmes for permanence and timely graduation (P&GOs) and administrative processes [46,49,50].

#### *2.2. Context of Dropout and Permanence in Higher Education in Colombia*

Student desertion and permanence as indicators of quality in higher education in Colombia began to be of interest to the state in 2003, with the implementation of the first strategies for the prevention and mitigation of desertion and the achievement of permanence [51]. As a result of these initial efforts, there was a need to expand the study of student dropout and permanence through accurate and reliable information, and the National Education Ministry (NEM) consolidated both the state information system SPADIES (Sistema para la Prevención de la Deserción de la Educación Superior in Spanish) and various public policies. Simultaneously, the national academic community became interested in the study of these educational events.

In the case of the state, public policies aimed at preventing and mitigating dropout have been designed and implemented jointly with HEIs. Thus, the state has taken on the role of funder for students, providing educational credits and scholarships [11,52]; in addition, HEIs have focused on strengthening competences, as well as developing Early Warning Systems (SAT in Spanish) and P&GOs to identify and support students at risk of not completing their educational programme [53]. As a result of these efforts and according to SPADIES data [54], in the first semester of 2021, the dropout rate of the system was 7.6%, while for the second semester it was 12.8%. While the dropout rates, as presented in Figure 2, are below those observed in the Latin American and OECD region, when analysing the situation of training institutions and programmes located in rural areas, the reality is different.

Thus, for the year 2016, it was estimated by the NEM that the dropout rate by cohort in rural areas was close to 50%, both for technical and technological levels, as well as for the university level [55]. However, at the national level, in these areas, the dropout rate varies between departments or states. An example of this is the departments of Chocó, La Guajira and Putumayo, where the dropout rates for technical and technological programmes were 91.3%, 73% and 71.2%, respectively, while for university programmes the departments with the highest dropout rates were Putumayo, La Guajira and Arauca, with rates of 80.2% and 55.6% for the latter two [56]. In both cases, these departments are characterised by high levels of social disparity [57].

**Figure 2.** Dropout rate in the Colombian higher education system for the years 2017 to 2021. Adapted with permission of [54]. Copyright 2022 SPADIES.

However, public policies implemented by the state to prevent and mitigate the effects of dropout and achieve permanence in the rural student population are characterised by being non-differential, based on the financing and support of HEIs [11]. This is due to a certain extent to the lack of information on what is happening in the education system in these areas, a situation that is not exclusive to Colombia, but which is also present in other countries, as Byun et al. [27] and Castleman and Meyer [30] state, since government information systems do not incorporate the rurality variable, and there is a lack of academic interest in researching dropout and permanence in this student population [25].

Having said that, in the case of the research developed in Colombia on the desertion and retention of rural students in higher education, only three studies have been carried out. The first was carried out by Rueda et al. [58], who determined that rural students who are at greater risk of dropping out were characterised by a low level of maladjustment or adaptation to university life, as well as belonging to single parent families and with severe or moderate family dysfunction. The second, which was developed in virtual education training programmes, established that academic variables do not influence the events of desertion or permanence, whereas conjugal status (related to family commitments), age, social status, work commitments, parents' education level and sort of work, the student's pay and sort of work relationship, as well as the number of individuals who depend on the family's income, do influence the events of desertion or permanence [11]. Finally, the third assessed the potential of student dropout in higher education to widen social gaps in rural Colombia, as it is the student and his or her family who bear the greatest costs associated with these events [21].

#### **3. Methodology**

To fulfil the objective of this article, which is to identify the individual, academic, socioeconomic and institutional variables that influence the dropout and permanence of rural students in higher education, a quantitative cross-sectional study was carried out, following the parameters established by Sedgwick [59] and Cvetkovi´c-Vega et al. [60]. The sample, instruments and explanatory variables are described below, as well as the data analysis and modelling.

#### *3.1. Sample*

For the present study, a non-probabilistic, non-intentional sampling was defined, so that the selection of information-rich cases was sought, using Patton [61] as a theoretical reference for the selection criteria of the participants, of which three were established. The first is to be linked to an undergraduate training programme (technician, technologist or undergraduate); the second is to express the intention to drop out or remain in the training programme; and the third is to be in or come from a rural area. As this was a non-probabilistic sample, support was requested from the RUPPEGO network (Red Universitaria por la Permanencia Estudiantil in Spanish), which groups 37 HEIs at the national level. Based on the above, the final sample was 269 rural students, of whom 131 reported having the intention to drop out and 138 to remain in education. The sample size was similar to that of previous studies, such as those developed by Guzmán et al. [11], Contreras [62] and Oasi et al. [63]. Although there is parity between the sample of the present study and others developed previously, it is clarified that the results presented here should not be generalised, due to the nature of the type of sampling selected. Table 1 presents the general characteristics of the study sample.


**Table 1.** General characteristics of study participants.

\* The values shown are in Colombian pesos. For conversion purposes, 1 USD is equivalent to 3950 COP, as of 30 March 2022.

The Study Participants Living in the Departments of Antioquia, Bolívar, Cundinamarca, Caquetá, Nariño, Guajira and Chocó.

#### *3.2. Instruments and Explanatory Variables*

An online self-reporting questionnaire was used to collect the data. The questionnaire was developed ad hoc based on the theoretical models proposed by Tinto [64], Barragán and González [16], Kemper et al. [65], Guzmán et al. [25], Segovia-García et al. [9], Heublein et al. [66], as well as Aina et al. [67], among others. The questionnaire was divided into six sections. The first sought to obtain informed consent and authorisation to participate in the study, and collected data from the students on their intention to drop out or remain in the training programme, as well as the type of programme they are studying; the second collected data on the variables of the individual determinant; the third focused on the socio-economic determinant variables; the fourth assessed the academic determinant variables; the fifth was related to the institutional type variables; and the last sought to confirm the student's rurality condition. Table 2 shows the variables analysed by the questionnaire and their theoretical contribution. To support this, the advances in the field of study related to

rurality and in those variables not dealt with in previous research on the rural student population were taken, as well as the basis of the research developed on other types of students. Similarly, Table A1 in the Appendix A presents the instrument and the coding of the study variables.



\* Corresponds to variables not addressed in the literature on higher education dropout among rural students.

In relation to the reliability of the instrument, an internal consistency analysis was carried out for each of the four determinants assessed, using the Cronbach's Alpha statistic (α). In this way, α was considered moderate in the event that its esteem was between 0.40 and 0.60, satisfactory between 0.60 and 0.80, and high when it was above 0.80 [77]. Additionally, it was decided to eliminate the explanatory variable in the determinant in question if this improved the value of α. Table 3 presents the reliability of the applied questionnaire.

Based on the results in Table 3 and to ensure the reliability of the questionnaire, variables that improve the value of α were eliminated from the analysis, both for the selection of statistical tests to be used and for the analysis of the results. Thus, in the case of the individual determinant, variable I1 was eliminated so that α was considered moderate (0.58); for the socio-economic determinant S1, S11 and S14 were eliminated so that α was acceptable (0.60); for the academic determinant A2, A4 and A15 were eliminated so that α was acceptable (0.701); and, finally, for the institutional determinant, IES1 and IES8 were eliminated so that α was acceptable (0.781).


**Table 3.** Reliability of the self-reporting questionnaire.

\* The value is negative due to a negative average covariance between elements. These breach the assumptions of the reliability model; however, by removing some element, this value may fit the reliability model. \*\* α-SE corresponds to the value of α if the element is removed.

#### *3.3. Data Analysis and Modelling*

With the data collected, due to the nature of the data and the purpose of this study, where rural students are categorised, we proceeded to identify the variables that influence the decision to drop out or stay in higher education. For this purpose, the Mann–Whitney U test was used because the data did not fit a normal distribution (see Table 4), and it facilitated the comparison of independent populations, in this case the students who expressed the intention to drop out or to stay in the training programme. The existence of statistically significant differences between the two groups of students for the study variables was present when the *p*-value was less than 0.05 [78].


**Table 4.** Kolmogorov–Smirnov normality test.

\* The degrees of freedom (gl) were 269. \*\* Normal distribution is rejected with *p*-value < 0.05.

With the explanatory variables in which statistically significant differences were identified, we proceeded to compare the way in which the groups behaved in relation to these variables, therefore modelling based on clusters or classification was chosen, since this allows the description of groups with homogeneous characteristics based on the study variables of a particular event or phenomenon [79]. In this sense, cluster modelling assumes that individuals share a common distribution of characteristics, while different individuals follow a different distribution [80]. That is, a study population has a finite number of n distributions, and the purpose of clustering is to take such a mixture and analyse it into simple components and estimate the "membership probabilities" [79].

This type of modelling has both supervised and unsupervised techniques. Since there are no previous studies on the rural population to establish how students cluster, both those who wish to drop out and those who wish to stay in higher education, hierarchical cluster modelling was used. This type of modelling, being in the unsupervised category, does not require an underlying statistical model. Ward's technique was chosen to create the model because it minimises the sums of squares of each variable's deviations from the mean, allowing for homogenous groups of people. Furthermore, the squared Euclidean

distance interval was used to determine similarities and differences across observations, and data values were normalised to minimise the impacts of the questionnaire scales.

To establish differences between clusters, the Mann–Whitney U statistic was used if the number of clusters to be extracted was two, or, if the number was greater than two, the Kruskal–Wallis statistic was used. In either case, differences were considered statistically significant when the *p*-value was less than 0.05 [78]. Finally, descriptive statistics were used to identify the individual, socio-economic, academic and institutional characteristics that influence dropout and retention among rural students.

#### **4. Results**

Regarding the statistically significant differences between rural students with the intention to drop out or to remain in the higher education programme, it was identified that the explanatory variables I9, I15, S15, A12, A13, A14, A16, A17, IES2, IES3, IES5, IES6 and IES7 were those in which the participants in the sample differed from each other. Table 5 presents the results of the Mann–Whitney U test.

**Table 5.** Mann–Whitney U test results between students with intention to drop out and with intention to stay.


\* Difference of medians with *p*-value is accepted <0.05.

Taking the variables in which statistically significant differences were identified as a reference, it was found that the (male) parents of students with the intention of dropping out had a lower educational level. At the same time, this group of students most frequently expressed that work and family obligations reduced the time they spent on their education. The need to move to study in a place other than the place of origin was more frequent in the group of students with the intention of dropping out. In terms of academic performance, students who indicated their intention to stay considered their academic performance to be outstanding or excellent.

In the case of academic preparation at previous levels of education, students intending to drop out most frequently stated that they were not adequately prepared for higher education. In addition, there is a higher level of dissatisfaction in the choice of training programmes among this student population, as well as a lack of access to technological resources for the correct development of their training programme.

However, for the variables of the institutional determinant, students with the intention of dropping out presented greater difficulties in communication with HEIs, as well as in attention from administrative staff. Similarly, this group of students consider that the technologies (e.g., virtual campus, specialised software and hardware) acquired by the institution are not necessarily the most appropriate, as they present greater dissatisfaction. The situation described above is the same in relation to their perception of the bibliographic resources (e.g., books or databases) that HEIs have. In relation to teaching, students with the intention of dropping out presented higher levels of dissatisfaction with the attention given by teachers to doubts and concerns, as well as the way in which the contents were taught. Table 6 presents the response counts for each of the student groups.


**Table 6.** Response counts among students with intention to drop out and to stay.


**Table 6.** *Cont.*

\* No, these are students who intend to stay. \*\* Yes, these are students with the intention to drop out. The variable codes are presented in Table A1.

Total 138 131 100% 100%

Never 3 9 2% 7% Occasionally 36 47 26% 36% Always 99 75 72% 57% Total 138 131 100% 100%

IES7

#### *4.1. Dropout in Rural Higher Education*

With the explanatory variables for dropout and permanence in which statistically significant differences were identified, for the study population that reported wanting to drop out, we proceeded to develop the cluster-based model. Thus, the total of 131 cases that made up the sample section were validated for the development of the hierarchical cluster. The cut-off was made at the rescaled distance 20 (see Figure 3), thus forming two clusters, the first with n = 45 (CD1) and the second with n = 86 (CD2).

**Figure 3.** Dendrogram. Note: The *x*-axis represents the cases of students with intention to drop out, and the *y*-axis represents the combination of rescaled distance clusters.

Regarding statistically significant differences between clusters, the results of the Mann– Whitney U test are presented in Table 7. Differences were identified in the explanatory variables I9, I15, S15, A14, A16, IES2, IES3, IES4, IES5, IES6 and IES7.

Both clusters were characterised by low levels of parents' education. Thus, for CD1, 20% of its members reported that their father had no education at all, 62.2% had completed primary school, 11.1% had completed secondary school and only 2.2% had completed their undergraduate degree. CD2 members indicated that 9.3% had not completed any level of education, 53.5% had completed primary school, 17.4% had completed secondary school and 9.3% had completed an undergraduate degree, while 10.5% indicated that they did not know their own father.


**Table 7.** Mann–Whitney U test results for CD1 and CD2.

\* Difference of medians with *p*-value < 0.05 is accepted.

Regarding work obligations, for CD1, 68.9% stated that these interfere with their educational process, while for CD2, the percentage was lower at 52.3%. In relation to the need to move from their place of origin to another city or municipality in order to be able to study, 42.2% of CD1 indicated having to do so. On the other hand, only 4.7% of CD2 students reported this situation. In the case of satisfaction with the choice of training programme, 66.7% of students in CD1 said they were satisfied, while 94.2% of students in CD2 said they were satisfied with their choice of training programme Concerning the availability of the necessary tools to carry out the work left in class (e.g., computer, internet, computer programs), 44.4% of the students in CD1 indicated that they did not have them, while only 9.3% of CD2 did not.

However, regarding the evaluation of the communication processes with HEIs, 88.9% of CD1 and 66.3% of CD2 indicated that it was not easy to communicate with HEIs. In line with the above, 82.2% of CD1 members perceive that HEI officials do not attend to their needs, and 46.5% of CD2 members perceive that HEI officials do not attend to their needs. In terms of the tools (e.g., databases, software, etc.) available to HEIs, 77.8% (CD1) and 18.6% (CD2) of students consider that these are not adequate. In relation to the training process, CD1 members tend to have perceptions that teachers do not deal with their doubts in a timely manner (84.4%), as well as that they do not impart the content in a simple way (82.2%). In these same aspects for CD2, 37.2% reported that teachers do not address their doubts, while 42.2% felt that they did not impart the content in a simple way.

#### *4.2. Permanence in Rural Higher Education*

In relation to the students who indicated that they wanted to remain in the undergraduate programme, it was identified that they conglomerate into two clusters (cut-off at the rescaled distance 20). Thus, the first cluster consisted of n = 99 (CP1) and the second of n = 39 (CP2). Figure 4 presents the dendrogram.

Regarding the statistically significant differences between clusters, Table 8 presents the results of the Mann–Whitney U test; differences were identified in the explanatory variables A12, A13, A14, A16, IES2, IES3, IES4, IES5, IES6 and IES7.

**Figure 4.** Dendrogram. Note: The *x*-axis represents the cases of students with intention to stay, and the *y*-axis represents the combination of rescaled distance clusters.


**Table 8.** Mann–Whitney U test results for CP1 and CP2.

\* Difference of medians is accepted with *p*-value < 0.05.

In relation to the differences identified, 71.7% of the related students in CP1 indicated that they considered their GPA to be outstanding or excellent, while for CP2 it was 82.1%. Regarding the perception of the students' preparation for entry to higher education, 65.7% of CP1 stated that their teachers had prepared them adequately. In the case of CP2, only 52.3% considered that their teachers had prepared them adequately for entry to HEI. At the same time, CP1 students reported being satisfied with the choice of the training programme in which they are enrolled, while for CP2 only 74.4% were satisfied with the training

programme. Finally, 81.8% of CP1 and 69.2% of CP2 considered that they carry out their training activities on time.

However, in the case of the institutional explanatory variables, 49.5% of CP1 and 89.7% of CP2 stated that it was never or occasionally easy to communicate with the HEI. Consequently, 74.4% of CP2 stated that HEI administrative staff never or occasionally attend to their requests. On the other hand, 74.1% of CP1 assessed that the administrative staff of the HEIs did attend to their requests and concerns. As for the technologies (e.g., virtual campus, specialised software and hardware) used by the HEI where they are studying, 90.9% of CP1 and 53.8% of CP2 considered them adequate. Regarding the bibliographic resources (e.g., books or databases) held by HEIs, 88.9% of CP1 members considered them to be relevant for the development of their academic activities, while for CP2, 66.7% did not consider them to be appropriate. Finally, regarding institutional processes related to teachers, 89.9% of CP1 and 12.8% of CP2 reported that teachers dealt with their doubts and concerns in a timely manner. Similarly, in regard to the way in which teachers teach the contents of the subjects, the perception of 89.9% of the members of CP1 was positive, however, for CP2, only 25.6% agreed with it.

#### **5. Discussion and Conclusions**

As presented in the results section, it was found that student permanence and dropout in the rural population is influenced by different explanatory variables. Thus, the findings of the present study concerning the individual determinant identified, firstly, that dropout is related to the educational level of the father, which is a discrepancy with previous studies, such as those of Guzmán et al. [11], Barbosa-Camargo et al. [81] and Lundetræ [82], because dropout is usually related to the educational level of the mother. This may be the result of the influence of other variables not evaluated in the present study, such as the cultural factors of these populations, or the low inclusion rates of this gender in the educational system [83]. Secondly, work and family obligations make it difficult for rural students to remain in the education system, which is consistent with the study by Hart and Vender [28], who reported the relevance of this variable as a predictor of dropout in the rural student population.

In the case of the socio-economic determinant, it was established that there are no statistically significant differences in variables (type of housing, socio-economic stratum, access to public services, being a beneficiary of state subsidies, income level and financing of studies) that are traditionally conclusive in the dropout or permanence of other types of students, such as urban students [9,27]. In view of the above, it was found that students with the intention of dropping out most frequently expressed the need to move from their place of residence to pursue their education.

For the academic determinant variables, it was established that permanence in higher education is related to education at previous levels, coinciding with the studies of Choi and Park [18]. Similarly, the satisfaction of rural students with the training programme is a deterrent to the intention to drop out. Unlike the study developed by Guzmán et al. [11] for a rural student population in undergraduate programmes in virtual mode, in the present study, by linking the face-to-face mode, it was observed that the academic variables do have an impact on the events of desertion and permanence. Finally, in the explanatory variables of the institutional determinant only, no statistically significant differences were detected between the typology of students in relation to institutional welfare plans and extracurricular activities. The above is contrary to the results presented by Warner [84] and Nishat et al. [70].

With regard to the internal comparison between the groups of students (with the intention of leaving or remaining), it was determined that, in the case of CD1, this is associated with lower educational levels of the father, greater interference of work and family obligations with studying and lower evaluations with regard to having tools for the development of their work, satisfaction with the training programme, the ease of communication and attention with the HEI, the technologies and resources provided by the institutions, the attention of the teachers and the simplicity of class teaching. For CD2, the main associated characteristic was the need to move from their place of origin to pursue their studies. However, in the case of permanence, when analysing the clusters, it was identified that there is no incidence of individual and socio-economic variables in this event, contrary to the findings of Georg [40], Guzmán et al. [85] and NEM [38]. CP1 was characterised by higher self-perceptions in relation to institutional variables, while CP2 was characterised by higher self-perceptions in relation to academic variables.

In light of what has been stated, it should be noted that the present study provides new insights into the events of dropout and retention in higher education for rural students by addressing variables that had not been previously addressed, such as marital status, type of housing, access to public services, state benefits, methods of financing their studies, dropout in other previous academic programmes, time of entry to higher education, number of subjects taken, attention from HEI administrative staff, the role of the teacher and extracurricular activities. The findings provide an opportunity for academics to further study these events and for public and institutional policy makers to modify current policies and create new ones in order to mitigate and prevent students' dropout, and consolidate their permanence at the educational level.

As a result of what has been described, it is necessary to recognise that policies that transcend over time are needed, with the aim of ensuring educational quality, reducing dropout indicators and increasing permanence rates in the rural student population. Considering that some of the variables that must be addressed for this purpose are not modifiable in the short or medium term, state efforts are required to improve the educational levels of parents, reduce the pressure of family and work obligations on students, improve academic performance prior to higher education, and support HEIs in adapting education to rural areas, especially when technologies are not adequate for this student population, among other factors.

To continue advancing in the research on dropout in rural higher education, it is necessary to verify how the variables studied behave in other contexts, both in developing and developed countries, in order to continue the discussion of the findings presented here. On the other hand, based on the modelling, it is necessary to verify the causal relationships between the determinants that explain the intentionality of permanence or dropout, in order to be able to propose strategies to prevent and mitigate student dropout in rural areas. On the other hand, the academic community is invited to continue deepening the study of this educational problem in rural populations, since it is not known what other variables analysed in other scenarios could influence the decision to stay or drop out.

However, for future research, some limitations of the methodological structure must be overcome, such as the transversality of the article, the sample size of the analysis groups, among other things. In addition, the results of the study must be interpreted from the limitations of the statistical analyses developed and the modelling technique selected, considering that all the variables analysed have the potential to explain dropout or retention in rural students studying in higher education; however, the variables that showed statistically significant differences are catalysts of these educational events. Finally, some of the findings presented here require further study, such as the limited influence of individual, socio-economic and academic variables on dropout or retention in rural higher education; hence, it is imperative to establish and deepen the causes of this absence.

**Author Contributions:** Conceptualization, A.G. and S.B.; methodology, A.G. and S.B.; software, A.G. and S.B.; validation, A.G. and S.B.; formal analysis, A.G. and S.B.; investigation, A.G. and S.B.; resources, A.G. and S.B.; data curation, A.G.; writing—original draft preparation, A.G. and S.B.; writing—review and editing, A.G., S.B. and F.C.-V.; visualization, A.G. and S.B.; supervision, S.B. and F.C.-V.; project administration A.G. and S.B. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of University Corporation of Asturias.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to the current Colombian laws that require the signing of a data transfer contract between the University Corporation of Asturias and the applicants.

**Acknowledgments:** The University Corporation of Asturias, whose support covered the cost of the publication, and Cecilia Carabajal, who, with her unconditional support, made the style corrections and translation of this article.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**

**Table A1.** Self-report questionnaire.


**Table A1.** *Cont.*



**Table A1.** *Cont.*


**Table A1.** *Cont.*



#### **References**


## *Article* **A Different Approach to Evaluation in Early Childhood Curriculum: Learning Stories**

**Gülüzar ¸Sule Tepeta¸s Cengiz <sup>1</sup> and Ahmet Altında ˘g 2,\***


**Abstract:** Many tools are used in child assessment during early childhood to support the development of children and plan the educational process. Originating in New Zealand, "learning stories" is a technique of observation that enables the assessment of children during early childhood. A learning story is the documentation by a teacher (or parent) of what a child (or group of children) is observed doing in an early childhood program. The adult must know the child well to properly assess a child's learning. Learning stories are narratives constructed from structured observations designed to provide a collective perspective on a child's learning. Observations obtained using this method are reinterpreted as stories, then analyzed and used as a basis for planning. Teachers collect 'critical events' or moments that seem important to a child. By analyzing a few of them through a narrative, they try to reveal the child's learning path and the pattern of their learning tendencies in the form of stories. The "learning stories" method, used in many countries across the world in early childhood, is not used in early childhood education in Turkey. In this context, this study was carried out to introduce the method to educators, families and researchers working in the field of early childhood, and to present the important points that should be considered during practice in the preschool period. This study was planned as a case study, which is a qualitative research method. The study group consisted of a teacher working in the preschool period (48–60 months) and a child in the classroom. The study included observations of the preschool teacher and examples of the learning stories created using these observations, followed by the analysis of the interview conducted with the teacher. It can be argued that the learning stories method is an alternative assessment method that can be used to evaluate the curriculum in early childhood education in Turkey, as is the case in many countries.

**Keywords:** learning stories; early childhood curriculum; program evaluation

#### **1. Introduction**

In New Zealand, educators use learning stories to assess children's progress in early childhood. This assessment tool is also a narrative tool with which teachers observe the children as they play and learn, in order to record their experiences in the classroom and school based on these observations [1]. The stories created through these observations are transformed into narratives that are intended to be shared with children and families. Learning stories serve as a pedagogical tool to discover children's strengths, assess and examine children's learning processes, and to help them reflect on their role in the process [2].

By focusing specifically on the positive aspects of children and their role as learners, developing their abilities by highlighting what children are able to do and supporting them to progress step by step by setting goals, the learning stories technique is a teaching and learning approach that differs from other methods, which usually present the negative aspects to children and families, emphasizing what children cannot learn and what they cannot do. At the same time, the learning stories technique differs from various other

**Citation:** Cengiz, G. ¸S.T.; Altında ˘g, A. A Different Approach to Evaluation in Early Childhood Curriculum: Learning Stories. *Sustainability* **2022**, *14*, 11218. https://doi.org/10.3390/ su141811218

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Leandro Silva Almeida and Silvia Fernández Rivas

Received: 10 August 2022 Accepted: 5 September 2022 Published: 7 September 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

assessment methods, since it enables families and their children to participate actively in the assessment process and supports assessment by individual and family narratives [3,4].

Learning stories do not apply a predefined and defined schema to children's actions they focus on the starting point of children's behavior and how children develop these behaviors in the classroom. For this reason, it focuses on what can be done and what can be improved, instead of negative messages. In this context, it has a constructive and positive perspective [5]. Focusing on children's learning tendencies prevents children from focusing on their developmental areas and developmental support based on only one area. For example, children are evaluated not only on their skills such as running, jumping, or just counting and addition, but also on how they use these skills in their games and in observable areas. This approach tries to consider all the interests, personalities, and developmental integrity of children [6].

Learning stories are utilized in many countries today (Sweden, Norway, Germany, Ukraine, etc.). In addition, learning stories are among the assessment tools used in alternative learning approaches in early childhood, such as the Reggio Emilia Approach and Forest Pedagogy, which have come to the forefront in recent years with their different perspectives on education [7].

The way learning stories appeal to children, the narrative language used in the narrative, aims to develop children's capacity for self-recognition and reflection [8]. These stories are intended to ensure that children become individuals who are aware of their own abilities and can self-regulate [9,10].

The learning stories assessment approach is a form of observation recorded in the form of a narrative story and the documentation of what is obtained through observations. As children explore through play, teachers listen, watch, and take notes on what they do. They form opinions by taking photos or recording their notes with videos from time to time. Through the learning stories explained with this evidence, teachers create a path to share with families what they observe about their children's development. Teachers take on the role of observer and author while reflecting on children's actions, words, and expressions in the process. The story they write is a positive text about children's strengths, good ideas, learning adventures and tendencies. Various steps must be followed when creating a learning story, as listed below [11,12]:


Since the games are born from the child's ideas, interests, or curiosity, teachers should focus on the games that children initiate when they document their learning stories. Discovering these moments and developing them is the important part [13]. Children's engagement, focus, and deep connection with play are important in learning stories. In this context, teachers should diligently try to capture these moments. The teacher should think about the child's relationship with their friends, materials, or environment and try to expand these processes. The deep focus on these processes, where learning tendencies are revealed and goals are set that will enable the child to expand these processes, is the most valuable aspect of learning stories [14].

The learning stories can be used to talk and acquire the following points with children in the following early childhood period [8,15]:


Using learning stories plays an important role in showing children that their discoveries are valued and that their play and ideas are respected. This has a special value for both children and their families. The process encourages children to review their learning adventures and to feel proud of their teacher's focus on their adventures [3].

Using learning stories provides an opportunity for teachers to plan these processes by paying attention to the developmental goals of children while developing their programs. It provides support for teachers to expand their education programs and curricula in this context by focusing more on the developmental differences and individual development of children. It provides an opportunity for all teachers, staff, and families working with the children to get to know them better. It helps families become aware of their children's characteristics, to get to know their children better and to understand how to guide them. With the help of images and videos, the teachers create a guide for families, children, and all educators who will meet the child in the future [16].

It is thought that learning stories are appropriate for all ages in early childhood. This method can be used at any time of the day when children are involved in any learning activity [17]. The documentation includes recording, photographing, and taking notes of a child's learning stories over some time by a teacher. These documents are then analyzed by the teacher to describe one or more of the child's five typical learning areas: "Attention", "Participation", "Persistence in the face of difficulty or uncertainty", "Expressing an idea or an emotion", and "Taking responsibility" [15]. These five learning trends are well aligned with the New Zealand early childhood learning curriculum. Carr [15] recommends adhering to the following rules for a learning story to better reflect children's tendencies in the learning process: "definition", "documentation", "discussion", and "decisions".

Although learning stories are thought to be effective in improving learning and learning experiences in early childhood education in Western countries such as New Zealand, the United Kingdom, Germany, Iceland, etc., this method is still unknown in many parts of

the world. Over the years, learning stories have also received criticism from some scholars. These criticisms are often related to the fact that learning stories alone are not sufficient to evaluate a child. There are also criticisms that suggest it should be used with many tools, not as a tool on its own.

#### **2. Materials and Methods**

This study aimed to examine the use of the learning story method in the curriculum of a preschool teacher in Turkey, who received training on the use of the "learning stories" assessment method, which is not widely used in this country in early childhood, and to present a sample practice.

#### *2.1. Study Design*

This study was designed as a case study, which is a qualitative research approach. A case study is an in-depth study of one or more situations by reviewing, examining, researching, and interpreting them [18]. According to another definition, a case study is defined as an approach in which the researcher examines one or more situations in a specific time frame by using multiple data collection tools, namely observation forms, interviews, document reviews, reports, etc. The most important purpose of the researcher in case studies is to observe and present the specific situation or situations to be discussed.

#### *2.2. Participants*

This study discussed the case of a preschool teacher who worked with 48–56-monthold children in the preschool period at the time of the study. This teacher was selected because the "learning stories" method was already used in children's assessments in the classroom in line with the curriculum. The selected preschool teacher had received various pieces of training on the subject before the study. Since the study topic has almost no applications in the preschool period in Turkey, the practices of this preschool teacher were deemed valuable and suitable for the case study.

#### *2.3. Procedure*

The learning story assessment method used in this study included observing children, supporting observations with evidence, recording and collecting data, and analyzing documents. The "Learning Stories Observation Form" was used to collect the data obtained in this study. This form included a section for teachers to write their observations, a section to present visual evidence regarding their observations, a section for teacher assessment, a discussion section to assess in regard to the curriculum, and a section to plan further activities. In addition, the form included a family opinion section to take the opinions of the families about the learning stories. The observation form with these specified areas was developed by examining all of the learning story forms in the literature, making translations, and adapting them according to the program in our country. Observations about various activities related to the curriculum were turned into learning stories by the preschool teacher. Examples from the observations made within the scope of this study and the preschool teacher's views on the learning stories were also presented descriptively in the findings.

#### **3. Results**

The study was designed as a case study because it included the unique experiences of a preschool teacher who, after undergoing various pieces of training, set out to use the learning stories method in the classroom as an assessment tool and to establish a link between the method and the curriculum. The findings are presented in two main parts. The first part includes the data on the Learning Stories Observation Form used by the teacher (N.S.) in practice, and the second part presents N.S.'s views on the use of learning stories.

The participating preschool teacher, N.S., prepared the learning stories by following these steps during implementation.

#### *3.1. Planning the Learning Process*

In the preliminary meeting, N.S. explained the measurement assessment tools used in the classroom for preschool children and their relationship with the curriculum in the following words:

"*In the preschool period, monthly learning processes are prepared in addition to daily learning processes. I prepare my monthly plan first and select some acquisitions and indicators according to the children's needs. Afterward, I take care to address these in different activities by dividing them over the month according to various developmental areas. I make notes about whether these acquisitions have been achieved in the assessment section included in the observation part or under the activity plans. I do not write my observations in any other specific form. If something catches my attention, I take notes; I try not to forget it. I examine the acquisitions with a general observation and create the developmental assessment forms at the end of the semester. Other than that, I don't use a different assessment method."*

According to the statements of N.S., it can be argued that the observations in the preschool period are made based on acquisitions and indicators and in a general manner, and detailed and planned observations are not often preferred.

After the preliminary interview, N.S. was asked to plan the daily education flows and list them for a daily flow, considering what should be attended to (to be observed) about the children in these flows. While preparing this list, the teacher was told to pay attention not only to the indicators and concepts but also to many issues in line with the curriculum. N.S. created the following list (Table 1).


**Table 1.** Preschool teacher N.S.'s list of observations that can be made in activities to be carried out in and out of the classroom for a daily flow.

After thinking about the list, the preschool teacher N.S. detailed the issues that could be observed according to the daily flow and started to focus on the observation steps. Beginning to discover the relationship between the learning process and assessment, N.S. started to gain experience about which aspects required attention in the development of learning stories and in the assessment while planning activities.

#### *3.2. Observation and Data Collection*

N.S. was asked to make assessments in the form of Learning Stories for the selected children after developing the curriculum and making the relevant preparations. No specific child or activity was recommended to the teacher to focus on during observations. The teacher was asked to structure the process based on the training previously received on Learning Stories. In the preliminary interview, N.S. expressed the following views on observations and data collection.

*"Making observations is a very open-ended concept in preschool. In my opinion, the openended nature of what will be observed or when it will be observed makes it rather difficult to use observation sometimes. After a while, making observations may be regarded as taking a glance, or simply watching children's games. When observation is needed in my classroom, I sometimes have difficulties regarding whom to observe and how to make observations. I believe it would be sufficient to observe whether the child in question has acquired the acquisition indicators in the curriculum."*

In the preliminary interview, N.S. reported that it was sufficient to observe only the acquisitions and indicators, in short, the written indicators related to the curriculum in the preschool period. Further in the process, when asked to convey the experiences about observation following the trainings and the practices about learning stories, N.S. expressed the comments presented in Section 2.

After the preliminary interview, N.S. conducted many observations with the participation of the family, using the learning stories observation form in accordance with the curriculum for the selected children. Considering the frequency of implementation of these observations, it was found that they were made daily for one or two children. N.S. filled the observation tools at the end of each day using the notes and photos taken in the classroom and presented them to the families and children in accordance with the implementation steps. Learning stories were shaped with their contributions. Some sample observations developed by N.S. in this regard are presented below.

#### **THERE IS A CURIOUS CHILD IN THE LIBRARY! Child's Name: U . . . C . . . Observer/Author: N.S. Observation Date: 11.01.2022**

**Photo(s) Story** Today we all went to the library together. UC, you love books, reading and anything related to books. Therefore, the idea of going to the library made you very happy and excited. I didn't think there were many children's books in the library, but I didn't share this thought with you. You were very happy to

gratifying to see this.

While many of your friends were running around instead of spending time with the books, you were studying them with all your attention. You took a few books to share with your friends and ran to sit next to them. You can see it in the Photo 2. You handed over some books to them and you looked like as if you knew how to read. You started reading the sentences in the books and telling them to your friends (imitating reading). You tried to get the attention of your friends by showing them pictures. You can see this in Photo 3.

be in the library but surprised that there were no children's book in the library. You ran and started to examine the colorful books, especially the animals, on the shelves as in Photo 1. You started taking books that were interesting to you from the shelves and sharing them with your friends. It was very


#### **U.C's opinions about the learning story The family's opinions about the learning story**

Teacher, I liked the story very much. You wrote about me. I couldn't read the books to my friends B.H and M.H, because I don't know how to read. I just showed them the photos and told them about the microbes in the photos. I talked about the work of scientists. I wish there were children's books in the library then I could talk about them as well. I could explain them about bacteria, organisms, with photos that were better suited to children. Because I love living things.

I'm so glad you discovered U.C's interest in books. He has been introduced to many children's books since his infancy. We take care to read him an average of 4–5 books every day. Now, the request to read a book comes from him, not from us. U.C tried to explain things to his best friends today, talked about books, and I think that made him very happy. When he got home today, his only complaint was that there weren't enough children's books in the library. I think it's time we introduced him to children's libraries. I had thought that he would not enjoy the library trip very much, but his interest in the subject made me very happy.

#### **I'M WORKING IN THE WORKSHOP WITH MATER˙ IALS FROM THE NATURE Child's Name: U.C. Observer/Author: N.S. Observation Date: 03.03.2022 Learning Outcomes**

**Photo(s) Story**

Today, it was the day to make creative designs. We went to the loose part workshop in our school together. There you had the opportunity to freely examine and explore the materials and make the designs you wanted. You (UC) studied the materials really carefully with your friends. I think the place caught your attention. While you were designing, you were particularly interested in corks and you took a very close interest in them. You exchanged ideas with your friends and showed them the materials you chose. The screws of different sizes especially got your attention.

You made your design using screws, beads, corks. You really like to make designs. I like it when you make an array and tell me about it. You designed some kind of vehicle during the activity today. This vehicle was a machine that could go back and forth, helping you with the housework when needed. You placed the beads and other materials one by one and they all had a task. I admired that you focused on your design so carefully and that you created a really different design.


Teacher, I love the loose parts workshop, it is fun to design there. I wish we always went there. There are shiny stones, beads, and many more. Yes, I had built a house. I also made a design. I'm going to show these to my mom.

#### **U.C's opinions about the learning story The family's opinions about the learning story**

UC has always told us that he loves the loose parts workshop at the school. Especially lately, I have been observing that he has been making designs with materials such as kneading materials, buttons and beads at home. He finds them and brings them to tells me about them. He recently made me the following designs. He said these were a snail house and a mandala. He had a bit of difficulty while making the mandala, but I supported him. Thus, we had some quality time together. The fact that he brings together unrelated and different materials to create something really impresses me.

The opinions obtained in the last interview from the preschool teacher, N.S., regarding the learning stories are provided below:

*"When I first heard about the learning stories, it seemed to me as if it were a very hard method to apply in the classroom. Then I thought about the form given to me and started to make observations to apply it to my student. Frankly, these observations highly impressed me, because I knew my student U.C., but I realized that I didn't know him that much when it comes to details about him. Thanks to the observations, I started to focus more on the capabilities and abilities of U.C. This was the aspect that I really liked."*

*"I was exciting to gather the evidence to design the learning story. I realized that the pictures I took must thus have meaning. We take pictures of children in the classroom and sometimes we upload them to our web page or share them in communication groups, with the permission of the parents. However, I usually took these pictures to disseminate the activity. Thanks to the learning stories, I started taking photos to provide evidence for the observations about U.C. This showed me that the pictures I took could be more useful. While explaining the pictures, it became very easy to write the story."*

*"I think one of the best aspects of learning stories is that they support you in communicating with the family. Thus, the relationship with the family starts to include more trust, because with learning stories, you write the story of the child and it is as if the*

*family is experiencing what happens in the classroom through your words. Sometimes the family has important observations at home, and by adding their observations, the family contributes these observations to ensure that the learning story is multifaceted. Sometimes, these stories provide an opportunity for families to get to know their children better."*

*"While writing the learning stories, filling in the section about the relevance for the MoNE program was the most challenging aspect for me. It was challenging to concretize the learning outcomes and write their relevance to the program, because although some observations are very valuable, they do not define a behavior or an indicator in the program. In this case, it is difficult to write this section."*

#### **4. Discussion and Conclusions**

It was determined that the findings of the study had parallels with the literature. During the interview, the preschool teacher N.S. expressed the following opinion: *"When I first heard about the learning stories, it seemed to me that it was a very difficult method to apply in the classroom".* Similarly, in the literature, Blaiklock [19] and Zhang [20] state that learning stories are long, and that writing them is comprehensive and sometimes time-consuming. However, it is also reported that the learning stories method brings a comprehensive and detailed perspective to the development of children. In the same vein, Blaiklock [17] explains that there are some shortcomings across the world regarding the learning stories technique and adds that this method should be explained in more detail, that more work should be done on where, when, and how often learning stories should be applied, and that guidance should be provided on the collection of obtained results. Similarly, in their study on the problems encountered by teachers in using learning stories, Davis, Wright, Carr, and Peters [21] found that teachers do not have a good grasp of children's developmental stages and their competencies in these stages. This shows that teachers sometimes fail to mention and address the important events in children's development. The participating preschool teacher, N.S., emphasized the relevant difficulties as expressed in the literature with the following statement: *"While writing the learning stories, filling in the section about the relevance for the MoNE program was the most challenging aspect for me. It was challenging to concretize the learning outcomes and write their relevance to the program, because although some observations are very valuable, they do not define a behavior or an indicator in the program. In this case, it is difficult to write this section".* Examination of the sample learning stories prepared by the teacher shows that there is a detailed narrative process with an average length of 2 or 3 pages.

Preschool teacher N.S. He replied to the interview questions as follows: "*I was excited to gather the evidence to design the learning story. I realized that the pictures I took must thus have meaning. We take pictures of children in the classroom and sometimes we upload them to our web page or share them in communication groups, with the permission of the parents. However, I usually took these pictures to disseminate the activity. Thanks to the learning stories, I started taking photos to provide evidence for the observations* ... ". When the literature is examined, Aschermann, Dannenberg, and Schulz [22] state that the memory of children's development is quite classy, that is, teachers forget most things. For this reason, he says that photographs can be used as concrete materials to follow the development of children. He stated from time to time that a single photograph does not make any sense, and that a series of photographs of children is required to create a story. The teacher who participated in the study also stated that learning stories is a good way for the children to blend the photographs with their observations. Reese and Cleveland [23] emphasized that purposeful and focused photographs are important materials in following the development of children. Takacs stated in Swart and Bus [24] that purposeful development photographs, which can be used both in print and digitally, are an important part of portfolios and developing assessment tools.

The participating preschool teacher, N.S., also noted the many valuable and fieldspecific aspects of the learning stories. For example, with the following statement, the

teacher expressed the role of learning stories in family training and communication: *"I think one of the best aspects of learning stories is that they support you in communicating with the family. Thus, the relationship with the family starts to include more trust"*. In the literature, in their study about the importance of the learning stories approach in communicating with the family, Carr and Lee [1] revealed that families receive better and more detailed information about their children in this way. Southcott [25] stated that the learning stories approach strengthens the bond between the children's families and the school and includes their families in the learning processes of the children. Similarly, Cowie and Mitchell [26] also showed this approach as an important tool in maintaining a healthy and strong bond between the families and the school in which the learning stories approach is used.

According to the findings obtained in the study, N.S. emphasized the contribution of the learning stories to getting to know the students in more detail and to discovering their characteristics while creating their learning stories by stating *"I knew U.C., but I realized that I didn't know him that much. Thanks to the observations, I started to focus more on the capabilities and abilities of U.C. This was the aspect that I really liked".* In the same vein, comparing the use of the curriculum and assessment methods of Northern European countries such as Norway, Sweden and Finland, and the learning stories method, also known as the New Zealand method, based on teachers' opinions, Alvestad and Duncan [27] found that the learning stories method is a very inclusive method in assessing and explaining the evidence gathered by preschool teachers. They also reported that it is an important assessment tool that can contribute to the individual portfolios of many children with differences, even in different age groups. Emphasizing that this assessment method highlights differences, Dunn [28] indicates that it is an effective form of assessment not only in early childhood education but also for children with special needs. In this way, it is possible to evaluate children's abilities in a detailed way by being aware of their abilities at every step. In their study, Nyland and Alfayez [29], experimented with the use of learning stories with Saudi Arabian children and they revealed that the use of learning stories is an inclusive way to recognize and explore the individual characteristics of children living in Saudi Arabia.

There is evidence in the literature that children and teachers are both positively affected by learning stories. Carr and Lee [1] and Reese [30] stated that children listen to their own learning stories and perceive themselves like a fairy tale hero, and they are curious about the new activities and stories that this fairy tale hero will experience. This way, they are motivated to learn and demand more stories about themselves from their teachers. Similarly, Reese, Gunn, Bateman, and Carr [30] reported that using the learning stories method improves teachers' language skills. They found that these texts, which were examined in terms of the linguistic reflection of the interactions with children, the organization of the text, the conversion into story format, vocabulary variety and the complexity of expressions, developed the language structures of the teachers in a positive way.

In their study, Reese et al. [30] reported that learning stories read to children support children's language development, and that this development is reflected in speaking, word diversity, word count, and the length of sentences that are used by students, similar to the development observed in teachers. Picken and Milligan [31] stated that learning stories should be used not only in early childhood, but also in later grades such as middle school and high school, and presented some examples in this context. They argued that such a method, which is based on presenting children's performances individually by valuing their individual development, should become globally widespread.

**Author Contributions:** Conceptualization, G. ¸S.T.C. and A.A.; methodology, G. ¸S.T.C. and A.A.; validation, G. ¸S.T.C. and A.A.; writing—original draft preparation, G. ¸S.T.C. and A.A.; writing—review and editing, G. ¸S.T.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **User Preference Analysis of a Sustainable Workstation Design for Online Classes: A Conjoint Analysis Approach**

**Ma. Janice J. Gumasing, Ardvin Kester S. Ong \* and Maria Angelica D. Bare**

School of Industrial Engineering and Engineering Management, Mapúa University, 658 Muralla St., Intramuros, Manila 1002, Philippines

**\*** Correspondence: aksong@mapua.edu.ph; Tel.: +63-(2)-8247-5000 (ext. 6202)

**Abstract:** The impact of the COVID-19 pandemic has caused the sudden shift of the interactions between students and teachers from the four corners of the classroom to fully online learning through the workstations. By using a conjoint analysis approach, this study aimed to evaluate the preference of undergraduate students from the Philippines on the workstation design attributes during the pandemic. Seven attributes were examined through orthogonal design: the type of gadget, keyboard, mouse, earpiece, desk, kind of chair, and light device. Through a purposive sampling approach, a total of 315 undergraduate students from the Philippines were gathered. Through an online survey with two holdouts, the respondents voluntarily responded to 51 stimuli produced by IBM SPSS using a 7-point Likert scale. Type of gadget (37.925%) was found to be the most significant attribute preferred by the students, followed by the type of mouse (28.345%), kind of chair (14.840%), type of keyboard (7.548%), earpiece (7.177%), light device (2.109%), and desk (2.056%). It is worth noting that the preferred workstation design combination is the personal computer, mechanical keyboard, wired mouse, headset, height-adjustable table, high-back office chair with neck support, and floor lamp. This study is the first to use a conjoint technique to examine undergraduate students' preferences for workstation design attributes during the COVID-19 pandemic. Finally, by applying the attributes and design developed in this study, the conjoint method can be used and expanded to evaluate the workstation design attributes of other courses and even employees worldwide.

**Keywords:** workstation design; online classes; user preference; conjoint analysis

#### **1. Introduction**

Due to the impact of the COVID-19 pandemic, universities across the world were prohibited from conducting face-to-face classes and were mandated to have online learning set up to resume the curriculum [1,2]. With the sudden shift to e-learning back in 2020 as a response to the escalating active cases of COVID-19, students were not prepared in the first few months of the lockdown [3]. This shifting has introduced new challenges, causing instructors, students, and parents to become more stressed [4]. Two years into the COVID-19 pandemic, people were left with zero options and were resilient as they managed to adjust to the remote learning and work-from-home setup. Taking the Philippines as an example to represent one of the third-world countries, Magsambol [5] forecasted a total of 44,609 state university and college students who would not enroll for the academic year 2020–2021 due to several reasons such as lack of gadgets, no proper workstation setup, financial problems, and fear of contamination. On the same grounds, as of late 2020, 13% or 4.4 million school-age Filipinos did not enroll that year, wherein 1.7 million were aged 18–20 [6].

However, as face-to-face classes were yet to be seen at that period, students and their parents started purchasing gadgets and workstation equipment to adapt to the new learning setup [7]. Flores [8] reported that in the recent survey conducted by Social Weather Stations (SWS), 58% of enrolled students were able to acquire the required gadgets for an

**Citation:** Gumasing, M.J.J.; Ong, A.K.S.; Bare, M.A.D. User Preference Analysis of a Sustainable Workstation Design for Online Classes: A Conjoint Analysis Approach. *Sustainability* **2022**, *14*, 12346. https://doi.org/10.3390/ su141912346

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Leandro Silva Almeida and Silvia Fernández Rivas

Received: 31 July 2022 Accepted: 21 September 2022 Published: 28 September 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

online class, 27% of them had their equipment, 12% had purchased their own, 10% had borrowed, 9% were given, and 0.3% were rented. Furthermore, 79% of those who bought or rented equipment for distance learning got a smartphone, 13% got a desktop or laptop, 5% got a TV, and 3% received a tablet. In addition, Ziady [9] reported that online sales of furniture companies such as IKEA surged up to 45% during the pandemic as people converted and made a workstation for online classes and work tasks in their houses.

Consequently, the definition of a workstation for students changed due to the current setup. A workstation for online classes is an area in the students' houses that allows them to attend courses virtually and accomplish school tasks. Obeidat and Al-Share [10] mentioned that during online learning, the interactions between students and instructors take place in the workstation, which serves as the location for teaching and learning activities related to design. Not only does the effective interior design of the workstation improve the functionality of such a learning environment but it also boosts the confidence of the individuals who will be participating in the teaching and learning process. Having a separate workstation for online classes alone benefits students because it increases their productivity and satisfaction staying indoors while attending classes, resulting in reduced perceived academic stress and mental health issues [11,12]. However, various attributes and considerations must be considered when designing an efficient and effective workstation for students, especially since they have different situations at home. The attributes of a workstation are the type of gadget used, kind of keyboard, mouse, and earpiece, type of desk and chair, and light device, as these objects are commonly used by students and significantly affect their performance [13,14].

In developing countries, the online learning setup became quite a challenge among students and faculty members. Prasetyo et al. [15] considered the evaluation of online learning platforms during the COVID-19 pandemic through an extended technology acceptance model and the Delone and McLean IS success model. It was seen that the perception of usefulness and perceived ease of use were primary factors for students to accept and utilize a specific platform. Students highlighted that the overall usability of platforms would provide the ease of utility to help them perform better during online learning. In India, Bast [16] presented how the perception and preferences of students would be affected based on where they were residing. Similarly, Muthuprasad et al. [17] explained that students in India preferred smartphones for their online learning education. However, Ong et al. [18] showed such challenges as laboratory activities that cannot be fully accomplished and appreciated during online learning. In addition, Pradana and Syarifuddin [19] highlighted the challenges of online learning in Indonesia. Their study showed how insufficient access to online teaching and learning were evident: most importantly, learning facilities, equipment, and students' study place were not available and inefficient. This provided a basis for challenges in the online learning setup that affects student's learning.

In the Philippines, Ong et al. [20] utilized conjoint analysis to assess the preference for online education among industrial engineering students at various educational levels during the COVID-19 pandemic. However, the attributes used in the study included only students' preference for online delivery of classes, including the delivery type, interface template, term style, final requirement, use of Coursera, practice sets, and delivery platform. Similarly, Ong et al. [21] also used the same methodology to determine senior high school students' preference for online learning, which focused only on six attributes: delivery type, assigned tasks, evaluation, use of virtual laboratory, interface layout, and delivery platform. However, the various attributes identified have their levels, offering the same function but different utilization, efficiency, and comfort that affect the students' preferences.

Students' preference for their workstations influenced their intent to purchase the attributes mentioned. Considering what the students prefer is one approach to assessing their satisfaction and engagement during online classes. Cakiroglu et al. [22] revealed that considering the students' preferences positively motivated them and related to their academic achievements. Likewise, a convenient and user-fit workstation design is essential to creating a stress-free, task-efficient, and comfortable place for students to study [11,12,23]. Supporting this, Xiao et al. [24] discovered an association between the chance of developing new health problems during the pandemic and having an adjustable workstation setup. As technology has progressed over the past few decades, the workstation setup considerably caters to users' convenience, further shaping how people and society develop [25]. In addition, ergonomics was brought into the picture, ensuring a suitable fit in terms of comfort and safety among humans and their tools [26]. Disregarding ergonomics could lead to designs that are uncertain to be economically successful because they do not cater to the users' needs. According to Mohamed Makhbul et al. [14], a non-ergonomic workstation can cause stress to a person. Nirmal et al. [27] discovered that online education has ergonomic concerns; hence, students and teachers should arrange for adequate interventions such as proper setup, equipment, and breaks to ensure health and safety.

Multiple studies on workstation design have emerged in the past months due to the developing health and social risk to students and office workers during the pandemic. However, past researchers mainly focused on evaluating the different equipment used during online classes and proposing ergonomic designs for desks, chairs, keyboards, mice, etc., to mitigate the risks separately. Despite the availability of studies about workstation design, there is insufficient literature focusing on the workstation design preference based on students' perspectives and experiences. In the Philippines, a study conducted by Jajoo et al. [13] focused on analyzing the current setup of work-from-home employees and designing a comfortable office workstation. Their results showed that adjustable chairs and tables are necessary to promote comfort and work efficiency. Furthermore, illumination through a table lamp, light-painted walls, and a comfortable temperature through a portable fan will also affect their work.

Since considering the students' preferences during online classes significantly affects their performance, this study utilized a conjoint analysis approach. Conjoint analysis is a standard method used in market research to determine the consumers' and users' decisionmaking regarding their preferences [20,21]. Despite the insufficient literature utilizing the conjoint analysis to measure the students' preferences on their workstation setup during online classes, few studies evaluated the students' preferences regarding online learning. Given this, no previous studies have used a conjoint analysis method to examine user preferences for workstation design during the COVID-19 pandemic.

Utilizing a conjoint analysis approach (CAA), this study aimed to determine the combination of workstation design attributes most preferred by undergraduate students in the Philippines during online learning only. In particular, the present study considered seven attributes: type of gadget, keyboard, mouse, earpiece, desk, chair, and light fixture. CAA with orthogonal design was used to evaluate the users' preference for the workstation design. Lastly, the study further examines the specifications (attributes) and levels to elaborate on the preference of students in online learning workstation design.

As a result, the present study recommends a convenient workstation design based on the users' preferences. The results also give leverage to the marketing strategy of gadget developers, business owners, and retailers, since they will be able to recognize and understand which attribute of a workstation design the users value, among others, as well as the significance of each attribute on their decision-making [21]. Business owners need to understand their customers' purchase intentions and match their preferences to help the business competitively to stay in the market. Finally, the study's findings may be utilized by students, teachers, and employees in the same online setup, globally, even in a post-pandemic setup, as universities are offering at least blended or even fully online setup.

#### **2. Methodology**

#### *2.1. Conceptual Framework*

As seen in Figure 1, four phases were used to achieve the objectives of this study. Initially, the researcher determined the attributes and levels of a workstation design through related literature. Next, the researcher generated the stimulus or combinations of levels per attribute by utilizing the orthogonal design in SPSS. After that, the respondents evaluated

each combination presented to them through an online questionnaire as the study's data gathering. Through preliminary runs, 150 random samples were obtained to validate the combinations considered. A Pearson correlation value of 0.90 was obtained and found to be valid [20,21]. Full dissemination of the questionnaire using a 7-point Likert scale was conducted.

**Figure 1.** Conceptual Framework.

#### *2.2. Data Gathering*

This study aimed to accumulate data from users of workstations during online classes, particularly the undergraduate students in the Philippines, regardless of their course. The target respondents were undergraduate students, because they are the group who spend more time in their workstations due to numerous online activities, such as discussions, meetings, online presentations, and tasks [28,29]. In the Philippines, undergraduate students are usually 18 to 26 years old [30]. Female and male groups of undergraduate students' anthropometric measurements do not have significant differences [31], which is favorable in developing an ideal workstation design for the said group level. Hence, only 18- to 26-year-old undergraduate students were considered for this study.

As cited in the study of Li and Wang [32], the minimum sample size required in conjoint analysis research is 75. Based on that, the study required at least 315 participants to rate the stimuli generated by SPSS 25. Furthermore, due to the COVID-19 pandemic, the responses were obtained through online survey using Google Forms dispersed through various social media platforms, as Sethuraman et al. [33] recommended. The questionnaire was open from April 2022 to May 2022 to give ample time to students to answer the online survey. Using a purposive sampling approach, the survey was intended for undergraduate students who were currently enrolled through online learning. Lastly, since Filipino undergraduate students were the target respondents, data collection was conducted in the Philippines and not restricted to a single geographic area.

#### *2.3. Demographics*

Through purposive sampling, 315 Filipino undergraduate students participated in this study. As seen in Table 1, most respondents were aged 20–21 years (65.40%), 39.37% were male, 58.41% female, and 2.22% preferred not to mention their gender. Furthermore, most of the respondents were located in National Capital Region (NCR) (44.44%) and Calabarzon (Region IV-A) (40.00%). In sum, 76.83% were enrolled in a private school and 23.17% in a public school. The courses of the respondents were engineering (45.71%), business (17.78%), social sciences (13.33%), health sciences (11.11%), architecture and design (6.35%), formal sciences (1.59%), natural sciences (0.95%), public administration (0.95%), agriculture (0.63%), media and communication (0.63%), humanities (0.32%), education (0.32%), and transportation (0.32%). Most of the respondents' height (in cm) ranged from 155.76 to 170.50 (51.11%), and 141.00 to 155.75 (30.16%). Lastly, most of the respondents' weight (in cm) ranged from 36.00–57.00 (48.57%), and 57.01–78.00 (40.32%).



#### *2.4. Conjoint Design*

Table 2 presents the attributes and levels of a workstation design for online classes usually considered by the users. Specifically, the study considered seven attributes, namely, type of gadget (personal computer, laptop, or tablet), type of keyboard (ergonomic keyboard, multidevice Bluetooth keyboard, mechanical keyboard, or membrane keyboard), type of mouse (ergonomic mouse, Bluetooth wireless mouse, USB wireless mouse, wired mouse, or touchpad), earpiece (headset, headphone, earbuds, earphones, or speaker), desk (height-adjustable table or sitting desk), kind of chair (high-back office chair with neck rest, mid-back office chair, or task chair), and light device (desk lamp, floor lamp, or room light).



The attribute type of gadget refers to the electronic device the students primarily use to attend online classes and perform their tasks. Emerson et al. [34] identified desktop or personal computers, laptops, and tablets as essential tools for communication and task-making of users. Several studies have found that different technological aspects of the type of computing facilities, such as gadgets, can affect student performance, which suggests that differences in the gadgets available to students may be significant. According to Ong et al. [21], students found that testing devices with larger screens were easier to read. In a study conducted by Prasetyo et al. [15], the researchers discovered differences in the levels of performance on online tests achieved by various devices in at least one state. Furthermore, the intention of students to purchase a particular gadget to utilize for an online class depends on three factors: financial capability, availability of the product, and preference [34]. Thus, the type of gadget significantly affects the students' school performance, since they have different technological capabilities.

The second attribute, the keyboard type refers to the students' device to directly input data or text into their gadget [35]. According to Ling [36], users' familiarity and proficiency with a particular type of keyboard could affect their efficiency and performance on their tasks. This is because the degree of familiarity with the keyboard or other input devices used in the assessments may affect the speed and accuracy of answering questions. It has also been discovered in prior studies that familiarity with computers, which includes knowledge of and experience with the input device, affects writing performance [37,38]. Thus, the type of keyboard students use during online learning significantly affects their efficiency, accuracy, and effectiveness in writing assessments. The present study considered four levels: ergonomic keyboard, Bluetooth keyboard, mechanical keyboard, and membrane keyboard or the standard desktop keyboards as the type of writing device during online learning. These keyboards may have the same function, but differ in providing efficient key layout, typing comfort, and convenience to users, affecting their work efficiency.

Furthermore, like a keyboard, a mouse is a primary tool in a workstation to input data to a computer, which is why it is considered the third attribute in this study [35]. According to prior studies, the student interaction data, especially mouse movement data, could help better model and understand students' learning behaviors [39,40]. Mouse movement trajectories, which include the mouse interaction timestamp, mouse event type, and mouse coordinates, can predict student performance in interactive online question pools, as stated by Wei et al. [41]. In the present study, two types of mouse attributes were identified: wired and wireless. However, according to Betts [42], various improvements in mice concerning convenience and efficiency have emerged in the last few years, yet they were not subjected to study. These levels include an ergonomic mouse, Bluetooth wireless mouse, USB wireless mouse, and laptop touchpad. Thus, these levels were used for the mouse attributes in the study.

The fourth attribute, earpiece, refers to the aural device usually plugged into a speaker port. The students use this to listen and communicate during online classes. Kozlowski's [43] study mentioned that hearing device affects the user's productivity since they have different acoustic signal values. According to Tangkiengsirisin and Kalra [44], the earpiece used for online learning can affect the learning process and students' social presence. Additionally, the earpiece can be a communication medium, which would affect the quality and effectiveness of the communication during online learning. In the present study, the levels identified for this attribute are the headset, headphones, earbuds, earphones, and speaker.

The desk is the fifth attribute considered in this study. This refers to the particular table the students set up their devices. According to Emerson et al. [34], the desk's height affects the overall performance of the users, especially if it is not fit for them. In a study that was carried out by Shen et al. [45], the authors investigated the effect of ergonomic desk design on the improvement of motor accuracy in the writing performance of students. The design and dimensions of furniture, such as desks, have been shown to affect the students' physical responses and their performance in a study conducted by Castellucci et al. [46]. It was determined that one of the most critical factors in improving the physical responses of some students was ensuring that the dimensions of the students' furniture were compatible with the students' anthropometric characteristics. The use of high furniture, sit–stand furniture, and tilt tables and seats are all design dimensions that contribute to positive effects. Therefore, the kind of desk that students use when they are engaged in online learning demonstrates a positive impact on their performance as well as their physical responses. Two levels of desk attributes are considered for this study: height-adjustable table and sitting desk.

"Chair" refers to the furniture on which the students sit during online classes. There have been studies done on student seating from a variety of viewpoints. One of the areas of emphasis is seating preference, which focuses on the reason students choose specific seats and how this choice affects their performance [47]. Similarly, the study conducted by Gumasing et al. [26] discovered that workstation design featuring comfortable chairs, tiered seating, and appropriate lighting resulted in higher levels of overall student satisfaction. Emerson et al. [34] also mentioned that chair height and material are fundamental factors to consider when purchasing the equipment since they can increase pressure on the user's popliteal space is too high or the lower back too low. Thus, the identified levels of computer chairs used in the present study included high-back office chairs with neck rest, mid-back office chairs, and task chairs, following the study of Gumasing et al. [26].

Lastly, the type of light fixture is the seventh attribute considered in the present study since lightning affects the contentment and productivity of students while executing their school tasks [13]. The lighting system has been shown in previous research to benefit the students' ability to concentrate. Consequently, emphasis on how essential proper lighting is to the educational process has been studied [48]. A study conducted by Singh et al. [49] also found that lighting had a significant impact on the students' ability to concentrate and their overall performance. Illumination levels ranging from 250 to 500 lux were associated with higher levels of student concentration, resulting in higher test scores and improved overall performance. In most cases, research on illumination has focused on how different lighting conditions affect the performance of various tasks. However, in the present study, students' preferences regarding lighting levels were investigated, taking into account a wide range of human responses to lighting, such as comfort, aesthetics, and performance. Thus, the levels considered for lighting attributes in the present study are desk lamps, floor lamps, and room light.

#### *2.5. Statistical Analysis*

Through the orthogonal design of the SPSS Statistics software, the study generated stimuli or a set of combinations of the considered levels per attribute in the study. The orthogonal design was used to ensure a manageable quantity of stimuli. Initially, the software generated an optimal amount of 49 stimuli, to which two holdout cases were added to verify the accuracy of the results. By comparing how effectively conjoint utilities estimate outcomes from the holdout cases, the internal validity of the conjoint task was assessed [50]. The participants were presented with 51 stimuli (Appendix A) that were evaluated by a 7-point Likert scale, with 1 representing "strongly disagree" and 7 representing "strongly agree".

#### **3. Results**

Table 3 presents the utilities and the average importance score among the attributes of workstation design for online classes. According to Hair et al. [51], the utility estimates signify how much a user values a particular level of attributes, while the importance score indicates how important an attribute of a workstation design is to a user. The greater the influence on user preference, the higher the utility estimate and importance score.


**Table 3.** Utilities and Average Importance Scores.

Based on the importance scores, results showed that the most significant attributes for users are the type of gadget (37.925%), followed by the type of mouse (28.345%), kind of chair (14.840%), type of keyboard (7.548%), earpiece (7.177%), light device (2.109%), and desk (2.056%). Specifically, in the first attribute, type of gadget, the users preferred personal computer (0.401), followed by laptop (0.273) and tablet (−0.673). Second, mechanical (0.099) was most favored by the users within the type of keyboard attribute, succeeded by multidevice Bluetooth (0.009). Third, for mouse types, Bluetooth wireless obtained the highest utility score of 0.204, followed by USB wireless (0.170). Fourth, for the earpiece attribute, users desired a headset (0.134), succeeded by headphones (0.014). Fifth, within the desk attribute, users preferred a height-adjustable table (0.029) to a sitting desk (−0.029). Sixth, a high-back office chair with a neck rest (0.189) was most favored in terms of type of chair. Lastly, for light device, the floor lamp obtained the highest utility score of 0.034, followed by the desk lamp (−0.009).

The ranking of 51 stimuli evaluated by the respondents is presented in Appendix B. With a total utility score of 1.017, it is evident that combination 4 was the most preferred by the users. The attributes under this combination were a personal computer, mechanical keyboard, wired mouse, headset, height-adjustable table, high-back office chair with neck support, and floor lamp. On the other hand, it is observable that the users' least-favored combination was the workstation design in combination 2, with a total utility score of −1.356. This combination consisted of a tablet, ergonomic keyboard, touchpad, headset, sitting desk, task chair, and floor lamp.

Table 4 represents the validation of the results. Akoglu [52] used Kendall's tau and Pearson's R values to determine the reliability of the results and the relationship between actual and predicted preference. Values greater than or equal to 0.8 indicate strong and consistent results for each tool. Results showed that the Pearson's R-value obtained in this study is 0.978 (greater than 0.80) [32], which indicates a strong relationship between the actual and estimated preferences of the workstation users. Additionally, Kendall's tau value of 0.978 and Kendall's tau coefficient for holdouts of 1.000 denote internal consistency among the responses [20,21].

**Table 4.** Correlation.


#### **4. Discussion**

The conjoint analysis of the different stimuli showed that the most preferred stimulus of a workstation design by undergraduate students was a personal computer, mechanical keyboard, Bluetooth wireless mouse, headset, height-adjustable table, high-back office chair with neck support, and floor lamp with a total utility score of 1.090. On the other hand, with a total utility score of −1.741, the least favored was a tablet, membrane keyboard, touchpad as the mouse, speaker, sitting desk, task chair, and floor lamp.

The type of gadget was the most significant attribute considered by the users, with an importance score of 37.925%. The most favored was a personal computer, whereas the least favored was a tablet. A workstation's primary feature includes the gadget, since it is the student's medium to attend classes and perform school tasks. Prasetyo et al. [15] discussed that using gadgets is one of the advances the education industry has adopted due to technological advancements for teachers and students. Supporting this, Marpuah et al. [53] discovered that gadgets play a vital function as a source of learning, enabling the learning process to be comfortable for students during online classes. In this case, it is better for the students to use a personal computer, which may be due to the processing power, physical size, and memory. This creates an effective and efficient learning process that increases student-learning outcomes.

Lade et al. [54] found that students have different opinions on utilizing other gadgets for an online class because these devices have additional technical capabilities and usability, especially when students need to download various school-related applications. Although Ozok et al. [55] found that tablets are more versatile and portable for students to use, they also concluded that tablets have technical limitations and can compromise output quality. Hence, a personal computer or laptop is better used for heavy tasks, especially since most respondents were engineering students, who usually need applications for plate-making, coding, data analysis, etc. [56].

Next, the second-highest attribute influencing the users' decision was the type of mouse (28.345%). Users preferred Bluetooth wireless (0.204), followed by USB wireless (0.170), wired (0.131), ergonomic (0.095), and touchpad (−0.599). A mouse is another vital component in the workstation since it is utilized for data input [35]. Betts [42] stated that gadget developers took advantage of technological advancement to improve devices' convenience and efficiency, especially in an online setup. Consequently, many people now have access to seamless data transmission to several devices because of the Bluetooth feature [21]. Furthermore, it has become one of the favored approaches for wireless communications due to its versatility and effectiveness as a connectivity mechanism. Joshi et al. [57] stated that a mouse is more challenging when it contains additional electronics. These additional electronics may be extra buttons, additional weight, and wires that may tangle upon movement. Supporting this, Conroy et al. [58] found that users prefer a lighter mouse because their computer activities become faster and more accurate. An external mouse improves productivity levels and guarantees all-day comfort for users [59]. This could be supported by the result that users would not prefer the touchpad, which limits movement and navigation effectively.

The kind of chair was the third-highest attribute considered by the users (14.840%). The most preferred was the high-back office chair with neck rest (0.189), succeeded by the mid-back office chair (0.042), and task chair (−0.231). The chair is essential in a workstation design because this improves the students' concentration during online classes and task making [60]. A high-back office chair with a neck rest is considered ergonomic because it provides additional support to the upper-back area [61]. Al-Hinai et al. [60] found out that an ergonomic chair design is significant in ensuring students' comfort and proven usability. Students in an online class setup prefer ergonomic furniture because it lessens their stress and improves motivation [62]. Suzuki et al. [63] stated that poor ergonomics and posture due to inefficient chair designs could cause musculoskeletal discomfort to students, especially when seated for long hours. If not given intervention, this can further develop into other medical problems, particularly in the upper extremities. This further validates that students would not prefer a regular task chair.

Fourth, keyboard type was also an attribute considered based on the preferences of the consumer (7.548%), and the most valued type was mechanical (0.099), then multidevice Bluetooth (0.009), ergonomic (0.008), and membrane (−0.115). With a more significantly lower score compared to the top three attributes, keyboards are more of an extra device, but can still affect a student's productivity and efficiency. Like a mouse, a keyboard is an essential element in a workstation because this is utilized to enter data into the students' devices [35]. Mechanical keyboards are increasing in popularity among computer users because they enable faster typing and require less physical effort to activate the key switch due to the additional tactile and auditory feedback from the key-switch design [64,65]. The findings in the study of Pham and Kelling [65] demonstrated a statistical difference between flexor-muscle effort on a mechanical keyboard and a standard membrane keyboard. Considering the online class setup, a mechanical keyboard is preferred by students because it is easy to use, efficient, and aesthetically pleasing [66]. Usually, sounds upon typing and placement of a keyboard affect student's preferences in choosing this device.

With an importance score of 7.177%, the earpiece was considered the fifth-most important attribute evaluated based on user preferences. Valued the most under this category was the headset (0.134). On the other hand, the least favored were the speakers (−0.069). According to Kozlowski [43], since each acoustic device has varying acoustic signal values, it influences the user's productivity. Since their invention, headphones/headsets have

become a significant user audio device [67]. They outperform loudspeakers in terms of portability, privacy, and cost. Users prefer headphones, since they are designed to cover the earlobe or block the ear canal, effectively muffling outside noise [67], consequently, improving the students' focus and creating an immersive listening experience [68]. On the other hand, Zelechowska et al. [67] found that using speakers during online classes can result in increased self-consciousness and distraction. Hence, to improve the clarity of communication between the teacher and students during online courses, using a headset instead of the device's built-in microphone and speakers is a better option. This presents better input and output quality, especially when conveying communication between student and teacher.

The findings show that light devices and desks were the users' most minor value considerations, with importance scores of 2.109 and 2.056, respectively. Among the light devices, the most favored was the floor lamp (0.034), followed by desk lamp (−0.009) and room light (−0.025). To provide users with more suitable illumination, lights specifically designed for a working environment have increased in value in the market [69]. Jajoo et al. [13] and Singh et al. [49] showed that supplementary lights are substantial because they provide more illumination to the users, making them more focused. Hence, a floor lamp and desk lamp can affect the students' visual comfort and aids in increasing the students' productivity and satisfaction during online classes. At the same time, poor lighting conditions can have adverse effects on students [70].

Lastly, the least-considered attribute by the users was the type of desk, wherein a height-adjustable desk was favored over a sitting desk. Nevala and Choi [71] suggested that working in a sitting posture was less efficient and put more significant pressure on the workers' upper extremities than working in a standing posture. A height-adjustable desk allows the users to sit or stand depending on the situation that best satisfies their comfort. The height-adjustable workstation can lessen sedentary habits and inactivity in an online class [71,72]. Although Aschenberger et al. [62] stated that students prefer ergonomic furniture in an online setup, the findings of this study showed that the undergraduate participants valued the kind of chair more than the desk. This is because their upper extremities are more prone to musculoskeletal discomfort if there is no proper back support during the hours of sitting [27].

#### *4.1. Practical Implications*

The findings showed that the type of gadget and mouse significantly influenced the users' preference for a workstation design. Since the respondents are primarily engineering students, who usually need to download and utilize different software in classes, they considered the type of gadget, particularly personal computer, to be the most significant attribute. Due to its capacities, the gadget alone plays a crucial role in the students' learning process. Additionally, students' intention to purchase different gadgets can still be affected if gadget developers initiate a transformative technology that would benefit the gadget and the users. The second-most favored attribute was the mouse. This attribute influenced users' preferences because intensive mouse use has a proven relationship with increased risk of upper-extremity musculoskeletal disorders. Hence, a user-fit mouse must be given importance.

The results of this study can serve as a baseline in determining students' preference in workstation design, since this is the first study to determine workstation-design attributes during online classes. When preference is considered, this study may serve as a foundation for improving students' engagement. Since the findings present a business opportunity, gadget developers must take these into account. These attributes significantly influence the users' purchase decisions, so we recommend considering highly the different levels found from the results. Gadget and furniture sellers could grow their business if they evaluate the quality of the equipment they offer in relation to the students' preferences.

It is suggested that combinations of the different devices may be placed as displays among workstation industries to promote and highlight the resulting preferences among students. This will help buyers imagine their setup at home upon choosing several devices. Placing moderately significant devices between highly significant devices may encourage buying among consumers. We suggest that the type of device should be at the forefront, followed by desks, keyboards, mouse, earpiece, lamps, and kind of chair, so that consumers can see all different devices before completion. This kind of setup would promote buyers to consider all necessities with the top three significant attributes between less significant attributes to heighten their intention to purchase the items.

#### *4.2. Limitations*

Despite strong findings and results of this study, several limitations are still present. First, this study considered only the preference of different devices utilized for online setups among undergraduate students in general. Clustering different courses and programs of students is suggested. This may result in differences in output of preference depending on the course or programs. Through k-means clustering, marketing segmentation may help deduce the findings that may also be utilized by industries. The survey was distributed online due to the COVID-19 pandemic. Since no control could be made for the location or type of university (public or private), it is suggested that comparative preference analysis may be conducted to uncover distinctly differences among private- and public-school students, their location, and even type of residence. This may provide other findings with enough data collected, since universities (public and private) in the Philippines provide different services. Similarly, as discussed by Bast [16], students living in urban and rural areas also provide different perspectives to online learning alongside their preferences. In addition, anthropometric measurement may also be considered to compare and contrast based on levels presenting significant preferences. Lastly, since workstation-design attributes and levels considered in the study were based on the common gadgets and furniture used during online classes, future research including other attributes and considering different levels that may influence users' preferences is recommended. Different attributes must be taken into account because they could influence importance rankings. Future research may therefore elaborate on the conclusions of preferences for workstations.

#### **5. Conclusions**

The sudden shift to e-learning due to COVID-19 compelled parents and students to convert a space in their houses and make a workstation for online classes. A workstation is an area in the students' houses where they complete schoolwork and, most importantly, attend classes. This study utilized the orthogonal design of the conjoint analysis approach to determine users' most preferred combination of workstation attributes: the type of gadget, keyboard, mouse, earpiece, desk, chair, and light device. Using purposive sampling, 315 undergraduate students participated in the online questionnaire comprising 51 stimuli.

The findings showed that the type of gadget was the most considered attribute affecting user preference, followed by the type of mouse, kind of chair, type of keyboard, earpiece, light device, and desk. Under the type of gadget attribute, the most preferred was the personal computer, while Bluetooth wireless was the most favored type of mouse. The kind of chair users favored most was the high-back office chair with a neck rest. Mechanical obtained the highest utility score for type of keyboard and headset for the earpiece. For the light, the most desired was the floor lamp. Lastly, a height-adjustable table was the most favored type of desk.

This study is the first to examine user preferences for workstation design since the pandemic. The results will benefit gadget developers, business owners, and retailers regarding user preferences for different attributes of workstation design. In particular, the results of this study will help industries understand the importance of each feature to users' decision-making and which workstation-design attributes users value the most. The findings and suggestions of this study may be considered by different sectors of industries to create marketing strategies for consumers. Lastly, the study's findings may be utilized by students, teachers, and employees in the same online setup globally, even in a postpandemic setup.

**Author Contributions:** Conceptualization, M.J.J.G., M.A.D.B. and A.K.S.O.; methodology, M.J.J.G., M.A.D.B. and A.K.S.O.; software, M.J.J.G., M.A.D.B. and A.K.S.O.; validation, M.J.J.G., M.A.D.B. and A.K.S.O.; formal analysis, M.J.J.G., M.A.D.B. and A.K.S.O.; investigation, M.J.J.G., M.A.D.B. and A.K.S.O.; resources, M.J.J.G. and M.A.D.B.; data curation, M.J.J.G. and M.A.D.B.; writing original draft preparation, M.J.J.G. and M.A.D.B.; writing—review and editing, M.J.J.G., M.A.D.B. and A.K.S.O.; visualization, M.J.J.G., M.A.D.B. and A.K.S.O.; supervision, A.K.S.O. and M.J.J.G.; project administration, M.J.J.G. and A.K.S.O.; funding acquisition, M.J.J.G. and A.K.S.O. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Mapua University Directed Research for Innovation and Value Enhancement (DRIVE).

**Institutional Review Board Statement:** This study was approved by Mapua University Research Ethics Committees (FM-RC-22-17).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in this study (FM-RC-21-11).

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author.

**Acknowledgments:** The authors would like to thank all the respondents who answered our online questionnaire. We would also like to thank our friends for their contributions in the distribution of the questionnaire.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**


**Table A1.** Stimulus.


**Table A1.** *Cont.*


**Table A1.** *Cont.*


**Table A1.** *Cont.*

#### **Appendix B**

**Table A2.** Stimulus Rank.



**Table A2.** *Cont.*


**Table A2.** *Cont.*


**Table A2.** *Cont.*

#### **References**


## *Article* **One Step Back or One Step Forward? Effects of Grade Retention and School Retention Composition on Portuguese Students' Psychosocial Outcomes Using PISA 2018 Data**

**Joana Pipa \* and Francisco Peixoto**

CIE-ISPA (Centre for Educational Research-ISPA), ISPA-Instituto Universtiário, Rua Jardim do Tabaco 34, 1149-041 Lisbon, Portugal

**\*** Correspondence: jpipa@ispa.pt

**Abstract:** Grade retention is a common practice applied to academically struggling students within the Portuguese context. Studies investigating the psychological experiences of grade-retained students are still scarce. In addition, most studies tend to neglect the multilevel nature of the school context. This study examines the effects of grade retention in grades 1–9 on Portuguese students' psychosocial outcomes by the age of 15, using PISA 2018 data. Using a quasi-experimental design through full matching, we reduced the bias between 1362 retained and 4189 promoted students in relevant background variables. Results from the multilevel models showed that retained students, by the age of 15, present lower task orientation and school belonging. In addition, we found that the high retention rates negatively relate to students' reading self-concept, task orientation, and school valuing and that school retention rates moderate the relationship between students' retention and the psychosocial variables considered. Overall, these findings suggest detrimental effects of grade retention and that grade retention also affects the promoted peers of retained students.

**Keywords:** grade retention; psychosocial outcomes; school retention composition; quasi-experimental methods; multilevel modelling; PISA

#### **1. Introduction**

When students struggle to meet the academic expectancies or goals established for a specific grade level, one option is to allow them more time by retaining them in the same grade [1,2]. This type of response to deal with students' heterogeneity, in terms of academic competencies, is one of the most discussed educational strategies [3]. Furthermore, each country usually uses grade retention rates as a measure of educational quality and equity [4–6].

Accompanying the long debate around grade retention effectiveness, most countries decreased their retention rates during the last decade. However, in countries such as Belgium, Luxembourg, Germany, Spain, and Portugal [6], grade retention is still a common strategy used to deal with students' low achievement [5,7–9]. This paper focuses on data from Portugal, where grade retention rates are among the highest in the Organization for Economic Co-operation and Development (OECD) countries. In these educational systems, grade retention is believed to bring several benefits to struggling students, such as giving them more time to develop and mature and to catch up on the learning materials [5,10,11]. In addition, retaining students in grades is believed to improve teacher effectiveness by creating more homogenous classrooms, and the threat of being retained might boost students' motivation to work harder [7,10,12].

However, opponents of grade retention argue that grade retention harms students' motivation, confidence, and sense of self-worth because, as they will go over the same curriculum once more, they are deprived of intellectual challenges and meaningful learning [13]. Moreover, retained students are detached from their peer group and will face a

**Citation:** Pipa, J.; Peixoto, F. One Step Back or One Step Forward? Effects of Grade Retention and School Retention Composition on Portuguese Students' Psychosocial Outcomes Using PISA 2018 Data. *Sustainability* **2022**, *14*, 16573. https://doi.org/10.3390/ su142416573

Academic Editor: Antonio P. Gutierrez de Blume

Received: 8 November 2022 Accepted: 6 December 2022 Published: 10 December 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

new classroom of younger students, leading to a decreased perceived competence and a sense of failure [10]. Finally, being retained brings extra economic and opportunity costs for the educational system, students, and families [11,14].

Being retained could constitute a rather negative psychosocial experience for students [15–17] and has been pointed out by children and adolescents as one of the most stressful life events [15], affecting their motivation and self-confidence [10,13,17–19]. Nevertheless, despite its relevance for educational success and adjustment, studies considering the effects of grade retention on students' psychosocial outcomes, such as self-concept, motivation, or engagement, have received less attention from the research community [20]. In addition, as advised by other scholars, school context plays a crucial role in shaping students' self-beliefs and experiences [9]. In particular, school retention composition, i.e., the proportion of repeaters in a given school, was associated with academic and psychosocial outcomes, namely, students' peer relationships, self-concept, school belonging, and enrolment in post-secondary education [21–24].

Therefore, this paper investigated whether Portuguese students with a history of grade retention differ from their promoted same-age peers in psychosocial outcomes. Moreover, we aimed to study whether these differences could also be exacerbated or not in schools with a more significant proportion of retained students. For this study, we relied on the available data from the Programme for International Study Assessment (PISA) 2018 regarding the Portuguese context.

#### *1.1. Previous Research on the Effects of Grade Retention*

Research on grade retention effectiveness has grown tremendously, especially since 2010 [20]. Recent meta-analyses and systematic reviews estimated an overall null effect of grade retention [1,20,25]. However, these studies found that the impact of grade retention is highly dependent on the context where it occurs (e.g., country, state, and educational system), the timing of grade retention, the timing of follow-up (e.g., short-term studies vs. long-term studies), and the variables observed (e.g., academic achievement vs. psychosocial or school career) [20,25].

The existing studies considering psychosocial outcomes are far less conclusive than those investigating academic achievement outcomes, suggesting positive, negative, or nonsignificant effects [3,20,25]. These inconclusive findings could be attributed to the broader nature of the term 'psychosocial outcomes', covering different variables that could be differently affected by grade retention [20,22]. Additionally, short-term studies tend to present more positive results [10,14,18,22,26,27].

#### 1.1.1. Effects on Students' Academic Self-Concept

Students' academic self-concept, defined as students' self-perception of competence in specific academic-related domains (e.g., reading self-concept), plays a significant role in school adjustment, achievement, and educational success [9]. To explain the effects of grade retention on students' academic self-concept, researchers have often referred to the big-fish–little-pond effect [28]. This effect posits that students compare their own school-related accomplishments with those of their classmates, and this frame of reference act as the base for their self-concept development [9,28,29].

Based on this framework, one could expect that retained students would develop more positive academic self-beliefs, at least during the retention year, because their frame of reference would comprise their younger grade mates with less academic experience [9,29]. On the other hand, however, some scholars also claim that being retained jeopardises students' perception of competence because they may perceive that being retained constitutes a personal failure that makes them less competent and capable [3,11].

Thus, the empirical evidence also shows mixed effects of grade retention. Longitudinal studies assessing grade retention in primary [18] and lower secondary education [14,18,29–31] found either positive effects on math and academic self-concept [14,29,30], or adverse effects on language, math, and academic self-concept [18,31], during the retention year. Conversely, more adverse long-run findings emerged from cross-sectional studies investigating effects in lower [26] and upper secondary education [17,22]. In contrast, longitudinal studies generally revealed nonsignificant effects on students' academic, language, and math selfconcept during lower and upper secondary education [26,29–32]. Positive long-run effects were found in only one study using international PISA data [9].

#### 1.1.2. Effects on Students' Goal Orientations

Students' motivation has received much attention from researchers as it is recognised for its critical role in students' academic behaviour and performance [33]. Students' goal orientations have become one of the largest research fields in motivation and are characterised by students' reasons or purposes for engaging in certain achievement behaviours [34]. The PISA in 2018 assessed two kinds of these reasons, or orientations, to engage in academic tasks: task orientations and self-enhancing ego orientations [35]. In task orientation, students engage in a school-related task (e.g., a reading task) to develop and acquire knowledge or master a new skill. On the other hand, when students pursue a self-enhancing ego orientation, the aim is to demonstrate competence and outperform others [35,36].

Considering the effects of grade retention on motivational outcomes in general, one may expect, on the one hand, that grade retention acts as a 'boost' in students' motivation since they will finally experience success and will receive positive feedback from teachers [10]. Additionally, students may view the retention year as a second chance to master the learning content or even an opportunity for a fresh start [31,37].

Despite its relevance within the educational context, to our knowledge, studies investigating the effects of grade retention on students' goal orientations are still scarce. The existing studies mainly suggest that students with a retention history demonstrate less adaptive motivational profiles in lower secondary education, even before being retained [31], and during the retention year and beyond [19].

#### 1.1.3. Effects on Students' Sense of School Belonging and Valuing

Apart from students' individual motivation, their social exchanges within the school context, particularly with their peers, teachers, and the broader school community, are essential agents in shaping their motivation at school [38]. This social aspect of school motivation is commonly known as students' sense of school belonging [38–41]. Students' sense of school belonging can be understood as students' feelings of being accepted, respected, valued, and supported by their peers and the broader school community [38–40]. In addition, these feelings of belonging are also associated with valuing school and school success, the two components of students' participation at school [39].

When students are retained in a grade, they lose their peer group. This experience of a broken relationship can trigger feelings of isolation and alienation from school [10,21] due to failure to satisfy the need to establish and, especially, to maintain stable relationships with others [41,42]. In addition, grade retention is an explicit form of academic failure. This stigma of failing a grade and not being good enough academically can make establishing new and positive relationships in a new and unfamiliar peer group even harder [21]. These feelings and experiences may thus lead to a greater sense of school disaffection and feelings of being an outsider from school and not connected with the school community [10,21].

Empirical studies investigating the effects of grade retention on students' sense of belonging suggest that grade retention does not improve students' sense of school belonging. Longitudinal studies showed predominantly adverse effects in both primary [27] and lower secondary education [43]. Cross-sectional studies revealed mainly adverse effects of retention on school belonging [22], particularly those using international PISA data [23,44]. To our knowledge, the effects of grade retention on students' school valuing as it is operationalised here are inexistent.

#### *1.2. School Retention Composition*

Empirical studies recognising the importance of school context in studying grade retention, although less common, have been growing during the last years, suggesting its crucial role in moderating the relationship between individual grade retention and academic and psychosocial outcomes [12,21–24,32,45–47]. In addition, Van Canegem et al. [22] posited that the context where grade retention occurs might be crucial to clarify divergent findings from previous studies.

The number of retained students attending a particular school can impact students' psychosocial outcomes in two ways. First, a direct effect of school retention composition is expected through the so-called spillover effects of retained students on their non-retained peers [22]. Spillover effects of grade retention have been less considered in grade retention research, despite constituting a big concern for families and educators [20]. Retaining students in a particular classroom may negatively affect the classroom climate and instruction and, therefore, the learning of non-retained classmates [24,48].

Second, school retention composition may moderate the relationship between individual grade retention and students' psychosocial outcomes. Accordingly, retained students might present more negative behaviours and feelings of being stigmatised in schools with low rates of grade retention. These feelings might, thus, exacerbate the impact of grade retention on students' self-concept and sense of belonging, for example [21–23,45].

The empirical studies that have addressed the impact of school retention composition on students' psychosocial factors showed that students from schools with many repeaters tend to be more likely to misbehave at school have a lower number of friends, and lower levels of academic self-concept and school belonging [21–23,45]. In addition, some studies showed more favourable results of grade retention in schools with higher retention rates [21–23].

#### *1.3. The Portuguese Context*

The Portuguese school system, along with other southern European countries (e.g., France, Italy, and Spain), offers a common core curriculum for all students until 9th grade, and grade retention is the primary strategy applied to deal with academically struggling students and to promote homogeneity inside the classroom [7]. In Portugal, 24% of 15-year-old students have reported being retained at least once during their school career, being largely above the OECD average of 11% [6]. These rates may reflect the 'culture of retention' mentioned in several studies, meaning that educators believe that grade retention is beneficial for students over and above the recommendations of the international educational community and even national legislation [5,8]. Currently, the national legislation states that grade retention in Portugal should only be an 'exceptional measure' when promoting the student to the next grade compromises the acquisition of new learnings [49]. Thus, grade retention decision falls on schools and teachers who, except for transition years, have the autonomy to define in which specific circumstances grade retention will be applied.

Although it is a widespread practice in Portugal, studies using Portuguese data are still scarce, especially in using adequate methodologies and considering variables beyond student achievement [50,51]. To our knowledge, only a limited number of studies have focused on the effects of grade retention on students' psychosocial outcomes, such as students' self-concept and motivation. These studies showed predominantly adverse effects [18,19,47].

#### *1.4. The Present Study*

In summary, the following considerations guided this study: (a) the findings that the characteristics of each school system moderate the effects of grade retention; (b) the limited number of empirical studies evaluating grade retention effects within the Portuguese context (and to some extent considering other countries with a similar educational system); (c) the mixed empirical evidence of the effects of grade retention on students' psychosocial

variables driven by the differential nature of each construct; and (d) the importance of school retention composition in clarifying the relationship between grade retention and academic outcomes.

Hence, we aimed to investigate the effects of grade retention on a group of psychosocial variables. Specifically, using a large-scale assessment and applying a same-age comparison approach, we examined (1) whether students who had experienced grade retention at least once between grade 1 and grade 9 differed from their same-age promoted peers in reading self-concept, goal orientations, and students' sense of school belonging and valuing; and (2) whether the nature of these effects can differ according to the proportion of retained students attending a school, i.e., school retention composition.

#### **2. Materials and Methods**

#### *2.1. Data and Participants*

Starting in 2000, PISA became one of the largest and most prominent large-scale assessment studies in education. Every three years, PISA assesses reading, math, and science competencies acquired by 15-year-old students. In addition, PISA gathers information regarding student and family background information and various psychosocial variables, including academic self-concept, motivation, and school engagement.

This study used data from 5932 Portuguese students who participated in PISA 2018, attending 276 schools. Since this study focused on grade retention effects, we excluded students who had missing information (5%, *n* = 308) in the grade retention variable from the data. Thus, further analyses were based on 1362 retained (24%) and 4262 promoted students (*M*age = 15.73, *SD*age = 0.29, 50% boys), attending between 7th and 11th grade. PISA uses a two-stage process regarding the sampling procedure to obtain a representative sample of students from each country and economy. First, schools are randomly selected from a complete list of schools containing the student population of interest. Second, 35 15-year-old students from each school are randomly selected to fulfil the questionnaires [52].

#### *2.2. Measures*

In this study, we focused on PISA data considering students' reading self-concept, goal orientations, and school belonging and valuing. In addition, we used information regarding students' social background and school characteristics retrieved from students' and school principals' questionnaires. All continuous measures were standardised to have a mean of 0 and a standard deviation of 1, and categorial measures were coded as dummy variables to facilitate interpretation. Concerning the validity and reliability of the measures used, internal consistency ranged from α = 0.74 to α = 0.88 (see Table 1), revealing acceptable levels of reliability, as was also referred to in the PISA report [44].


**Table 1.** Means and standard deviations for retained and promoted students, internal consistency, and correlations between the outcome variables.

\*\*\* *p* < 0.001.

#### 2.2.1. Grade Retention

In the PISA questionnaire, students were asked whether they had ever repeated a grade in ISCED I, II, or III. This study considered grade retention responses regarding ISCED I or II. In these variables, 0 means that a student never repeated a grade during ISCED I or II, whereas 1 means that a student has repeated it at least once. In addition, school retention composition was derived from responses to grade retention in ISCED II and III variables since many schools offer both these two levels of education and were operationalised as the percentage of retained students in each school.

#### 2.2.2. Reading Self-Concept

To assess academic self-concept in reading, students were asked on a 4-point scale (strongly disagree to strongly agree) whether they perceive themselves as good readers, whether they are able to understand complex texts, and whether they read fluently [53].

#### 2.2.3. Goal Orientations

Students' task orientations, or learning goals as mentioned by PISA [53] (p. 215), were measured on a 5-point scale (not at all true of me to extremely true of me), asking to what extent they have the goal of learning and master class-related materials. Likewise, to assess self-enhancing orientations or attitudes toward competition, as referred to by PISA [53] (p. 215), students answered three items on a 4-point scale (strongly disagree to strongly agree), asking whether they enjoy working in competitive environments and whether they have the goal of outperforming others.

#### 2.2.4. School Belonging and Valuing

To assess their sense of school belonging, students answered six statements, such as 'I feel like an outsider (or left out of things) at school' and 'I feel like I belong at school' [44] (p. 130), on a 4-point scale (strongly disagree to strongly agree). Students also completed three items to assess how much they value school. In these statements, students were asked whether they agree (strongly disagree to strongly agree) that trying hard at school would help them obtain a good job or help them be accepted into a good college [44].

#### 2.2.5. Students' Social Background and Competencies

We considered several individual background variables retrieved from the students' questionnaire and the measurement of competencies that were related to grade retention [9,14,50,54–57]. Specifically, we retrieved information considering students' age, gender, immigrant background, language spoken at home, index of economic, social, and cultural status (derived from parents' highest level of education, parents' highest occupational status, and home possessions), home educational resources (i.e., household possessions and the number of books at home), and parents' emotional support (self-report measure where students where asked whether they feel supported by their parents) [53]. In addition, we considered students' PISA scores in reading, math, and science.

#### 2.2.6. School Context

We also integrated some school-related components as covariates considered in the literature related to grade retention [24,54–57]. Specifically, we retrieved information regarding school type (public vs. private) [53]. The proportion of participating students attending public schools does not differ from the true proportion of Portuguese students attending such schools (88% in both cases) [4]. In addition, school composition in terms of the index of economic, social, and cultural status and immigrant background, obtained by aggregating students' responses, was also considered.

#### *2.3. Data Analysis*

#### 2.3.1. Handling Missing Values

In most observational studies, participants often leave one or more questions unanswered. Researchers have been encouraged to deal with incomplete datasets in recent years by applying imputation methods. In this study, we applied multiple imputations by chained equations using the MICE package in R [58], generating five completed datasets and allowing ten iterations. The proportion of missing values in the variables used in the

analyses ranged from 0.1% to 10%. Subsequent analyses were conducted in each imputed dataset and then aggregated [59].

#### 2.3.2. Group Comparison Strategy

Applying an experimental methodology to estimate the causal effects of grade retention is not theoretically or ethically attainable, as students cannot be randomly retained or promoted. Nevertheless, the literature on grade retention effects draws attention to the importance of establishing comparability between the intervention (i.e., retained students) and comparison (i.e., promoted students) groups [1,20]. Thus, propensity score matching methods are often applied to reduce selection bias by achieving a balance between treatment and comparison groups regarding background characteristics related to both the treatment and the outcome [60,61]. These methods have also been widely used in international studies, such as those using the PISA data [61]. We opted to apply the full matching technique among the different matching methods, using the MatchIt package in R [62]. The full matching technique is considered a more sophisticated and flexible matching method and has the advantage of not discarding any observation (as does, for example, the one-to-one nearest neighbour technique). Full matching forms a series of matched sets (subclasses) containing at least one treated and one comparison subject [63]. After creating these matched sets, each comparison individual receives a weight proportional to the number of treatment individuals present in each set [62]. In subsequent outcome analyses, these weights are introduced in weighted regression models [62]. Full matching techniques are recognised to be efficient in maximising the similarities between treatment and comparison individuals in each matched set [63] and have been successfully used in grade retention effects research [31]. The final step of preparing comparison groups for the outcome analysis was assessing the balance between treated and comparison subjects across the covariates used [63]. This assessment was performed considering [62,63] (a) standardised mean differences below 0.25; (b) variance ratios between 0 and 2; and (c) graphical inspection. In this study, we include in the propensity score matching a series of background characteristics and interaction terms, described in Section 2.2.5. The selection of these variables was theoretically based, as mentioned previously.

#### 2.3.3. Outcome Analysis

The effects of grade retention on psychosocial components were estimated using multilevel models since students are nested within schools [9]. Thus, a series of hierarchical linear models were computed for each outcome, considering four stages (Model 0 to Model 3). Model 0 or the 'null model' was estimated with no predictors to examine the amount of variance in the outcomes that is explained by either student or school levels. Model 1 was estimated by entering students' background variables, reading, math, and science scores, school context variables, and grade retention variables. In Model 2, school retention composition was added to the previous variables. Model 3 assessed the interaction between grade retention and school retention composition. In estimating these models, we followed Stuart's [63] recommendations of combining matching and regression methods by including in the regression models the predictors and controlling variables previously considered in the matching procedures. We used weighted regression models, as already mentioned, considering the weights obtained after matching. Finally, in every model, we additionally checked for multicollinearity using the variance inflation factor (VIF). Using a threshold of 10 indicating a strong correlation between the independent variables, our results supported its independence, with VIF values ranging between 1.02 and 4.84.

#### **3. Results**

#### *3.1. Covariate Balance across Retained and Promoted Students*

Before conducting the outcome analysis, we assessed whether the selection bias on background variables was reduced through full matching. It is first worth mentioning that we observed extreme weights in some comparison individuals, suggesting difficulties in finding suitable matching individuals for these observations [63]. Hence, we discarded the observations with extreme weights above the 99th percentile (*n* = 73). Further analyses were based on 1362 retained students and 4189 promoted students.

Results from the full matching revealed adequate balance. As shown in Figure 1, the absolute standardised mean differences between retained and promoted students in the background variables decreased considerably after matching, with all standardised mean differences below 0.25. In critical, highly related to retention, background variables, such as students' ESCS, the standardised mean differences were reduced to 0.01 standard deviations.

**Figure 1.** Absolute standardised mean differences between retained (*n* = 1362) and promoted students (*n* = 4189) before and after full matching.

#### *3.2. Effects of Grade Retention*

This section presents the effects of grade retention on the outcome variables assessed. Table 1 shows descriptive statistics in the outcome variables for the retained and promoted groups and correlations between outcome variables.

To improve readability, in Table 2, we present only the results from Model 3, i.e., the model testing both single effects of grade retention and school retention composition and interaction effects between these variables. In the following sections, we briefly describe the findings from the remaining models. Detailed information regarding coefficients from Model 0 to Model 2 for each outcome variable can be found in Appendix A.

#### 3.2.1. Effects of Grade Retention on Reading Self-Concept

The intraclass correlation coefficient (ICC) obtained from Model 0 (these coefficients are only described in the text; calculations are possible from the tables in Appendix A) suggests substantial variance between schools in reading self-concept (ICC = 0.20). Thus, these results support the multilevel analysis. Considering the main effect of grade retention, the results were consistent across the three models; the relationship between grade retention and reading self-concept was nonsignificant. Conversely, Model 3 revealed a negative relationship between school retention composition and reading self-concept (b = −0.07, *p* < 0.05), suggesting that students in schools with a larger proportion of repeaters tend to feel less competent in reading.


**Table 2.** Multilevel unstandardised parameter estimates from Model 3 for the outcomes assessed.

Note: Standard errors are in parenthesis. <sup>a</sup> Index of economic, social, and cultural background; <sup>b</sup> Home educational resources; <sup>c</sup> Parental emotional support. \* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001.

In addition, the results showed a combined effect of grade retention and school retention composition (b = 0.08, *p* < 0.001) on this outcome, revealing that the higher proportion of repeaters in school attenuates the effects of grade retention on students' reading self-concept. Figure 2a illustrates the nature of this interaction effect, showing that in schools with a high number of repeaters, repeaters tend to present a higher perception of reading competence than promoted students.

#### 3.2.2. Effects of Grade Retention on Goal Orientations

Considering students' task orientation and self-enhancing orientation, the ICCs from Model 0 revealed that 19% and 16% of the variance in students' goal orientations is attributable to schools. The results from all models showed that retention is related to lower levels of task orientation (b = −0.12, *p* < 0.001; b = −0.11, *p* < 0.001; b = −0.16, *p* < 0.001; Models 1 to 3, respectively). Considering the effects of school retention composition, we found an effect of this variable only in Model 3 (b = −0.13, *p* < 0.001) when the interaction term was entered. The moderating effect of school retention composition in the relationship between individual retention and task orientation (b = 0.13, *p* < 0.001) showed that, on

the one hand, grade retention affects students' task orientation less when they attend a school with a higher number of retained students. On the other hand, an inspection of this interaction considering retained and non-retained students showed that for retained students, being in a school with high retention rates does not affect or even slightly improve their task orientation and, for promoted students, being part of such a school negatively affects their task orientation (see Figure 2b).

**Figure 2.** Graphical representation of the interaction effect between grade retention and school retention composition that resulted from multilevel hierarchical models (Model 3). Note. (**a**) Reading self-concept; (**b**) task orientation; (**c**) school belonging; (**d**) school utility value.

3.2.3. Effects of Grade Retention on School Belonging and Valuing

For school belonging and utility value, the ICCs again showed substantial variability between schools in these components (ICC = 0.23 and ICC = 0.18, respectively). The results for school belonging showed in both models that grade retention is related to lower feelings of school belonging (b = −0.09, *p* = 0.004, for Models 1 and 2; b = −0.12, *p* < 0.001, Model 3). In addition, consistent with the previous results, individual retention was found to be less detrimental to students' sense of belonging when schools present many repeaters (b = 0.09, *p* = 0.001). Moreover, promoted students attending a school with a high rate of retained students present lower sense of school belonging (see Figure 2c). For students' sense of school utility value, this component was only affected by grade retention and school retention composition when the interaction term was considered (individual retention: b = −0.03, *p* = 0.324, Model 1; −0.03, *p* = 0.415, Model 2; b = −0.08, *p* = 0.011, Model 3; retention composition: b = −0.12, *p* < 0.001). Again, school retention composition was found to moderate the effects of individual retention (b = 0.14, *p* < 0.001), showing that grade retention rates at the school level tend to attenuate the negative relationship between grade retention and school utility value. Moreover, in line with previous findings, promoted students attending a school with a high rate of retained students present lower sense of school utility value (see Figure 2d).

#### **4. Discussion**

In Portugal, grade retention is still a prevalent practice applied to academically struggling students, irrespective of being considered an 'exceptional measure'. Therefore, based on data from PISA 2018, we aimed to explore the effects of grade retention on students' reading self-concept, goal orientations, and school belonging and valuing by employing methods that consider several background variables related to grade retention. The use of full matching [63] and the available data on several individual and contextual variables allowed us to reduce the differences between promoted and retained students and to estimate more rigorous effects of grade retention by disentangling the impact of potential confounders, as previously recommended [1,10,25].

#### *4.1. Effects of Grade Retention and School Retention Composition on Psychosocial Outcomes*

In this study, we found that retained students, by the age of 15, are less oriented to master academic-related tasks and have a lower sense of school belonging and valuing. These results are in line with previous studies, suggesting a detrimental effect of grade retention on motivational variables and school belonging [17–19,22,23,26,44]. Although the cross-sectional nature of this study limits our potential to make causal statements, our findings, together with the previous longitudinal and retrospective studies, suggest that retained students could engage in a negative cascade and that grade retention could leave an irreversible mark on students' motivation and engagement [15,16]. We additionally have explored the effects of repeating more than once by conducting separate analyses for students retained only once (n = 778) and students retained at least twice (n= 584). The analyses considering students retained only once revealed similar findings for all the outcomes. On the other hand, repeating more than once had positive effects on academic self-concept, negligible effects on task orientation and negative effects on students' selfenhancing orientation. The effects of school retention composition remained unaltered. Detailed results of these analyses are available on request from the first author.

These results are supported by the social goals framework, e.g., [42], stressing that students are most likely to engage in a context that provides opportunities to meet the social goal of establishing personal relationships with teachers and peers. Thus, in the case of retained students, by losing their reference peer group and friends, they could experience feelings of not being accepted, respected, or valued within their school community. In addition, students' motivation to learn is also affected by the fulfilment of social goals [42]. In this study, this is visible in the low levels of task orientation and school utility value presented by retained students, suggesting that these students are less oriented to develop learning task-related skills and gradually devaluate school learning.

Regarding students' reading self-concept, our results seem to unravel a more complex picture than individual grade retention effects alone. For this variable, we have found a more substantial effect of school retention composition over and above the impact of individual retention on students' perception of competence. These findings could also be interpreted in light of the big-fish–little-pound effect [28], which posits the school context's prevalence in shaping students' self-concept.

Indeed, one of the most notable findings of this study is the impact that school retention composition exerts on students' self-concept, motivation, and engagement. Overall, the larger the share of retained students in a school, the lower the self-perception of competence in reading, task orientation, and sense of school valuing. In addition, the interaction effect between individual retention and school retention composition suggests that the retention rates do not affect students equally. In the case of our study, non-retained students were those that presented more negative outcomes by attending a school with a large proportion of repeaters. Thus, the adverse effects of grade retention also have significant implications for classmates of retained students and the broader school community, supporting previous findings [21,22,45,48,64].

These findings seem to reflect one of the major concerns of educators and parents—that sharing a classroom with repeaters by disrupting classroom instruction negatively affects the academic outcomes of their non-retained peers [65]. Thus, interventions, such as ability grouping, may sound attractive to ensure non-retained students' academic success. However, as previous studies showed, educational systems that preconise tracking and ability grouping in classes present the most detrimental effects of grade retention on students' development [6,20]. Moreover, retention rates and individual retention are related to students' background and achievement, meaning that schools with a large proportion of repeaters are most often attended by socially disadvantaged and low-achieving students [5,24,65], and these are simultaneously the characteristics that put students in a more vulnerable position to being retained [50,54,56]. Therefore, the most vulnerable students are, simultaneously, more likely to be retained and to share a class/school with many repeaters [48], resulting in greater inequalities among these students [24]. Here, tracking and ability grouping will only exacerbate this effect and, consequently, student disparities. Finally, at a macro level, the PISA 2018 data revealed that countries and economies presenting higher grade retention rates generally showed lower levels of reading performance and lower levels of equity in education [6].

#### *4.2. Limitations and Future Directions*

The contributions of the findings from this study are not without their limitations. First, we must mention the cross-sectional and retrospective nature of the presented data that prevents us from investigating developmental trajectories regarding students' academic self-concept, goal orientations, sense of belonging and valuing, and establishing causal relationships. In the case of grade retention research considering Portuguese data, longitudinal studies regarding grade retention effectiveness would be very important given the limited number of studies within this context and the high retention rates. Further investigations assessing Portuguese retained students' developmental trajectories of academic and psychosocial aspects are needed to clarify the effectiveness of this practice.

Second, and linked with the previous limitation, it was impossible to disentangle which grade students were retained considering the data used in this study. Thus, we could not estimate the potential long-term effects of grade retention more precisely. Considering the specific grade when students were retained is important not only from a developmental perspective but also due to findings from previous studies, where grade retention effects differed according to the grade where they were retained, e.g., [66].

Although some studies do not support the claim that repeating early or later grades produce differential effects on outcomes, from a developmental perspective, it would be important to consider and provide such information when estimating grade retention effects.

Third, concerning the moderating effect of school retention composition, our operationalisation of this variable is limited because only a small number of students from each school participated in the PISA assessment, and non-identification of the participating schools prevents us from obtaining school retention rates from official records. Based on this, one should interpret our findings regarding school retention composition with caution. Given its notable contribution, we encourage researchers to consider this vital variable in future studies assessing grade retention effects.

Fourth, and finally, we assumed the broad definition of grade retention, considering it as a single and universal treatment. Additional interventions coupled with grade retention during the retention year and beyond, such as additional support and educational services provided to retained students, were not considered. Past research showed that positive effects could emerge when retention is coupled with other treatment sources [20]. Moreover,

we did not investigate the potential moderating effects of students' background characteristics. Some researchers suggest that students from certain subgroups or presenting specific features might benefit from grade retention [20,54]. Although these aspects were beyond the scope of this study, we encourage researchers to collect information about 'what happens' during the retention year and test the effects of grade retention according to some students' characteristics in future studies. These two considerations will be very informative either to research or to practice.

#### **5. Conclusions**

The findings of this study suggest that grade retention is not an effective practice. Specifically, our results indicate that grade retention is related to lower levels of motivation and engagement, two valuable conditions for school success. Moreover, it was reported that grade retention affects retained students and could also be detrimental to their peers attending the same school.

Based on our findings, we cannot support the use of grade retention as an effective intervention for struggling students. Instead, we first recommend early identification of at-risk students, monitoring their academic and psychosocial development, and providing additional support to avoid grade retention. Valbuena et al. [25] listed numerous evidence-based and cost-effective policies, practices, and interventions that are alternatives to retention, such as tutoring, summer schools, and multi-age grouping. Likewise, students' academic competencies and psychosocial development should also be considered when deciding to retain a student, not only school marks. Furthermore, in the case of retaining a student, both these competencies should be monitored and supported equally, ensuring that the 'second chance' given to the students will not be harmful to their academic and psychosocial growth. To the broader school community, we recommend monitoring and reducing school retention rates since they affect the whole student community. At last, we advise educators and policymakers to continuously consider the psychosocial components of learning when debating the effectiveness of grade retention and its related policies and norms.

**Author Contributions:** J.P.: Conceptualisation, methodology, formal analysis, investigation, data curation, writing—original draft preparation, visualisation, project administration, funding acquisition. F.P.: Conceptualisation, writing—review and editing, supervision, funding acquisition. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Portuguese national funds via Fundação para a Ciência e Tecnologia (FCT) under grant [SFRH/BD/132195/2017] awarded to the first author, and grants [UIDP/04853/2020] and [UIDB/04853/2020] awarded to CIE-ISPA.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Publicly available datasets were analysed in this study. These data can be found here: https://www.oecd.org/pisa/data/2018database/ (accessed on 18 March 2021). Our computations of the data are available upon request from the first author.

**Acknowledgments:** The authors wish to thank Timo Van Canegem for the valuable comments and suggestions provided for the manuscript.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**

**Table A1.** Multilevel unstandardised parameter estimates for reading self-concept.


Note: Standard errors are in parenthesis. <sup>a</sup> Index of economic, social, and cultural background; <sup>b</sup> Home educational resources; <sup>c</sup> Parental emotional support. Standard errors are in parenthesis. \* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001.




**Table A2.** *Cont.*

Note: Standard errors are in parenthesis. <sup>a</sup> Index of economic, social, and cultural background; <sup>b</sup> Home educational resources; <sup>c</sup> Parental emotional support. Standard errors are in parenthesis. \*\* *p* < 0.01, \*\*\* *p* < 0.001.


**Table A3.** Multilevel unstandardised parameter estimates for self-enhancing orientation.

Note: Standard errors are in parenthesis. <sup>a</sup> Index of economic, social, and cultural background; <sup>b</sup> Home educational resources; <sup>c</sup> Parental emotional support. Standard errors are in parenthesis. \* *p* < 0.05, \*\*\* *p* < 0.001.


**Table A4.** Multilevel unstandardised parameter estimates for school belonging.

Note: Standard errors are in parenthesis. <sup>a</sup> Index of economic, social, and cultural background; <sup>b</sup> Home educational resources; <sup>c</sup> Parental emotional support. Standard errors are in parenthesis. \* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001.

**Table A5.** Multilevel unstandardised parameter estimates for school utility value.



**Table A5.** *Cont.*

Note: Standard errors are in parenthesis. <sup>a</sup> Index of economic, social, and cultural background; <sup>b</sup> Home educational resources; <sup>c</sup> Parental emotional support. Standard errors are in parenthesis. \* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001.

#### **References**


## *Article* **Teacher Self-Regulation and Its Relationship with Student Self-Regulation in Secondary Education**

**Fabiola Sáez-Delgado 1,\*, Yaranay López-Angulo 2, Javier Mella-Norambuena 3, Catherine Baeza-Sepúlveda 4, Carolina Contreras-Saavedra <sup>5</sup> and Gisela Lozano-Peña <sup>5</sup>**


**Abstract:** Self-regulation is relevant to understanding the teaching–learning process; however, few studies have focused on teachers' self-regulatory processes. The aim of this study was to characterize and analyze the relationship between teachers' and students' self-regulation. The design was crosssectional and correlational. The sample consisted of 1481 participants (students *n* = 1123 and teachers *n* = 358) from 25 secondary schools in 17 cities of the Biobío region of Chile. In students, self-regulatory strategies were found to be deployed only half of the time; women were more self-regulated, and there was no difference in the levels of self-regulation according to grade. Teachers, it was found, almost always self-regulate their teaching, and the variables that influence their self-regulation are motivation, gender, and age, explaining between 25% and 28% of the variance. Positive and small correlations were evidenced between teacher role disposition with learning performance (*rho* = 0.10, *p* < 0.05) and teacher role self-evaluation with both learning performance and self-evaluation of learning in their students (*rho* = 0.12, *p* < 0.05). This study provides relevant evidence and proposes changes that could have a positive impact on teacher training and improve current teaching–learning practices in Chile, which would contribute to the quality of education.

**Keywords:** teacher self-regulation; student self-regulation; secondary education; quantitative study

#### **1. Introduction**

Available evidence related to factors that impact academic success highlights the theory of self-regulated learning as a central element to understanding the different performance trajectories during the school stage [1,2]. It has even been shown that self-regulatory skills predict academic success when controlling for intelligence [3].

Particularly, the level of self-regulation of students in secondary education has become a central focus of attention since its relevance for successful academic performance has been confirmed [1,4]. It empowers students for the demands of higher education [5,6], it provides students with the fundamental competencies for lifelong learning [7], and it develops the ability to overcome learning difficulties arising from an unfavorable environment [8].

#### *1.1. Background on Self-Regulation of Learning in Schools*

SRL is defined as self-generated and student-directed thoughts, emotions, and behaviors that are cyclically adapted to achieve academic and personal goals [9]. SRL is an active, recurrent, critical, and reflective process, where students approach tasks in a

**Citation:** Sáez-Delgado, F.; López-Angulo, Y.; Mella-Norambuena, J.; Baeza-Sepúlveda, C.; Contreras-Saavedra, C.; Lozano-Peña, G. Teacher Self-Regulation and Its Relationship with Student Self-Regulation in Secondary Education. *Sustainability* **2022**, *14*, 16863. https://doi.org/10.3390/ su142416863

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Leandro Silva Almeida and Silvia Fernández Rivas

Received: 21 November 2022 Accepted: 13 December 2022 Published: 15 December 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

purposeful, strategic, and self-directed style, which contributes to the individual's sense of self-determined learning [10].

Although it is possible to identify different models of self-regulation, they share certain components and coincide in understanding SRL as a cyclical process which involves macroand micro-phases in its development, highlighting three main phases: the first phase of disposition where the task is analyzed, the study is strategically planned, and motivational beliefs are deployed; a second phase of performance where the student performs the task and monitors and adjusts his strategies if necessary; and, finally, the evaluation phase where the student performs a self-evaluation of his results regulating and adapting strategies for next performances [11].

It has been widely demonstrated that self-regulated students achieve better academic performance [12–16], have better self-efficacy beliefs for the task [17] and intrinsic value motivational strategies [18], and demonstrate a goal-oriented approach to learning [19]. Self-regulated students are able to set goals proactively, monitor their learning intentionally, use study strategies effectively, respond to personal feedback adaptively, and reach goals more quickly, and they are also more motivated to maintain their efforts to learn [20]. In addition, research has shown that self-regulated learning also has positive effects on other variables, decreases academic stress, increases subjective well-being [21], and decreases behavioral problems [3].

Conversely, low self-regulation leads to unsatisfactory outcomes such as academic failure, e.g., repeating a course or achieving low final grades [22], failure to achieve the established learning demands of school [23], and making inexact self-assessments of performance that are consequently unlikely to be modified [24]. In this sense, it is of concern that secondary school students show inadequate levels of self-regulation [23]. Therefore, this metacognitive competence is seen as a key and determining factor for improving performance at the secondary-school academic level [2].

#### *1.2. From Self-Regulation of Learning to Teacher Self-Regulation*

Although there is empirical evidence focused on students' self-regulated learning, few studies have focused on teachers' self-regulatory processes considering that it is a central actor of the context and learning dynamics that happen in classrooms [25,26]. In this sense, specifically, the available research has focused on the epistemological beliefs that teachers have about SRL [27,28], knowledge about this competency [29], self-efficacy to instruct SRL in the classroom [30], teachers' beliefs about feedback practices in relation to self-regulation [31], assessment practices that help improve self-regulation [32], teachers' practices in specific disciplines or subjects [33], the potentiality of teachers' instruction in self-regulation [34], and direct or indirect instructional practices of teachers to drive the improvement of their students' self-regulation [35]. The emphasis of existing studies involving teachers has been on the different variables associated with promoting or fostering self-regulation in students and not on the level of self-regulation of the teachers themselves [35,36], and to a lesser extent on whether their level of self-regulation relates to the self-regulation of their students.

#### *1.3. Self-Regulated Teachers*

Just as self-regulation helps students take responsibility for their own learning, it can also be expected to help teachers in their own professional development. Teachers have to constantly deploy many strategies to achieve teaching goals [37].

Teacher self-regulation (TSR) refers to the teacher's own self-regulated strategies executed in the teaching environment to achieve the set pedagogical objectives [38]. Consequently, teacher self-regulation can be seen as an active process where teachers direct and maintain their metacognition, motivation, and strategies for effective instruction. This definition is consistent with Zimmerman's theoretical model of self-regulation of three cyclical moments corresponding to disposition, performance, and self-reflection [9]. That

is, for each teaching session, the teacher plans his class, executes the teaching–learning process, and concludes with an evaluation of his teaching [37].

In the disposition phase, which includes processes necessary for the preparation of the action, teachers establish specific objectives to be achieved by the end of a class. Based on the objectives and considering the nature of the content, the characteristics of students, and the available resources, teachers select the appropriate instructional methods. Teachers need to apply many strategies to achieve the pedagogical objectives and, therefore, they need to self-regulate their teaching and maintain high levels of professional motivation [39].

Self-regulated teachers evaluate the effectiveness of their instruction based on comparison to their previous performances, student feedback, and whether or not the goals set at the beginning of the class were achieved. As a result of these evaluation processes, they develop behavioral, cognitive, and affective responses that will influence future action, planning, and execution processes in a cyclical manner. The literature describes that effective teachers are those who achieve self-regulation processes of their teaching, which can stimulate their beliefs to select appropriate actions that lead to the successful execution of their tasks and professional performance [37]. Thus, a self-regulated teacher is aware of why, when, where, and how pedagogical knowledge and skills can be learned and applied in the classroom. Teachers also have a strong sense of personal responsibility for their teaching; they exercise metacognitive control of continuous reflection on their pedagogical practices and the effort deployed to meet challenges [40]. Self-regulated teachers exhibit adequate functioning in the teaching exercise, and they build knowledge before and after the teaching session through cyclical self-regulatory processes that allow them to improve with each class [37].

#### *1.4. The Present Study*

Undoubtedly, promoting an autonomous learning model requires a corresponding teaching model, which consequently necessitates appropriate teachers of cognitive and meta-cognitive strategies in the service of teaching. Undoubtedly, promotion an autonomous learning model requires a corresponding teaching model, therefore, teachers are required to apply cognitive and metacognitive strategies during teaching [41]. First, teachers must be self-regulated, and then they need to learn how to effectively teach their students to self-regulate their study and learning processes [42].

Although self-regulation has been highlighted as an important part of a teacher's role in the school experience [43], the self-regulatory processes of teachers, and whether these are related to the regulatory processes of their students, are still unknown. In general, evidence on secondary-school teachers' self-regulation has been investigated from a qualitative approach [44], and in the case of quantitative studies, these have focused on adding elements for understanding self-regulatory factors that help explain variability in students' performance [45]. For example: the relationship between teachers' reflective practices, their self-regulation, and their teaching experience has been examined [46]; teachers' emotional regulation while teaching has been studied from their students' perceptions [47]; and the interrelationship between teacher self-regulation and teachers' knowledge for teaching has been explored [40]; however, a study examining teachers' self-regulation and their students' self-regulation has not been found. Although it has been argued that the construction of self-regulation includes steps, processes, and components that integrate and interact with each other, providing positive learning outcomes [48], there is little evidence of the interaction specifying relationships in teacher–student self-regulatory processes [49]. It is noted that this approach to the phenomenon is limited; therefore, studying how the learning process is related to the teaching process from the perspective of self-regulation has become a central concern in formal instructional situations [50]. Additionally, it is important to specify sociodemographic characteristics for an adequate inference of results.

Due to the background presented, this study aimed to respond to the theoretical and empirical gaps regarding teacher and student self-regulation, considering the importance of these variables on the success of students' academic performance, in particular given that few investigations have included both participants. The specific research questions were:


#### **2. Materials and Methods**

The present study is based on the positivist paradigm, implementing a study with a quantitative approach and a cross-sectional, correlational design.

#### *2.1. Participants*

The sample consisted of students and teachers from public secondary schools in the Biobío region of Chile. The sampling was non-probabilistic, specifically "convenience sampling".

Specifically, the sample for objectives 1 and 2 was composed of 1123 students from 25 schools in 17 different cities, with a mean age of 15.42 years (SD = 1.64); 578 (51%) were women, 527 (47%) were men, and 18 (2%) preferred not to declare their gender. Regarding educational level, secondary education in Chile has a first, second, third, and fourth grade. Of the total, 319 students (28%) were in first grade, 419 (37%) were in second, 295 (26%) were in third, and 90 (8%) were in fourth. The sample of teachers was composed of 358 participants from 20 schools in 14 different cities, with a mean age of 39.08 years (SD = 10.99); 218 teachers (61%) were women, 139 (39%) were men, and 1 (0%) preferred not to declare their gender. In relation to work experience, the mean was 12.09 years (SD = 10.14) of experience, with respect to the type of employment contract: 241 teachers held permanent contracts, 112 teachers held fixed-term contracts, and 5 teachers held fee contracts. Regarding the workload, respondents had a mean of 39.84 hours (SD = 6.68) of contract per week.

Finally, to achieve the third objective of this study, the data of students and teachers who interacted with each other were cross-checked, and the teachers and their respective students were verified. For this objective, the sample of students was 721 participants with a mean age of 15.42 (SD = 1.77); 355 students (49%) were women, 353 (49%) were men, and 13 (2%) preferred not to declare their gender. The sample of teachers was 133 with a mean age of 38.43 years (SD = 9.85); 73 teachers (55%) were women, 59 (44%) were men, and 1 (1%) preferred not to declare their gender. Both teachers and students were from 12 schools in 10 different cities.

#### *2.2. Instruments*

#### 2.2.1. Teacher Self-Regulation

The instruments to measure the teachers' self-regulation phases scale were designed and validated in the context of the FONDECYT Project 11201054-ANID-Chile. Based on an analysis of the literature and theory of teacher self-regulation [38,39,51,52], a first proposal of items was obtained. It then went through the process of validation by expert judges, followed by cognitive interviews and a pilot application for exploratory analysis. In this design and validation process, the international guidelines for test development in research were considered [53]. The result of this process concluded with an instrument of 3 scales: disposition, performance, and self-evaluation of teacher roles. Instrument dimensions were confirmed in this study. The teacher role disposition scale was composed of 5 items and measured the frequency with which teachers used self-regulatory strategies to prepare their teaching. An example item is, "I plan learning activities according to the needs of my students", and the internal consistency was adequate (α > 0.80; Ω > 0.88). The teacher role performance scale was composed of 5 items and measured the frequency with

which teachers used strategies to monitor their teaching. An example item is, "I monitor my teaching strategies used and adjust them if necessary", and internal consistency was adequate (α > 0.84; Ω > 0.86). Finally, a teacher role self-evaluation scale was composed of 5 items and measured the frequency with which teachers reflect on the results obtained at the end of their academic processes. An example of this item is, "I self-evaluate whether my teaching practices were effective", and internal consistency was adequate (α > 0.84; Ω > 0.87). For these three dimensions that make up the instrument, the response format was a 7-point Likert-type scale, where 1 is always, 2 is almost always, 3 is frequently, 4 is half the time, 5 is seldom, 6 is almost never, and 7 is never (See Appendix A). The instrument went through the process of evaluating its factorial structure by means of confirmatory factor analysis (CFA) where three models were tested: (1) a model of 3 independent factors; (2) a multilevel model of 3 factors and 1 of higher order; and (3) a multilevel model of 3 related factors and 1 of higher order. Of these models, the second one showed fit indices according to the fit indices proposed in the literature, i.e., an RMSEA ≤ 0.07, the CFI and TLI > 0.92, the SRMR < 0.08 [54], and the chi-squared [55] (see Table 1).

**Table 1.** Confirmatory factor analysis fit index of the teacher self-regulation instrument.


Note. TSR: teacher self-regulation. X2: chi-square goodness-of-fit statistic. CFI: comparative fit index. TLI: Tucker-Lewis index. RMSEA: root mean square error of approximation. SRMR: standardized root mean square. DWLS: weighted least squares.

#### 2.2.2. Teacher Motivation

A brief scale of teacher motivation was designed and validated in the context of the FONDECYT Project 11201054-ANID-Chile. It followed the same construction and validation process as the teacher self-regulation instrument. The unidimensional scale consisted of 5 items, and measured the intrinsic causes of teaching practices, an example of which is, "I am motivated to improve my teaching practices", and the internal consistency was adequate (α > 0.87; Ω > 0.90). The response format is a 7-point Likert-type scale, where 1 is always, 2 is almost always, 3 is frequently, 4 is half the time, 5 is seldom, 6 is almost never, and 7 is never (See Appendix B). The instrument went through the process of evaluating its factorial structure by means of confirmatory factor analysis (CFA) where three models were tested: (1) a model of 3 independent factors; (2) a multilevel model of 3 factors and 1 of higher order; and (3) a multilevel model of 3 related factors and 1 of higher order. Of these models, the second one showed adequate fit indices according to the fit indices proposed in the literature, i.e., an RMSEA ≤ 0.07, the CFI and TLI > 0.92, the SRMR < 0.08 [54], and the chi-square [55] (see Table 2).

**Table 2.** Fit index of the confirmatory factor analysis of the teacher motivation instrument.


Note: X2: chi-square goodness-of-fit statistic. CFI: comparative fit index. TLI: Tucker-Lewis index. RMSEA: root mean square error of approximation. SRMR: standardized root mean square. DWLS: weighted least squares.

#### 2.2.3. Self-Regulation of Learning in Students

To assess self-regulation of learning, scales were used to measure the phases of the cyclical process of student self-regulation. The original version was created for secondaryschool students in Ecuador [56] and then adapted and validated for Chilean students of the same academic level [57]. Both studies showed adequate psychometric properties. This instrument, in its original version validated in Chile, has 34 items in total which

are distributed in three scales consistent with the self-regulation process proposed by Zimmerman (2000) (disposition: 7 items; performance: 16 items; and self-evaluation: 11 items). However, in the present study, the decision was made to apply a new abbreviated version of the instrument. When changes were made to the scales, their psychometric properties were again studied in the context of the first stage of implementation of the FONDECYT 11,201,054 research project.

The adjusted instrument consisted of the same three subscales, but the number of items was reduced. The disposition learning scale was composed of 5 items and measured the frequency with which students use self-regulatory strategies to prepare their study, an example of which is, "Before starting to study, I make a schedule to organize my study time", and internal consistency was adequate (α > 0.79; Ω > 0.82). The learning performance scale was composed of 6 items and measured the frequency with which students use strategies to monitor their study based on established pre-planning that includes the goals to be achieved, an example of which is, "While studying, I check if my strategies are effective and adjust them if necessary", and internal consistency was adequate (α > 0.87; Ω > 0.91). Finally, the self-evaluation learning scale was composed of 5 items and measured the frequency with which students reflect on the results obtained in any school task or test, an example of which is, "When I finish my study, I self-evaluate whether I met my proposed goals", and internal consistency was adequate (α > 0.85; Ω > 0.87). For these three dimensions that make up the instrument, the response format is a 7-point Likert-type scale, where 1 is always, 2 is almost always, 3 is frequently, 4 is half the time, 5 is seldom, 6 is almost never, and 7 is never (See Appendix C). In this study, the new version of the instrument went through the process of evaluating its factorial structure by means of confirmatory factor analysis (CFA) where a unifactorial model was tested and showed adequate fit indices (see Table 3).

**Table 3.** Fit index of the confirmatory factor analysis of self-regulation of the learning instrument.


Note: SRL: self-regulation of learning. X2: chi-square goodness-of-fit statistic. CFI: comparative fit index. TLI: Tucker-Lewis index. RMSEA: root mean square error of approximation. SRMR: standardized root mean square. DWLS: weighted least squares.

#### *2.3. Data Collection Procedure*

All procedures followed the ethical guidelines for research with human subjects. The research was approved by the Ethics and Bioethics Committee of the Universidad Católica de la Santísima Concepción. Specifically, the procedures were as follows: (a) meetings were coordinated with the directors of the Municipal Education Administration Departments (DAEMs) responsible for public schools in cities belonging to the Biobío region in Chile, and directors were asked for authorization to conduct the study; (b) then a meeting was coordinated with the directors of each secondary school, and an invitation was extended for their participation; (c) after that, a meeting was coordinated with the management team of the participating schools to invite teachers, students, and their parents to participate, as well as to send links to the surveys for teachers and students, and in the case of parents, to send a link to read and review the informed consent form. In the case of teachers and students, the link was sent in its first section, and before answering the scales, they had to answer an informed consent and assent, respectively; (d) the instruments were applied in the first academic semester of 2022 (between March and June).

The links were generated in the SurveyMonkey technological tool. Once the data collection process was completed, the responses were downloaded into an Excel spreadsheet for data analysis preparation. A data collection protocol was standardized for all schools, following similar procedures for both teachers and students.. Finally, activities were carried

out to benefit the participating schools. These consisted of the presentation of a global report of the results and a workshop on topics agreed upon with the school's management team; in all the schools The workshops dealt with the socioemotional competencies of the teachers.

#### *2.4. Data Analysis Procedure*

In order to respond to the first objective of this study, which referred to the evaluation of differences in the phases of self-regulation of learning according to gender and level (grade) in students, a two-way ANOVA analysis was proposed [58]. Prior to its performance, it was verified that there were no extreme outliers, and then the assumption of normality was evaluated for the distribution of each variable according to comparable groups. For this, the Kolmogorov–Smirnov test was used with Lilliefors coding [59]. Then the assumption of homogeneity of variance was evaluated; for this purpose, the Levene test was used [60].

To respond to the second objective of this study, referred to as the association that sociodemographic variables (age, gender, years of teaching experience, type of employment contract, workload, and teaching discipline) and teacher motivation have on the phases of self-regulation of teaching work, multiple linear regressions were proposed for the selection of variables. The mixed stepwise method was used to test the fit of the models; normality of the residuals was evaluated with the Kolmogorov–Smirnov test with the Lilliefors modification; then homoscedasticity of the residuals was evaluated using the Breusch–Pagan test [61]; multicollinearity was evaluated using the variance inflation factor (VIF) [62]; and finally, autocorrelation was evaluated using the Durbin–Watson test [63].

To respond to the third objective of this study, referred to as the relationship between phases of self-regulation of student learning with phases of self-regulation of teaching, first the normality in the distribution of data in each variable was evaluated. For this, the Kolmogorov–Smirnov test with the Lilliefors modification was used; because in all cases it was not possible to assume normality in distribution of data, Spearman's correlation test was performed.

Non-representative groups were excluded from the analysis. Specifically, in the case of RQ1 that incorporated the gender variable, people who preferred not to declare gender were excluded from the analysis because of their low representation (n = 18). All analyses were performed in RStudio version 4.2.2.

#### **3. Results**

*3.1. Results of Objective 1: Evaluation of Differences in Students According to Gender and Grade for Each SRL Phase (RQ1)*

To evaluate differences in phases of SRL in students according to gender and school level, the two-way ANOVA assumptions were reviewed. The review of outliers showed that for the disposition phase there were seven outliers and for the performance and selfevaluation phases there were five outliers each, but in all these cases none was an extreme outlier. Then, normality in the distribution of the data was checked for each group to be compared, and the Kolmogorov–Smirnov test with the Lilliefors modification was applied. In the group of fourth-grade men, it was possible to assume normality in the disposition (W = 0.98, *p* = 0.59) and performance (W = 0.97, *p* = 0.25) phases. In the case of the selfevaluation phase, the groups of fourth-grade men, W = 0.97, *p* = 0.36, and fourth-grade women, W = 0.95, *p* = 0.06, could be assumed normal. Then, the homogeneity of variances was evaluated using Levene's test. The disposition phase showed F(7.1097) = 0.87, *p* = 0.53; the performance phase showed F(7.1097) = 0.96, *p* = 0.46; and the self-evaluation phase showed F(7.1097) = 1.69, *p* = 0.11; therefore, in all three phases homogeneity of variance could be assumed.

In relation to the means of the phases of self-regulation of learning in students, it could be observed that, for each of the phases of disposition, performance, and self-evaluation, women had higher scores than men, the highest mean for women being in the disposition

phase (M = 4.87), while for men the lowest mean was for the self-evaluation phase (M = 4.12) (see Table 4).



Note: SD: standard deviation.

Regarding the descriptive analysis considering gender and grade, the mean for women is higher for all phases of self-regulation, with women in the fourth grade of secondary school having the mean average (M = 4.68 and M = 5.05); see Table 5.


**Table 5.** Descriptive statistics of student self-regulation according to gender and grade.

Note: SD: standard deviation.

Regarding the inferential analysis used to determine differences between groups, the two-factor ANOVA for disposition to learn showed that only the effect of gender was significant F(1.1097) = 28.81, *p* < 0.05. With this result, we proceeded to perform a T-test to analyze disposition to learning according to gender, which was significant T(1080.4) = 4.917, *p* < 0.001, d = 0.30. Therefore, it can be assumed that women have a significantly higher mean in their disposition to learn compared to men.

The two-factor ANOVA for the performance of learning also showed that only the effect of gender was significant F(1.1097) = 14.6, *p* < 0.001. With this result, the T-test was applied to the performance phase according to gender. The test was significant T(1089.8) = 3.87, *p* < 0.001, d = 0.23; therefore, it can be assumed that women have a significantly higher mean in the performance phase compared to men.

The two-factor ANOVA for the self-evaluation phase of learning also showed that only the effect of gender was significant F(1.1097) = 8.39, *p* < 0.01. Then a T-test was applied that was significant T(1094.5) = 2.94, *p* < 0.01, d = 0.18. Therefore, it can be assumed that women have significantly higher means for the self-evaluation phase of learning compared to men.

Finally, it can be stated that none of the phases of self-regulation of learning varied as a function of the participants' school level (see Table 6).

#### *3.2. Results of Objective 2: Evaluation of the Association between Teacher Motivation and Sociodemographic Variables on TSR Phases*

In relation to the means of TSR phases, it was observed that, as in the case of students, women teachers had higher average scores for phases of disposition, performance, and self-evaluation, with the highest mean for women being in the disposition phase (M = 6.15) and, on the contrary, the lowest mean for men being in the self-evaluation phase (M = 5.77) (see Table 7).


**Table 6.** Results of the two-way ANOVA analysis on differences in student self-regulation according to gender and grade level.

Note: \* *p* < 0.05.

**Table 7.** Descriptive statistics of teachers' self-regulation phases according to gender.


Note: SD: standard deviation.

*3.3. Association between Teacher Motivation and Sociodemographic Variables in the Teacher Role Disposition Phase (RQ2)*

To evaluate the association between teacher motivation and sociodemographic variables in the teacher role disposition phase, a multiple linear regression model was utilized. The model generated using the mixed stepwise method was as follows:

#### *formula* = *disposition* ~ *motivation* + *gender* + *age*

The model was significant, F (3.353) = 21.45, *p* < 0.001, R2 = 0.15. Next, the normality of the residuals was evaluated, which was significant, D = 0.08, *p* < 0.001, so normality of the residuals could not be assumed. Next, the homoscedasticity of the residuals was evaluated, which was significant, BP(8) = 41.53, *p* < 0.001; therefore, constant variability of the residuals cannot be assumed. Next, multicollinearity was evaluated, with the variable "gender", vif = 1.01, being the highest value. Finally, autocorrelation was evaluated, which was not significant, DW = 2.11, *p* = 0.31, so we assume that there is no autocorrelation in the model. Considering that the first model generated did not comply with some of the assumptions, we proceeded to incorporate a motivation variable as a polynomial, given that the visual inspection of the relationship between this variable and the teacher role disposition phase did not follow a linear relationship. The formula of the model was as follows:

#### *formula* = *disposition* ~ *poly*(*motivation*, *2*) + *gender* + *age*

The model was significant, F (4.352) = 30.02, *p* < 0.001, R2 = 0.25, being a model that improves the R2. To confirm this improvement, the BIC was used, M1 BIC (5) = 823.05, M2 BIC(6) = 783.92, which confirmed the improvement of the model. Then, the assumptions were evaluated, and the normality of the residuals was significant, D = 0.09, *p* < 0.001. The homoscedasticity test was significant, BP (4) = 25.25, *p* < 0.001, and the VIF values were between 1.01 and 1.04, so they were acceptable. Finally, the autocorrelation, DW = 2.10, *p* = 0.36, was evaluated; it was not significant. With this background and a visual inspection of waste distribution, this is considered a good model for high levels of disposal; this may be because data of this variable present significant skewing to the left. Therefore, the

results show that, for the teacher role disposition phase, the variables that have a significant influence are motivation, gender, and age, explaining 25% of the variance (see Table 8).


**Table 8.** Regression models on variables that influence the phases of teacher self-regulation.

Note: a significant *b*-weight indicates the semi-partial correlation is also significant. *b* represents unstandardized regression weights. *Sr*<sup>2</sup> represents the semi-partial correlation squared. LL and UL indicate the lower and upper limits of a confidence interval (CI), respectively. \* indicates *p* < 0.05. \*\* indicates *p* < 0.01.

#### *3.4. Association between Teacher Motivation and Sociodemographic Variables on the Teacher Role Performance Phase (RQ2)*

To evaluate the association between teacher motivation and sociodemographic variables on the teacher role performance phase, a multiple linear regression model was utilized. The model generated using the mixed stepwise method was as follows:

#### *formula* = *performance* ~ *motivation* + *workload*

The model was significant, F (2.354) = 36.11, *p* < 0.001, R2 = 0.16. The normality of the residuals was evaluated, and the resulting test was significant, D = 0.08, *p* < 0.001, so normality could not be assumed. Next, the homoscedasticity of the residuals was evaluated, which was significant, BP(2) = 29.58, *p* < 0.001, and the variance inflation factor (VIF) for each predictor was vif = 1.01 for the two variables. Finally, the autocorrelation test was not significant, DW = 2.13, *p* = 0.19, so it was assumed that there is no autocorrelation in the model.

Considering that the first model generated did not comply with some of the assumptions, we proceeded to incorporate the polynomial of the motivation variable. With this modification, workload was no longer significant, leaving a simple linear regression model. The resulting model was as follows:

#### *formula* = *performance* ~ *poly(motivation, 2)*

The model was significant, F (3.353) = 44.58, *p* < 0.001, R2 = 0.27, being a model that improves the R2. To confirm this improvement, the BIC was used, M1 BIC (4) = 860.53, M2 BIC(5) = 818.01; therefore, model improvement was confirmed. Then, assumptions were evaluated, and the normality of the residuals was significant, D = 0.09, *p* < 0.001. The homoscedasticity test was not significant, BP (3) = 7.85, *p* = 0.05, and finally the autocorrelation was evaluated with the Durbin–Watson test, DW = 2.12, *p* = 0.22, which was not significant. With this background and a visual inspection of the distribution of the residuals, this is considered a good model for high levels of performance; this may

be because the data of this variable present significant skewing to the left. Therefore, the results show that only the motivation variable was associated with teacher role performance, explaining 28% of the variance (see Table 8).

#### *3.5. Association between Teacher Motivation and Sociodemographic Variables on the Teacher Role Self-Evaluation Phase (RQ2)*

To evaluate the association between teacher role motivation and sociodemographic variables on the teacher role self-evaluation phase, a multiple linear regression model was utilized. The model generated using the mixed stepwise method was as follows:

#### *formula* = *self-evaluation* ~ *motivation* + *gender* + *age* + *workload*

The model was significant, F(4.352) = 17.8, *p* < 0.001, R2 = 0.16, and the test for normality of the residuals was significant, D = 0.07, *p* < 0.001; therefore, normality of the residuals cannot be assumed. The results of the homoscedasticity test of the residuals were significant, BP(4) = 16.30, *p* < 0.01; therefore, constant variability of the residuals cannot be assumed. Regarding variance inflation (VIF) all predictors presented a VIF between 1.01 and 1.03. Finally, autocorrelation was tested, which was not significant, DW = 2.15, *p* = 0.93, so it was assumed that there is no autocorrelation in the model.

Considering that the first model generated did not comply with some of the assumptions, we proceeded to incorporate the polynomial of the motivation variable. With this modification, workload was no longer significant, so it was eliminated from the model. The formula was as follows:

#### *formula* = *self-evaluation* ~ *poly(mot,3)* + *gender* + *age*

The model was significant, F (5.351) = 23.95, *p* < 0.001, R2 = 0.25, being a model that improves the R2. To confirm this improvement, the BIC was used, M1 BIC (6) = 899.37, M2 BIC (7) = 866.21; therefore, the improvement in the model was confirmed. The normality test of the residuals was significant, D = 0.08, *p* < 0.001; the homoscedasticity test was not significant, BP (5) = 7.47, *p* = 0.19; the variance inflation factor (VIF) for all variables took values between 1.01 and 1.04; and finally, the autocorrelation DW = 2.10, *p* = 0.84 was evaluated, which was not significant. With this background and a visual inspection of the distribution of the residuals, this is considered a good model for high levels of execution; this may be because the data of this variable present significant skewing to the left. Therefore, the results show that, for the self-evaluation of the teacher role, the variables that have a significant influence are motivation, gender, and age, explaining 25% of the variance (see Table 8).

#### *3.6. Result of Objective 3: Evaluation of the Relationship between TSR Phases with SRL Phases (RQ3)*

In order to respond to the third objective of this study, a descriptive analysis of the variables of interest was first carried out, where it can be observed that the means of the teachers are higher than those of the students; for teachers, the performance phase had the highest mean (M = 6.13) and the self-evaluation phase the lowest (M = 6.07). On the other hand, for students the disposition phase presented the highest mean (M = 4.68) and the self-evaluation phase (M = 4.25) the lowest. That is, teachers self-report that they almost always implement strategies for self-regulation of their teaching, while students deploy self-regulatory study strategies half of the time (See Table 9).

Finally, to evaluate the correlation between TSR phases and SRL phases, the assumptions of normality were first verified for all variables. In all cases, the Kolmogorov–Smirnov test was significant, so Spearman's correlation test was performed.

Table 10 shows that there are positive and small significant correlations between the teacher role disposition with learning performance (*rho* = 0.010, *p* < 0.05) and the teacher role self-evaluation with learning performance scale, and with self-evaluation of study in their students (*rho* = 0.12, *p* < 0.05).

**Table 9.** Descriptive analysis of the phases of self-regulation in teachers and students.


**Table 10.** Correlation analysis between TSR phases with SRL phases.


Note: M and SD are used to represent mean and standard deviation, respectively. \* indicates *p* < 0.05. \*\*\* indicates *p* < 0.001. TSR: teacher self-regulation. SRL: self-regulation of learning.

#### **4. Discussion**

This section presents a discussion of findings of the study, reflects on the importance of self-regulatory processes in students and teachers to contribute to educational quality, and specifies limitations and future research.

#### *4.1. RQ1 Discussion: Differences in Students According to Gender and Grade for the SRL Phases*

The results for students revealed that only "half of the time" do they deploy selfregulatory strategies to study and learn. It was evident that students do not plan, monitor, and self-evaluate their study as necessary to successfully meet academic demands. Fortunately, research has pointed out that, although we talk about self-regulated subjects as states of the student, they are not genetically based characteristics or personality traits over which the individual has no control, but rather the result of psychosocial processes; therefore, it is multifactorial at the intrapersonal and interpersonal level of interaction of individual variables as well as contextual variables [64]. Among the students' own variables can be considered, for example, age, gender, ethnicity, and prior knowledge, while among the contextual variables can be considered teachers' instructional methods, activities, interactions, and classroom dynamics; therefore, self-regulation is not only allowed or limited by personal cognition and motivation, but is also privileged, encouraged, or discouraged by contextual factors [65]. Thus, it can be understood that self-regulation is a dynamic and interactive process, which could be affected by daily instruction and practice through multiple experiences in different contexts [66].

In students, the results showed that women are more self-regulated than men, given that in all self-regulatory processes including disposition, performance, and self-evaluation they had significantly higher means. An investigation of Mexican students found statistically significant differences in ARA concerning gender in favor of women, specifically in repetition, critical thinking, and metacognitive strategies; however, no differences were found in the dimensions of planning, organization, time management, and environment [67]. On the other hand, the results also showed that none of the phases of self-regulation of learning varied according to the school level (grade/grade) of students. In other words, from the time they enter the first year through the fourth year (last year of secondary education) they have a similar level of self-regulation. This could indicate, on the one hand, the quality of change that self-regulatory processes have when they are used to intervene [10,20], and, on the other hand, the need to implement programs that develop this competence before entering a university [68]. This is also consistent with

previous research that has shown that university students do not have a sufficient level of self-regulation to face challenges adequately at the university level [69].

#### *4.2. RQ2 Discussion: Evaluation of the Association between Teacher Motivation and Socio-Demographic Variables on TSR Phases*

In teachers, the results showed some valuable predictors of self-regulatory processes in the teaching practice. Teacher role disposition is predicted by motivation, gender, and age, explaining 25% of the variance; teacher role performance is predicted only by motivation, explaining 28% of the variance; and teacher role self-evaluation is predicted by motivation, gender, and age, explaining 25% of the variance. This result is interesting because it allows us to identify some variables involved in teacher self-regulation. It is relevant considering that previous studies have revealed that high levels of teacher self-regulation are directly associated with good pedagogical results. For example, a study on a sample of 76 English teachers in Iran revealed that there is a significant correlation between their level of selfregulation and their success (r = 0.52, *p* < 0.05); moreover, the model containing total self-regulation scores can predict 27% of teachers' success [70]. On the other hand, in the different phases of teacher self-regulation, motivation was a predictor. This is consistent with previous studies that have shown that teacher professional motivation has a positive and significant effect on teacher performance [71], which in turn is also recognized as a significant predictor of their students' interest in the subject [72].

#### *4.3. RQ3 Discussion: Evaluation of the Correlation between the SRL Phases with TSR Phases*

Regarding the correlation between the phases of self-regulation of student learning and TSR phases, it was first possible to demonstrate in the descriptive analysis that the means for the different phases of self-regulation are higher for teachers compared to their students. At this point, it is important to highlight that teachers perceive themselves as self-regulated, since they self-report that they almost always implement strategies for selfregulation of their teaching. This is consistent with previous research results; for example, a study involving 106 teachers from rural and urban primary and secondary schools in Switzerland revealed that teachers have a positive self-concept about their own competence to self-regulate their learning [39].

The results also evidenced that there are positive and small significant correlations between the teacher role disposition with learning performance scale (*rho* = 0.10, *p* < 0.05) and teacher role self-evaluation with both the learning performance scale and self-evaluation of study in their students (*rho* = 0.12, *p* < 0.05). This is consistent with a study developed in Spain that confirmed that regulatory teaching (teaching effectiveness involving appropriately structured teaching and assistance to facilitate and induce self-regulated learning) was closely related to self-regulated learning. The correlation was moderate, but statistically significant (r = 0.082; *p* < 0.03) [73]. In fact, researchers claim that teachers' regulatory behaviors influence students to become better self-regulated learners [74]. It has been recognized in research that teachers' professional competencies are related to classroom practices, and that they also affect the academic and learning outcomes of their students [75]. This has implied that general frameworks on teaching professional competencies emphasize the need to expand these frameworks to include cognitive–motivational competencies beyond the disciplinary ones; however, the adaptation of general frameworks of teaching professional competencies specifically in SRL is incipient in the literature [39].

#### *4.4. Reflections on Self-Regulation and Quality of Education*

The promotion of self-regulation in the school environment becomes relevant since it is associated with better academic experiences and results, as well as general wellbeing of both students and teachers. From this perspective, SRL can contribute to the reduction of existing gaps and to the quality of education, as well as to the reduction of existing inequalities in learning, and, at the same time, increase the possibilities of access to opportunities for continuity of study and professional academic training.

It has been demonstrated that beyond the sociodemographic characteristics of vulnerability in schools, self-regulated students achieve control of their study and learning, with a sense of determination towards their personal and academic goals [76]. This shows the value of SRL; therefore, it is necessary to incorporate its promotion in an intentional way into the different public education policies that involve teachers and their students. Undoubtedly, this will allow support from a new area that supports the improvement of educational quality within schools.

The results of this research on self-regulation in teachers and students contribute to the challenge presented by the 2030 Agenda for Sustainable Development. The results are a relevant input to propose changes that will have a positive impact on the training and improvement of current teaching–learning practices in Chile. Undoubtedly, it would be beneficial to develop more specific training aimed at teachers to support them in the instruction of SRL within their classrooms and also to consider the work of the variables that affect their promotion or those self-regulatory variables that are diminished in the teachers' own self-regulation [77].

Evidence-based interventions support, on the one hand, the validity of working with reliable diagnoses, characterizations, and results carried out by competent professionals, and, on the other hand, adaptation to the specific sociocultural context. The development of SRL, in this educational instance, makes it possible to reduce school absenteeism, avoid early school dropout, and develop skills considered in the literature as transversal and enhancing human development [57].

This research is relevant in that it includes two of the most important agents in the teaching–learning process, i.e., students and teachers; therefore, it contributes to a more holistic understanding of the process of self-regulation of learning at an educational level that precedes university education, which prepares students to be more autonomous and to develop skills that allow them to positively face the adaptation to university education in their first year.

#### *4.5. Limitations of the Study*

In order to adequately interpret the results of this research, some limitations must be taken into consideration. First, as suggested in the literature [78], it is important to take into consideration the "constraints on generality" (COG); in other words, it is not possible to generalize the results to groups not represented in the study. This study focused on public secondary schools; therefore, the COGs are students and teachers belonging to private schools, elementary schools, universities, and from other regions of Chile.

A second limitation is that the data were collected during the first academic semester of 2022. This period coincided with the return to face-to-face classes in Chile after two years of online classes as a result of the COVID-19 pandemic. Therefore, the responses could be different in the next semester, since during this first semester both teachers and students had to adjust to this new post-pandemic scenario and, therefore, the data collection moment could have influenced the results of the present research, which could be further studied in future studies with a longitudinal scope. A third limitation refers to the type of instruments used corresponding to self-reporting scales, which, especially in the sample of teachers, could exhibit a social desirability bias in the responses. It could be questionable that they achieve a high self-regulation mean in the different phases of the teaching self-regulation process. There is precedent that the results of teachers selfreporting on components of their SRL promotion do not necessarily correlate with data from classroom observations of the teachers' own SRL instruction [35]. This is not unexpected; it can be explained by considering that self-reported self-regulation processes generally reflect teachers' perceptions of their overall and more habitual behavior, i.e., self-analysis over an extended period. On the other hand, classroom observations are characterized by showing teachers' behaviors in a single visit to the classroom. In summary, it would be desirable for studies to obtain data from both sources (self-reported and observed) to provide a more concrete perspective of the self-regulation of their teaching. A fourth limitation to consider

is the cross-sectional design of the research. This type of design allowed relationships to be established but did not allow causal inferences to be made. Therefore, no conclusions can be drawn about the direction of the effects of the interaction between the study variables, in this case, teacher and student self-regulation.

#### *4.6. Future Research*

Chronologically, it is possible to identify that progress has been made in the proposal of theoretical and empirical models to understand student self-regulation; then, progress was made in proposing different extracurricular interventions to improve student self-regulation; then, proposals for intracurricular interventions to improve student selfregulation were included. At this point, the focus is now on teachers as key actors in the promotion of self-regulation in the development of their classes. With the focus on teachers, variables that influence self-regulation promotion practices have been identified, such as epistemological beliefs, teacher knowledge in self-regulation, and self-efficacy for the promotion of SRL competence. However, although theoretically, it has already been argued in the specialized literature that the teacher's own self-regulation can play a determining role in the promotion of student SRL, given that it is considered a high-value opportunity to model SR strategies; however, there are still few studies that have moved in this direction, i.e., that confirm that teacher self-regulation influences student self-regulation [74]. Studies are needed that propose relationships of self-regulatory processes of teacher–student interaction processes in school [51]. This study moves in this direction and analyzes the teachers' self-regulation and their students' self-regulation.

The present research makes an important contribution to the specialized literature on self-regulation. It examined and situated the role of teachers and its correlation with students' SRL. However, there are unresolved issues that need further research. Some challenges include responding to the same limitations of this study, that is, to consider expanding the sample in terms of characteristics and groups to better understand what happens in different social contexts (public and private schools), educational levels (primary, secondary, and university education), diversity of students (gifted or with special educational needs), and other populations (regions of Chile and Latin America). Another advancement that is required is to consider longitudinal studies and explanatory models that consider the interaction of teacher and student regulation variables in order to explain the causality of the relationships. On this same point referring to study design, future research could implement mixed designs, complementing the results of quantitative data to enrich the understanding of comprehensive models of teacher and student self-regulation. It would also be interesting to incorporate other variables that could be influenced by teacher self-regulation, such as the academic results of their students. Finally, advancement in the design and application of multiple methods of measurement that allow for the triangulation of self-reported data with the real actions of teachers during the development of their classes is needed.

#### **5. Conclusions**

From the results and discussion of this study, it can be concluded that: (a) the study contributes instruments for measuring self-regulation of both students and teachers, and they are made available for other researchers and their respective adaptations to other social and educational contexts (see Annex 1); (b) in students, women are more self-regulated than men, and there are no differences in the level of self-regulation in the different grades of secondary education (first, second, third, and fourth); (c) the variables that significantly influence self-regulation of the teaching role are motivation, gender, and age; (d) teachers perceive themselves as self-regulated while students showed an insufficient level of selfregulation; (e) there are significant but small relationships between teachers' self-regulatory processes and those of their students; (f) further longitudinal and explanatory studies are required. In summary, it is concluded that this study has an impact on research and

education, making visible the importance of SRL in students and the training of teachers in TSR to advance the current challenges of a better education and a better society.

**Author Contributions:** Conceptualization, F.S.-D., Y.L.-A. and J.M.-N.; methodology, F.S.-D. and J.M.-N.; formal analysis, J.M.-N. and F.S.-D.; investigation, F.S.-D., J.M.-N., Y.L.-A., C.C.-S. and C.B.-S.; writing—original draft preparation, F.S.-D. and J.M.-N.; writing—review and editing, C.C.-S., C.B.-S. and G.L.-P.; data curation, C.C.-S., C.B.-S. and J.M.-N.; software, J.M.-N.; visualization, C.C.-S., C.B.-S. and G.L.-P.; supervision, F.S.-D. and Y.L.-A.; project administration and funding acquisition, F.S.-D. All authors have read and agreed to the published version of the manuscript.

**Funding:** To the FONDECYT Initiation Project N◦ 11201054 entitled "The reciprocal relationship between teacher self-regulation and student self-regulation of learning and academic performance. An explanatory model in secondary education".

**Institutional Review Board Statement:** The project has been approved by the ethics committee of the Universidad Católica de la Santísima Concepción.

**Informed Consent Statement:** Ethical instruments were applied to all study participants. Specifically, letters of authorization were requested from school principals, then informed consent was obtained from teachers. In the case of the students, an assent was obtained in addition to the consent of their parents. Finally, retribution mechanisms were considered for the participants.

**Data Availability Statement:** For further information, please contact the corresponding author.

**Acknowledgments:** The authors would like to thank the teachers and students who participated in this study.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A. Instruments to Measure Teachers' Self-Regulation Phases**

Response format (Formato de respuesta)




#### **Appendix B. Instrument to Measure Teacher Motivation**

#### Response format (Formato de respuesta)


#### **Appendix C. Instrument to Measure Self-Regulation of Learning Phases**

Response format (Formato de respuesta)



#### **References**


## *Article* **Perceived Social Support and Engagement in First-Year Students: The Mediating Role of Belonging during COVID-19**

**Jorge Maluenda-Albornoz 1,\*, José Berríos-Riquelme 2, Valeria Infante-Villagrán 3,4 and Karla Lobos-Peña <sup>5</sup>**


**Abstract:** Academic engagement and the conditions that favor it have become relevant in recent decades due to their relationship with academic performance, well-being, and university permanence. Variables such as perceived social support and sense of belonging are relevant aspects of social integration to promote engagement. Evidence shows both variables predicting engagement. In addition, the available evidence suggests this possible mediating role, which requires further analysis in freshmen in the emergency teaching context due to the COVID-19. The present study aims to evaluate the mediating role of sense of belonging in the relationship between perceived social support and engagement in university students. Results showed significant predictive relationships between social integration variables and engagement and showed the mediating role of sense of belonging in the relation between perceived social support and the three types of engagement. These results suggest the relevance for degree programs to consider these social integration variables as a key element for university freshmen.

**Keywords:** engagement; sense of belonging; social support; freshmen; university

#### **1. Introduction**

Starting a university degree is a difficult experience for students. They face finding themselves, constructing their identity, and trying to overcome the transition from secondary education to university education with the complications that this entails [1]. The student must learn the formal and informal codes and regulations of the institution in which they are located; they have to establish new interpersonal relationships—with peers, professors, and other officials—that allow them to have a social network within the university.

The construction of support networks was abruptly affected during the first year of the pandemic, with severe changes in different areas. This phenomenon significant impacted students, especially freshmen entering higher education [2]. Students were working with peers and teachers through electronic devices, trying to complete academic tasks and still enjoy a social university experience. These changes generated differences in the habitual norms of the initial university experience, since it was necessary to rethink teaching innovation strategies, change the programmed methodologies, and allocate more institutional resources; above all, there was a radical change in the methods of motivating and supporting students.

**Citation:** Maluenda-Albornoz, J.; Berríos-Riquelme, J.; Infante-Villagrán, V.; Lobos-Peña, K. Perceived Social Support and Engagement in First-Year Students: The Mediating Role of Belonging during COVID-19. *Sustainability* **2023**, *15*, 597. https:// doi.org/10.3390/su15010597

Academic Editor: Rosabel Roig-Vila

Received: 17 November 2022 Revised: 21 December 2022 Accepted: 27 December 2022 Published: 29 December 2022

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

The basic psychological need for relationships was affected by the confinement and isolation in this new academic space of interaction. Students were trying to resolve problems, communicate, and negotiate while working on teams with communication delays and lack of real physical contact. Recent research made during the pandemic analyzed social integration variables, finding that social interaction, participation and feedback, and close relationships with peers and teachers, were the most valued teaching–learning practices and had the greatest impact in this emergency virtual context [3]. At the same time, several studies have shown that confinement, isolation, and lack of social interaction decreased the motivation and participation of students [4,5].

In this scenario, where social integration variables have been compromised, student's engagement could be affected, given that engagement is built on the conception of the interactions that the student has with the educational context [6]. In this regard, it is important to consider engagement in first-year university students, since it has been shown to be important in academic variables such as retention and performance [7].

Few studies have analyzed the variables of social integration and their relationship with engagement in first-year students during confinement. This issue deserves attention, given that the confinement would have affected the variables of social integration and, therefore, would have impacted the development of engagement in this population.

#### *1.1. Engagement in University Context*

In the university context, engagement is a high motivational state that manifests itself in student behavior [3]. When students are in this motivational state, they show interest in educational activities, and they make an effort and dedicate time to learning [8–11]. By understanding engagement as a highly motivational state, it is possible to operationalize students' attitudes in three dimensions: behavioral, affective, and cognitive [10].

The behavioral dimension includes all the behaviors that are observed in students that are interested in learning [11], such as participation, collaboration, performance, and interest [12]. The cognitive dimension refers to the purpose of learning and is related to the thoughts, beliefs, and perceptions that students have regarding the effort that academic work requires. Therefore, it includes characteristics such as motivation, critical thinking, and understanding of complex ideas, among others [12]. Finally, the emotional dimension is related to the attitude and feeling of students towards the institution they belong to. Some criteria that are evaluated in this dimension are the perception of relations with peers, teachers, and the institution itself [12].

In the context of the pandemic and according to Junt and Lee (2018), in online education, behavioral engagement is related to asking questions and communicating through electronic devices; cognitive engagement is understood as the effort that learners make to develop specific skills or understand online subjects; and emotional engagement is related to the positive feeling towards teachers, students, and the subjects they take.

When students are "engaged", they participate and collaborate in academic, social, and extracurricular tasks linked to their learning. Therefore, they achieve positive academic results. Furthermore, when students engaged, they are perceived to have more energy and willingness to make the effort that is required to perform complex tasks and to develop complex skills [13,14]. Additionally, they tend to manifest effort, persistence, and prosocial behavior in class (behavioral engagement). Consequently, when students engage, they show positive characteristics such as high interest in the topics they are learning, enthusiasm, concentration, strategic learning, and self-regulation (cognitive engagement). On the contrary, when students are engaged, they show a decrease in the levels of anxiety and boredom (emotional engagement) [13,15,16].

Research on Study Engagement has shown relevance through its relationships with a several variables linked to the teaching–learning process in university students [17]. Study engagement has been related with burnout, present and future academic performance [18], wellbeing [19,20], academic satisfaction [20,21], dropout intention [22,23], and

dropout [24,25]. For these reasons, it is imperative to analyze the social integration variables that are related to engagement in first-year college students.

#### *1.2. Social Integration Factors Related to Engagement*

In the university context, social integration is related to the adaptation that students achieve in the institution through daily experiences [26]. Social integration variables such as perceived social support and sense of belonging have been deeply studied in the past few years [27,28].

On the one hand, perceived social support is defined as the evaluation of students regarding the resources the institution makes available for social support. This considers both the quantity and quality of the social support [29]. Therefore, perceived social support is a subjective and personal evaluation [30]. In this sense, the subjective element of the evaluation is key, since it will depend directly on the personal needs that students have concerning social support or social contact [31].

It has been demonstrated that in an educational environment, perceived social support is related to the level of engagement that students have [23]. It has been observed that perceived social support and the interpersonal relationships that students establish with their peers, teachers, and members of the university campus are fundamental to develop a sense of belonging [32]. Thus, the sense of belonging is related to the process of integration of a person with the organization they belong to. Consequently, a student that has a feeling of belonging towards the institution will also have a high perception of social support.

Sense of belonging refers to the perception that students have of being part of the educational organization [33]. Having a high perception or sense of belonging means that students feel valuable and respected in their own educational program [34–38]. Furthermore, perceiving a strong or special bond between students and others makes them feel that they are part of a community, even when going through difficult moments or when facing challenges [37]. Therefore, the sense of belonging is useful for students, since it means that they feel accepted within the educational system, especially by peers and teachers [39].

To develop a high level of sense of belonging requires to have positive interactions in a stable context, such that students can feel part of a community within the university [39]. Therefore, students who feel comfortable with the formal academic learning environment and the social and cultural environment inside the institution have a sense of belonging that translates into a desire to commit to their studies and to achieve their academic goals [40]. Since the sense of belonging is such an important variable for students' education, its influence on the level of engagement has been deeply studied during the pandemic, as there has been a lack of social interaction that could have a negative effect on belonging [3]. Moreover, during the pandemic, it has been shown that sense of belonging is a predictor of cognitive and emotional engagement [3].

Some authors have argued that in higher education institutions, sense of belonging is a fundamental process that emerges from a collective one and that promotes the development of engagement [38]. This is because belonging involves the development of a student's sense of identification with their university. Therefore, it can be inferred that belonging predicts the level of engagement in the academic context [41]. Belonging can be determined by the similarity and connection that students perceive with respect to their immediate academic community [42]. It has also been shown that sense of belonging is a predictor of behaviors such as respecting the rules or assuming more functions than the mandatory ones [43].

Although there is evidence of the relationship between perceived social support, sense of belonging, and engagement, there is still little evidence evaluating a mediating model of this relationship. It has been found that the support of peers and parents influences the sense of belonging that students have towards their institution and is linked to a greater institutional engagement [44]. This leads to propose a scenario where perceived social support influences engagement, and sense of belonging contributes to explain this relation by a mediating effect. This is consistent with studies that have evaluated the mediating role that sense of belonging has in the relationship between resilience and engagement in higher education students [45].

The present study aims to evaluate the mediating role of sense of belonging in the relationship between perceived social support and engagement in university students. The diverse evidence available suggests this possible mediating role, suggesting the need for analysis of first-year students during the mandatory confinement period.

#### **2. Materials and Methods**

#### *2.1. Design*

A cross-sectional associative-predictive design was used by testing mediation models [46]. Dependent variables were three dimensions of engagement (cognitive, emotional, and behavioral). Independent variables were perceived social support and perceived social isolation. The mediator of these relationships was the sense of belonging.

#### *2.2. Participants*

A convenience sampling method was used because of the restrictions imposed by the COVID-19 pandemic. Participants were 700 freshmen enrolled in 2020 in a Chilean university. The distribution of the sample was 280 men (40%), 418 women (59.71%), and 2 students who identified with another preference (0.29%). Student's age average was 18.4 years (SD = 1.7; minimum = 17; maximum = 32).

#### *2.3. Instruments*

The electronic questionnaire consisted of 27 items taken from 4 culturally and linguistically adapted instruments for the Chilean context. The response was made using a Likert-type scale of 1 to 7 points (1 = maximum disagreement; 7 = maximum agreement). The subsections of the questionnaire were as follows.

The University Student Engagement Scale consists of 15 items and was created by Maroco et al. [25]. This instrument has an adapted version for Chilean university students [24]. It measures engagement by 3 dimensions: Interest (5 items), Effort (5 items), and Participation (5 items). Adaptation and validation results in the university Chilean context showed good results for a bifactorial structure, with one general factor and three subfactors (RMSEA = 0.047 [95% CI: 0.040–0.055]; χ<sup>2</sup> = 210.276, *p* < 0.001; CFI = 0.967; TLI = 0.954) as well criterion validity and reliability (ω = 0.843; α = 0.841).

Membership factor was extracted from the Organizational Identification Questionnaire with Study Centers created by Yáñez et al. [47] and adapted to Chilean university students [39]. In this research the Belonging dimension was used and was made up of 4 items. The psychometric study in the adapted version showed good results (RMSEA = 0.028 [95% CI: 0.000–0.085]; χ<sup>2</sup> = 3.126, *p* = 0.20; CFI = 0.999; TLI = 0.999; RSMR = 0.005) as well as reliability (ω = 0.834; α = 0.815).

The perceived social support factor was inspired by the conceptual proposal of Biasi, Vicenzo and Patrizi [48]. This factor measures student's perception of having a supporting network in their educational community (degree), focusing on peers and professors, when it is needed. The factor was made up of 4 items, with good psychometric properties for a one factor structure (RMSEA = 0.072 [95% CI: 0.041–0.131]; χ<sup>2</sup> = 11.616, *p* = 0.003; CFI = 0.997; TLI = 0.992; RSMR = 0.011) as well as reliability (ω = 0.823; α = 0.798) [3].

#### *2.4. Procedure*

All new students enrolled in 2020 were invited to participate openly and voluntarily through an official email sent by the university's academic direction office and were equivalent to 16.78% of the total population.

For ethical safeguards, the Declaration of Helsinki was considered as a reference [49]. The students reviewed and signed an informed consent designed by the research team and approved by the Ethics Committee, which led them to the instrument in electronic format. The students who did not want to participate were excluded from the research. Participation did not imply any compensation and was voluntary. The data collection was made in the first semester of 2020.

#### *2.5. Analysis*

Data analysis consisted of testing mediation models following the proposed procedures by Baron and Kenny [46]. Indirect, direct, and total effects were tested using the Maximum Likelihood estimator with 95% confidence interval. Correlation analysis was made using Pearson's r coefficient. Plots were made considering parameter estimates. All the analyses were made using Mplus version 7.1.

#### **3. Results**

Before performing the analyses, the distribution of the different study variables was tested (Table 1). None of the variables met the assumption of normality, which is why Spearman's Rho coefficient was used to analyze the correlations.


**Table 1.** Descriptive Statistics.

The results showed statistically significant correlations between all the variables studied (Table 2). The observed correlations fluctuated between medium to strong values, where the strongest association was found between affective engagement and sense of belonging. On the other hand, the weakest relationship observed was between cognitive engagement and sense of belonging.

**Table 2.** Correlations between variables.


\*\*\* *p* < 0.001.

Following the main hypothesis of the study, the mediating role of sense of belonging in the relationship between perceived social support and engagement was tested. To test this hypothesis, three structural equation models were developed that evaluated the indirect effect of sense of belonging on the relationship between perceived social support and the different dimensions of academic engagement. In Table 3 total, direct, and indirect effect is reported.


**Table 3.** Standardized estimations of total, direct, and indirect effects.

\*\* *p* < 0.01, \*\*\* *p* < 0.001.

In the first model evaluated, results show that perceived social support predicts cognitive engagement in a statistically significant way (β = 0.194, z = 9.291, *p* < 0.001). When evaluating the mediating role of the feeling of belonging, it is evident that the relationship between both variables decreases in a statistically significant way, but the direct effect continues to be statistically significant (β = 0.14, z = 5.783, *p* < 0.001). In this way, it is possible to affirm that the feeling of belonging partially mediates (27.83%) the relationship between perceived social support and cognitive engagement (β = 0.054, z = 4.061, *p* < 0.001).

In the second model evaluated, the total effect of social support on affective engagement was statistically significant (β = 0.27, z = 12.196, *p* < 0.001). When assessing the role of sense of belonging, it is possible to assert that the indirect effect is statistically significant (β = 0.052, z = 2.482, *p* < 0.01) and, at the same time, the direct effect decreases but continues to be statistically significant (β = 0.218, z = 12.683, *p* < 0.001). Considering these values, it is possible to assert that we are in the presence of a partial mediation, where 19.25% of the effect of perceived social support on affective engagement can be explained through the feeling of belonging variable.

The third model evaluated analyzed the effect of perceived social support on behavioral engagement (β = 0.225, z = 9.652, *p* < 0.001). For its part, the indirect effect is statistically significant (β = 0.106, z = 6.911, *p* < 0.001), and the direct effect continues to be statistically significant (β = 0.119, z = 4.518, *p* < 0.001). These results show partial mediation, where 47.11% of the effect of perceived social support on behavioral engagement is channeled through the feeling of belonging.

#### **4. Discussion**

The pandemic forced higher education institutions to close due to mandatory confinement and start implementing online classes. The relationships and interaction at the university and inside the classroom changed, as well as the social and educational experience. Professors had to use new methodologies and institutions had to use new strategies to provide support to their students, especially those who entered the university in 2020.

The present study tested the mediating role of sense of belonging in the relationship between perceived social support and engagement in a sample of first-year university students during the start of the pandemic in order to contribute to understand relationships between these variables in the context described above.

The observed correlations allow us to indicate that there is a strong relationship between perceived social support and the sense of belonging, the first being a significant predictor of the second, in accordance with what was observed in previous research in a regular context and in virtual education during the pandemic [23,24,32]. This implies that, by perceiving greater social support, a greater sense of belonging is also felt. In this way, maintaining better relationships with peers and teachers affects belonging experienced by students.

Perceived social support showed a moderate relationship with three types of engagement and was also a significant predictor, in accordance with previously reported research in a regular context and in virtual education through the pandemic [23,24,50]. The link between perceived social support regarding behavioral and cognitive engagement may be because having support within an academic activity can encourage students to be interested, involved, and persevere in the academic tasks they face. The strongest relationship, however, was obtained with respect to affective engagement, which may be due to the importance of positive relationships in the classroom to generate favorable and satisfactory work environments that stimulate student motivation.

Sense of belonging showed a moderate and predictive relationship with respect to cognitive and behavioral engagement and a strong relationship with respect to affective engagement, in accordance with what was reported in previous research in a regular context and in virtual education during the pandemic [23,24]. The relationship observed between the sense of belonging with respect to cognitive and behavioral engagement may be linked to people's behavior when part of a reference group [42,43]. Students who maintain stronger ties to their degree and better align with social and academic degree interests may experience greater interest, engagement, and participation. The greater relationship between the sense of belonging and affective engagement, in line with what was previously stated, may be because a student that experiences sense of belonging develops a positive relationship with peers and teachers and perceives a better work environment. As a result, they perceive a different academic workspace, which can affect their level of motivation.

Regarding the mediating role of the sense of belonging in the relationship between perceived social support and the different types of engagement, it was observed that the sense of belonging mediated the relationship with respect to the three types of engagement. In all cases, the mediation was partial and explained a good part of the relationship between the variables. These results are indicative that there is a direct influence of perceived social support on the different types of engagement, but that part of the relationship is explained by the presence of a sense of belonging. In other words, the effect that a greater perception of social support exerts on the engagement occurs because it affects it directly but also because it influences the students to feel a greater sense of belonging.

In this way, a student exhibits greater behavioral (participation, adherence, etc.), cognitive (more effort and intellectual involvement), and affective (enjoyment, positive attitude, etc.) engagement because they perceive more support from peers and teachers, but also because this support makes them feel a greater sense of belonging to their educational community.

The sense of belonging mediated the relationship between social support and affective engagement to a greater extent than the other types of engagement. According to what was previously stated, this effect may be linked to a greater preponderance of positive social relations and a favorable socio-academic climate to produce positive effects on the affections of the students around their studies and academic tasks.

The results described and analyzed above suggest that, in an emergency virtual education context as described in this article, perceived social support is a predictor of academic engagement. In the same way, the sense of belonging is configured as a measure that allows partially explaining this relationship, showing how supportive relationships can influence commitment, but also that they can influence belonging, and through this, the motivational states of the students.

#### **5. Conclusions**

The present study aimed to evaluate the mediating role of sense of belonging in the relationship between perceived social support and engagement in university students. Results showed the mediating role of sense of belonging and additionally showed differences in the percentage of mediation for each type of engagement.

It can be concluded that the sense of belonging acts as a partial mediator of the relationship between perceived social support and academic engagement. In addition, it is observed that mediation is greater in the case of the relationship between perceived social support and affective engagement, which highlights the importance of the effect of social relationships on the motivational experience of the university student.

The relationships observed and the mediating role were appreciated in a context of virtual education, in which the way of establishing interaction with peers and teachers was mediated by technological supports, the use of software with different social possibilities, and delays in communication, among other factors. The results presented occurred under these conditions, which allows reaffirming the relevance of these variables in influencing the motivation of students in that context. However, it is not possible to determine whether these variables acquire greater or lesser relevance with respect to a regular context because the study is limited by the lack of a previous valid basis for comparison. This allows an interesting line of research to emerge to elucidate this question.

Another limitation of this study has to do with the sampling that reduces the sample to a portion of university students. This restricts the possibilities for analyzing differences by type of degree and type of university and/or generalizing the results to the university population. This type of disaggregation of the results emerges as a future line of research, as well as differences by semester of study and gender, among others, which could be relevant.

These results have possible relevant applications to higher education institutions and can be useful to guide the support provided by higher education institutions to promote the academic engagement of students in virtual educational contexts. Generating connection strategies between students can favor perceived social support and thus engagement. However, strategies that allow students to connect with each other and with their teachers, in an integrated educational community that allows students to feel part of it, can be a deeper and more powerful strategy that encourages academic engagement. On the other hand, interventions to promote post-pandemic adaptation should consider the strengthening of these constructs among students, especially new entrants and those who took online classes in their first two years of university.

**Author Contributions:** Formal analysis, J.B.-R.; Funding acquisition, J.M.-A.; Investigation, J.M.-A., J.B.-R. and V.I.-V.; Project administration, J.M.-A.; Writing—original draft, J.M.-A.; Writing—review & editing, J.B.-R., V.I.-V. and K.L.-P. All authors have read and agreed to the published version of the manuscript.

**Funding:** This publication received funding from Universidad San Sebastian through project VRID\_ DocI22/01u.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki and approved by Ethics Committee of the Research and Development Vice-presidency of Universidad de Concepción for studies involving humans (Certificate: CEBB 645-2020, April 2020).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Data supporting the reported results can be found at https://drive. google.com/file/d/127bIfoc3n6HK8fyMDJKQL-YzgCQzsp1W/view (accessed on 16 November 2022).

**Acknowledgments:** The authors are grateful to the Universidad San Sebastián for providing the necessary resources to publish this research through project VRID\_DocI22/01u.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

## *Article* **Do Virtual Campuses Provide Quality Education? A Study on the Perception of Higher Education**

**Ellián Tuero 1, Lucía Álvarez-Blanco 2,\*, Isabel C. Ayala-Galavis 1, Celia Galve-González <sup>1</sup> and Ana B. Bernardo <sup>1</sup>**


**\*** Correspondence: alvarezblucia@uniovi.es

**Abstract:** In recent years, there has been an increase in the use of technologies in all aspects of daily life, especially in educational contexts. Indeed, in most universities, using a virtual campus as a support for teaching is now a general practice, even in face-to-face teaching. However, although there are multiple studies on the quality of education and services provided by virtual campus platforms, as well as statistics on the use of the various tools and forums, very few studies have considered students' perceptions of the quality perceived by students concerning the pedagogical use that teachers make of virtual campuses. Even fewer studies have examined this as a function of what year the students are in. In order to examine this more thoroughly, an ad hoc questionnaire was applied to 783 students enrolled in the first three years at various universities in Spain, covering all knowledge areas. The results show that first-year students had more positive opinions of the quality of their teachers' virtual-campus practices than students in subsequent years. More specifically, those first-year students perceived greater encouragement and motivation from the teaching staff and more communication between teachers and students. These findings suggest the need for good teaching practices that consider motivation, communication and collaborative groups, not only during the first year, but also throughout university courses in order to ensure quality education.

**Keywords:** higher education; virtual campus; quality; virtual education; digital environments

#### **1. Introduction**

The challenge facing current university teachers in the 21st century is educating skilled professionals capable of adapting to the demands and requirements of a globalized society that is in constant flux and is dominated by information and communication technologies [1–3]. In this regard, in a context in which the amount of information seems unsurmountable [4], future graduates will have to develop skills that will allow them to learn autonomously, practically, and continuously, and maintain their engagement throughout their lives [5]. More specifically, and as established by the European Higher Education Area (EHEA), competencies such as personal initiative, individual responsibility, and particularly critical thinking, digital competence, and collaborative working are essential and have become more predominant in this new era [6,7].

Against this backdrop, universities have seen the need to revamp and adapt their traditional teaching methods in order to encourage the learning of these competencies beyond the walls of the traditional classroom [8,9]. In the last decade, and in response to the European convergence process, new, more suitable spaces have been added to university teaching [10], based principally on ICT [11,12]. This is the case with virtual campuses, the use of which as a teaching support is nowadays one of the most widespread practices in these institutions [13,14]. In fact, according to González-González and Infante-Moro [15] (p. 1), they have become a key part of contemporary higher education by allowing "continued lifelong education and learning, and making collaboration, expansion, the relationship

**Citation:** Tuero, E.; Álvarez-Blanco, L.; Ayala-Galavis, I.C.; Galve-González, C.; Bernardo, A.B. Do Virtual Campuses Provide Quality Education? A Study on the Perception of Higher Education. *Sustainability* **2023**, *15*, 620. https://doi.org/10.3390/su15010620

Academic Editor: Aras Bozkurt

Received: 14 November 2022 Revised: 19 December 2022 Accepted: 23 December 2022 Published: 29 December 2022

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

with society, and knowledge transfer possible". All of which are in line with the new teaching–learning paradigm demanded by the EHEA.

The use that different universities make of these virtual environments is so varied as is the terminology used within the field: web educational platforms, virtual spaces, virtual classrooms, etc.—that according to Urbina and Salinas [16] (p. 6), it is difficult to find unanimity in the scientific literature about how they are defined. Despite that, and according to these authors, there does seem to be a consensus that virtual campuses, rather than correlating to physical university campuses, are "web sites available to an educational community, with the facility to provide teaching resources, and communication and interaction functions". Nevertheless, although they were created to make the educational services provided by the universities more available—teaching materials, online library access, etc. [17]—currently, thanks to advances in educational information technology, there are many possibilities. These include a better accessibility from any internet-connected device [18,19], making the teaching–learning process possible in any context and at any time [20].

Apart from the above, virtual campuses are spaces which provide students with information relevant to their courses such as teaching guides, syllabuses and coursework exercises [21]. They also allow students more opportunity for interaction with teachers [22]. Students can consult their teachers more, exchange more messages (not only on academic topics) and receive more tutoring from their teachers by being able to manage these processes more easily and quickly given their essentially online nature [23]. In this regard, the possibilities of these more direct, varied, and educational interactions [24] in these qualification or evaluation instruments [25], make it possible for students to receive more continuous attention and allow a more individualized evaluation process, providing other means of communication and contact outside the traditional classroom [26]. Similarly, virtual campuses help students to interact more with their classmates by, for example, participation in forums and the use of interactive resources (Webquests, Blogs, Wikis, etc.) and tools in these online spaces which encourage collaborative learning [27,28].

Nonetheless, and even though there is this methodological versatility [14], researchers such as Area et al. [13] have warned of the domination of expository teaching in which the teaching role is that of a mere transmitter of knowledge. There are, however, professionals who encourage more active, autonomous, collaborative work from their students in the teaching–learning process, relegating their own roles to the background [29].

This being the case, and having confirmed that the success of these virtual spaces comes from the "multiple educational activities, from permanent accompaniment ( ... ) to the generation of affective ties of value and respect, that is, to human interaction" [30] (p. 53), the overall trend, nowadays, for teachers' use of virtual campuses is as simple support for—and even copies of—the activities in their traditional face-to-face classes [22]. These technological spaces are flooded with notes, class presentations, and an endless duplication of materials [19], which turns them into repositories of information and content [31], consequently failing to take advantage of the interactive and communicative potential there is for the individualization of learning [19].

In accordance with these ideas, what provides quality to teaching is not the virtual campus tools in and of themselves [23,32], but rather that "they acquire pedagogical value ( ... ) as mediating artefacts between the teacher and student or between peers which provides a unique virtual educational context facilitating interactive processes of co-construction of knowledge" [33] (p. 164). Only to the extent that the use of these virtual tools are in effect combining teaching-support-quality (in face-to-face or distance teaching), will higher educational institutions be pursuing excellence, educational efficacy [34], and consequently student motivation [35], satisfaction, and performance as indicators of the quality of their educational systems [36].

In this regard, the 2016 UNIVERSITIC report showed that universities should continue to develop, implement, and promote good practices related to virtual teaching [37]. This inevitably includes research and the systematic evaluation of the quality of the teaching in these technological learning environments, considering not only criteria based on the technological infrastructure, teaching methodologies used, and the results of learning, but also on users' opinions, especially student users [38]. Although in recent years there has been growing interest in the scientific community about the study of the quality of teaching practices via virtual campuses in higher education [39–41], there are few studies that have examined this from the student perspective [42].

#### *Objectives and Hypotheses*

In light of this, and bearing in mind that new students have different needs in terms of attention, monitoring, and evaluation of the learning process compared to more experienced students [8], the general objective of this current study is to analyse students' perceptions, by university year, of the teaching practices delivered through the virtual campuses as a support to provide quality to their university teaching. A more specific objective is to attempt to determine whether there are statistically significant differences in student perceptions of student–teacher interactions encouraged by teaching practices through the virtual campus. To that end, we have formulated the following research hypotheses: (1) first-year students will ascribe more importance to teachers' teaching uses of the virtual campus as a quality complement to their classes than students in later years, (2) firstyear students will ascribe more importance to teacher–student interaction-communication driven by the teaching use of virtual campus tools, and a final hypothesis that applies to the entire sample is (3) that there will be no statistically significant differences between the three academic years in students' perceptions about the teachers' promotion of motivation in interactions via the virtual campus.

#### **2. Materials and Methods**

In order to respond to the objectives, we performed an ex post facto study with a descriptive, inferential research design.

#### *2.1. Participants*

A total of 783 university students participated in the study. Three quarters were women (74.58%, *n* = 584) and one quarter were men (25.42%, *n* = 199). They were aged between 19 and 59 years old (M = 22.43, SD = 7.04). All were studying courses in various Spanish higher education institutions: the Universidad Nacional de Educación a Distancia (The National Distance Learning University)—UNED—(*n* = 112), and other universities in Spain (*n* = 31), with the vast majority studying at the University of Oviedo (*n* = 636). Most students were in the first year of their degree course (*n* = 381), followed by second-year (*n* = 256), and third-year (*n* = 146) students. The students were studying subjects that covered all knowledge areas: social and legal sciences (54.3%), engineering and architecture (3.8%), health sciences (37.2%), arts and humanities (4.1%), and science (0.6%).

Given the sociocultural and historical background of the students (i.e., the knowledge and information society characterized by the use of new technologies), 98% of the sample reported having internet-connected electronic devices with which they could access, among other things, the virtual campus during their courses of study.

#### *2.2. Instruments*

The data collection was via a questionnaire entitled "Analysis of university students' perceptions of virtual campuses in the European Higher Education Area". This was created ad hoc and validated in previous studies [43]. The reliability, in terms of internal consistency, was calculated using the Cronbach alpha, giving a value of 0.894, which according to O'Dwyer and Bernauer [44], is a more than acceptable value.

The initial questionnaire was made up of 44 items in eight blocks: Block A–Introduction and collection of sociodemographic data (9 items with semi-open responses); Block B– Availability of resources at home (5 items with yes/no responses); Block C–Teacher planning (4 items); Block D–Content (7 items); Block E–Methodology (5 items); Block

F–Communication (5 items); Block G–Evaluation (5 items), and Block H–Digital competence (4 items). Blocks C, D, E, F, G, and H were Likert-type responses with four options (e.g., 1 = completely disagree, 2 = disagree, 3 = agree, and 4 = completely agree).

We selected 12 items from various blocks for the current study in order to achieve our objectives, making up two dimensions of study. The 44 items assessed aspects of the teachers, students, and the virtual campus in general. Therefore, for this specific study, given its objective and hypotheses, 12 items were selected corresponding to what students might think about teachers' usage of the virtual campus, as well as the interaction that they could maintain through using this tool. The following table establishes the classification of these items with their respective dimensions (see Table 1).

**Table 1.** Dimensions and items selected to study university students' perceptions about whether teaching practices in virtual campuses provide quality to education.


Source: researchers' own work.

#### *2.3. Procedure*

For this study, an incidental sampling was used, with the intention that the sample be as diverse as possible. The researchers contacted teachers in various universities with easy access who were willing to collaborate, explaining the aim of the study in detail and the feasibility of applying the instrument to their students voluntarily and anonymously. Teachers were selected with the following inclusion criteria, those who: (a) preferentially taught undergraduate degrees, and (b) used virtual campuses in their subjects.

The teachers were responsible for administering the questionnaire to their own students following the researchers' instructions, and offering the students the option to participate in the study by completing the questionnaire online in the virtual campus of the teachers' subject. Before completing the questionnaire, the students were informed of the study objectives, its confidential nature, and the time needed to complete the electronic questionnaire, which was around 30 min.

#### *2.4. Data Analysis*

We used the SPSS v.24. statistical software, starting with a descriptive analysis (i.e., measures of central tendency and variability). Following that, the normality of the distribution was checked in order to select the comparative analysis to perform. According to the Kolmogorov–Smirnov statistic (df > 50 and *p* < 0.001 in all cases), we confirmed the existence of differences in the samples, meaning the data were not normally distributed.

Subsequently, we carried out non-parametric tests, performing the Kruskal–Wallis analysis to determine possible differences between the groups according to their academic year (i.e., 1st, 2nd, and 3rd) via the Chi-squared statistic. Effect sizes were determined using Cohen's *d*, with values between 0.20 and 0.49 indicating small effect sizes, values between 0.50 and 0.79 indicating moderate effect sizes, and values over 0.80 indicating large effect sizes [45]. Correlational analyses were also performed using the Pearson correlation coefficient.

#### **3. Results**

This section covers the two dimensions of analysis about university students' perceptions of the quality of teaching practices via the virtual campus as: (1) a complementary support resource which adds quality to the education received, and (2) a potential tool for encouraging teacher–student interaction.

As Table 2 shows, most of the students surveyed (M = 2.83; SD = 0.690) thought that the subject content in the virtual campus at their universities was up to date (73.9%), although there were some very different scores (with close scores for agree and disagree) in response to the activities being published on the virtual campus which encouraged the discussion of ideas, debate, etc., and teachers asking for an evaluation of the teaching and technical content of the subject (M = 2.36; SD = 0.767).

**Table 2.** Percentages, mean, and standard deviation for the items in Dimension 1: the campus as a complimentary support resource adding quality to the education received.


Source: researchers' own work.

In addition, referring to the teaching role, 60.4% of the students felt that the teachers had up to date, specialized training for successfully managing the virtual campus for their various subjects (M = 2.58; SD = 0.746), with a similarly high percentage of students reporting that teachers demonstrated a positive attitude towards using the virtual campus (68.1%, M = 2.71; SD = 0.749). Lastly, when asked about whether teachers gave guidance and advice through the virtual campus (M = 2.31; SD = 0.781), the majority disagreed or completely disagreed (57.5%).

The Kruskal–Wallis test was performed to determine whether there were statistically significant differences between the surveyed students depending on their academic year. The results are given in Table 3.

Significant differences were only found in one of the variables in this dimension, with first-year students more in agreement that activities are published on the virtual campus which encourage the discussion of ideas, debate, etc. (*χ*<sup>2</sup> = 15.33; *p* < 0.001; *d* = 0.264), with a small effect size. This was followed by second-year students (*n* = 256; mean rank = 368.93), with third-year students being least in agreement that there were these types of activities in the virtual campus (*n* = 146; mean rank = 355.09).

After reviewing the main results for the first dimension of the study, we continued with the scores for the second dimension (Table 4).


**Table 3.** Comparative analysis for Dimension 1: the campus as a complimentary support resource adding quality to the education received.

Source: researchers' own work.

**Table 4.** Percentages, mean and standard deviation for the items in Dimension 2: the campus as a potential tool for teacher–student interaction.


Source: researchers' own work.

As the table shows, in this second dimension the students' had a negative assessment of the potential of the virtual campus as a tool or vehicle promoting communication with teachers. More than three quarters of those surveyed (82.2%) reported that the fundamental content about which they received information from the virtual campus was about subject grades (M = 3.04; SD = 0.802), rather than other non-curricular aspects that would contribute to increased student motivation and involvement. In addition, and in relation to this communication, most subjects reported that teachers responded satisfactorily to questions and observations via the virtual campus (M = 2.70; SD = 0.722).

However, there did appear to be certain gaps in the nature of this communication, which was not frequent (M = 2.26; SD = 0.837), nor did it flow well (M = 2.23; SD = 0.823). In addition, the speed with which the teachers respond to questions and observations was rapid for only 54.3% of the sample. The details of these communications helps us understand that 68.3% of students reported that teachers did not encourage motivation in their interactions (M = 2.16; SD = 0.797).

Finally, following the Kruskal–Wallis test, significant differences were found in five of the six variables analyzed (see Table 5), with small effect sizes in each case.


**Table 5.** Comparative analysis for Dimension 2: the campus as a potential tool for teacher–student interaction.

Source: researchers' own work.

An overall examination of the items in Table 5, focusing on those where differences were seen, shows that the first-year students had higher evaluations of the campus as a tool promoting interaction with the teachers, especially compared to the third-year students who disagreed most strongly with the idea of a communicative process driven by their respective virtual campuses. The values of the mean ranks were higher for the first-year students in variables referring to communication with teachers that was frequent (mean rank = 415.06; *χ*<sup>2</sup> = 9.847; *p* = 0.007; *d* = 0.020); that flowed well (mean rank = 420.99; *χ*<sup>2</sup> = 13.91; *p* = 0.001; *d* = 0.024), was satisfactory (mean rank = 407.25; *χ*<sup>2</sup> = 7.769; *p* = 0.021; *d* = 0.017), and was a rapid response to student queries (mean rank = 427.76; *χ*<sup>2</sup> = 25.21; *p* < 0.001; *d* = 0.035). The first-year students also perceived better encouragement of motivation in interactions by teachers via the virtual campus (mean rank = 419.05; *χ*<sup>2</sup> = 12.58; *p* = 0.002; *d* = 0.024) than the second- or third-year students.

#### **4. Discussion**

Starting from the premise that a new student would need more support in their learning processes to be able to make the most of their entry into higher education, and with that, to properly adapt to the academic demands involved [46], the general objective of this study was to examine whether students' perceptions about the quality of the teaching practices delivered via virtual campuses would vary depending on which academic year they were in. In the first dimension of the analysis, we expected the first-year students to have higher ratings for teachers' teaching practices in these spaces as a supporting teaching quality (H1); however, the results from the variables in this dimension (items 1–6 in Table 3) did not support this hypothesis, although one of the variables (item 3 in Table 3) did exhibit statistically significant differences between the three years. This item referred to the didactic use teachers made of these platforms to publish activities which encouraged a discussion of ideas, debate, thought, and critical thinking, etc. This finding is in line with Bangert [47] who, in a study looking at master's students' evaluations of a nursing course, concluded that students positively evaluated the quality of activities set by teachers, via the virtual campus, promoting debate and discussion, which allowed for a better understanding of course content [48]; however, most of the students surveyed in our study exhibited the opposite opinion. This was what Area et al. [13] and Cisneros [14] found, demonstrating that generally, and regardless of the year the teaching took place in, the tendency was to use these spaces principally as mere repositories of information and, to a lesser extent, as a teaching resource for more active learning activities needing collaboration between students (e.g., discussion, exchange of ideas, etc.).

Continuing with the second dimension, referring to teachers' making the most of the interactive and communicative potential of the virtual campus, once again it was the firstyear students who perceived more encouragement of teacher–student interactions via this resource, which confirms our second hypothesis (see items 7–12 in Table 3). Considering the fact that many studies have confirmed that the transition to university produces significant academic, psychological, and social changes that can on occasion lead to insecurity and disorientation, better attention and monitoring by teachers becomes essential for better student adaptation to the university context. Aspects such as interaction, feedback, and tutorials during the first year promote better satisfaction with the teaching received. For that reason, our results show that from the perspective of the students surveyed, it was the first-year students who most felt that teachers maintained a frequent contact and flowing communication with them, and who responded quickest and most satisfactorily to their queries and observations. In this way, they understood that their teachers made better use of the interactive and communicative tools offered by the virtual campus (see items 7–11 in Table 5). This is along similar lines to the findings from Bangert [49] with master's students. That author found that a high percentage of students reported that their teachers were always accessible and not only responded rapidly to their queries, but also gave them supportive feedback, effective, personalized communication, exhibited high levels of concern that they learn, and encouraged them to perform their tasks better. However, our results contrast those found by Fariña-Vargas et al. [22] who, in a study in the virtual classrooms of La Laguna University, found that teachers who taught various classes made little use of interactive resources via the virtual campus, offering hardly any educational feedback. These results agree with those from Area et al. [13], who warned of minimal continued tutoring or teacher–student feedback. According to that study, a lack of this teaching feedback could be an incentive for students to participate less in academic activities, and consequently for a reduced interest and curiosity about their subjects [50].

Lastly, the third research hypothesis, within this same dimension, was rejected. Against our expectations, our results confirmed that there were statistically significant differences between the three academic years in the student perceptions about the encouragement of motivation by teachers in interactions via the virtual campus. Once again, it was the first-year students who exhibited the most appreciation of this encouragement compared to their second- and third-year classmates. These results contrast those found by Álvarez et al. [43], who reported in their study that 83.9% of undergraduate students surveyed, mainly in their first year, reported that their teachers did not motivate them enough in their interactions via the virtual campus. Nevertheless, one of the strategies to implement in order to achieve a standard of quality in teaching is for motivating university students via individualized attention which fosters and incentivizes their curiosity and desire to learn [51]. In order to do that, the teacher must be flexible in the face of the various challenges and concerns that students may present, and of course, ensure a friendly and harmonious working climate, encouraging participation in discussion forums through the creation of different types of debate. On similar lines, Rojas et al. [52] suggested that if good teaching practices were implemented via virtual campuses, it could notably increase student motivation, so that students would persist in their courses and be more likely to successfully complete their education.

Similarly, in research by Martínez et al. [43], it was first-year students who made the best use of the support tools and teaching materials the teachers made available on the various virtual platforms (e.g., teaching guides, syllabuses, notes, activities, etc.) and who used forums and bulletin boards more, with the latter being the space the teachers usually used to inform students about items of interest related to their subjects. In our study, on similar lines, the fact that the first-year students were the ones who demonstrated the best perceptions of teaching quality supported by the virtual tools provided by the campus may be precisely due to the fact that during the first year the teachers use them more, incentivizing their students to use them as well. This would be in line with Feliz [53] by indicating that these tools need significant teacher involvement to obtain full use of them; therefore, if teachers do not make proper use of the virtual campus, it will also be difficult for students to do so.

This study does have some limitations. It would have been interesting to have had a greater representation of students from private or fully virtual universities, as well as the teachers' own opinions, which would have allowed the identification of where opinions agree and differ, giving guidance to future informative and training action aimed at both agents in the teaching–learning process. It would also have been interesting to explore the possible influence of gender, knowledge area, and university type on the opinions collected. For future research, it could also be interesting to compare the students in their first and fourth years to analyze the differences between them, as well as those that could belong to similar areas or disciplines. There might be differences between those new students and those who are about to graduate in their perception about the use of the campus by teachers, as well as the importance of the use of the virtual campus for the students of a 1st year course. These considerations could be included in future research which, in addition to the above, could complement this positivist methodology with another interpretive paradigm that would provide information on what the items in the questionnaire mean individually to the surveyed students. In this respect, we would suggest discussion groups as an instrument, with the information treated by a content analysis.

#### **5. Conclusions**

In summary, and in light of the above, it seems essential that teachers deliver appropriate practice, through the support of the virtual campus, not only during the first year but also in subsequent years. In consequence, higher education institutions, and even more so the teachers within them, should be aware of the need to adapt these practices to the profiles of the students in the different academic years so that they can deliver quality teaching, which involves offering the students what they require: functional knowledge; strong, effective communication processes; and a continual updating of the subjects of digital skills and student tutoring. Besides that, future challenges to develop the quality of virtual campuses could include activities that improve the collaborative learning between students. In addition, it would be important for teachers to consider the full potential of the virtual campus and to take advantage of all the tools it offers through training and education in order to avoid using it merely as a repository, especially for those teaching first-year students, because as this study found, new students need more involvement from their teachers through the virtual campus.

**Author Contributions:** Conceptualization, E.T., L.Á.-B., I.C.A.-G., C.G.-G. and A.B.B.; methodology, E.T. and L.Á.-B.; software, E.T. and L.Á.-B.; validation, E.T., L.Á.-B., I.C.A.-G., C.G.-G. and A.B.B.; formal analysis, E.T. and L.Á.-B.; investigation, E.T., L.Á.-B., I.C.A.-G., C.G.-G. and A.B.B.; resources, E.T., L.Á.-B. and I.C.A.-G.; data curation, E.T., L.Á.-B. and I.C.A.-G.; writing—original draft preparation, E.T., L.Á.-B., I.C.A.-G., C.G.-G. and A.B.B.; writing—review and editing, E.T., L.Á.-B., I.C.A.-G. and C.G.-G.; visualization, L.Á.-B.; supervision, E.T. and L.Á.-B.; project administration, E.T., L.Á.-B. and I.C.A.-G.; funding acquisition, E.T., L.Á.-B. and C.G.-G. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Severo Ochoa Program of the Government of the Principality of Asturias, grant number BP20-116 (Mrs. Celia Galve González).

**Institutional Review Board Statement:** Ethical approval was not provided for this study because the ethics committee did not exist before starting the study. The ethics committee began operating in 2019, June, and the data was collected earlier; the "Acuerdo de 25 de junio de 2019, del Consejo de Gobierno de la Universidad de Oviedo, por el que se aprueba el reglamento del Comité de Ética en la investigación" that means, "Agreement of 25 June 2019, of the Governing Council of the University of Oviedo, which approves the regulations of the Research Ethics Committee", created the ethics committee (see the public resolution at http://sede.asturias.es/bopa/2019/07/23/2019-07307.pdf (accessed on 10 November 2022) for further explanation of the process). However, this study was carried out with consideration for the international protocols for scientific research, and in particular, in accordance with the requirements of the Declaration of Helsinki for research with human beings and Organic Law 3/2018, 5th of December, on the Protection of Personal Data and ensuring digital rights. In addition, we had the explicit permission of each participant to use their data for scientific research, with their anonymity and confidentiality assured. The participants provided their written informed consent to participate in this study.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

## *Article* **The Dropout of First-Year STEM Students: Is It Worth Looking beyond Academic Achievement?**

**Joana R. Casanova 1,\*, Adrián Castro-López 2, Ana B. Bernardo <sup>3</sup> and Leandro S. Almeida <sup>4</sup>**


**Abstract:** The expansion of Higher Education increased the diversity of students, with heterogenous characteristics, needs, and values. Institutions, intending to preserve the mission and the transformative potential of the tertiary level of education, are facing and implementing policies and practices that enhance success conditions, persistence, and avoid student dropout, in order to meet the goals for sustainable development of people and societies. The present study aims to analyze the impact of personal and academic variables on students' academic difficulties in adaption to HE, academic achievement, and dropout among first-year STEM students. From a cohort of STEM first-year students at a Portuguese public university, the participants numbered 1376. Applying the structural equation modelling, the results highlight the effect of the variables age, gender, scholarship, and grade point average on access to higher education, difficulties in adapting to higher education, and dropout decision. Understanding STEM students who have already dropped out can contribute to better identification of institutional actions to prevent and reduce its occurrence, especially in first-year students.

**Keywords:** higher education; first-year students; student adjustment; academic achievement; dropout

#### **1. Introduction**

In last decades we have witnessed a great expansion of Higher Education (HE), explained by the increasing acknowledgement of HE as a context with high transformative potential, which has the mission to provide opportunities for students' global development and training in technical and professional knowledge and skills (Harman, 2017). Frequently, students and their families invest in tertiary education as a chance to obtain a degree and thus widen their opportunities. This is especially true for people from socioeconomically disadvantaged backgrounds [1,2].

In this framework, the sustainable development goals (SDG) assumed by the United Nations highlight the scientific and social role of higher education in the global objectives of sustainable development, generalized to the entire population. In line with the SDG 4–Education and the SDG8–Decent work and Economic Growth objectives, it is urgent to provide lifelong learning opportunities for youth and adults (target 4.3), to substantially reduce the proportion of youth not in employment, education, or training (target 8.5), and to achieve full and productive employment and decent work for all (target 8.6), increasing the number of youth and adults who have relevant skills for employment, decent jobs, and entrepreneurship (target 4.4). The educational goals are also related with more healthy and sustainable lifestyles, promoting more democratic societies, based on the respect of human rights and the promotion of a culture nonviolence (target 4.7) [3].

Given this world context, the dropout of students from HE can be understood as a phenomenon that goes against the objectives of societies and governments [3]. Students

**Citation:** Casanova, J.R.; Castro-López, A.; Bernardo, A.B.; Almeida, L.S. The Dropout of First-Year STEM Students: Is It Worth Looking beyond Academic Achievement? *Sustainability* **2023**, *15*, 1253. https://doi.org/10.3390/ su15021253

Academic Editor: Antonio P. Gutierrez de Blume

Received: 11 November 2022 Revised: 4 January 2023 Accepted: 5 January 2023 Published: 9 January 2023

**Copyright:** © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

<sup>1</sup> Research Centre on Child Studies (CIEC), Institute of Education, University of Minho, 4710-057 Braga, Portugal

who drop out usually experience the non-fulfilment of a personal and professional project, with negative impact on their own expectations, but also on their family's expectations. The negative impact also occurs in HE institutions, since internal quality assurance systems consider not only the attraction of students, but also course completion rates as indicators of the quality. From an economic perspective, HE institutions lose public and family funding which are directly associated with the number of students who attend them. For all these reasons, institutions are becoming increasingly attentive in monitoring and predicting the phenomenon to develop institutional strategies and services to promote persistence and support students at-risk [4,5].

Dropping out can be understood as a complex and multidetermined phenomenon. According to V. Tinto [6], assuming an interactionist approach with dynamic and reciprocal interactions, dropout is the result of a gradual process of disengagement of the student from the course and/or institution. The students' personal characteristics, the formal and informal characteristics of the HE institution, and the characteristics of the community in which the students are inserted play an important role in the decision to remain or drop out [7–10]. In the same line, other authors point out that dropping out cannot be analyzed as a decision taken at a given moment, because it is essentially the result of a process of investment, in affective, cognitive, and behavioral terms, of students in academic activities, which is associated with the quality and degree of attachment of students to their course and institution [11–14]. For some students with low levels of maturity and autonomy, the transition and adaptation to HE is more challenging, especially for those who need to develop coping mechanisms to manage the demands and/or perceived lack of support from the institution [15–18].

In this article, considering STEM first-year students, we will analyze some of the variables identified in the research as relevant to explaining the student dropout. In the area of STEM, student dropout does not favor countries' development strategies and economic competitiveness [18]. Given the great importance of STEM skills in solving most everyday problems and exercising citizenship [19], the dropout of students in these undergraduate areas becomes socially more critical.

One of the variables assumed as relevant in the dropout process is related to the students' learning and academic performance levels [20,21]. In the first weeks, the student evaluates the levels of understanding of the curricular subjects or the effectiveness of lessons and the work done in various groups, for example [22]. When the levels of performance achieved are not representing the efforts made by the student, the first doubts about the course and higher education emerge and some intentions to drop out could also emerge [23,24]. The first tests or exams at the end of the semester may confirm such academic difficulties, and at that point the decision to drop out can be made. The student can weigh the marks obtained and the number of curricular units successfully completed to appreciate whether the investment made in the academic activities was compensated or not. In this sense, in our study, we considered the number of curricular units completed by first-year students as a possible predictor of dropout in line with research that points to academic performance as the closest determinant or the most relevant predictor of the dropout decision [12,25]. Thinking about dropout as a gradual disengagement process, achievement difficulties appear in our model as a variable related to the dropout decision.

On the other hand, research shows that several students experience difficulties in their transition and adaptation to higher education, and that these difficulties impair their academic performance and thus may be the origin of the dropout decision [12,16,26]. HE makes several demands and students do not always have the necessary skills and levels of autonomy to successfully overcome such demands [27–29]. Such students, without support, may progressively disengage from academic activities, exacerbating feelings and behaviors associated with academic maladaptation [30–32].

Research in the area indicates, for example, that first-year STEM courses have foundational curricular units in the areas of mathematics, physics, and chemistry, and students with poorer academic skills in secondary school may experience more difficulties in their

learning [32,33]. Problems in the academic adjustment of incoming students are not limited to difficulties in learning and academic achievement; some students arrive in higher education with insufficient levels of personal autonomy to manage new daily responsibilities or to establish relationships with new classmates and teachers [10,34,35], while others have unrealistic expectations about the course and HE, sometimes with poorly defined vocational projects. Finally, the institution and academic environment in ES is quite different and less welcoming than the reality experienced in secondary school, and students do not always have the personal resources to cope with less structured and more anonymous environments [15,36]. Due to its relevance, in our model, we introduced the difficulties of academic adaptation that students verbalize when entering HE as a factor that can impact academic performance at the end of first semester and present a direct or/and indirect impact on the decision to remain or leave during the first year [23,37,38].

Research also shows that other entrance characteristics of students can have an impact on their adaptation, performance, and permanence. One of these variables has to do with the grade point average (GPA) to access HE, which reflects the set of skills that the student has developed throughout his or her previous schooling. At the same time, and especially when we talk about STEM courses, this access average may condition the access of students to certain courses when the admission rules are determined by the seriation of candidates based on the access marks. In Portugal, the numerus clausus system determining the access to courses and institutions is based on the application mark of the students, weighting their school performance in secondary education and the marks in the entrance examinations to higher education. The grade point average (GPA) is assumed to be a predictor of academic success and explains around 20% of the variance of academic performance in the first year [10,25,39].

Another variable that could influence the students' adaptation to HE is that of socioeconomic resources, and the financial support policies for students (scholarships) may be an relevant to their access and permanence, particularly for students from low-income families. These financial supports help students' engagement, persistence, and higher completion rates [40–43].

Moreover, student age is a relevant variable to consider. Older students, being workers, seem to have a higher propensity to drop out than traditional students; they usually present a higher familiar level of cultural and economic capital that allows them to understand and become implicated in academic university life [40,44,45]; however, older students may have fewer skills in organizing their learning activities because they have been disconnected from their academic training for some time or because they need to manage studying with other social, professional, or family responsibilities, missing more classes and showing less availability for group work and losing the opportunity to ask questions of colleagues and teachers [46]. On the other hand, due to their older age and socioprofessional status, they may also be stereotyped by their peers, and even by teachers, about their role as students, complicating their academic adaptation. Students from low social backgrounds and students with no family tradition of attending HE tend to have lower skills in mathematics and reading, critical thinking, and aspirations, and have the perception of less parental support and encouragement [47]. The research points, therefore, to the need to investigate the difficulties experienced by "non-traditional" students when adapting to higher education, which, if not overcome, may be at the origin of low academic performance and higher dropout rates.

Finally, some differences in academic achievement and on dropout rates are associated with sex. Male students usually drop out more frequently and devote less time on academic activities and tend to be older [48]. For female students, age is not a determinant variable to explain dropout decision; more frequently female students exhibit more difficulties with social integration [12,38,49]. Sex-based differences between students' decision to drop out could be explained by that fact that female students are often more methodical, focused, and organized in planning their academic activities, more diligent in class and group work, more participatory in class, and seek more support from faculty and peers [17,50], obtaining better grades along academic trajectories and enhancing access to universities the degree of their preference. However, in STEM courses, the situation can be quite different and some authors present higher dropout levels in female students [51–53]. In this case, female students often experience a low sense of belonging, low self-efficacy, and weak STEM identity, largely because of the remaining stereotypes that such students do not meet the requirements of STEM courses [51,54]. For example, some research shows that sexist bias against women persists in teachers and students [55,56]. The dropout ratio by gender increase for students in scientific areas of graduation where their gender are underrepresented, such as in some engineering courses [57].

In sum, the present study aims to analyze the impact of personal and academic variables on students' academic difficulties in adaption to HE, academic achievement, and first-year dropout rate. It is also intended to analyze how this wide set of variables interact with each other (direct effects and mediated effects) in the students' decision to drop out during the first-year. Understanding the gradual dropout process by STEM students can contribute to the better identification of institutional actions to prevent and reduce its occurrence, especially in first-year students. For the reflective construct 'Adaptation difficulties', the moderating variables are illustrated in Figure 1.

**Figure 1.** Path diagram of gradual process of students' disengagement and dropout.

#### **2. Materials and Methods**

#### *2.1. Participants*

The study was carried out in a public university in the north of Portugal, in which 1376 students were enrolled in different STEM degree courses who had enrolled in higher education for the. According to the research by [58], for this type of analysis, they recommend having samples larger than 100 respondents. Therefore, our study fulfills this requirement. The majority of the students were male (59.9%), with a mean age of 18.51 years (SD = 2.52) with values ranging from 16 to 44 years. There are no significant differences in age according to gender (M*male* = 18.62, *SD* = 2.78; M*female* = 18.35, *SD* = 2.48). In the sample, 36.8% of students receive scholarship, reflecting social support by the government to students from less well-off households. Grade point average to access HE ranged from 10.00 to 19.64, with a mean value of 15.19 (*SD* = 1.83).

#### *2.2. Instruments*

This study considers part of the data from the research project on first-year dropout rate in HE students. The instruments used to collect the data/information contained in this study are presented.

Sociodemographic questionnaire. Students' information: participants' age (years), sex (0—female, 1—male); students' information on academic history and vocational options: grade point average of access to HE (from 10 to 20 points), degree in which the student enrolled (name of the degree).

Academic Services Report on academic situation and dropout. Academic situation on students' dropout rate or permanence (1 = permanence; 2 = dropped out), number of curricular units approved during students' first year (students take on 5 or 6 subjects per semester, totaling 60 ECTS credits per year), have benefited from a study grant (0 = no, 1 = yes).

Instrument to Explore Difficulties in Academic Higher Education Adaptation [59]. This assesses the difficulties anticipated by students regarding six situations of academic life: (i) adaptation to the institution (welcome, spaces and services, relationship with academic staff); (ii) learning (knowing how to study, participating in classes or completing assignments on time, academic results); (iii) interpersonal (making new friends, integrating into the class, participating in activities and social gatherings outside classes); (iv) economic (defraying daily expenses); (v) autonomy (living on your own, self-confidence, managing stress, taking on responsibilities on your own, missing your family); and (vi) vocational (not liking the course, finding out that the course is not what you expected, future job prospects). The answers are pointed on a 5-point Likert scale, from 1=no difficulties at all to 5=many difficulties. The present study will use all the difficulties except economic difficulties because the variable social grant was considered. The reliability of the scale scores was estimated by Cronbach's α coefficient (being 0.79).

#### *2.3. Procedure*

The study was conducted according to the ethical standards of research with human beings, following the guidelines of the Declaration of Helsinki and the Oviedo Convention, and, as part of a wider project, was presented and approved by the Ethics Committee of the HE institution where the participants first enrolled.

Initially, first-year students were invited to participate in the study at enrolment in the university. They were informed of the study objectives and gave their free, informed, written consent to match the data collected when they started their course (sociodemographic questionnaire). The confidentiality of the data was assured, and students were able to decline to participate or to drop out of the study at any time simply by communicating their wishes. In a second moment, at the beginning of the second academic year, data about academic achievement and the dropout rate from Academic Services (academic and dropout report) was collected and matched with the sociodemographic questionnaire. Only students for whom we had information about their HE entry and dropout status were considered in the study.

#### *2.4. Data Analysis*

Data were analyzed using the statistical program Stata v.14 [60,61]. The relationships hypothesized in Figure 1 are assessed through structural equation modeling (SEM), and mediation analysis is performed to estimate indirect, direct, and total effects of the potential mediators. Data were analyzed using the statistical program Stata v.14 [62–64]. The use of structural equations is derived from the variance–covariance matrix, so that a variable is measured with a series of observable measures that facilitate the analysis of the relationships between variables. This allows us to compare the model with other alternatives and to account for measurement errors [62]. In this case, we have used the Stata v.14 program to analyze the causal relationships between the variables.

#### **3. Results**

To carry out the analysis, internal consistency and discriminant validity were studied for the construct "Adaptation difficulties". In the case of internal consistency, composite reliability (CR) values should ideally be higher than 0.70 [62]. In this case, a value of 0.880 is obtained. Therefore, the results satisfy the internal consistency criteria. Likewise, convergent validity requires an average variance extracted (AVE) for each construct greater than 0.50 [62], which, in this case, has a value of 0.660 and therefore also satisfies this criterion. Afterwards, we calculate R-square for the coefficients (Dis1, Dif2, Dif3, Dif4, and Dif5) that explain the construct "Adaptation difficulties". In this way, the results were 0.52, 0.26, 0.54, 0.34, 0.20, and 0.25, respectively.

The goodness-of-fit indices were the χ2/df (ratio chi-square and degrees of freedom), comparative fit index (CFI), Tucker–Lewis index (TLI), root mean square error of approximation (RMSEA) and standardized root mean square residual (SRMR). The fit of the model was considered good for values of χ2/df < 5, values of CFI and TLI > 0.90, and values of SRMR and RMSEA < 0.08 [63–66].

#### *3.1. Difficulties on Adaptation to Higher Education*

Five situations (items) have been described to students in order to explore the level of difficulties experienced or anticipated by students in their first days at university. Table 1 presents the results on the 5-points Likert scale per item, including skewness and kurtosis of results distribution.


**Table 1.** Items results distribution (n = 1268).

Note. Min. = Minimum; Max. = Maximum; M = Media; SD = Standard Deviation; Skew. = Skewness; Kurt. = Kurtosis.

In all five situations, the minimum and maximum obtained cover the values of the 5-points Likert scale. More difficulties are being experienced and anticipated by students in learning and academic achievement, and inversely vocational difficulties tend to be lower. Skewness and kurtosis coefficients are above the unit suggesting a normal distribution of values. A standard deviation near unit in a 5-points scale can also be interpreted as an adequate variance coefficient.

Table 2 presents the goodness-of-fit indices of the global model that have been tested. The model includes a confirmatory factor analysis to identify a latent variable of students' adaptation based in five situations of potential difficulty and the relationships among variables to explain the students' dropout rate (path analysis).

**Table 2.** Model fit indexes.


All the goodness-of-fit indices assume values that are included in the intervals of suggested values. The 90% confidence interval for RMSEA is above 0.08.

In Figure 1, the direct and indirect effects of variables to explain dropout rate are present. Initially, results confirm the unidimensional structure of the five items of the questionnaire to assess students´ adaptation difficulties. All five items converge on the latent variable assumed in model, and the loadings values are higher than 0.50 (the vocational item presents the lower loading value which means that the career or professional questions are less important when students just start their graduation compared to other domains of potential difficulty).

#### *3.2. Direct and Indirect Effects on Dropout Rates*

The hypotheses proposed in this study were evaluated through the covariance based structural equation modeling (CB-SEM) approach [58]. For this approach, bootstrapping with 10,000 resamples was used to assess the significance of the path coefficients. Figure 2 represents standardized coefficients on the direct and indirect effects of variables in dropout explanation, the SEM results revealed a direct significant relationship between number of curricular units (academic achievement) and the variables included in the assessment model. In particular, there is a direct significant relationship between age and number of curricular units approved (β = −0.17, *p* < 0.001). Moreover, there is a direct significant relationship between having a scholarship and the number of curricular units approved in the first year (β = −0.12, *p* < 0.001). In addition, there is a direct significant relationship from GPA to access HE and the number of curricular units approved (β = 0.77, *p* < 0.001). However, the direct causal relationship between sex and number of curricular units approved is not significant (β = 0.2, *p* > 0.001).

**Figure 2.** Path diagram of students' disengagement and dropout (the dashed paths represent nonsignificant paths).

It is also observed that the input variables show certain causal relationships with the difficulties of adaptation to HE. In this sense, there is a direct significant causal relationship between age and adaptation difficulties (β = 0.039, *p* < 0.001), between sex and adaptation difficulties (β = 0.027, *p* < 0.001), and between GPA and adaptation difficulties (β = −0.022, *p* < 0.05). Nevertheless, the direct causal relationship between scholarship and adaptation difficulties is not significant (β = −0.045, *p* > 0.001).

Considering intermediate variables in model (adaptation difficulties and number of curricular units approved), it has been observed that there is a direct causal relationship between difficulties and number of subjects (β = 0.038, *p* > 0.001). Finally, the results show a direct causal relationship between the antecedents with the higher education dropout variable. In particular, there is a direct significant causal relationship among number of subjects and higher education dropout rate (β = −0.044, *p* > 0.001), between sex and higher education dropout rate (β = −0.045, *p* > 0.001), between scholarship and higher education dropout rate (β = −0.029, *p* < 0.001), and between GPA and dropout rate (β = −0.019, *p* < 0.001). However, the direct causal relationship between age and higher education dropout rate is not significant (β = 0.0048, *p* > 0.001), and the direct causal relationship between adaptation difficulties and dropout rate is not significant (β = −0.016, *p* > 0.001).

#### **4. Discussion and Conclusions**

The dropout of students in higher education appears to be a serious problem that negatively affects the expectations of students and their families, as well as goes against the goals of social and economic development of countries and societies, since such goals require highly qualified professionals [3,67]. In this sense, it is necessary to know the most relevant personal, curricular, and contextual variables in explaining dropout rates, particularly among first-year and STEM students, and to implement intervention programs to reduce their rate of occurrence [5,23].

Assuming a gradual students' disengagement and decisions prior to dropping out in STEM courses, the set of direct and indirect effects on intermediate variables (the adaptation difficulties and the number of subjects approved) and the outcome variable (dropping out), in the first instance, positive goodness-of-fit indices have been obtained comparing theoretical and empirical model tested. Considering difficulties anticipated or experienced, there was a direct significant causal association between age and gender with adaptative difficulties, with older and female students presenting a higher level of difficulty. One the other side, students with higher GPA to access HE present lower difficulties of adaptation to HE.

Considering the number of curriculum units approved, there was a direct significant impact of previous variables, like age, scholarship, and GPA to access HE. Older students, students without scholarship, and those with lower GPA are associated with lower numbers of curricular units approved in the first year. Previous research presented studies demonstrating academic achievement by older students, usually with many sociofamiliar responsibilities and less time for academic activities or lacking some basic curricular knowledge or study habits [12,46]. Moreover, students from socioeconomically and socioculturally more disadvantage groups, tend to present difficulties in their academic achievement and adaptation, a situation that can be mitigated for those who obtain a scholarship [41–43]. A scholarship can be an incentive to engage with academic activities and obtain a sufficient level of academic achievement, especially in systems (like in Portugal) when this later is a condition to maintain the scholarship. Finally, as expected from the research available, there is a direct significant relationship between GPA to access HE and the number of curricular units approved in the first year. GPA access to HE reflects a global set of competencies in terms of motivation, study habits, and previous academic achievement, being, in the literature, assumed to be the best individual predictor of first-year students' achievement on HE [27–29]. This aspect increases its relevance when we consider STEM graduation students because, during the first year, some subjects, such as mathematics, physics, and chemistry, are in continuity with the academic competency students have developed in secondary school [32,33]. Contrary to what has been found in other studies, no relationship has been found between gender and number of curricular units approved. Some authors mention that female students are more methodical or organized in planning their academic activities, being more active in classes and group works [12,17,23,50], what may reflect better academic achievement, but this was not found in this study. Some authors comment on the specific case of women in STEM courses, namely their low sense of

belonging, low self-efficacy, and weak STEM identity because of the remaining stereotypes against women [55–57]. Those negative experiences impact lower achievement and higher dropout rates by women in STEM courses [46,51,52].

To conclude, it is important to highlight a direct causal relationship between students´ level of difficulties in academic adaptation and the number of curricular units approved on first-year graduation. As expected, lower academic achievement is observed among students that experienced or anticipated more adaptation difficulties during the initial weeks on university [12,16,26]. Those difficulties are diverse; some of them are not directly related to academic achievement; in fact, students could be facing more immediate demands like the socioemotional needs of belonging—e.g., the need to identify people/peers that could provide social support—and could thus be less engaged or invested in learning process.

This study presents some limitations that can be mentioned in the interest of future developments. First, the sample was collected from a single university which reduces diversity in terms of the personal and contextual variables present on academic adaptation, academic achievement, and dropout causes. Secondly, dropout information was provided by Academic Services at the end of first year; it will be interesting to consider the students leaving university during the academic year and the specific characteristics they can present. This last aspect points to the relevance of considering dropout as a dynamic and continuous process in students' disengagement with course and institution; we recommend longitudinal data collection and analysis. Finally, taking first-year students from STEM, it is relevant to integrate into the research specific curricular aspects like academic background in math and sciences learning strategies students mobilize daily. HE institutions have the tradition of introducing in the first year propaedeutic subjects in those fields where several students present more learning and achievement difficulties. This highly established practice could be an important aspect to analyze and should be reflected on by institutions and course-directors; they could ensure that teachers who enforce the curricular units of STEM in the first year present nor merely scientific competence but also pedagogical and relational competences.

**Author Contributions:** Conceptualization, A.C.-L., A.B.B., J.R.C., L.S.A.; methodology, all authors; software, A.C.-L., J.R.C.; formal analysis, all authors; data curation, J.R.C., L.S.A.; writing—original draft preparation, J.R.C., A.C.-L.; writing—review and editing, all authors; funding acquisition, J.R.C., L.S.A. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was funded by CIEd–Research Centre on Education (SFRH/BD/117902/2016; projects UIDB/01661/2020 and UIDP/01661/2020) and CIEC–Research Centre on Child Studies (projects UIDB/00317/2020 and UIDP/00317/2020), Institute of Education, University of Minho, through national funds of FCT/MCTES-PT.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee of University of Minho (CEICSH035/2019).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

## *Article* **Teaching Marketing Research at the University Level—From Academic and Professional Perspectives**

**Javier de la Ballina \* and Silvia Cachero**

**\*** Correspondence: fballina@uniovi.es; Tel.: +34-985103918

**Abstract:** The evolution of marketing research (MR) has run parallel to marketing; however, nowadays, statistical techniques and data technologies are gaining more importance. The need for alignment regarding MR training between professors, students, and professionals is becoming increasingly urgent. This work continues on a double survey administered to professors and MR professionals in Spain to determine their proposals for adapting MR courses' format and content to the companies' current information needs. The results show that not only do professionals lead in terms of wanting changes to training, but also that these professionals are by no means extreme in their demands regarding university professors. The findings also show that, although there are significant differences in priorities at both the statistical and technical levels, the solution may be to combine and slightly adjust the current mandatory MR courses in business administration degrees. In addition, an elective course that develops training in new data and intelligent technologies for MR should be implemented.

**Keywords:** marketing research; higher education; evolution; university subject; professors; professionals; data; technology

#### **1. Introduction**

Companies are increasingly seeking information for decision-making in today's rapidly changing world. The importance of training in market research (MR) runs in parallel with companies' needs.

Current technological innovation has produced significant changes in the MR field, which is committed to digital, innovative, and transformative data processes or methodologies. These changes may affect the content, tools, and format of MR courses at the university level.

Firstly, statistical engineering has allowed for the development of software that can apply increasingly more complex techniques to both SQL (Structured Query Language) and NoSQL (non-relationally stored)-managed data [1].

New working technologies that are also are incorporated into this field in terms of both data collection and analysis, as well as data storage, include: big data, extract-treat-load (ETL), data warehouses [2], apps, and smartphones [3]. Moreover, it is essential to highlight the importance of neuromarketing, which has critical applications in research with virtual reality and artificial intelligence [4].

Furthermore, the abundance of microdata on all people has brought the critical question of ethics in MR and the management of social networks and bots to the table [5,6].

The traditional dichotomy between university training in MR and professional practice must adapt to this innovation context. The literature offers some studies incorporating new statistical or digital techniques. However, only some works provide a global vision of the convergence of both approaches [7].

This paper aims to provide empirical evidence for the reflections of Malhotra and Peterson [8]. They reflected on the influence of new smart technologies on MR research

**Citation:** de la Ballina, J.; Cachero, S. Teaching Marketing Research at the University Level—From Academic and Professional Perspectives. *Sustainability* **2023**, *15*, 1480. https://doi.org/10.3390/ su15021480

Academic Editors: Leandro Silva Almeida, Ana B. Bernardo, Adrian Castro-Lopez and Silvia Fernández Rivas

Received: 9 November 2022 Revised: 27 December 2022 Accepted: 5 January 2023 Published: 12 January 2023

**Copyright:** © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

School of Economics and Business, University of Oviedo, 33006 Oviedo, Spain

methodologies. However, they should have specified them and addressed the issue of necessary university training to use the new MR methodologies. Therefore, this work conducts a comparative study between university professors and professionals from large MR companies to determine the incorporation of new statistical and digital technologies into MR courses offered at the university level [6]. More specifically, this study also examines preexisting points in programs that may reduce or remove [9] the need to facilitate the adaptive change of MR.

This study has the following structure: first, it examines the literature related to the conceptual evolution of MR, emphasizing the new perspectives of "Intelligence" versus "Research". In addition, it reviews the literature related to teaching MR at the university level with a view to the perspectives of professors, students, and professionals. Second, it presents the methodology, the hypotheses for the empirical work, and the results of the statistical analysis of the data obtained through surveys administered to the professionals and professors. Finally, this article ends with the presentation of the study's conclusions and a discussion of their practical implications.

#### **2. Literature Review**

#### *2.1. About the Evolution of the Concept of Marketing Research*

The evolution of MR throughout history may be organized into a series of main stages. First, there is the initial stage, which consisted of the formal emergence of MR in 1910 with the creation of the first MR department in Curtis Publishing Company. A few years later, a university department was created in the Harvard Business School, thus beginning the training of students in MR [10].

The period between 1920 and 1940 represents an acceleration stage [11], with the term MR becoming more popular in the business and academic spheres with the relevance of sampling and questionnaire analysis techniques. In addition, telephone surveys were developed in this period, and the first panels appeared.

Between 1940 and 1960, the phenomenon of the 4Ps influenced the MR consolidation stage, which affected operations in MR and helped it become a fundamental activity for managers and decision-making. This decade also bore witness to the US government creating its first statistical offices that regularly gathered demographic, social, and economic data [10].

The 1960s and 1970s represent the beginning of the behavioral research stage [11]. This moment became necessary as the need to divide consumers into groups and segment the market arose in the marketing field. This stage developed consumer motivation studies using tools such as in-depth interviews and focus groups. This period promoted multivariate classification statistical techniques known as cluster analysis.

The office automation era began when the first computers and statistical packages were launched on the market [11]. Quantitative research began to take precedence. These technological advances allowed the use of attitude scales, factor and discriminant analysis, mathematical models, and simulation implementation. Researchers began first to introduce spreadsheets (first with Lotus, and later with Excel), thus generalizing the use of statistical software (from BMDP to IBM's SPSS) [12].

In the 2000s, how managed information was managed underwent a sudden and dramatic change with the development of the Internet [12], giving rise to the technological revolution stage. The widespread introduction of the Internet within the field of MR multiplied the options for accessing all types of data and carrying out market studies over time, which had previously been quite tricky.

This stage also saw the appearance of CRM (Customer Relationship Management), a database storing all interactions between a company and its customers [7]. This element led to sharing and maximizing the information available about a customer. All possible information about a given customer could be collected and accessible on any company device, allowing for new objectives regarding customer satisfaction and loyalty [5].

In the late 2010s, the rapid development of smart technologies began [13]. This change consisted of a new technological revolution that used smart intelligence as its basis to make both resources and media act and respond intelligently, learn from experiences, and be capable of solving previously unseen situations thanks to their data processing capabilities and interactions with the environment. Among these technologies, the following are noteworthy: big data, machine learning, blockchain, artificial intelligence, and, undoubtedly, nanotechnology [14].

The MR process changed with the development of digital transformation, which supported a transition toward new concepts such as business intelligence (BI), competitive intelligence (CI), and market intelligence (MI).

The development of smart technologies entailed perfecting and better-integrating data within organizations [7], which translates into better business information. Now appears business intelligence. BI integrates all the techniques based on computer or technological means to detect, deepen, and analyze data related to a company's activities to improve decision-making [10]. It is a decision support system that aims to enhance and optimize organizational performance and decision-making (see Table 1).

**Table 1.** Characteristics of business intelligence.


It is essential to note the differences between BI and CI. All information within the organization is the domain of BI, but information about current competitors is the domain of CI. Gibbons and Prescott [15] note that CI refers to the processing of obtaining, analyzing, interpreting, and disseminating information of strategic value about the industry and competitors, which is transmitted to decision-makers as soon as possible. CI allows organizations to make decisions to get ahead of their competitors [12].

BI has a broader perspective than CI, as it focuses on events beyond current competitors and the primary industry and attempts to predict and anticipate significant changes that may arise, both at the specific sector and overall market levels.

The second relevant term is MI. It seeks to collect all types of data that provide the company with a general and global vision of the market, not only of the current markets, such as in the case of CI.

The Academy of Market Intelligence (AMI) established [16] that MI seeks to obtain relevant results about a company's market through internal and competitive analyses supported with information. From the professional perspective, QuestiónPro [17] considers MI to be the information or data that an organization obtains from the industry in which it operates to determine its existing segmentation, penetration, opportunities, and metrics. MI is a strategy that uses internal and external data to supervise the market continually, rather than ad hoc, to improve competitiveness.

#### *2.2. Teaching Marketing Research at the University Level*

MR has always gone together with marketing, meaning that their implementation at the university level is connected. In the US, the presence of marketing at the university level dates to the beginning of the 20th century, supported by commercial investigation, which strongly focuses on sales. In Europe, and more specifically in Spain, the delay was considerable, with the discipline arriving several decades later [12].

There are five fundamental periods in the evolution of MR at the university level in the case of Spain [18,19]:


Bridges [21] was the first academic to take an interest in the need for change and adaptation in the teaching of MR. The fieldwork Bridges conducted with students began by detailing the students' difficulty with MR, despite them also considering a relevant course for their training. Bridges also found that interactive activities and group work caught students' attention more than conferences or independent work.

The author also reviewed the leading textbooks on MR and concluded that, despite the advances and changes in IT, the content primarily covered had stayed the same: use of secondary data, questionnaire design, sampling, data analysis, and presentation of results.

There is a debate between those who recommend dedicating more time to developing quantitative skills and those who choose to focus on qualitative skills. Those that make up the first group believe that there should be more focus on working with data and that data analysis is better to help managers to understand business problems [24]. However, those who belong to the other school of thought believe that numerical databases cannot explain consumer behavior, resulting in the increased importance of qualitative research [25].

Bridges [23] developed a later study based on in-depth interviews with MR professors from universities in North America. Bridges' results show that professors try to incorporate quantitative and qualitative research into their syllabi. However, the increase in statistical analyses, which is a consequence of the development of IT, forces them to choose which aspects are critical. However, with the same number of hours for the subject, the time allotted to cover the content decreased.

In said work, Bridges found five critical points in the design of the MR university course, which corresponded to the stages of a market study: determining the purpose of the study, establishing the focus (qualitative or quantitative), designing the study, collecting and analyzing data, and, finally, presenting the results.

The author also brought a new debate: the interest in working with secondary data. Some professors focus on available information, which students must locate and analyze, whereas others feel it is more critical to cover primary research in MR courses.

Finally, in Bridges' study, professors agreed on the reluctance of MR students to approach statistics. Despite the general use of statistical software (SPSS) in universities, professors were concerned because their students needed help understanding the necessary processes and, consequently, could not explain their results.

Nonis and Hudson [24] noted that MR students found it challenging to understand the contents of the course, given that it implied previous knowledge that, on occasion, must be added. It is an additional problem for professors because they must invest more time in covering the content of earlier courses. It leads to lower overall grades, a lack of satisfaction with the period, and changes in university majors. Bridges [21] studied a sample of 488 marketing course evaluations and concluded that the 48 MR courses were significantly worse than the rest.

Other authors have studied the professional perspective, that is, the relationship between teaching MR at the university level and the requirements needed to obtain a position as a market researcher.

For Kover [25], one origin of the problem was in academics' tendency to develop their MR syllabi to reflect their own needs and the skills they considered to be the most appropriate for the training. However, these choices may differ from companies' demands for a specialized job in the field.

Bellenger and Bernhardt [26] found that businesspeople considered it necessary to deepen quantitative analysis further and provide the course with a more practical perspective.

Bonoma [9] noted that one of the leading causes of conflict was the lack of interaction between professors and businesspeople in the sector, which resulted in academics not sharing their theories with professionals; consequently, these professionals did not understand them. There was also scarcely any communication in the opposite direction that pointed toward the industry's demands. It makes it even more essential to consider MR an eminently practical discipline.

In recent years, several studies [27] have shown similar findings and exemplified the gap between the opinions of both groups regarding the content, structure, and skills taught in the course. The study by Stern and Tseng [28] is worth noting. The purpose of the study was to further understand the perspectives of both groups on how to teach MR. To this end, the authors surveyed CEOs and professors in the US and Canada. Stern and Tseng found several interesting results:


Segal and Hershberger [29] used a sample of 610 MR-related job offers to assess the skills, experience, and knowledge levels required by the industry. The authors obtained exciting conclusions, such as the following:


Through a similar methodology of examining job offers, Benítez [30] studied the profile of the personal and professional skills required by MR companies. The study's results were similar to those obtained by Segal and Hershberger [29]. However, there were some quantitative differences: professional experience and university degree were at the top, with both having an equal weight (7.04%) and Excel skills being highlighted (4.23%).

Given the differences between professionals and academics in the MR field, both groups should establish a regular dialogue regarding the needs and depth needed for each technique, course topic, and method taught. As noted by Segal and Hershberger [29], it is more necessary in the MR field than in any other field to close the gap between academics and professionals, and for students to ultimately acquire the knowledge and skills needed for professional purposes during their time at university.

Finally, López [31], in a much more recent study, studied the opinions of students regarding their MR courses. The students valued all items positively (with a score above three on a Likert scale of one to five). However, teamwork was remarkably well-evaluated, as were the ability to apply the theories studied and the analysis of data and results.

Because of the problems found, some authors have proposed alternatives to the current design of the MR courses that are oriented towards trying to solve student weaknesses.

Nonis and Hudson [24] developed a proposal to improve the training given before starting the MR course. Specifically, they suggested creating a statistical basis by substituting the only statistical course available in BA degrees in the US (as it mainly occurs in Europe) with two other courses to be taught in the first two academic years. The first course would teach the fundamentals (Business Statistics), and the second one would focus on practical aspects (Applied Research Course) that include training in statistical software (SPSS).

Burns and Bush [32] proposed teaching the course using a staged approach. The objective would be to cover as many MR concepts as possible, but only for those students who had successfully passed a series of exams.

Kennett et al. [33] proposed an alternative method for evaluating students through a kind of MR portfolio. As students advanced through the syllabus, they would carry out projects related to each course theme. Moreover, this option would appeal to students who value experience and learning more than the traditional approaches.

In a much more recent study, Méndez and Ballina [34] analyzed the MR course programs in BA degrees in 76 Spanish universities, obtaining the following results:


Finally, it should be noted that since 2018, Spanish universities have introduced the first official master's degree programs in data with the names "Big Data", "Data Analytics", and "Business Analytics". In addition, since the academic year 2020–2021, data engineering degrees have been developed in Spanish Polytechnic Universities. It resulted in a quantitative advancement in terms of MR development. Moreover, it was also a qualitative advancement given that, for the first time, university training in business information was moving from business and enterprise faculties to the schools of engineering.

#### **3. Empirical Study—Academics vs. Professionals**

The overall objective of this study was to evaluate the significant changes that have taken place and continue to take place in terms of the availability and types of data. In addition, developing new analytical techniques should encourage changes in the structure and content of the MR courses taught in Spanish universities. Different phases or subobjectives have this goal (Figure 1):


**Figure 1.** Outline of the objective and subobjectives of the work.

Some of these issues are closely connected. For example, the introduction of changes to the content of an MR program's subjects will have to produce transformations in the methods of analysis required for such content. In addition, the perception of the need for changes by the subjects studied will relate to the new structure necessary for the "new" MR subject.

To this end, and following the previous literature, a structured field study was designed in two interrelated parts: a survey completed by MR professionals from different universities across the world, and another survey conducted by professionals from the primary MR and data companies in Spain, many of which are multinational (Table 2).

**Table 2.** Technical sheet of the study.


The design of the questionnaire considered the work of Bridges [21,23] and Méndez and Ballina [3] carried out with MR university professors. Logically, the study omitted the items related to methodology, teaching, and evaluation of the subject because they were not convenient for the group of professionals. We worked with ten questions: three general questions, two questions about parts that are "unnecessary" in the program, two questions about the need to "add" ICT data, two questions about the relevance of multivariate statistical techniques, and an open question for additional input (Table 3).


#### **Table 3.** Questionary Structure.

Lm: Likert/metric scale.

In accordance with the previous literature, four main hypotheses were developed (Figure 2):

**Figure 2.** Hypothesis Scheme.

**H1.** *The need to change current MR course programs is positive for professionals and negative for academics*.

The classic works of Kover (1976) and Bellenger and Bernhardt (1977) support H1. They note that professors design MR programs according to their needs and preferences and even avoid interaction with MR companies (Bonoma, 1988).

#### **H2.** *Professionals, not academics, support removing concepts in current MR course programs*.

Univariate analysis of statistics develops H2A. Thus, Nonis and Hudson (1999), Burns and Bush (2010), and Kennett et al. (2010) advocate for the need to remove reiterative studies in BA degree courses.

Two different approaches have H2B: one direction of authors who support more qualitative research (Evans, 2002; Paas, 2019) versus another who call for the further development of multivariate analytics (Benítez, 2016; Bridges, 2020; Segal and Hershberger, 2006 and Stern and Tseng, 2002).

**H3.** *Professionals rather than academics propose the change toward the term "intelligence" instead of "research"*.

**H4.** *The importance of the MR course, as measured in the number of credits, is increasing for professionals and remains the same for academics*.

Finally, H3 and H4 run parallel to each other. Early contributions by Malhotra and Peterson (2001), and, more recently, Ghorbani et al. (2022), are aligned, as they suggest adapting the term used to describe the field to the new trends in the professional sector, thus promoting the expansion of the content and, therefore, the weight of the MR university course.

#### **4. Results**

The data were processed using various comparative statistics. IBM SPSS v.27 was the software used.

First, the results presented in Table 4 are the final distribution of the sample regarding the professionals and academics and the distribution of the sample by major countries.

**Table 4.** Composition of the final sample.



**Table 4.** *Cont.*

The reliability of the Likert scale used for the 45 variables of the questionnaire (see Table 3) shows satisfactory Cronbach's alpha results above 0.0 (Table 5):

**Table 5.** Scale Validation.


First, regarding the H1 hypothesis, the analysis evaluates the mean differences in the value professionals and academics each give to the demand for change in MR programs. The *t*-test for independent samples indicates that there is a slightly higher, but significant, mean for the case of the professionals. The analysis considered three issues regarding changes to the program: removal, adding multivariate statistics, and adding ICT. The results of the *t*-tests showed significant differences between the three cases. However, whereas the mean for "adding aspects" (ICTs and multivariate statistics) was higher in the professional group, according to H1, this is not the case with the question of "removing aspects," where the academics presented a higher mean (Table 6).

**Table 6.** Mean difference *t*-test for independent samples.


To address H2, a deeper study into the different options to "remove aspects" from the program is needed. A simple discriminant analysis (SDA) was carried out in which the dependent variable was academics vs. professionals. Nine of the 16 options studied were significant (stepwise statistics and Wilks' lambda), which, by relating their coefficients to the centroids of each type of individual, indicates that (Table 7):

• Academics (centroid +) favored removing (in this order): qualitative techniques, specific applications of MR, univariate statistics, report writing, and experimentation. • Professionals (centroid −) favored removing (in this order): sampling, panels, databases, and basic concepts.

**Table 7.** SDA over "remove" from the MR program.


After applying the SDA technique for the case of "adding" ICT techniques to the course program, using the same statistical process, seven significant options out of the 13 studied were identified (Table 8):


**Table 8.** SDA over "adding ICT" to the MR program.


Other ICTs proposed in an open question by the professionals were dynamic dashboards, audio matching, and bots (Table 9).

**Table 9.** Other ICT techniques to add to the program.


The third SDA applies to the case of multivariate techniques. In addition, it used stepwise statistics and Wilks' lambda protocol. There were ten effective techniques out of the 13 considered, indicating that (Table 10):



**Table 10.** SDA over "adding multivariate statistics" to the MR program.

Moreover, professionals proposed to incorporate another technique that mainly highlighted the case of sector-specific techniques and decision trees (Table 11).

**Table 11.** Other statistical skills to add to the MR program.


Concerning H3, the proposed future name for the MR course, another *t*-test studied the differences in means, which presented the following results (Table 12):


Concerning the term "business intelligence", there were no significant differences, although the values were lower than in the rest of the cases.

Finally, analysts performed a chi-square test in terms of H4 and regarding three outputs: the weight of the course, the format, and the number of credits. The test concluded that there were significant differences between the two types of sampling units (Table 13):



**Table 12.** Mean difference *t*-test for independent samples.

**Table 13.** Differences (Oi-Ei) chi-square test.


In summary, the hypotheses to accept or reject are:


#### **5. Discussion**

This study shows the necessity to provide students taking MR courses with more previous knowledge, which professors see as an opportunity to take more time to develop their academic programs.

However, professors suggest a further reduction of the current content. The debate regarding training in qualitative research is still open, and the ability to communicate with reports divides both interest groups.

Both professors and professionals recognize the introduction of complex statistical methodologies and new smart techniques, although the professionals are significantly more noteworthy in this case. The case of big data is critical, perhaps due to the effect that it has on the development of more master's and bachelor's degrees related to it that are unrelated to MR training.

In any case, the results do not suggest mutually exclusive positions; in fact, it is quite the contrary. One reason is that professionals continue to advocate for the term "market", and professors opt for "research". However, above all, this is because, whereas professors support a significant restructuring in the contents of mandatory MR courses, professionals propose an increase of content, which would allow training in new techniques and concepts through an additional elective course. As such, there is a proposal to work on aligning both perspectives in universities.

#### **6. Conclusions**

MR has experienced an evolutionary process parallel to marketing at the academic and company levels. However, we are now at a point of enormous expansion in all aspects related to data management and even the use of MR to get closer to the general functions of business management.

This is due to the increased complexity and the significant momentum of change in today's markets. Moreover, it also can be explained by the constant search for precision in MR methodologies and techniques. Thanks to software development, the advances in the use of complex statistical techniques have combined with the result of qualitative research, neural network techniques, and the possibility of working with NoSQL data.

Today, smart technologies are driving a new revolution, mainly due to big data, artificial intelligence, and virtual reality developments. Universities are experiencing an increasing divide between teaching MR and developing new master's and bachelor's degrees in data engineering.

Several authors have noted the gap between the MR programs taught at the university level and the knowledge and skills that professionals and market research institutions need today. It is due to the significant difference in adaptation times with which universities, especially public universities, operate. The age of university professors, the search for comfort zones in their teaching practice, as well as the difficulties in recruiting technicians specialized in new technologies provide support and act as factors preventing a transition in the short term.

Three agents participate in MR training: professors, professionals, and students. Each agent has their objectives, which are only sometimes in line with the teaching–learning training that should characterize the teaching of MR courses in universities.

According to the referenced works, students show an increased interest in MR course content and related skills. However, they are concerned about difficulties arising from increased statistical knowledge demands.

It is necessary to have a balanced model built based on the needs of professionals. The distances could be wider than they are, though. There is enough consensus among the agents in maintaining a main structure of the MR subject very similar to the current one, both in fundamentals and concepts, with the addition of a new complementary subject being the way to approach the desired balance.

#### *6.1. Academic and Business Implications*

The first and most important aspect of this work is aligning the knowledge interests of professionals and companies with university-level training. The professional sector demands more complex statistical techniques and new technological tools at the university level. Moreover, the right path is not to create master's or bachelor's degrees in data engineering, which is distant from what MR should be. This approach is becoming increasingly popular in university training, which may result in data technicians needing knowledge that is useful or applicable to companies or management technicians who need help handling new data.

#### *6.2. Limitations and Future Research*

The main limitation of this work has been the restriction of working with surveys. In addition, it has been subject to the existence of a great diversity of MR programs in each of the universities considered. Academics must learn more about the new technologies and their MR possibilities. This issue affects the results obtained. Something similar occurs in the case of qualitative MR, which, although better known, has less use in practice in universities. It may have led to some bias in the results.

Meeting the challenges for a future research agenda is essential. As noted in the literature, the gap between practitioners' positions and academics remains and requires further research. The first tasks should relate to broadening the pool of stakeholders, first with students and second with new centers and university professors, linked to big data or data engineering degrees. From the perspective of the MR subject programs, it would be relevant to consider the existence of a hierarchy in smart technologies in their professional application, as well as in the confluence of qualitative and quantitative data of

the NoSQL type, which is hardly developed at the university level. Finally, there needs to be an essential fill in the knowledge gap in the study of the complementary tools that students should work on within the MR profession, whether related to the management of logic, statistics, or computer programming.

**Author Contributions:** Conceptualization, J.d.l.B.; Methodology, J.d.l.B.; Software, S.C.; Validation, J.d.l.B.; Formal analysis, J.d.l.B. and S.C.; Data curation, S.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Data Availability Statement:** https://drive.google.com/file/d/1J70pCzkJGvCG7k6O7xDjwhjD4 WWIsyBF/view?usp=share\_link (accessed on 1 November 2022).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

## *Article* **The Role of Critical Thinking in Predicting and Improving Academic Performance**

**Silvia F. Rivas 1,\*, Carlos Saiz <sup>1</sup> and Leandro S. Almeida <sup>2</sup>**


**\*** Correspondence: silviaferivas@usal.es

**Abstract:** Today's world demands new ways of dealing with problems and different ways of preparing for them. Some studies argue that these new demands also require new skills. Critical thinking (CT) involves a set of skills that are entirely relevant to today's adaptive needs. In this study, we explore the extent to which CT serves to both account for and improve academic performance. To do this, we measured the CT skills of a number of undergraduate students, along with their university admission grades and average course grades. We successfully established the structural validity of the general construct of CT, along with a strong relationship between CT measurements at two different timepoints (at the beginning and end of studying a critical thinking subject area) and the admission grade and average grade variables. These data lead us to reflect on using CT level as a suitable assessment of academic performance. We also consider the limits of our findings and their implications.

**Keywords:** higher education; critical thinking; academic performance; assessment

#### **1. Introduction**

The world has probably seen more changes in recent decades than at any time since the industrial revolution. These changes demand substantial transformation in society in general, and particularly in our educational institutions. This transformation is characterised, above all, by increasingly complex problems, caused, in large part, by changes brought about by information and communication technologies (ICT) and economic competitiveness in a globalised world. All this means that today's issues are a mix of the social, the professional and the personal. There used to be a clear line separating these types of issues and situations, but today, it is very much blurred.

These changes mean that people today must increasingly develop their abilities to make sound decisions and solve problems effectively. At the same time, today's complex real-world problems are not solved by following a model of intelligence based primarily on classic IQ or logical reasoning [1–4]. Socio-emotional dimensions of intelligence, along with competences in creative and critical thinking, are fundamental today in the cognitive exercises of problem-solving and problem-finding [5–9].

Tackling the problems of today's world requires us to employ more critical thinking (CT) skills, as they seem to be specifically designed for these problems, given that they involve skills and strategies that are more generalisable or transferable than others [10]. CT skills being mainly horizontal and contextualised competences means that we can use them equally effectively in many situations. The flexibility and adaptability of CT to different situations and contexts makes it an excellent candidate for handling today's changes and new demands. In this sense, education and training at all levels need to adapt much more quickly to these changes. In particular, university education is called upon to change further in order to train young people in their transition to adulthood [11]. Universities need to transform themselves more than other institutions in society if they are to progress, lead the necessary change and produce professionally trained, mature, socially responsible adults.

**Citation:** Rivas, S.F.; Saiz, C.; Almeida, L.S. The Role of Critical Thinking in Predicting and Improving Academic Performance. *Sustainability* **2023**, *15*, 1527. https:// doi.org/10.3390/su15021527

Academic Editor: Antonio P. Gutierrez de Blume

Received: 7 November 2022 Revised: 10 January 2023 Accepted: 10 January 2023 Published: 12 January 2023

**Copyright:** © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

However, there seems to be a worrying disconnect between academia and the real world. Companies are increasingly demanding more transversal or horizontal competences. For example, they need not only qualified biologists or engineers, but also professionals who, from their specialist area (vertical competences), solve problems in different work contexts, make decisions individually and collectively and communicate their results in an argumentatively precise manner [12]. These general skills in solving decontextualised problems, making good decisions and arguing persuasively are not the focus of today's higher education. As we noted above, developing these competences requires a different conception of our cognitive machinery and the skills that are really fundamental. Additionally, and no less important, it requires an awareness of the deficiencies and limitations of that machinery.

Higher education today suffers from three significant ills [12]. As implied earlier, the first of these stems from the fact that the real world has changed faster than universities have. The second relates to the fact that the predominant model is still the administration of accumulated knowledge and not the management of learning, in terms of the process of acquiring knowledge and developing competences. The third is that training is still mainly vertical or disciplinary and rarely horizontal or cross-disciplinary. We have already noted that in the real world—the world of work—companies and some institutions require qualified people who can perform in very different contexts and solve problems equally well in any of them. Our students lack sufficient preparation for working with real problems in different contexts—in other words, the widespread application of knowledge and its inter-domain practice is rather limited, at least in comparison to what it needs to be. With these three dominant characteristics, graduates are unlikely to be qualified to perform and solve problems in complex, new, changing contexts.

New times and new problems require new tools and new strategies. Today's world has become so complicated that the need for lifelong learning is now a given. One real and challenging change is therefore the ongoing preparation of professionals and citizens. The consequence of this is the issue of institutions offering courses throughout people's lives. This brings us to a reflection that has yet to be properly addressed: being aware of the differences between teaching, learning and training helps us to better address this issue. If it is not possible to offer education for all ages, yet lifelong learning is needed, how can this be achieved? In our view, the answer is primarily training, and not just apprenticeships. Training implies student autonomy and almost complete independence. This means that they know where to go for documentation; know how to apply reliable criteria when evaluating what they find; and are able to reach their own conclusions and explanations about the problems or questions that interest them. This reflection therefore relates to the question of where training can or should be given today.

However, it is important to note that some strategies are particularly effective in lifelong or ongoing training. People must learn how to learn, and learn to develop behavioural, cognitive and motivational strategies that regulate their ongoing acquisition of knowledge and skills. At the same time, with a greater emphasis on training than on learning, people need to be involved in processes that encourage their autonomy, initiative and responsibility at personal, professional and social levels. Three UNESCO reports, spaced roughly two decades apart [13–15], show attempts to adapt to the changes that have taken place at each point in time.

The latest UNESCO report "retires the Delors report and redraws a new horizon for education" [16]. According to Sobhi Tawil, a leading member of the commission that drafted the report: "In the current global debate on the future of education, there are two parallel currents. The first concerns the role of education in the post-2015 international development agenda. The second relates to the way in which the transformation of society globally impacts on our approach to education and learning..." [17]. The second current is that which today requires training in terms of learning to understand, to do, to be, and to be with others.

If changes are made in educational methodologies by giving greater prominence to learning processes and focusing on greater application and cross-domain practice, then higher education can be made to take on the development of critical thinking as its main objective [12]. Indeed, today's changes and demands are greater than in past decades, and how we deal with them needs to be through competences and strategies that can be employed to handle this complexity.

CT offers what is needed in terms of how to progress, train and generalise. Being able to change faster entails developing collaborative communities of enquiry that participate in solving real projects together. In training, it is important to consider different methods that facilitate the acquisition processes, such as working on tasks involving not only comprehension but also production, aimed above all at acquiring knowledge based on explanation, and applying this knowledge in different contexts. In order to be able to generalise in different situations or contexts, it is essential to develop CT's fundamental competences—such as explanation (search for causality), decision-making and problemsolving. These skills are domain-independent, meaning they are required skills that need to be used throughout any situation or context [12].

Before specifying the research problem and the hypotheses, it is worth remembering the conceptual approach published in different works, some of which have been cited above. We understand that "to think critically is to reach the best explanation for a fact, phenomenon or problem, in order to gain insight and solve it effectively" ([18], p. 27). Knowing or solving a problem requires knowing what causes are responsible for certain events or problems. If the explanation is sound, we can choose the best course of action or the best option to resolve the situation effectively by bringing about the desired change. Therefore, it is the explanation that determines the decision and the solution and, finally, the change and well-being or achievement (a full description of our approach can be found at [19]). If improving critical thinking is our goal, we believe that this is the best way to achieve it. Therefore, our intervention aims to achieve change through the development of the aforementioned skills. This is the intervention framework for the development of such skills (for more information about the intervention, see [18,20–22]).

To be able to ascertain whether CT skills are being properly developed or can effectively predict performance in higher education, we need reliable, valid assessment tools. Fortunately, this assessment is available. As an example, we used a CT test— PENCRISAL [23,24]—to assess a number of first-term psychology undergraduate students, during the academic years 2011–2012 to 2015–2016, as part of the CT subject area, which forms part of a subject. PENCRISAL assesses five dimensions of critical thinking: deductive reasoning (DR), inductive reasoning (IR), practical reasoning (PR) or argumentation, decision making (DM) and problem solving (PS). We also obtained students' average academic grades and average university entrance grades. In addition, we considered some of the main dimensions of achievement motivation, using the Manassero Achievement Motivation scale [25]—although these were not used in this study beyond the procedures for monitoring course performance. With this information, we organize this article as part of our study of PENCRISAL's structural and criterion validity as a tool for assessing critical thinking in higher education students.

As we described at the beginning, the complex problems of today's world cannot be solved with a classical model of intelligence; these changes force citizens of this century to increasingly develop their ability to make solid decisions and to solve problems effectively. These competencies are precisely part of the fundamental CT competencies, widely studied in [18,19,26–28]. However, there are still a lack of studies that prove that the CT model is better than the classic model of intelligence in solving the problems of today's world [27]. This study aims to fill this gap. Therefore, one of the objectives of our work is to demonstrate that CT predicts academic performance well, something that has not yet been proven. The other objective of our study is to demonstrate the structural validity of our CT assessment test. We must be sure that our measurement instrument measures what it says it measures and that, in addition, it predicts the performance of our university students. For this, we

used a sample of more than 600 students, in order to test the construct validity. Furthermore, we used a sample of more than 200 students who underwent a CT instructional program, in order to test the predictive validity.

Thus, the purpose of our study is to test the structural and criterion validity of our CT assessment test and, therefore, to show the degree to which CT can account for academic performance. It is not our intention here to deal with conceptual developments or intervention procedures. However, we have cited our own works and those of relevant authors so that the interested reader can delve into these topics there. Let us now describe the methodology used in our study.

#### **2. Method**

#### *2.1. Participants*

The sample for this study was composed of first-year undergraduate students at the University of Salamanca studying Psychology of Thought (a CT subject area that explains what CT is). These students were assessed at the beginning and the end of this course, which focuses on developing critical thinking skills. Data were collected for 5 years, from the academic years 2011–2012 to 2015–2016, within the CT subject area.

The sample consisted of 682 students, most of whom were women (60.2%), aged 18 to 35 (*M* = 19.02, *SD* = 2.17). To examine the structural validity of the PENCRISAL test, a sub-sample of 242 students with academic performance information was taken to assess criterion validity. The majority of this sub-sample were also women (51.2%), with ages ranging from 18 to 28 (*M* = 18.67, *SD* = 1.38).

#### *2.2. Instruments*

Critical Thinking Test. To measure critical thinking skills, we applied the PENCRISAL test [23,24]. This consists of 35 situations relating to production problems with an openended response format. It has five factors: deductive reasoning, inductive reasoning, practical reasoning, decision making and problem solving; each factor has 7 items. There is currently a Peruvian adaptation [29] and a Brazilian-Portuguese adaptation [30].

The items in each of the factors reflect the most representative structures of fundamental critical thinking skills; these are briefly described below. The items that make up the deductive reasoning factor assess the most important forms of reasoning: propositional reasoning (four items) and categorical reasoning (three items). Formal reasoning is less frequent than practical and inductive reasoning, but is used to some extent. The inductive reasoning factor includes: (1) causal reasoning (three items); (2) analogical reasoning (two items); (3) hypothetical reasoning (one item); and (4) inductive generalisations (one item). The decision-making dimension assesses the use of general decision procedures, which requires making accurate probability judgements and using appropriate heuristics to make sound decisions. Two general situations are included here, whereby we need to proceed in a certain way in order to reach the best decision. In the other five situations, we need to identify the main heuristics and the biases they produce. Lastly, as with the decisionmaking items, the problem-solving items are divided into *general* problems (four items) and *specific* problems (three items); these will require specific solution strategies to be employed. Both the decision-making and problem-solving factors promote the performance of general decision and solution processes, with the aim of stimulating the use of the strategies necessary for planning around a problem. Meta-knowledge and the awareness of thought processes are where action is planned, directed, organised and regulated.

The format of the items is open-ended, meaning that participants must respond to a specific question by adding a justification for their answer(s). For this reason, standardised grading criteria are used to assign values ranging between 0 and 2, depending on the quality of the response. The test provides a total score for critical thinking skills and five other scores for the five factors. The range of values is 0–70 as the maximum total score for the test, with 0–14 points allocated for each of the five dimensions. Reliability measures demonstrate adequate levels of accuracy according to the scoring procedures, with the lowest Cronbach Alpha values at 0.632 and a test–retest correlation of 0.786 [23]. The PENCRISAL test was administered as a computerised version, via the internet, using the evaluation platform: SelectSurvey.NET V5: (https://www.classapps.com/product\_ssv5 .aspx (accessed on 1 May 2016)). PENCRISAL is described in more detail in [24].

*Academic performance.* For the analysis of criterion validity, some of the participating students' academic records were collected—specifically, their *average academic score* and *university admission score*. The latter was obtained by adding together the average baccalaureate score (weighted at 60%) and average university entrance exam grade (40%). This gives the admission grade, which needs to be at least 5 points for students to be accepted into university, and can be up to 10 points.

#### *2.3. Procedure*

The study was conducted over five academic years—from 2011–2012 to 2015–2016 with first-year psychology undergraduates at the University of Salamanca. The CT instruction programme was applied during the second term as part of the Psychology of Thought course (CT subject area). To obtain a baseline of students' performance in critical thinking, we administered the PENCRISAL test one week before the start of the course (pre-treatment measurements: PENCRISAL\_M1). The CT programme then ran for four months. A week after the end of the course, a second measurement was taken (post-treatment measurement: PENCRISAL\_M2) of critical thinking skills, using the same test. Students were informed of the aims of the study and gave their informed consent for their data to be used for research purposes and to be analysed anonymously as part of this study. The information on students' university entrance grades and their average grade at the end of the degree was provided by the faculty's administrative services team.

#### **3. Results**

Table 1 shows the distribution of students' scores from the five PENCRISAL subtests by adding together the marks for their respective items. Along with the minimum and maximum values, we show the mean and standard deviation, skewness distribution coefficients and kurtosis. These values take into account the two timepoints when PENCRISAL was applied, i.e., at the beginning (pre) and at the end (post) of the course.


**Table 1.** Distribution of the students' scores in the five subtests at the two evaluation timepoints.

(Pencri: pencrisal test; DR: deductive reasoning; IR: inductive reasoning; PS: problem solving; PR: practical reasoning or argumentation; DM: decision making).

As Table 1 shows, the students' scores for all five critical thinking subtests were low, with scores of zero or just one point. The post-course scores were higher than the precourse scores, meaning that at the end of the course, some students had improved their performance in the subtests. The only exception was the problem-solving (PS) subtest, as

the average at both timepoints was very similar, and even slightly higher in the pre-course measurement (before starting the subject). In addition, the skewness and kurtosis indices for the distribution of the results in the sample fit a Gaussian distribution, as they were always less than unity. Lastly, the standard deviation of the results for the five subtests rose in the post-course measurements. This means there was more heterogeneity in the students' scores at the end of the course, which may be because some students benefitted from the critical thinking lessons and others did not. This did not occur in the PS subtest, as the variance values were similar in pre-course and post-course measurements.

To analyse the changes in students' scores in each subtest, pre- and post-course, we calculated the mean differences using the t-test for dependent samples (paired-samples t-test). To appreciate the magnitude of the difference between two moments, Cohen's d was estimated. Most of the differences are statistically significant: PencriDR (t = −15.209, df = 681, *p* < 0.001, d = −0.582); PencriIR (t = −12.067, df = 681, *p* < 0.001, d = −0.483); PencriPR (t = −18.237, df = 681, *p* < 0.001, d = −0.698); and PencriDM (t = −15.847, df = 681, *p* < 0.001, d = −0.607). In all four of these subtests, Cohen´d presents near 0.50 or a medium effect size [31]. The *p* and d values indicate clearer improvement in the scores in four PENCRISAL subtests. In the problem-solving subtest, there was a slightly higher mean in the pre-course test, but it was not statistically significant and a very small effect size was observed (t = 1.326, df = 681, *p* = 0.093, d = 0.051).

Bearing in mind one of the central objectives of our study, Table 2 shows the results of PENCRISAL's structural validity. Taking the test authors' theoretical model, the indices of fit for the unidimensional model using the AMOS programme are presented [32,33]. This is because each of the five tests assesses a different dimension or cognitive process relevant to critical thinking, all adding up to one overall score. In other words, the PENCRISAL test proposes a measure of general critical thinking ability by taking the diversity of cognitive aspects presented in the five subtests: deductive reasoning, inductive reasoning, practical reasoning, decision making and problem solving. As pre-course and post-course data were available, Table 2 shows the indices of fit for these two test timepoints (PENCRISAL\_M1 and PENCRISAL\_M2), indicating the confidence intervals (CI) for the RMSEA coefficient.


**Table 2.** Fit indices for the one-dimensional model of PENCRISAL.

There were good indices of fit for both applications of PENCRISAL, which were slightly higher at the end of the course (timepoint 2). In both cases, CMIN/DF was below 5.0, the TLI and CFI indices were above or very close to 0.95, and RMSEA was below 0.08 (in the pre-course measurements, the RMSEA 90% confidence interval increased to 0.09). These indices are within the required parameters [34]. One potential improvement for the PENCRISAL model at the beginning of the course is indicated by the RMSEA upper confidence interval being below 0.08 if we correlate the errors for the deductive reasoning and inductive reasoning subtests. We will return to this in the discussion.

These indices highlight the fact that the five subtests combine to give one overall score—understood here as critical thinking. The contribution of each subtest to the overall factor is shown in Figure 1, with the two timepoints separated.

The factor weightings of some subtests increased between the start and end of the course, although in two cases, they remained the same (deductive reasoning) or decreased (practical reasoning). The deductive reasoning subtest was less strongly related to the latent critical thinking variable at both assessment timepoints. Both the inductive reasoning and deductive reasoning subtests were less strongly related to the general critical thinking factor than the other three subtests (practical reasoning, decision making and problem solving).

**Figure 1.** Weights of the five subtests in the general factor from PENCRISAL for the two test timepoints.

Turning to the second objective of this study—criterion validity—Table 3 shows the distribution of students' results from the pre-course and post-course PENCRISAL tests, admission scores and final average course scores. In addition to maximum and minimum values, the means, standard deviations, skewness and kurtosis, and correlations between variables are included in the table. The PENCRISAL score is the sum of the students' scores in the five subtests, and these results are from a subsample of 242 students.

**Table 3.** Distribution of the PENCRISAL results in the criterion variables and their relationships.


Legend: \* *p* < 0.05; \*\* *p* < 0.01; \*\*\* *p* < 0.001.

All of the correlations were statistically significant—which is in line with our second objective. The values for the correlations between the criterion variables were the highest, which may indicate intellectual baseline differences and cognitive improvements over the years. These data require further consideration, which we cover in the following section.

#### **4. Discussion and Conclusions**

Overall, the results are in line with our initial proposals. Critical thinking (CT) can account reasonably well for academic performance. Additionally, the test used to measure CT shows strong unidimensional structural validity, with an overall CT factor supported by the core dimensions of CT (deductive reasoning, inductive reasoning, practical reasoning, decision making and problem solving). In addition, the statistically significant correlations between the total scores in the pre-course test and post-course test and the criterion variables support our belief that CT can be a good predictor of academic performance.

Having an assessment of the level of CT before and after studying the topic provides useful information on the potential for improving these basic CT competences. With one exception, we were able to observe an increase in post-course versus pre-course scores in most factors or subtests. This reinforces the idea that CT can be improved with training and practice. Only in regard to the problem-solving aspect (where the scores from the post-course test were no better than for the pre-course test) was this improvement not seen. Bearing in mind the sample of classes measured (2011–2016) this lack of improvement is probably due to the way this aspect operated at the time. Decision making and problem solving both employ general strategies that are hard to separate. Additionally, at the time the study took place, the activities used to improve these skills were not yet able to distinguish between them sufficiently well. We have since managed to eliminate this overlap.

There are some less robust data that need to be substantiated. A correction was made to the pre-course measurement to reduce the RMSEA index in order for it to better align with the nature of the sample. The DR and IR subtests are more formal skills than the others and this makes them less sensitive to change because they are more difficult to apply and generalise. This results in a weaker correlation with the overall CT factor. It is important to remember that the CT test items are all problems that need to be answered by using, applying and generalising those specific processes; formal processes are less flexible and less easily modified for use outside their essentially algorithmic domain.

The indices of fit support the existence of a unidimensional model, which also worked better at the end of the course; the improvement at the end of the course may reinforce all dimensions of CT as a whole. Good performance is not possible without all CT core competencies working together. This may explain the unidimensional nature of the PENCRISAL test in terms of its structural validity.

In terms of criterion validity, we found significant relationships between CT and academic performance, and an even stronger relationship between this criterion variable and the university admissions measurement. As we noted above, these data go in the direction that we expected, based on our initial approach. However, there was no relationship between CT and university entry requirements. One way to interpret this lack of relationship may be the fact that the level of CT measured at the start of the course was fairly low in relation to the test's reference standards. Moreover, whilst post-course measurements of CT demonstrated an increase in the level of CT, they were still too limited to capture the relationship that should exist with the entry requirements. CT skills are complex and require significant levels of expertise in order to be able to capture correlations with measures of a different intellectual nature—such as those that may underlie a university entrance score, which is the result of several years of schooling. In a study into how permanent the change in CT is following teaching and measured four years later, we saw greater improvement than immediately after the intervention. This improvement is attributable not only to the CT programme, but also to the experience and education gained throughout those four years at university [20], although this is not always the case (see [35]). We can therefore say that in this study, the relationships between CT and academic performance are easier to see (as they are the result of experience and education) than those between CT and the cut-off or selection measures for university admission, which capture a more stable and experience-independent threshold.

A number of implications emerge from our study. First, in order to be able to use CT levels as a predictor, we need levels that are above the average of the test scales used. Second, it is possible to improve CT competencies using the right instructional tools and given sufficient practice and procedural work. Lastly, we need to take CT measurements in order to ascertain our starting point, and thus, see what has been achieved. If we believe that today's world requires these complex skills, we also need to ascertain the degree to which they are available from the outset. Without such an assessment, we cannot see how far we have come.

**Author Contributions:** Conceptualization, S.F.R., C.S. and L.S.A.; methodology, S.F.R. and L.S.A.; software, S.F.R. and C.S.; validation, S.F.R., C.S. and L.S.A.; formal analysis, S.F.R., C.S. and L.S.A.; investigation, S.F.R., C.S. and L.S.A.; resources, S.F.R., C.S. and L.S.A.; data curation, S.F.R. and L.S.A.; writing—original draft preparation, S.F.R., C.S. and L.S.A.; writing—review and editing, S.F.R., C.S. and L.S.A.; visualization, S.F.R. and C.S.; supervision, S.F.R. and C.S.; project administration, S.F.R. and C.S.; funding acquisition, S.F.R. and C.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

## *Article* **Lessons Learned from the Educational Experience during COVID-19 from the Perspective of Latin American University Students**

**Karla Lobos 1,\*, Rubia Cobo-Rendón 2, Diego García-Álvarez 3,4, Jorge Maldonado-Mahauad <sup>5</sup> and Carola Bruna <sup>6</sup>**


**Abstract:** The COVID-19 pandemic impacted the educational context. University students were exposed to an educational transition from a face-to-face context to emergency remote teaching (ERT). This change affected the educational experience of students and teachers in general, and impacted their educational performance, as well as their emotional and mental health, among other aspects. However, learning from the successes during the ERT and reflecting on good and bad practices will allow us to configure effective learning scenarios that respond to the new normal. The objective of this paper is to describe and present the lessons learned during ERT from the experience of university students in Latin America who have already returned to face-to-face instruction. The study used a qualitative inductive approach and a phenomenographic design. The sample consisted of 640 undergraduate students (63% women) of higher education who experienced online education during the year 2021 and a face-to-face modality during the first semester of 2022, belonging to universities in Chile, Venezuela, and Ecuador. The results suggest that new learning scenarios should consider specific pedagogical practices, including active, collaborative, meaningful, and problembased strategies, together with a diversity of feedback practices. It is concluded that the ERT brought good practices that should guide university educational policies.

**Keywords:** higher education; COVID-19; university students; quality education; virtual education

#### **1. Introduction**

Since January 2020, when the World Health Organization (WHO) classified the coronavirus disease SARS-CoV-2 or COVID-19 as a pandemic, it brought with it radical changes around the world [1,2]. The educational context was not exempt, especially in terms of higher education, impacting students in emotional, psychological, and educational aspects [3–5].

University students were exposed to an educational transition from face-to-face instruction to emergency remote teaching (ERT). In some cases, this transition lasted for more than two years [6]. With the aim of adapting face-to-face courses to an online teaching modality, educational institutions were subjected to the need to change their traditional teaching methods based on lectures to the adoption of various information and communication technologies [7,8]. However, ERT was implemented quickly and in an unplanned fashion. There are aspects that clearly distinguish virtual training, characterized by careful planning to offer a quality learning experience to students. In order to do so, courses follow a predefined instructional design, which seeks to address the heterogeneity of the students

**Citation:** Lobos, K.; Cobo-Rendón, R.; García-Álvarez, D.; Maldonado-Mahauad, J.; Bruna, C. Lessons Learned from the Educational Experience during COVID-19 from the Perspective of Latin American University Students. *Sustainability* **2023**, *15*, 2341. https://doi.org/10.3390/su15032341

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Leandro Silva Almeida and Silvia Fernández Rivas

Received: 28 November 2022 Revised: 30 December 2022 Accepted: 17 January 2023 Published: 27 January 2023

**Copyright:** © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

via the careful planning of synchronous and asynchronous work, integrating technological platforms with enriched content (e.g., hypermedia resources, study and evaluation guides), as well as strategies that promote interaction between teachers and students [9,10].

Virtual education is effective when it results from careful instructional design and planning, considering the target audience and using a systematic model for its implementation [7]. Therefore, this implies planning a variety of interconnected elements (e.g., technology, tutorial model, communication, monitoring and follow-up, evaluation), which produce specific forms of teaching that have to be adapted to the educational objectives [11]. However, during this stage of the pandemic, teachers were not prepared to redesign the courses and redefine their instructional, pedagogical, technological, assessment, management (academic and technical), support and ethics approaches [12].

Currently, in this "post-pandemic" stage, in which students and teachers from higher education institutions are returning to campuses, attention is being given to how to restructure teaching and learning processes to transition from a traditional expository model to a hybrid model that takes the best of face-to-face and virtual instruction, and that responds to the new demands made of those involved. On the one hand, students demand greater preparation from teachers to design scenarios of greater flexibility, with students partly managing their own learning processes, at their own pace and with opportunities for work continuity [13]. On the other hand, teachers demand from universities accompaniment, training, and both technological and methodological tools to face "the new normality". Thus, universities need to satisfy these demands, which is considered a complex task at the pedagogical, organizational, infrastructural, legal, economic, and even sociological level [13–15].

#### *1.1. Virtual and Hybrid Learning*

The virtual classroom is identified as a space for educational innovation, which is characterized by being flexible, integrating content through a diversity of formats and generating spaces for reflection through different learning activities [16]. However, for students to achieve planned learning, emphasis is placed on the need for student motivation and autonomy as a critical success factor [16,17].

Garrison [18] proposes that the construction of knowledge in virtual environments of teaching–learning, such as virtual classrooms, occurs through the development of a community that is characterized by the presence of three components: (1) pedagogic, (2) social and (3) cognitive [19,20]. The first, pedagogic, considers all aspects of the design, planning, facilitation, interaction, and management of teaching and learning processes [21–23]. The second, social, refers to the interaction between participants in the classroom to promote collaborative learning, in which participants feel affectively and cognitively connected to each other, with open communication and the ability to express themselves [23,24]. The third component, cognitive, refers to all activities and learning materials that involve the autonomous construction of knowledge by the student, and that allow the development of competencies such as critical thinking, the use of communication skills, reflection, and disciplinary skills, among others, based on the resources and activities available in the virtual classroom [23,24]. Of the three components, the most important is the cognitive [20].

Virtual university training, if systematized (and not as a reactive effort in the face of emergencies), has been shown to have important advantages in the training process. Empirical evidence establishes that virtual education provides greater opportunities for the cognitive development of students, particularly for those under the conditions of social vulnerability [25]. Audiovisual media represent an opportunity to support learning processes; these facilitate and grant greater learning opportunities to students [26]. In general, students highlight the benefits of virtual learning, such as being able to learn at their own pace, developing strategies to self-regulate their learning process, greater responsibility, leadership and teamwork capacities, flexibility to interact asynchronously, and continuous access to learning materials, among other desirable outcomes [16,27–29].

In addition to the advantages of online training, recent studies analyze its effectiveness in terms of learning achievements, and compare it with traditional face-to-face learning and even hybrid approaches. One analysis is presented in the study by George, et al. [30]. The authors conducted a literature review on online learning from January 2000 to August 2013, in which they studied 59 learning experiences with 6759 students enrolled in online courses. The most relevant conclusions of this study suggest the following: (1) online learning is equivalent to or better than face-to-face learning in terms of knowledge acquisition and skill development; (2) online learning helps in the development of technological, educational, communicative, research and communication management skills; and (3) online learning encourages the development of self-regulation strategies for learning, which are key to the autonomous development of the student throughout their life.

Despite benefits also being reported during online education undertaken in the pandemic, as well as many shared experiences, emphasis has been placed on the specific challenges for institutions and countries according to socioeconomic, experiential and other factors [31]. Identifying and focusing on successful practices is an opportunity to enrich virtual education in the present and future.

Today, higher education institutions are at an inflection point [32], at which they must decide between going back to traditional teaching or adapting and promoting innovation in the teaching and assessment methods, integrating methods, mechanisms, and spaces. The international trend is to promote the development of educational practices for highquality virtual teaching [33], presenting blended learning (BL) as one of the main means to transform higher education to the current post-pandemic scenario [15].

BL is an ambiguous term that can present adopt definitions depending on the author and the context in which it is applied [34]. So and Brush [35] define BL as "any combination of teaching systems methods, including face-to-face (F2F) instruction with asynchronous and/or synchronous computer technologies." Dziuban, et al. [36] refers to BL as "a pedagogical approach that combines the learning possibilities of the online environment, rather than a relationship of delivery modalities". Other researchers, such as Koper and Tattersall [37], use the term in a more technological way to refer to the combination of F2F with technologysupported activities, as well as pure online learning, to enrich educational experiences.

BL is a teaching modality that combines in a "fair" way the best of the face-to-face instruction and distance education. Its implementation implies the definition of a technological didactic strategy, in which the advantages of both contexts (face-to-distance) are taken into account and combined in a fair way according to the recipients, the learning outcomes, the characteristics of the teachers, the context, etc. [38]. An ally of BL is the learning management system (LMS) that allows for collecting evidence regarding the learning process through virtual classrooms [39,40].

#### *1.2. Objective of the Study*

Learning from the ERT during the pandemic is an opportunity without precedent, given that never before has there been a scenario that forced students, teachers and institutions to think of a new way of approaching the massive modernization of teaching and learning, through the incorporation of techno-pedagogical tools, consistent with the changes on a global scale that the digital age has generated in our societies and that place technologies at the service of people.

With globalization and the development of technology, university institutions have found in hybrid and virtual education a key element to generate new scalable learning spaces that meet the demands of today's society [41]. Within this framework, postpandemic virtual education in university contexts could contribute to the Sustainable Development Goals (SDG) [42]. Goal 4 seeks to ensure inclusive, equitable and quality education, and to promote lifelong learning opportunities for all [42,43]. Therefore, higher education institutions can potentially contribute to equality for present and future generations by identifying the successes of the ERT, without ignoring the difficulties and negative consequences caused by the pandemic.

In this context, the main objective of this study is to describe and present the lessons learned about the successes of ERT from the experiences of university students in Latin America who have already returned to face-to-face learning, and whose perceptions are not biased by the uncertainty that characterized virtual education during the pandemic. This study highlights the challenge of addressing the features of the teacher, students and content that favor greater motivation and better learning in virtual environments, and that would enable higher-quality post-pandemic university education based on a blended learning modality that includes the benefits of the ERT [15].

#### **2. Materials and Methods**

An inductive qualitative approach and a phenomenographic design were used. This design aims to identify and systematize ways of thinking by which people understand, experience, conceptualize and perceive aspects of reality [44]. In addition, hermeneutically oriented text analysis was used to understand the meaning and significance of the written discourses regarding the experiences during pandemic education [45,46].

#### *2.1. Participants and Instruments*

The sample consisted of 640 students—403 women (63%) and 237 men (37%) at the undergraduate stage of higher education who experienced online education during the year 2021 and transitioned from confinement due to COVID-19 to the face-to-face modality during the first semester of 2022, belonging to universities in Chile, Venezuela, and Ecuador.

The written discourses were obtained from answers to three open-ended questions as part of an adhoc questionnaire called "Questionnaire on university educational experiences in the return to face-to-face classes after the COVID-19 pandemic", which was built following a literature review, expert judgment, cognitive interviews and pilot applications to assess appearance and format validity and ethical validity. The experts determined that the indicators of the instrument do not cause harm to the participants. This type of instrument is recognized as necessary in the literature around education for responding more precisely to the research objectives, in addition to being one of the most broadly used types of instruments.

The questionnaire contains 3 sections, each containing 19 closed questions with response alternatives, and another section with 5 open questions [47]. The instrument was applied at the end of the first academic semester of 2022 when the students had already experienced the return to face-to-face education. Of the 5 questions, 3 were used for this study: (a) From your experience with virtual education, what teaching strategies or practices increased your academic motivation and why? (b) From your experience with virtual education, what resources or activities used by your teacher were effective for yours learning and why? (c) Which were the benefits or advantages of online education?

#### *2.2. Procedure and Ethical Considerations*

Three open-ended questions were developed as part of the application of the Online Education Experience survey [48]. We sought to explore students' perceptions about the pedagogical practices that motivated their online learning, as well as the resources and activities they consider most effective for online learning, and the benefits of online education for the training of higher-level professionals. The students' discourses were obtained through non-probabilistic sampling using the snowball technique.

Administrators of the universities to which the researchers had access were asked to facilitate the application of the survey to their students. The invitation to participate was sent via institutional e-mail to all students taking undergraduate courses at their universities. From the returned surveys, only those in which participants had answered all three open-ended questions were selected. The regulations and ethical principles for the research were considered: informed consent, the safeguarding of the information in codified databases, and, especially, the assurance of the confidentiality of the information.

#### *2.3. Analysis Plan*

Data analysis was carried out through three main steps: (1) the review of the information, (2) the identification of the units of analysis, and (3) categorization. Through the constant comparison method, the information was coded and analyzed simultaneously. First, the meaning of the unit of analysis was analyzed, and a category was generated. Then, another unit of analysis was analyzed and compared with the previous one, and if it did not correspond, it was associated with another. Finally, the categories generated by the same unit of analysis were compared.

The coding and reordering categories are iterative processes based on the permanent comparison of data, readings, and re-readings. The coding of the written discourses continued until the theoretical saturation of the generated categories was reached, that is, until the point at which new data no longer offered new information [49]. Using descriptive analysis, meanings associated with the educational experience during the pandemic were obtained. This analysis allowed us to identify the main components and to hierarchically organize these contents [45].

#### **3. Results**

With the objective of describing and understanding the experience of Latin American university students with ERT from the perspective of the return to face-to-face learning, the analysis of the discourses of students from Chile, Ecuador, and Venezuela was performed.

The use of discourse analysis allowed us to characterize the three topics set out by the researchers (pedagogical practices that motivate virtual learning, the most valuable resources, and activities for virtual learning and the benefits of virtual education), and to add an unforeseen one, which refers to strategies that favor participation in synchronous classes via videoconference.

#### *3.1. Pedagogical Practices That Motivate Learning in Virtual Education*

The students reported a series of pedagogical practices that they consider beneficial to their motivation for e-learning, and that facilitate the use of classroom resources and activities. These can be divided into the characteristics of instructional design, interaction, teaching and learning strategies, and how the teacher approaches student performance.

#### 3.1.1. Instructional Design

The students' reports have allowed us to establish the specific characteristics of the instructional design of the virtual classroom that favor motivation to interact with its contents (Table 1).

A classroom with a logically ordered structure in terms of the organization of materials, that responds to a planned learning cycle and that is coherent with the subject programs favors a greater willingness to learn. The quantity, diversity, and availability of resources and activities in the virtual classroom are also mentioned as relevant to the motivation of using them, as opposed to classrooms with few materials of the same type.

Availability refers to the fact that the classroom allows permanent access to all the resources and activities of the subject. Another element highlighted by the students is the inclusion of breaks in the planning, which would provide a space to consolidate learning and rest, favoring virtual learning.

The variety of assessment processes that arise in virtual education and the possibility of having more time for them is considered by students as an element that motivates their study, as well as the inclusion of formative evaluations that allow them to approach the expected performance without sacrificing their grades.

Finally, the quality of the audiovisual resources used in virtual education would also influence motivation in this teaching modality, together with the incorporation of breaks in the academic planning.


**Table 1.** Instructional design strategies that motivate learning in virtual education.

3.1.2. Student–Teacher Interaction

Regarding the interaction between teachers and students, students' express appreciation for those teachers who exhibit intentional actions to stimulate participation, who are attentive to their learning rhythms, and who are willing to provide help and address doubts when students request it, giving the interaction a pedagogical role focused on improving learning (Table 2).

On the other hand, students declare that the availability of their teachers in virtual education is greater, as they show themselves to be more flexible, understanding, and attentive to their emotional state.


**Table 2.** Characteristics of teacher–student interaction.

From the students' perspective, another motivating element of virtual education is that it would provide them with more opportunities and a variety of communication channels to interact with their teachers, as well as their classmates.

#### 3.1.3. Teaching strategies

Students referred to specific teaching and learning strategies that would favor their learning in virtual education. These are presented in Table 3, divided into active, collaborative, and meaningful learning strategies.

In active learning, the intentional integration between synchronous and asynchronous material and various information and communication technologies with gamification elements stands out. Regarding collaborative learning, academic and social interaction among students is expressed as central, as reflected in group work, debates, and the use of shared documents to construct knowledge among students.

The teacher promotes the application of knowledge through exercises, laboratories, case analyses, and examples. Moreover, superior experiences are offered when based on the use of meaningful and problem-based learning strategies, and these are positively valued by students.


**Table 3.** Teaching and learning strategies that motivate virtual learning.

#### 3.1.4. Performance Approach Strategies

Students distinguish between individual, group, public and anonymous feedback, as ways for teachers to approach their performance and favor their motivation to learn. Additionally, they identify two moments for the provision of effective feedback—one at the beginning of the class when reporting on previous performance, and another immediately after a learning activity has been executed.

As for the content of the feedback, the use of instructive and evaluative messages would favor greater motivation for learning in students (Table 4).

The instructive messages deal with two aspects: (a) the ability/capacity displayed by the students in coherence with the learning goals, and (b) the work procedures considered desirable for certain learning activities.

Students distinguish between individual, group, public and anonymous feedback as ways for teachers to approach their performance that would favor their motivation to learn.

On the other hand, motivation to learn would increase when the teacher, in their feedback, recognizes successes through praise or prizes, and conveys high expectations and the usefulness and importance of their students' contributions and performance to the educational process.


**Table 4.** Teaching strategies for approaching student learning achievement.

#### *3.2. Most Effective Resources and Activities for E-Learning*

Students reported a series of virtual classroom resources and activities that favored their learning in the virtual education context (Table 5).

The learning resources they recognized as effective are recorded classes, infographics, and videos. In the latter, they differentiated between documents, instructional videos, solved exercises, and the use of movies or television series relevant to the contents of the class.

As for the learning activities reported as practical, the following are identified: immediate response systems, virtual games, surveys, forums, private classrooms for group work, online documents, interactive whiteboards, and simulations. In particular, the forums are divided into two types—academic content, and another of a general nature for the clarification of doubts, both of which are promoted by e-learning.


**Table 5.** Most effective resources and activities for e-learning.

In addition, the students' discourses present the use of resources and activities in an integrated manner, in which the former contributes to the development of the latter, identifying learning resources linked to the performance of the group and individual activities and resources associated with immediate response systems.

#### *3.3. Benefits of Virtual Education*

Students reported a series of benefits of virtual education that can be divided into learning, teaching, and personal benefits (Table 6).

The learning benefits can be classified into individual and group learning benefits. In the first, students reported improvements in their capacity for autonomy and time organization, and the deployment of more effective learning strategies. In the latter, they

reported better opportunities for collaborative work, improved peer coordination, and more opportunities to work with students.



As for the benefits for teaching, virtual education would improve the teacher's organization and allow for the faster distribution of information to their students. At the same time, they state that virtual education allows the teacher a better educational response to

the needs of the students with attention to their learning pace and more timely feedback regarding their performance.

In terms of personal benefits, students report savings in economic resources, more comfortable study environments, better food and sleep habits, and the facilitation of household and childcare tasks.

#### **4. Discussion**

The transition to the pandemic has provided an opportunity to think about how to continue the transformation of university education to a sustainable model for the next generations [50]. In this sense, the present study aimed to understand the experience of university students in Latin America, particularly in Chile, Ecuador, and Venezuela, with emphasis on the qualitative perceptions of different key elements of the ERT formative experience from the point of view of the experience once they returned to face-to-face classes.

The study is based on the premise that during the pandemic, students recalled previous classroom education in a manner that was biased by an idyllic retrospection [51,52], a coping strategy that human beings use to adapt to difficult situations, in which the past is perceived as more favorable, minimizing the negative aspects, while the present is viewed with greater pessimism. This cognitive bias in the interpretation of the ERT experience did not allow students to adequately value the benefits and opportunities of virtual classrooms, their materials, and activities. On the contrary, the reports issued by the students in this study were elaborated 3 months after the return to face-to-face classes, granting the possibility of a more adjusted evaluation of their experiences with virtuality.

#### *4.1. Pedagogical Procedures to Motivate Learning*

Student motivation during re-entry to face-to-face education after ERT due to the COVID-19 pandemic was impacted by instructional design, teacher interaction, teaching strategies, and performance approach strategies

#### 4.1.1. Practices Associated with the Instructional Design

The organization and virtual classroom structure of resources and activities in the virtual classroom are perceived by students as a favorable teaching aspect, giving relevance to pedagogical intentionality that reflects an anticipated instructional design. Instructional design is the process of creating the learning experience by conjugating diverse elements to ensure more effective planned cycles or paths in particular learning situations [53]. All the teacher's efforts made during the ERT, in the form of providing tools, knowledge, and skills to manage, create and develop virtual teaching processes, lead to the challenge of framing them in a good instructional design [54].

The benefits indicated by students regarding a virtual classroom with a logical structure are a global and integrated perception of the subject, similar to a mental map, consistent with the academic programs, which encourages them in decision-making and reflecting on what contents could be made more or less complex, allowing us to understand the academic load and estimate the need for and distribution of time, among other actions. This helps us develop better study strategies and meta-cognitive evaluation processes [55,56].

The variety and quantity of virtual classroom materials are mentioned as important by students. In this context, the different instructional design models support the students' viewpoints, all of them indicating a common planning process that promotes the search for a variety of resources and activities to respond to different needs and learning styles, as well as estimating the dedication required in achieving the competencies set out in the learning objectives [54,57].

The planning of assessment processes within the instructional design is considered relevant by the students. In the context of authentic assessment [55], the students' state that in virtuality, they had a greater variety of opportunities for assessment focused on improving the quality of learning, instead of just a final judgment of their performance [58].

A longitudinal study conducted by Panadero et al. [59] on university students found that assessment processes generate the greatest negative emotionality in students, and seriously impact their academic motivation, as they are characterized by being summative and not associated with the promotion of self-regulated learning skills. On the contrary, formative assessment involves students in the performance of realistic tasks that give them a clear idea of their knowledge, skills, and abilities, and provides teachers with feedback regarding their teaching methods, forcing both to self-evaluate their strategies and make decisions about them.

The assessment activities described by the students are more consistent, i.e., they are part of the daily life of the classroom, in which mistakes are valued, and expectations that improvement is possible via a more constructivist approach to the teaching and learning process are transmitted [59].

Other relevant elements of instructional design expressed by students are the quality of audiovisual materials and a planning process that includes breaks. Regarding the former, there is evidence that supports the need to ensure the quality of audiovisual material while selecting and/or creating, since this determines the future level of use, in addition to impacting their motivation and satisfaction with the course. On the other hand, breaks have a more protective function over mental health, and a learning-leveling function, providing the opportunity to renew energies. In this context, the study carried out by Lo and Liu [60] reports that students noticed greater fatigue, a lack of concentration and visual problems when the online educational periods were very long, so they suggest that universities and instructors should consider shorter educational periods with a greater number of breaks. In curriculum planning [60], this period could represent an opportunity to consolidate learning and provide activities for self-care and/or bonding among students, teachers and directors, all of which are aspects reported as favorable for the quality of educational processes.

#### 4.1.2. Practices of Teacher–Student Interaction

The role of the teacher is considered as key in the students' discourses, with a large part of the responsibility for the students' engagement in their learning falling on their ways of interacting. Following the transactional distance theory, in asynchronous education models, students experience a sense of separation, a psychological and communication void, which is attributed to the high dropout rates in online courses. Therefore, behaviors such as the promotion of participation, the permanent availability to answer questions and clarify doubts, the demonstration of concern for their learning rhythms and their associated emotional states, would shorten this transactional distance, in which the teacher is perceived as an understanding, flexible and close figure.

One element that stands out is the perception that in the ERT, the availability of teachers was greater. This availability may be due to the fact that in virtual environments the channels and means of communication are diversified, increasing the opportunities for interaction, both synchronous and asynchronously. The use of chat, forums, written audio and gestural messages (emoticons), among others, favors interaction. These communicational tools also favor a more youthful audience, which brings students closer to a more dialectic and collective learning, in which the student can even become a passive actor, that is, a spectator of the communication of others.

#### 4.1.3. Teaching Strategies

The students' perceptions allowed us to identify strategies previously reported as more successful because they respond to a constructivist and socio-constructivist approach. In particular, these include active, collaborative, and meaningful learning models.

Active learning involves highly complex reflective and cognitive processes that can only be developed if students play an active role in the construction of knowledge [61]. This interpellation towards students' active participation would be given by pedagogical practices such as the intentional linkage between resources and activities, between

synchronous and asynchronous material, and the incorporation of gamification in the learning processes [61].

Regarding gamification, which emulates the video games that are so popular among young people, thus considerably increasing their motivation, this approach also involves feedback elements such as the comparison of their performance with themselves and with their peers, which favors metacognitive evaluative processes and the self-regulation of learning [62]. These may not necessarily favor self-efficacy, but guide one on the path towards it [63]. Despite these potential benefits, the study by Ng and Lo [64] warns that gamification can negatively affect student motivation, and interrupt their participation in learning activities when training has not been safeguarded. Technological and technical aspects are relevant to its effective implementation. The authors of these studies suggest that to improve the effectiveness of gamification, the digital literacy of both students and teachers should be ensured, and to promote enthusiasm for it, the teacher must conduct a brief information session on the results of the gamification activity at the end of each online class session.

In the sociocultural learning model, collaborative processes for the construction of knowledge are central. Designing learning activities as social experiences generates individual benefits (improves communication skills, self-confidence, develops critical thinking skills, stimulates autonomy and responsibility) and collective benefits (fosters social interaction, promotes inclusion and diversity, encourages commitment, develops planning skills and goal setting) [65,66]. According to students, collaboration in learning could be promoted by group work using virtual tools such as web-based documents, private classrooms and virtual black-boards, among others, as well as the development of debates, plenary sessions, and discussion groups through applications such as forums and videoconferences.

One of the main components of meaningful learning is the use of previous knowledge for cognitive activation in the face of new learning, according to which the latter must be perceived by students as "meaningful", i.e., useful, functional, and representative of everyday life [67]. The teacher who gives examples and shares experiences, while asking for them, and who promotes the realization of exercises and applications of the contents will favor greater student motivation and involvement, more lasting learning, a better classroom climate, and more participative and democratic students, thus improving the educational results [68]. In addition, the exercise activity reported by students is related to problem-based learning (PBL), which offers benefits such as promoting critical thinking and knowledge integration skills, as well as using them in real time.

#### 4.1.4. Practices for Addressing Students' Learning Achievement

How the teacher influences their students' performance is considered relevant by the students. Effective feedback encourages students to assume a more active and central role in their learning process by clarifying their performance and raising awareness of what and how they learn. Recognizing the affective and motivational world of the student helps when proposing more appropriate pathways to improve their performance [69]. Thus, through feedback, focus shifts to the student, allowing him/her to be an autonomous learner, while the teacher acts as a mediator. The model that supports this approach to performance is referred to as feedback for learning [70].

Regarding the means to deliver feedback, teachers' messages have been shown to have a greater effect on students' motivation, with two favorable types of messages having been identified—instructional and evaluative [58,71]. The instructive component refers to the teacher indicating concretely and precisely what the students did well or what elements should be improved to achieve the learning goals. The evaluative component refers to accompanying the instructive feedback with a message of praise and/or rewards for success, or general encouragement. Students declare that the use of this type of messages by their teachers transmits high expectations and a feeling of being useful and important to the educational space, given that their performance nurtures the learning of all participants [55].

The messages that have a positive impact on students are those that are situated at the level of the value interaction between teacher and students, and that comply with the following conditions: (a) being expressed with positive language; (b) focused on strengths and not on shortcomings; (c) appearing personal by including accurate and differentiated learning achievement information that makes them credible to students; (d) are part of daily classroom interactions promoting a culture of continuous improvement, and (e) being timely.

Regarding the latter, students report two moments of greatest effectiveness for feedback—one at the beginning of the class, reporting on the previous performance, and one immediately after the execution of an activity that reflects their learning. The former makes it possible to activate previous knowledge and use the information as a diagnosis; the latter is contingent, avoiding gaps or lags that would affect later performance [72]. In both cases, feedback is not only at the service of the students, but also at the service of the teacher, by providing him/her with relevant information for adjusting his/her teaching.

The students mention as favorable both individual and group feedback, and the role of anonymity in the evaluation space stands out. Anonymity allows one to avoid exposure in front of their peers at times when they must acknowledge mistakes or failures, thus reducing negative emotions and contributing to a greater willingness to make sustained efforts when faced with more complex tasks.

#### *4.2. Resources and Activities Considered the Most Effective for Learning*

The learning resources that students recognize as facilitators of their learning are recorded classes, infographics, and videos. The recorded classes favor the taking of notes, the clarification of doubts and a better understanding of the content provided by being able to review it several times. Infographics synthesize the relevant content with a strong visual support that makes it more didactic [73]. On the other hand, videos offer enough empirical evidence, making them a successful technological mediation tool and highly valued by students; the variety of types of videos could be associated to the particularities of the learning results they pursue [74].

Regarding virtual classroom learning activities, there is evidence that both immediate response systems and surveys favor learning processes by providing opportunities for the activation of prior knowledge, diagnostic, formative and summative assessments, and acting as tools that support the development of mastery pathways [54]. In addition, the available applications (menti, Kahoo, Forms, etc.) are of low complexity for both teachers and students, and allow great economy in correction times, along with favoring more timely feedback.

Forums offer spaces for consultation, and the exchange of opinions and experiences among students that would fulfill the function of the democratization of knowledge by stimulating more constructivist learning that is accessible to all those involved [15]. Research developed with 5000 Chilean students indicates that the use of forums favors academic performance up to the third year of studies, with a moderating effect in which the use of forums is more favorable for those students whose entry conditions are associated with higher risks of abandonment. This leveling effect offers an important contribution to combating early dropout in higher education.

Private classrooms, online documents and interactive whiteboards are technological means that favor collaborative learning [75], as well as the better coordination of the time and place of group work, favoring the systematization of information since in all three it is possible to leave an audiovisual record of the progress.

The use of simulators is a more complex teaching strategy, but one with important benefits, the main ones being the early exposure of students to real training situations without risks (use of substances, procedures with people, etc.) and the possibility of giving immediate feedback using parameters that report achievement levels and information on performance, and propose corrective measures [76].

Finally, the integration of resources and activities, as well as the linkage with individual and group learning activities, both relate to the fact that these virtual tools are designed with pedagogical intentions, and span from Information and Communication Technologies (ICTs) to Learning and Knowledge Technologies (LKTs). The TKTs require teachers capable of integrating technology into the pedagogy of their discipline, a challenge theorized by the Technological Pedagogical Content Knowledge Model or TPACK Model, which proposes this integrative competence as the highest and most desirable level of teacher preparation [77]. Along the same lines, the Community of Inquiry Model or COI model also indicates the central and preponderant role of the teacher in the definition of quality criteria for virtual education [18].

#### *4.3. Benefits of Virtual Education*

The most reported benefits of virtual education in the pandemic were related to the equity and flexibility of this modality [6], and, to a lesser extent, to benefits for educational, attributing this decrease to the emergency condition of virtual education in confinement. Coincidentally, students report personal benefits associated with these elements, but also identify essential benefits for learning and teaching [32,78].

In terms of personal benefits, rest, home comforts, financial savings, security related to the avoidance of social dangers (theft, transportation accidents, etc.), improved eating and sleeping habits, and the possibility of collaborating with household chores and, in some cases, parenting, give students a perception of well-being associated with virtual education that is consistent with previous studies. It is important to point out that although these studies support this vision of the students, at the same time, they report the sacrificing of indispensable learning processes that are not possible under the virtual modality, and that have led to the generation of pedagogical lags and more superficial levels of learning.

The benefits for learning are expressed at two levels: individual and social. The individual benefits are associated with improvements in self-regulated learning skills. Following the Zimmerman Model [79], a greater number of skills have been developed for the pre-discipline stage of study (organization of time, activities, group work, etc.); secondly, in the execution stage, they report an amplification of their learning strategies and metacognitive monitoring processes driven by virtual tools that provide greater opportunities for feedback in a more timely manner. As regards the third stage, evaluation, there are no clear reports that allow us to account for it.

At the social level, benefits have been reported associated with greater opportunities for collaborative work, better coordination among students, and more opportunities to work with students other than the ones they would have chosen in the classroom. On the contrary, at the beginning of the pandemic, this was one of the areas worst evaluated by students, who had the expectation that virtual education would not allow them to have quality relationships with their peers [47]. After the first year of the pandemic, a longitudinal study with university students sought to establish to what extent the students' expectations had changed because of their experiences of virtual education, and although the perception of the quality of the relationship with peers improved with respect to the initial measurement, it continues to tend towards the negative pole.

The positive evaluation expressed by the students in this study may be because they have already experienced the return to face-to-face education, so an adjustment is made in the parameters of comparison, with a more real and less idealized evaluation of the academic social interactions that pertained during the pandemic, in which isolation and confinement sharpened the need for social contact and social relations. Along with this, the advancement in teachers' technological competencies is likely to have consolidated at the end of the ERT period, which could have given them more and better experience of interactions with peers and professors.

In the area of teaching, the students report that virtual education also had a positive impact on the teacher's self-regulation skills, and declarations that allude to the better order, organization, risk anticipation behaviors, etc., attest to this. A self-regulated teacher provides more and better structures for students to organize their learning, transmitting greater security to their students who can anticipate and feel more control over their educational processes, and this allows the teacher to provide the student with a better balance of the academic load by being able to distribute resources, learning activities and evaluation processes in a better way [72].

A second benefit for teachers is related to their better educational response to students' needs [80]. The diversification of teaching strategies with a variety of technological devices, the increase in the channels and frequency of communication, as well as the incorporation of formative evaluative processes into daily classroom interactions would justify this perception of students that virtual education gives the teacher greater possibilities to deploy timely and relevant support actions.

The evaluation of the perceptions of Latin American students has provided a broad view with relevant information to consider in the re-structuring of post-pandemic teaching, a challenge taken up by many higher education institutions that intend to move towards improving the quality of education by identifying learnings from the ERT.

The limitations of this study are related to the fact that the analyses carried out do not consider the variables that make the participants very different, such as the type of degree they study, the year they are studying in, the country they belong to, etc. The short period of time assigned to collect the information was also an obstacle, since the perceptions of the students had to be fresh but still close to the time of ERT; without this restraint, more students could have participated.

Future research could address the learning of the ERT considering differences between different disciplines, in the level or year of study of the students, in the types of higher education institutions, and between countries, to characterize information, and to gain further insights and identify effective solutions that respond to particular realities. Going deeper into the characteristics of the teachers who promote these practices would substantially enrich the results and enable us to begin to investigate the characteristics of each resource and learning activity in greater depth, enabling us to specify what makes them effective and favors their replicability.

#### **5. Conclusions**

This study describes the lessons learned during the ERT from the experience of university students in Latin America who have already returned to face-to-face learning. The lessons learned point out that new learning scenarios should consider pedagogical strategies in relation to instructional design; active, collaborative, meaningful, and problembased learning teaching strategies; and a feedback process that considers different types and moments of delivery, which includes performance information, recognition, praise, and encouragement. The design or selection of resources and activities must consider elements perceived as effective by students associated with visual attractiveness, interactivity, and gamified materials. Finally, we consider not only the personal benefits that contribute to the equity and flexibility of this educational modality, but also pay attention to those benefits related to the development of transversal competencies for learning and teaching, which are considered criteria of educational quality in higher education.

It is concluded that ERT introduced successful practices that should be incorporated into educational policies, teacher training processes, and student teaching and learning processes, offering all educational actors the opportunity for improving experiences in higher education.

**Author Contributions:** Conceptualization, K.L. and R.C.-R.; methodology, K.L.; software, K.L.; validation, K.L. and R.C.-R.; formal analysis, K.L. and D.G.-Á.; investigation, K.L., R.C.-R., D.G.-Á. and J.M.-M.; resources, K.L.; data curation, K.L.; writing—original draft preparation, K.L., R.C.-R., D.G.-Á., J.M.-M. and C.B.; writing—review and editing, K.L., R.C.-R., D.G.-Á., J.M.-M. and C.B.; visualization, K.L., R.C.-R., D.G.-Á. and J.M.-M.; supervision, K.L. and R.C.-R.; project administration, K.L.; funding acquisition, K.L. and J.M.-M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the Vicerrectorado de Investigación de la Universidad de Cuenca, Ecuador.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki for studies involving humans.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The original contributions presented in this study are included in the article, further inquiries can be directed to the corresponding authors.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Sustainability* Editorial Office E-mail: sustainability@mdpi.com www.mdpi.com/journal/sustainability

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel: +41 61 683 77 34

www.mdpi.com

ISBN 978-3-0365-6737-2