**Ensuring Quality Education and Good Learning Environments for Students**

Editors

**Ana B. Bernardo Adri ´an Castro-L ´opez Javier Puente Leandro S. Almeida**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editors* Ana B. Bernardo University of Oviedo Spain

Leandro S. Almeida University of Minho Portugal

Adrian Castro-L ´ opez ´ University of Oviedo Spain

Javier Puente University of Oviedo Spain

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *Sustainability* (ISSN 2071-1050) (available at: http://www.mdpi.com).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-2444-3 (Hbk) ISBN 978-3-0365-2445-0 (PDF)**

© 2021 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**



### **About the Editors**

**Ana B. Bernardo Guti ´errez** has a PhD in Psychology at University of Oviedo. She is lecturer in Developmental and Educational Psychology at University of Oviedo. Since 2000, she has belonged to the research group ADIR (School Learning, Difficulties and Academic Achievement) of the same university.

**Adri´an Castro-L ´opez** has a PhD in Business Administration at University of Oviedo. His current research focuses on marketing, artificial intelligence and education. He has published more several papers in international journals and distinguished with several research awards.

**Javier Puente** has a PhD in Industrial Engineering at University of Oviedo. He is a full professor in the Business Management area. He has published more than 50 publications in high-impact international journals.

**Leandro S. Almeida** has a PhD in Psychology and is a full professor of Educational Psychology at the Institute of Education (University of Minho, Portugal). He belongs to the Research Center of Education in UMinho.

## **Preface to "Ensuring Quality Education and Good Learning Environments for Students"**

New technologies are radically changing our current personal and social lives. In the information society in which we live, we collect information and make decisions based on the virtual resources available. Despite the increased quantity and quality of such resources, there are still problems. One of these is their lack of standardization among countries and the social groups within each country. The use and development of new technologies and virtual environments is a common feature in all areas of social life, including in the educational context, but the aforementioned differences across countries and population groups raise questions on sustainability, especially in the light of the Sustainable Development Goals (SDGs) defined by the United Nations for the year 2030.

In recent years, higher education systems and learning environments have evolved alongside society and changes arising in the environment, affecting their sustainability over time. For example, the COVID-19 pandemic highlighted the vulnerability of these systems worldwide, fundamentally due to the difficulty found in adapting to new technological teaching resources (focused on new online teaching platforms), the complex integration of these into lecturers' teaching and research tasks, and the impact that a virtual environment can have on student learning. In this regard, initiatives to prepare lecturers and students for the appropriate use of new technologies and virtual teaching–learning environments must be reinforced. Indeed, immersion in new teaching methodologies demanded by new technological environments can lead to a paradigm shift in the teaching–learning process, affecting both policies for the development of educational systems and the quality of the curricula taught, as well as student satisfaction.

This book reflects on important factors regarding the sustainability of traditional education systems complemented by these new virtual learning environments. In this context, important research questions arise in different areas. For instance, concerning higher education institutions and their quality systems, is it possible to ensure the planned SDGs are met and the feelings of all stakeholders involved in the teaching–learning process are being taken into account? Furthermore, can the educational dimension be strengthened in disadvantaged social environments to promote employability and reduce poverty? Regarding the new teaching technologies being incorporated into educational systems and their impact on learning, are there clear guidelines to identify teaching technologies capable of promoting quality e-learning, minimizing absenteeism and dropout? Are these new tools capable of motivating and satisfying students adequately, providing them with constantly updated knowledge? Finally, are they able to avoid undesirable aspects, such as a lack of teacher–student interaction, poor technical and instructional support or insufficient collaboration for problem solving/group projects? Do they allow for inclusive practices for students with disabilities, and can mobile devices act as a catalyst for improvement in specific learning processes, such as languages? All these questions, in addition to many others, illustrate the diversity and complexity of potentialities and problems that may occur in the rapidly changing and unpredictable near future.

In summary, this Special Issue of *Sustainability*, "Ensuring Quality Education and Good Learning Environments for Students", consists of a compendium of 13 articles that provide interesting reflections on the above issues and raise the need to monitor the positive and negative factors that are observed in various empirical studies on sustainability in education.

The present volume seeks to expand the research field, which arises as a result of the challenges that the educational system currently offers us.

> **Ana B. Bernardo, Adri´an Castro-L ´opez, Javier Puente, Leandro S. Almeida** *Editors*

#### *Article*

### **Longitudinal Analysis of Teacher Technology Acceptance and Its Relationship to Resource Viewing and Academic Performance of College Students during the COVID-19 Pandemic**

**Rubia Cobo-Rendon 1, Karla Lobos Peña 1,\*, Javier Mella-Norambuena 1,2, Nataly Cisternas San Martin <sup>1</sup> and Fernando Peña <sup>1</sup>**


**Citation:** Cobo-Rendon, R.; Lobos Peña, K.; Mella-Norambuena, J.; Cisternas San Martin, N.; Peña, F. Longitudinal Analysis of Teacher Technology Acceptance and Its Relationship to Resource Viewing and Academic Performance of College Students during the COVID-19 Pandemic. *Sustainability* **2021**, *13*, 12167. https:// doi.org/10.3390/su132112167

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Javier Puente and Leandro Almeida

Received: 30 September 2021 Accepted: 29 October 2021 Published: 4 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

**Abstract:** Due to COVID-19, teachers quickly changed their courses from traditional face-to-face modality to emergency remote teaching (ERT), relying on learning management systems (LMS). In this simple prospective design study, we analyzed the relation of the level of teachers' technological acceptance at the beginning of ERT (March 2020) considering three variables: the time spent by teachers in the LMS during that semester, the percentage of LMS's resources their students viewed during the semester, and the final academic performance of the same students at the end of that semester (September 2020). This study included 251 teachers (57% male) and 12,185 students (45% male). We measured the teachers' level of acceptance with the Spanish version of the Questionnaire Technology Acceptance Model (TAM). We found that the relation between the teacher's acceptance and their time spent on the LMS was significant and positive (rho = 0.24, *p* < 0.001). In addition, teachers' perception of LMS's easiness is related to the percentage of educational resources their students utilized (rho = 0.26, *p* < 0.001). Finally, we found a relation between the usefulness dimension of the TAM to the academic performance of the students at the end of that semester (rho = 0.18, *p* < 0.01). Considering these results, we discuss practices for implementing quality education.

**Keywords:** TAM Model; learning analytics; academic performance; higher education; COVID-19

### **1. Introduction**

Due to the rapid changes produced worldwide by the COVID-19 pandemic in educational institutions [1], teachers at all educational levels had to make changes in the way they taught, relying on the use of technology [2,3]. Specifically, university teachers had to quickly transition from face-to-face teaching to emergency remote teaching (ERT) in order to assure academic continuity [4,5]. ERT is characterized by being developed in virtual environments with urgent and scarce planning with the goal of maintaining the continuity of educational training during a crisis [5].

In this scenario, teachers were faced with the untimely need to quickly modify their programs to an online modality. University institutions also had to make decisions and favor this modality by incorporating virtual tools and learning management systems (LMS) [6,7]. However, for these tools to play a positive role in the quality of teaching, teachers require new skills and readiness for the use of technological resources that allow them to take full advantage of their benefits [8].

#### *1.1. Technology Acceptance Model (TAM)*

In recent decades, several theoretical models have been proposed that seek to evaluate the willingness to use technological resources and the processes of acceptance and use of technology by teachers. Among them is the technology acceptance model TAM proposed by Davis (1989). Its objective is to explain the reasons that users have for using technologies [9]. This model has become one of the most interesting for researchers and technology designers in recent years, being used in various contexts where technology is implemented; for example, in the evaluation of the mobile experience [10,11], in the implementation of technology in health areas [12] and in the use of technological devices [13]. The TAM model has been undisputedly the most tested and validated model in different contexts and studies, confirming its robustness and ability to predict technology adoption in users [14,15].

This model is explained from the Theory of Reasoned Action [16]. This theory is based on the premise that the perception of a person will determine his/her attitude and behavior, i.e., the attitude and behavior that a person may have about the use of a particular technology is influenced by the perceptions that the person has about it [17]. In this sense, in the approach proposed by Davis, the level of use of a technology depends on the users' perceptions regarding the ease and usefulness of this technology for the performance of a particular task [18]. These two aspects (ease and usefulness) influence the user's attitude and intention to accept a technological system or device [15]. Therefore, these factors are two of the main extrinsic motivations (beliefs) that influence an individual's acceptance and use of technology. The two variables are positively correlated (i.e., a user-friendly website is more likely to be perceived as useful) and both influence an individual's attitude toward the use of a given technology. Likewise, under this model, attitudes toward technology use are positively correlated with the behavioral intention to use it, which, in turn, is positively correlated with actual use [18]. All this makes the TAM a powerful tool for describing technology adoption by teachers in higher education [15].

The use of technology for teaching and learning processes is usually referred to as digital learning [19]. In this context, technology or educational technology involves implementing technology-based software or applications for delivering learning resources and implementing learning activities in face-to-face, online or hybrid classrooms [20]. The role of educational technology in the COVID-19 pandemic has been fundamental. Teachers could implement the emergency remote teaching modality thanks to learning management systems LMS, like Moodle or Canvas, and videoconferences systems like Zoom or Teams. These two technologies are among the most used during the pandemic period. However, once educational institutions start implementing face-to-face modality again, it is expected to observe new technologies that satisfy the blended and hybrid modalities [21].

In the educational setting, a systematic review analyzed published research from 2003 to 2018 on the application of TAM including 71 papers, [22]. Indicated that TAM is a credible model to facilitate the evaluation of various learning technologies [22]. The core variables of TAM, perceived ease of use and perceived usefulness, affect the acceptance of learning with technology [22]. Researchers also reported that a small percentage of the articles analyzed (6%) focused on teachers [22]. In another systematic review regarding teachers and students [23], researchers analyzed 50 publications that aimed to investigate the acceptance of technology at that level. In the results, the authors indicate that most of the research using the TAM model refers to education linked to information and communication technologies and how this model could identify relevant elements from the point of view of teachers as well as students [23].

A meta-analysis developed on the TAM model in teachers, particularly for pre-service and in-service teachers, sought to synthesize 124 correlation matrices from 114 empirical TAM studies (*n* = 34,357 teachers) to test the fit of the TAM and its versions. They analyzed data from teachers in schools, universities, or technology acceptance in educational contexts, most of the participants were from Asia or America. The authors found that the TAM explains well the acceptance of technology in teachers [15].

In the case of university teachers, some studies on TAM report that the measurement and evaluation of this model is beneficial for the prediction of teachers' intentions of implementing learning management systems [24,25]. In the same context, recent reports regarding the use of LMS in higher education institutions, in which responses from both students (*n* = 584) and teachers (*n* = 42) were analyzed, reported different perceptions between students and teachers regarding the use of LMS, so particular research is necessary considering the type of user [25].

Now, faced with the context of the ERT due to the pandemic, in order to study how the TAM model research was used in this scenario, we conducted a brief systematic review following the PRISMA model [26]. For this case we reviewed the Web of Science (all collections), EBSCO (all collections) and Scopus databases, using the keywords of "technology acceptance" AND "COVID-19" OR "coronavirus" OR "2019-nCoV" OR "SARS-CoV-2" OR "COV-19" AND "higher education" OR "college" OR "university". A total of 30 studies were identified. In phase 2, 17 duplicate papers were eliminated, identifying a total of 13 original investigations. In phase 3, We are selected 11 papers what included the keywords in their title and abstract, of which eight full manuscripts were accessed. Finally, in phase 4 we used the selection criteria (research in the context of COVID-19, on technological acceptance in higher education), concluding with a sample of eight papers (See Supplementary Material).

From the analysis, we found that three studies seek to assess specific elements of online education of the TAM model, such as the use of Zoom, Microsoft Teams and Google classroom as videoconferencing platforms [27–29]. The remaining papers focused on online education in general [30–34]. Regarding the research participants, most were university students (*n* = 7) and only one research focused on higher education instructors or teachers [34]. The latter is qualitative, and its authors describe some exploratory findings on the level of acceptance of the teachers. Teachers stated that virtual tools were useful to promote the fulfillment of the objectives of their courses, the realization of evaluative processes, and instructional methods. Nevertheless, they did not neglect the challenges that ERT represented in terms of interaction, evaluation, and development of their pedagogical practices [34].

In conclusion, the TAM model (a) is useful for the evaluation of the level of technological acceptance with respect to specific technological tools and also with respect to the online teaching modality in general in the context of ERT, (b) is positively associated with the satisfaction and participation of users in digital platforms, (c) it successfully explains the factors that predict the use of e-learning and, (d) allows to distinguish different user roles in regarding technology (such as student and teacher).

#### *1.2. Educational and Learning Analytics in Higher Education*

In higher education, the collection and use of university data has expanded dramatically over the past two decades [35]. One of the ways to assess how faculty technology acceptance beliefs impact the educational experience of students in virtual learning environments is through the exploration of data that describes the behavior of users in online platforms. This is known as academic analytics and learning analytics.

On one hand, academic analytics refers to digital information that is used to manage an institution to benefit data-driven decision-making [36]. These analytics are nourished by the information provided by institutional systems about undergraduate programs, subjects, teachers, and students [37]. On the other hand, learning analytics refers to the process of measuring, collecting, analyzing, and reporting student-centered data with the goal of improving the student's educational experience [36].

Understanding teacher factors that might drive student academic performance is becoming an important topic of research in Educational Psychology. In general terms, both researchers and educational authorities are interested in identifying elements that could contribute to improve the quality of education [38]. In this sense, the United Nations included education as one of the relevant topics within the sustainable development

objectives. Among the specific objectives proposed, the aim is to promote safe and effective learning environments for all students, as well as improving teaching skills [39]. The use of data provided by academic and learning analytics will allow the implementation of effective strategies to improve the quality of education. In this sense, analytics are a powerful mechanism to help students, instructors, teachers, designers, and developers of learning systems to better understand educational processes and predict students' needs and performance [36,40].

Academic and learning analytics enables the improvement of teaching and learning processes through the analysis of educational data available in the LMS [40], which are software developed exclusively to manage the teaching and learning process [25]. From an LMS, a digital fingerprint can be obtained that describes the behavior both teachers and students (connection time, access to resources, participation in forums, etc.) [41]. One of the most widely used LMS in Higher Education is Canvas In structure LMS. It was created in 2010 and it is currently considered the fastest growing LMS worldwide [42]. It is used by about 30% of higher education institutions in the United States [43]. Research on LMS usage by faculty indicates that compared to other LMSs, Canvas was the most popular system. Moreover, research reports that teachers perceived ease of use and user satisfaction. The features of this LMS reported as relevant are related to the possibilities of structuring the class, creating assignments, uploading files, and other aspects that distinguish it other LMSs [44,45].

#### *1.3. The Present Study*

The COVID-19 pandemic forced universities to change the ways in which they taught. In this case, online education was considered as an effective sustainable learning solution in the conditions of ERT and in the conditions that may arise in the future [46,47]. The sustainable development is defined as meeting the needs of current generations without compromising the ability of future generations to meet their needs [48]. The online education focuses on promoting student participation, encouraging critical analysis, regulation of their learning pace and other elements of the educational experience, thus satisfying the needs of each student [2,49,50]. However, to be able to respond to this, one of the factors that guarantees success in the development of online education of quality is the paper of the teacher [47,51].

Understanding the processes of acceptance of technology by university professors is a relevant task because it allows a better understanding of the possible mechanisms that exist for the implementation of e-learning in higher education [15]. From sustainable development's perspective, university institutions need to implement strategies to transform the way in which education is it developed today [52]. In this sense, for teachers to be able to implement digital resources such as LMS during university training processes, it is necessary that they possess beliefs of usefulness and ease of use of these tools, influencing their pedagogical practices [53].

Additionally, analyzing the process of acceptance of technology also allows evaluating how teachers' beliefs could impact the development of quality learning experiences in students. The present study aims to contribute to the proposals of the various declarations on sustainable development regarding the importance of universities being able to provide teachers with resources that contribute to the education of their students and, therefore, to society [48,52].

For this reason, the general objective of this research is to analyze the relationship between the level of technological acceptance of teachers at the beginning of the ERT semester (T1) with (1) the time spent by teachers in the Canvas LMS, (2) the percentage of resources viewed their students in the LMS, and (3) the final academic achievement of the same students at the end of that semester (T2).

To this end, we set the following specific objectives:


We propose the following hypotheses:

**Hypothesis 1 (H1).** *Teachers with higher levels of technology acceptance at (T1) will have greater time spent on the Canvas LMS at the end the ERT semester (T2).*

**Hypothesis 2 (H2).** *The teacher's level of technological acceptance (T1) is positively related to time spent on the Canvas LMS at the end the ERT semester (T2).*

**Hypothesis 3 (H3).** *The teacher's level of technological acceptance (T1) is positively related to the percentage of resources viewed their student in the LMS Canvas at the end the ERT semester (T2).*

**Hypothesis 4 (H4).** *The teacher's level of technological acceptance (T1) is positively related to the student's academic performance at the end of the ERT semester (T2).*

Teachers' pedagogical beliefs play a key role in their pedagogical decisions about how to integrate technology into their classroom practices and how to do so [54]. Some research shows that when teachers present high levels of technological acceptance with an LMS, the possibility of using it increases [18,24,25], evidence that supports H1. In addition, it has also been shown that teachers' beliefs can intervene in the educational experiences and performance of students during the development of their courses [55], evidence that we seek to deepen with H2 and H3.

#### **2. Materials and Methods**

The present research corresponds to a simple prospective non-experimental design [56]. We analyzed the TAM model in teachers who initiated the design and development of ERT courses due to the COVID-19 pandemic (T1). We related this variable to the teacher's time in the Canvas LMS (T2), to the students' level of visualization of educational resources, and the academic performance obtained by students at the end of the courses (T2).

#### *2.1. Participants*

A total of 251 university teachers participated in this study. These belonged to a public university in Chile. From the total of participants, 143 were male (57%), with a mean age of 48.17 years (SD = 11.17). Of the teachers participating in this study, 126 (50.2%) have a doctorate degree and 79 (31.7%) have a master's degree or a specialty degree. The scientific area where they teach, the years of teaching experience and the type of teaching day we are described in Table 1. These teachers designed and implemented a total of 487 online courses during the ERT due to COVID-19.

From learning analytics, it was possible to obtain the percentage of interaction and academic performance of 12,185 university students who had taken classes with any of the teachers in the sample and that agreed to participate in the study. From the total of students, 6703 were women (55%) and the average age was M = 22.96 (SD = 2.82). From the total of students, 23.62% were students in their first academic year. Table 2 shows the description of the sociodemographic variables of the students according to the scientific area to which they belong.


**Table 1.** Descriptive statistics on the area of performance, years of experience and time of dedication of the participating teachers.

Note: M: Median; SD: Standard Deviation; *n* = 251

**Table 2.** Descriptive statistics of the sociodemographic variables of the student's participants according to the scientific area to which they belong.


Note: M: Median; SD: Standard Deviation; *n* = 12,185.

#### *2.2. Instruments*

#### 2.2.1. TAM Model in Teachers: Perceived Usefulness and Ease of Use

A Spanish version of the Measurement Scales for Perceived Usefulness and Perceived Ease of Use (TAM) designed by Davis (1989) was used to evaluate people's perception of usefulness and ease of use of devices or software in digital environments. This scale consists of 12 items distributed in two dimensions: perceived usefulness and perceived ease. The former has six items oriented to evaluate the user's perception of the benefit of the device or software in improving productivity and performance of their work. The latter is composed of six items which aim to evaluate the perception of competence in the use of the device or software and the ease of integration with the user's work. Both dimensions are answered on a Likert-type scale with five response options ranging from strongly disagree = 1 to strongly agree = 5. Scores above 3 are considered as a high level of technological acceptance. In the English version, adequate internal consistency indices were identified (α = 0.98 for usefulness and α = 0.94 for ease of use) as well as high convergence, discriminant, and factorial validity (Davis, 1989). For this study, adequate internal consistency indices were found (α = 0.93 for usefulness and α = 0.93 for ease of use and α = 0.94 for the complete scale). We confirmed an adequate fit of the model with two factors proposed by the authors of the scale (X2(53) = 154.647, *p* < 0.001; CFI = 0.956; TLI = 0.945; SRMR = 0.050; RMSEA = 0.078 [0.064, 0.092], *p* < 0.001.

#### 2.2.2. Teacher Academic Analytics: Time Spent

For the evaluation of teacher behavior in the Canvas LMS, we analyzed the variable of time spent on platform from the academic analytics. For this purpose, the teacher's log records were extracted [57]. With this data we worked with two variables:

Average session time: where each session is calculated as the time between interactions (or events associated with a timestamp) in the Canvas LMS, with a cut-off point of ten minutes. If the user, in this case the teacher, does not perform any action within ten minutes, the session is finished. This is used to calculate the average time spent on platform by teachers per session.

Average session time: to have a standardized measure of the connection time considering the characteristics of the courses, we added the duration of all sessions and divided it by the number of credits SCT of each subject taught by the teacher during the ERT 2020 period. Thus, the calculated time is assigned to 1 credit. The SCT credits are designed based on the estimated time dedicated by the student to achieve the learning outcomes. One SCT credit is equivalent to a range of workloads between 24 and 31 h by the student [58].

#### 2.2.3. Student Learning Analytics

In the case of student analytics, we extracted the data associated with the number of visualizations of educational resources that were available in virtual classrooms of the participating teachers' courses. For this purpose, the information from the students' logs was analyzed. This information was used to generate the variable of percentage of resources viewed:

The percentage of resources viewed: for each course, we considered the resources with which at least one student interacted, this gave us the total number of resources. Then, for each student, we obtain the number of resources that they interacted with. Finally, the percentage is calculated as the number of resources that students interacted with over the total number of resources on platform.

#### 2.2.4. Academic Achievement

The grade point average (GPA) in the first semester of 2020 was obtained from the academic record of the university and considered as the students' academic performance. In Chile, the GPA is constructed on a scale from 1.0 to 7.0 points. The grades from 6.0 to 7.0 correspond to an academic performance considered as "excellent." The grades from 5.0 to 5.9 are labeled as "good" grades, while 4.0 to 4.9 are defined as "satisfactory." Last, grades from 1.0 to 3.9 are "unsatisfactory," which means the student failed the course [59].

#### *2.3. Procedure*

This research was endorsed by the Ethics Committee of the participating university, corroborating the ethical criteria for research with human beings. The application of the instrument on the TAM model of the teachers was carried out in digital format by sending it to the institutional e-mails in a single opportunity. The link to answer the questionnaires was available during the first three weeks of March 2020 (beginning of the first ERT academic period). Participating teachers responded to the questionnaire after reading and signing the consent form. Information on teacher academic analytics and student learning analytics were obtained at the end of the semester from the Canvas platform. The students' academic performance was obtained from the university's official information recording platforms. At the beginning of the semester, the participating students signed a consent form authorizing access to their academic information for the development of the research.

#### *2.4. Statistical Analysis*

To obtain the information on the academic analytics of teachers and student learning in the Canvas LMS, the Canvas Data portal service was used. The information was obtained from the Log table or request table. Python programming language was used, using an algorithm to review the information provided by the URL. We used the information provided by the "TimeStap" to identify the time of interaction of the teacher with the Canvas LMS. For the analysis of the resources viewed by the students we used regular expressions to identify the educational resource linked to the URL.

Descriptive analyses were performed on the study variables. The assumption of normality was verified. In this case we used the Kolmogorov-Smirnov test with the Lilliefors modification since the total sample size of teachers was larger than 50 participants. We applied the Levene's test to verify the constant variance between groups (homoscedasticity) [60].

Since the assumption of normality was not met, nonparametric statistical procedures were used. Spearman's rho was used to evaluate the relationships between variables. For the evaluation of differences by groups of high and low technological acceptance, we used Yuen's robust test and the one-way ANOVA test for trimmed means for statistical analyses employing more than two groups [61,62]. The method proposed by Algina et al. (2005) was used for the effect size analysis of the results [63].

In the case of the analysis for the low technological acceptance group, since it had fewer than 50 participants, normality was verified by means of the Shapiro Wilk test [64]. It was possible to verify that the data in all cases did not follow a normal distribution (*p* < 0.001). The data analysis was made with R software version 4.1.0 (18 May 2021).

#### **3. Results**

The general objective of this work was to analyze the relationship between the level of technological acceptance of teachers at the beginning of the ERT semester (T1) with (1) the time spent by teachers in the Canvas LMS, (2) the percentage of resources viewed their students in the LMS, and (3) the final academic achievement of the same students at the end of that semester (T2). To address this objective, we will present the results according to the specific objectives described in the Section 1.3.

#### *3.1. Technological Acceptance of the Canvas LMS and its Relationship to Time Spent Teacher during ERT by COVID-19*

To respond to the first specific objective, we describe in Table 3 the measures of central tendency and dispersion of the variables studied. We identify a high level of technological acceptance by teachers regarding the use of the Canvas LMS (average > 3). At a specific level, the results indicate that, on average, teachers perceived greater ease (M = 3.82) than perceived usefulness (M = 3.63) concerning the Canvas LMS.

**Table 3.** Descriptive statistics on technological acceptance and academic analytics of teachers during ERT due to COVID-19.


Note: Min: Minimum; Max: Maximum; M: Median; SD: Standard Deviation; Mdn: Median; \* hourly connection time; \*\* hourly connection time as a function of an SCT; *n* = 251.

> We evaluated the differences in the level of Technological Acceptance of the teachers, considering the scientific area where they work (OECD area). We did not find statistically significant differences in teachers' scores for each group (F (5,49.42) = 1.78, *p* = 0.13). The same occurred when analyzing the dimensions of Perceived Usefulness (F (5,49.36) = 2.29, *p* = 0.06), and Perceived Ease (F (5,49.43) = 2.0489, *p* = 0.09).

> Concerning academic analytics, we evaluated the teachers' time spent in the Canvas LMS, we found that teachers spent an average of 13.36 h in the virtual classroom for each credit (SCT) their subject had during the semester. However, since the standard deviation found in the data was substantial (SD = 17.82), we incorporated the median information indicating a dedication of Mdn = 8.0 h per credit (SCT) of the subject during the semester as a better indicator.

> We evaluated the possible differences with respect to the weighted time on the platform according to the OECD area where the teachers work. We found statistically significant differences between groups (F (5,52.01) = 3.23, *p* < 0.05). The Lincoln posthoc test indicated significant differences between teachers working in Agricultural Sciences (M = 10.37; SD = 9.48) vs. Engineering and Technology (M = 30.62; SD = 35.67; *p* < 0.05), between Natural Sciences (M = 10.8; SD = 9.93) vs. Engineering and Technology (M = 30.62;

SD = 35.67; *p* < 0.05). Between Social Sciences (M = 11.15; SD = 14.56) vs. Engineering and Technology (M = 30.62; SD = 35.67; *p* < 0.05), and between Humanities (M = 9.85; SD = 10.31) vs. Engineering and Technology (M = 30.62; SD = 35.67; *p* < 0.05). In the case of teachers in Medical and Health Sciences (M = 13.74; SD = 18.19), they showed trend differences with the group of teachers of Engineering and Technology (M = 30.62; SD = 35.67; *p* = 0.051). Teachers in the Engineering and Technology spent more time on the LMS than teachers working in other scientific areas. With respect to the average time of the sessions, we found no statistically significant difference between the groups of teachers according to OECD area (F (5,49.11) = 2.83, *p* = 0.06).

When analyzing the teachers' scores in terms of high (average > 3) and low technology acceptance scores. we found that 38 teachers fell into the category of low technology acceptance of the Canvas LMS (see Table 4). In the case of the teachers with a low technology acceptance, most of them were male (*n* = 24). When confirming the existence of statistically significant differences. We found that teachers in this group had shorter connection times and shorter sessions than teachers with high technological acceptance. Finding answering H1. Another interesting finding was that the teachers in this group were older than the teachers with high acceptance of the use of the Canvas LMS.

**Table 4.** Categorization of teachers considering the level of technological acceptance with the Canvas LMS.


Note: M: Median; SD: Standard Deviation; AKP effect: effect size; *n* = 251. \* *p* < 0.05. \*\* *p* < 0.01. \*\*\* *p* < 0.001.

In Table 4 we show the connection time of teachers and the sociodemographic characteristics of each group. We evaluated the possible differences in the age of teachers according to the scientific area where they work (OECD area). We found statistically significant differences (F (5,51.44) = 4.71, *p* < 0.01). The Lincon posthoc test [62], indicated significant differences between teachers of Agricultural Sciences and teachers of Medical and Health Sciences (*p* < 0.01), teachers of Humanities (*p* < 0.05) and teachers of Engineering and Technology (*p* < 0.05). In this case, teachers who teach in Agricultural Sciences courses are older than the rest of the groups identified.

Regarding the relationship between the level of technological acceptance of the teacher (T1) and the connection time used by teachers in the LMS (T2). We found a positive and statistically significant association in the dimensions and the total of the TAM model (evidence answering H2). A moderate and positive correlation is observed with the perceived ease dimension (rho = 0.30. *p* < 0.001). Which indicates that the greater the perceived ease of use of the LMS. The greater the connection time of the teacher in the LMS (see Table 5).


**Table 5.** Spearman correlation on the relationship between technology acceptance and teacher and student analytics with the Canvas LMS.

Note: \*\* *p* < 0.05. \*\* *p* < 0.01. \*\*\* *p* < 0.001.

#### *3.2. Technological Acceptance to Teacher's LMS Canvas and Its Relationship to the Percentage of Resources Viewed and Student Academic Achievement in the End ERT by COVID-19*

To respond to the second objective (see Section 1.3). We calculated the average percentage of resources viewed by students. Which was 45.64% (SD = 20.23). Regarding the usefulness and ease perceived by teachers about the Canvas LMS. A moderate and positive correlation was observed between the ease perceived by the teacher and the percentage of resources viewed by students (rho = 0.26. *p* < 0.001). Regarding teacher-perceived usefulness, it had a slight correlation with students' academic performance (rho = 0.18. *p* < 0.01). These results respond to hypotheses H3 and H4 raised in this study (see Table 5).

#### **4. Discussion**

Remote education in emergencies prompted teachers to incorporate virtual tools in their teaching processes as the only alternative to provide continuity in higher education training. One of the most used virtual tools by higher education institutions were LMSs. Although they had been incorporated before the pandemic, their use was scarce among both teachers and students. The purpose of this study was to analyze the relationship between the level of technological acceptance of teachers at the beginning of the ERT semester (T1) with (1) the time spent by teachers in the Canvas LMS, (2) the percentage of resources viewed their students in the LMS, and (3) the final academic achievement of the same students at the end of that semester (T2).

#### *4.1. Technological Acceptance of the Canvas LMS and its Relationship to Time Spent Teacher during ERT by COVID-19*

In general terms, teachers express a good level of technological acceptance regarding using the LMS Canvas for their courses. University teachers perceive the LMS as a tool of low difficulty, wherein they have learned to use it in an intuitive and self-taught way to conduct their online classes during the ERT. In addition, they perceive that this tool has been useful to develop their teaching, facilitating the completion of pedagogical tasks in less time.

This information indicates that the choice of LMS to implement online education in an untimely manner and without prior preparation was a wise decision by higher education institutions. Furthermore, LMSs have a user-friendly design that allows teachers to adapt to them quickly. Although various research [65,66] point out that the ERT does not meet the standards of proper online education, it appears that the teachers were able to carry out their classes thanks to the LMS that provided them with the appropriate functionalities. This was enough for them to achieve the objective of giving continuity to their academic programs during a first adaptation period.

When teachers perceive the LMS as easy to learn and use, they spend more time working on it. This finding is congruent with the theory about TAM since teachers' perception of LMS use is influenced by their acceptance of the technology [16–18] and these results respond to hypotheses H1 raised in this study.

Therefore, it would be necessary for higher education institutions to generate strategies that favor the perception of ease of use of the LMS, such as tutorials, help desks, and short training sessions. In this way, by resolving quickly and efficiently the obstacles that teachers face when learning how to use an LMS, the institutions would be promoting its use for teaching and avoiding the use of alternative tools outside the LMS. Although using tools outside the LMS may have advantages, they have the significant disadvantage of not having a system for tracking and storing data. making it challenging to make pedagogical decisions based on learning analytics.

LMS offers various tools in the same virtual environment, such as discussion forums, assessments with various types of questions, structuring of the virtual classroom, enriched text editors, and external tools. These tools make it possible to design learning paths that seek to respond to the needs of students [67]. LMS also allow the implementation of active learning methodologies, which could be sustainable learning spaces, in which students could go beyond conceptual knowledge, encouraging critical and socially responsible thinking in favor of society and industry [50]. The diversity of tools available in LMS favors teachers' attempts to integrate the synchronous with the asynchronous, delivering a complete learning experience for the student, where both modalities complement each other. Therefore, a teacher who perceives the LMS as easy to learn will positively integrate virtual tools into his or her pedagogical practices, which will positively impact student learning. Thus, in the institutional choice of the LMS, it is essential to consider the simplicity of the interface that teachers will face for the development of the educational task since the ease of user interaction with the interface plays a fundamental role in teachers' perception of the platform [24,25].

Consistent with other research, we found that older teachers have a lower level of technological acceptability [68]. Therefore, the time on the platform is substantially less than that of younger teachers. This result reflects the generation gap regarding the use of educational technologies [69].

On the other hand, we evaluated differences in the variables studied according to the scientific area where the teachers teach. Concerning technological acceptance, we found no differences. But we did find differences regarding the time spent by teachers in the LMS. In this case, teachers in the engineering and technology area spent more time than other teachers. These differences show the need to provide differentiated and focused support to the teachers.

Despite this, we were unable to assess teachers' level of experience and familiarity with the LMS before the ERT. It is important to consider how the acceptance of LMS during the COVID-19 pandemic differs between teachers with and without previous experience, since teachers with experience with these platforms, presented greater behavioral intention to use them. Consequently, it has been reported that for teachers with little experience, institutional support is important to encourage their use [14]. Thus, it is crucial to take institutional actions that promote less resistance and greater enjoyment in using virtual tools, particularly regarding the use of LMSs, especially considering the projection of a post-pandemic blended learning education [70,71].

#### *4.2. TechnologicalAacceptance to Teacher's LMS Canvas and Its Relationship to the Percentage of Resources Viewed and Student Academic Achievement in the End ERT by COVID-19*

From the students' perspective, they visualize more virtual classroom resources when their teacher perceives the LMS as easy to learn and use. Furthermore, students achieve better academic performance (GPA) when their teachers perceive the greater usefulness of the LMS to develop teaching and promote better learning in them. These findings point to the importance of attitudes and beliefs that teachers transmit to their students about the LMS in developing their online courses [55]. For example, if a teacher complains or devalues the functionalities of the LMS by conveying to the student that they are not very useful and challenging to use, they are probably discouraging its use. In the same context, the fact that the teacher's perceived usefulness of the LMS is related to better performance, even if the relationship is small, it indicates that the LMS could be promoting more effective

teaching practices to improve student performance. In the future, routine measurements should be conducted to understand the LMS-related needs of teachers to offer specific training considering the characteristics of the courses and the teachers who teach them [43].

The strength of the results of this research lies in the contribution that they can offer to the investigation of technological acceptance in university teachers, a context in which few studies have been developed [22]. The results also give more insight into how teacher data can be analyzed to improve the learning and teaching processes. In our case, we analyzed teacher analytics provided by LMS, an analysis that has been developed to a limited extent [67]. Using information from academic and learning analytics is relevant for research on the incorporation of technology in teaching and institutional decision-making that can facilitate this process [36,37].

However, the present study had several limitations:


Future research could evaluate how the teacher's level of technological acceptance is related to the use of the different tools offered by the LMS. Likewise, it should be studied how the diversity of resources used for teaching and the type of course designs in the virtual environment intervene in the educational experience and the academic performance of students. For this, it is important to consider more detailed analyses using learning analytics and academic analytics. In addition, it could be evaluated how the personal characteristics of teachers (gender, age) and other variables linked to online education (level of experience, knowledge, and training) could intervene in the level of technological acceptance of teachers in higher education.

#### **5. Conclusions**

From the results found, we conclude that most teachers indicated having high beliefs of technological acceptance of the LMS Canvas during the ERT due to COVID-19. They report higher beliefs of ease compared to usefulness of the tool; these findings were similar in teachers from various scientific areas.

A small percentage of teachers reported low technological acceptance; these teachers were less connected to the Canvas LMS and were older. We also concluded that teachers' connection times varied according to the scientific area in which they teach. Teachers in the Engineering and Technology spent more hours on the LMS. However, when we compared this time considering the SCT of the subjects taught, we found no differences, which allows us to conclude that the characteristics of the courses could be variables that moderate the time spent by teachers in the LMS.

We conclude that the greater the teacher's perceived ease of use, the higher the percentage of resources viewed by students, and in a weak but significant way, the greater the teachers' perception of usefulness, the higher the academic performance of their students.

University authorities have an important role in the implementation of tools and policies that promote the quality of education, as well as the accompaniment of teachers during these processes. These aspects allow the promotion of an effective and meaningful pedagogical practice by teachers during the development of online education. Teachers can be transforming agents of teaching methods and generate environments that allow active participation of students and an environment that responds to their needs within a socially responsible and sustainable institutional framework.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/ 10.3390/su132112167/s1, Table S1. Matrix of information extracted from the papers selected in the review.

**Author Contributions:** Conceptualization. R.C.-R. and K.L.P.; methodology. R.C.-R. and J.M.-N.; software. J.M.-N. and F.P.; validation. J.M.-N. and F.P.; formal analysis. R.C.-R., K.L.P. and N.C.S.M. investigation. R.C.-R., K.L.P. and N.C.S.M.; resources. K.L.P. and N.C.S.M.; data curation. J.M.-N. and F.P.; writing—original draft preparation. R.C.-R., K.L.P. and N.C.S.M.; writing—review and editing. R.C.-R., K.L.P. and N.C.S.M.; visualization. J.M.-N.; supervision. K.L.P. and N.C.S.M.; project administration. R.C.-R. and K.L.P.; funding acquisition. K.L.P. and N.C.S.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Unidad de Fortalecimiento Institucional of the Ministerio de Educación Chile. project InES 2018 UCO1808 Laboratorio de Innovación educativa basada en investigación para el fortalecimiento de los aprendizajes de ciencias básicas en la Universidad de Concepción.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Ethics Committee of University of Concepción (protocol code CEBBE-656-2020, date of approval April 2020).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Further inquiries can be directed to the corresponding author/s.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **Cross-Cultural Pragmatic Competence in an EFL Context for a Sustainable Learning Environment: A Case of Northern Cyprus**

**Ben Bardis 1,\*, Fato¸s Silman <sup>1</sup> and Behbood Mohammadzadeh <sup>2</sup>**


**Abstract:** A review of literature on pragmatic competence reveals that less attention is given to the enrichment of cross-cultural pragmatic awareness in the classroom. The study focuses on the need to survey and discuss communicative dynamics in classroom situations and the importance of enhancing cross-cultural pragmatic competence in the English as a foreign language (EFL) context. We believe that enhancing students' cross-cultural pragmatic competence will provide students with a sustainable learning environment, which is crucial for the quality of education. The investigation involves a mixed approach of quantitative and qualitative methods by using a questionnaire for EFL students and interview guide questions for both EFL students and teachers. For quantitative analysis, the participants involved in this research include 200 EFL students of different nationalities, and for the qualitative analysis, the participants include 10 teachers and 20 EFL students in a university in North Cyprus. The study's findings indicate that students lack cross-cultural pragmatic knowledge and must be provided with classroom activities for social interaction and to develop explicit and implicit communicative competencies in EFL as a target language. Findings further reveal that EFL students are not endowed with cross-cultural pragmatic competence by the EFL teachers who provide little attention to pragmatic knowledge and classroom activities in the EFL context.

**Keywords:** cross-cultural pragmatic competence; social context; communicative dynamics in classroom situations; EFL context

#### **1. Introduction**

*Background Study*

There is a close relationship between language and culture, which requires the knowledge of cultural norms in an act of effective communication. Thus, language is the basis of culture since one depends on the other. In this sense, language must be taught alongside cultural norms and values in order to help speakers of diverse languages communicate across cultures. In addition to the knowledge of the language, EFL students should be provided with the features of pragmalinguistics and socio pragmatics in order to enable them to communicate effectively in English, as they need to communicate in a social context. This requires EFL teachers to take a step further in creating cross-cultural pragmatic awareness with the aim to help their learners enhance their cross-cultural pragmatic competence. This awareness is significant for an effective and sustainable communication environment, which will help avoid pragmatic failure in cross-cultural communication [1].

For an act of effective communication, EFL learners of different cultural backgrounds should be endowed with the knowledge of cross-cultural pragmatics by introducing it in classroom situations. Mey [2] describes pragmatics as the use of language in human communication, which means it is essential for interlocutors to be familiar with the conditions and conventions created by the members of a social community. In other words, pragmatics reflects the conventions and social norms manifested in a language as the

**Citation:** Bardis, B.; Silman, F.; Mohammadzadeh, B. Cross-Cultural Pragmatic Competence in an EFL Context for a Sustainable Learning Environment: A Case of Northern Cyprus. *Sustainability* **2021**, *13*, 10346. https://doi.org/10.3390/su131810346

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Javier Puente, Leandro Almeida and Marc A. Rosen

Received: 16 July 2021 Accepted: 11 September 2021 Published: 16 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

constituents of sociopragmatics. In a similar manner, LoCastro [3] states that cross-cultural pragmatic competence is the ability to use language effectively in a social context. As for the necessity of the cross-cultural features, Koester and Lustig highlight that "cross-cultural study is one in which researchers compare a particular concept in two or more cultures whose members are having intra-cultural experiences". An "intercultural" study involves interaction among people from two or more cultures" [4]. In divergent global societies, interlocutors of diverse languages may not share the same cultural norms. This results in miscommunication due to the norms of different cultural backgrounds in the social milieu. For this reason, it is essential for EFL teachers to help their students enhance cross-cultural pragmatic competence. In contrast to the traditional pragmatic competence, Taguchi states that "the surge of transnationalism and multiculturalism, has inevitably challenged the traditional notion of pragmatic competence" [5]. For Mulyana [6] and Dash [7], cross-culture is described as an intercultural relation by representatives of different cultural backgrounds for communication harmony through L2 in social circumstances. As a result, it renders it difficult for those with cross-cultural pragmatic competence to exchange information and communicate appropriately in another language.

With respect to this, few studies have been published in this field of pragmatics involving students in Northern Cyprus. One of these few studies conducted in Northern Cyprus focuses on the pragmatic competence of Kurdish EFL undergraduate students in Northern Iraq in the use of request and apology strategies [8]. In another research study by Mohammadzadeh et al. [9], they looked into how Turkish ELT students in the Faculty of Education at Cyprus International University comprehend "Conversational Implicatures". Considering these studies, the researcher discovered a gap in the study of pragmatics in a university with multilingual students from more than 100 different countries. In this study, the focus is on both students' and teachers' cross-cultural pragmatic competence, necessary for successful communication in a social context. This is another aspect of the present study, which makes it original and contributes to the related field.

Timpe [10] and Levine [11] point out how language usage is dynamic in classroom situations, especially in interlingual and multilingual settings, and point out that pragmatic competence involves cultural norms and values. Rafieyan [12] presumes that the cultural characteristics of a target language as in the EFL context could be an important factor for establishing the level of pragmatic dynamics of L2 learners in the target language. In another study, this is exemplified by Röver [13] as "it is hot over here" and may intricate conflicting interpretations of the literal meaning of the expression in different contexts as it may imply "a complaint of a situation, or a weather condition" or a "request for a fan" or "a suggestion to move to a different environment. In this concept, the pragmatic competence involves the comprehension of "what is said, what is heard and what is meant" for successful communication [3]. In this context, the research of Dash [7] suggests reinforcing oral and written communicative activities that are in line with pragmatic knowledge in order to arouse cross-cultural awareness in classroom circumstances.

This study highlights EFL learners' perception and communication dynamics associated with cross-cultural pragmatic competence. It also involves the interaction of EFL teachers and learners in classroom situations where students face critical challenges of cross-cultural differences such as cultural shocks, misunderstandings, disagreements, intimidation and misinterpretation. This study aims to show the significant role of crosscultural communicative competence among EFL learners in using words and expressions in line with cultural norms and values. The study also aims at ascertaining EFL learners' competence regarding cultural norms for effective interaction in English. The study seeks to find answers to the following research questions:


4. What are the personal views and experiences of both EFL teachers and their students on cross-cultural pragmatic in classroom situations?

#### **2. Theoretical Framework**

*Perception on Cross-Cultural Pragmatic Competence in EFL Classroom Context*

The literature review in this study entails some essential definitions of cross-cultural pragmatic competence, the important perceptions of EFL teachers and learners concerning cross-cultural pragmatic competence and the key challenges of cross-cultural pragmatic competence in the EFL context. Students' awareness of cross-cultural pragmatics is essential in the EFL context. Integrating cross-cultural communication in EFL classes helps students develop the ability to communicate in a social context [7]. It helps avoid some challenges such as misconception, misinterpretation and misunderstanding during the interaction of two or more different cultures. Thus, teaching and learning a target language must not be separable from its cultural norms and values. Hence, it is not sufficient for the EFL teachers to teach only the grammar part of the target language, but they must also help students enhance cross-cultural pragmatic competence awareness. Jie [14] claims that language teachers must focus on three points when teaching culture in EFL classes: (a) helping students become familiar with cross-cultural differences; (b) helping students tolerate each other's culture by considering the cultural norms and values in the target language; and (c) helping students to learn the target language along with its culture. Teachers' task of making students understand the different cultural norms and values in the EFL context is essential for learners to understand what seems polite in one culture and impolite in a different culture. Thus, using a word or an expression may be morally acceptable in one culture and not in another. In this perspective, Bouchard [15] affirms that the challenges of cross-communication often occur when the felicity conditions are not respected.

The sociolinguist, James [16], describes cross-cultural pragmatic competence as using words and expressions concerning societal norms and conventions of the target language. Thus, cross-cultural pragmatic competence is the ability to use language by acquiring the meaning of words and expressions through context and sociocultural embedding. Chen et al. [17] conducted a discourse completion test (DCT) for Taiwanese and American students at the university level; the finding indicates different cultural resentments from the different nationals that seem to be concerns raised in social circumstances due to cultural differences and misinterpretations. With respect to this, Taj-eddin Mogadam [18] and Benadla [19] discovered the need to bestow communicative competency on EFL learners in classroom circumstances and highlight a teaching approach called "competency-base" in order to rekindle L2 learners for effective communication in a social context. Moreover, cross-cultural communication actively takes solid ground in the tourist industry, where Borni [20] proclaims in his research that the English language by cultural norms and values tends to promote tourism by drawing both the US and the UK close to Algeria and most African countries. By using an example, Guo [21] indicates some similarities in crosscultural behavior between American students and Chinese students with respect to direct and indirect refusal to request and order by "nodding or shaking one's head" as a response and may signify a direct or indirect acceptance or concern in language interpretation. On gender, Maccoby [22] explains that "gender differences generally show the higher performance of women on verbal tasks and men on spatial tasks". Thus, this assumption implies that female interlocutors are more indulged in thorough communication and, as such, are favourably correlated with more affirmative responses to the target language (TL) or L2, as well as EFL. In the domain of gender issues in communication, the author Maccoby [22] lays emphasis that female communicators develop more interest in language learning. With respect to this concept, oral communicative skills must be built up among males and females by using numerous functional platforms of pragmatic routines in order to innovate conversational situations involving the native and the non-native speakers taking part in role-play activities that can establish cross-cultural understanding among the EFL learners. The author confirms that, in gender differences, male communicators, as

well as language learners, could demonstrate less optimistic manners than female language learners who show a higher level of interaction with others.

Equally, in the EFL classroom context, the research of Gulzar [23] draws our attention to the interaction between Pakistani students and teachers in terms of code-switching from mother tongue (L1) to EFL (English as a Foreign Language) and vice versa by cultural socialization. Corresponding to the classroom situation, Bensen and Cavusoglu [24] also drew our attention to the effectiveness of EFL classroom code-switching in Northern Cyprus that could impart on students' level of communication with regard to the pragmatic context. Admittedly, the research of Ishihara and Cohen [25] sensitizes on the academic preparedness of EFL teachers and how practical it seems to deliver cross-cultural pragmatic knowledge to the EFL learners for communicative competency.

Morgan [26] and Ishihara and Cohen [25] bring to light the exposures of classroom interactive activities involving communicative tasks such as role-plays, pair-work, reading original English materials, presentation in English language and imitating native English speakers' pronunciation in order to develop cross-cultural communication and to enhance pragmatic competence of EFL learners. In another development, Wu [27], Weeger [28] and Sunal [29] raised concerns on strategies involving the basic principle of Communicative Language Teaching (CLT), which is a student-centered approach for enhancing explicit and implicit communicative competencies of EFL. By using an example, in an interactive conversation involving immigrants, "is Pope a Catholic?" by [13], a declaration is aroused that is not part of the conversation but is suddenly used as a definite answer to a question, and the meaning implies "Yes, of course" or "It is so" or "Yes indeed" or "No doubt about that". These associative meanings introduce intricate pragmatic challenges, leaving cross-cultural communicators in suspense.

Therefore, with all the concerns raised in the literature review, the purpose of the paper is (a) to examine the dynamics of cross-cultural pragmatic competence among EFL learners, (b) to examine the perception of EFL learners in cross-cultural pragmatic competence, (C) to examine the key challenges in cross-cultural communication among EFL learners and (d) to examine the personal views and experience of both teachers and students of cross-cultural pragmatics in the classroom situation.

#### **3. Method**

This part of the research study presents the mixed methods of quantitative and qualitative analyses deployed in the study. Denzin [30] affirms that mixed-methods research (MMR) must practically project pragmatism as a prototype for social research and must be eventually detached from the philosophical foundations of pragmatism. Henceforth, the method includes the instruments and population, data collection instruments, data collection procedure and data analysis method.

#### *3.1. Instruments*

The instruments for the investigation comprise a questionnaire and interview questions. The questionnaire and the interview questions have been tested on pilot basis. The questionnaire has been administered on pilot basis to twenty five (30) undergraduate students in their first year comprising 13 females and 12 males from various faculties of physical sciences and social sciences who voluntarily responded to the questionnaire. Moreover, 5 teachers and 10 students responded to the interview questions and 40 min to 45 min were spent on each respondent. The pilot test provided us feedback for restructuring the questionnaire and the interview questions in terms of content, instructions and sentence structure after contacting three experts in the field of this research for advice and to determine the reliability and validity of the findings in this study. The questionnaire for the investigation involves three parts. Part one comprises the demographic information. Part two comprises the questionnaire, and it contains three sections (A, B and C). Section (A) is composed of ten questions (1–10) of the multiple-choice discourse completion test (MDCT) for meaningful interpretation. It is relevant to research question 1: "What is the competency

level of EFL learners on comprehension of Pragmatic competence in regard to their gender, language group (mother tongues), fields of studies and age?" The cross-cultural pragmatic competency level test of 10 questions is marked over 100; each question is for 10 marks. An example from the questionnaire is the following.

Simon asks about his fiancée Paulina from his friend David:

Simon: "You know. I've been trying to know if you went out with Paulina."

David: "Paulina is not really my type."

\*What does David probably mean?

A. He is not sure of his emotions

B. He is talking bad about Paulina as he does not like her

C. Paulina is somehow his best type

D. They did not go out.

Section B comprised five items: questions (1–5). It is relevant to research question 2: "What are the perceptions of EFL learners on cross-cultural pragmatic competence?" Moreover, Section C comprised five items, questions (6–10), and they are relevant to research question 3 of the paper: "What are the key challenges to EFL learners in crosscultural communication?" Moreover, the answers relative to questions (1–5) in section (B) and questions (6–10) in section (C) are chosen from the options (A, B, C and D) that seem suitable to each item, for example, 1 = SA: Strongly Agree; 2 = A: Agree; 3 = N: Neutral; 4 = D: Disagree; 5 = SD: Strongly Disagree. Hence, part one and part two of the questionnaire are relevant to the study's quantitative analysis. An example from the questionnaire includes the following: "Cross-cultural communicators experience emotional interference due to clashes of norms and values of different cultures.

In addition, part three includes interview questions relevant to the qualitative analysis related to research question 4: "What are the personal views and experiences of both EFL teachers and their students on cross-cultural pragmatic in classroom situations?" There are four questions composed for EFL teachers, and another four questions composed for EFL learners. The interview was conducted separately for the teachers and learners respondents and the students' respondents. For reference, both the questionnaire and the interview question guides are found on the Appendix A of this study. An example for teachers include the following: "What are the challenges or benefits your students face in cross-cultural communication in classroom situations?" Example for Students: "What are the challenges or difficulties, or benefits your students face in cross-cultural communication with student colleagues in the classroom?

#### *3.2. Participants*

The participants are 200 respondents comprising 88 male students and 112 female students categorized under positive sciences (94 students) and social sciences (106 students) from different continents: Africa, Asia, Europe, America and others. For the participants of the interview section, category (A) comprises 10 teachers, and category (B) presents 20 students. The abbreviations or the acronyms (FT) represent female teachers and (MT) for a male teacher. There are 6 female teachers and 4 male teachers. Category B also involves 20 international students with different cultural backgrounds such as African Francophone, African Anglophone, African Arabic, Asian Arabic and European Turkish cultures. Therefore, the abbreviation (SP) represents student participants, and the participants are numbered from 1 to 20.

#### *3.3. Data Collection Procedures*

The process of answering the questionnaire by students was administered under the permission and control of the EFL lecturers on 7 May 2019. Subsequently, permission was granted, and convenient dates were arranged to precisely conduct both the teachers' and students' interviews on the 9 and 10 May 2019. The interviews were conducted in the English language for 10 EFL teachers and 20 EFL students. Moreover, 45 to 50 min was spent on each participant. The dates for taking the data seem appropriate since

it was a revision week and most of the students were present and preparing for their end of semester exams in June 2019 and also presumably might have experienced some cross-cultural interactions with one another from the beginning of the semester.

The data of this research were collected in 2019. With the COVID-19 pandemic process that started in 2020, our university, similarly to other universities, has switched mainly to online education. However, due to economic reasons, most of the international students stayed in North Cyprus and did not return to their homeland. Therefore, face-to-face education continued with these students. The researchers believe that this study still has validity in 2021 since interactive classroom activities for developing cross-cultural communication and enhancing the pragmatic competence of EFL learners continued to have significance. Moreover, our university hosts students from more than 100 different countries; therefore, our multicultural campus is an ideal milieu for students to experience cross-cultural communication and to learn from each other.

#### *3.4. Data Analysis*

The study uses mixed-method and comprises both quantitative and qualitative analyses. Consequently, the data analysis presents, in the first place, quantitative analysis involving the first three research questions. However, the device of ANOVA (Analysis of Variances) by Fraenkel et al. [31] is used to analyse the differences among the group by means of involving the following: students' competency level with regard to "Gender" on Table 1; students' competency level with regard to "Language group (mother tongues)" on Table 2; students' competency level with regard to "Field of studies" on Table 3 and students' competency level with regard to "Age?" on Table 4. Fraenkel et al. [31] confirmed that ANOVA (Analysis of Variance) is a testing tool that describes and explains the dynamics involving various categories of scenarios that occur in their simplest structures of groups: Gender group, Language group (mother tongue) and Age group. We used ANOVA to observe the differences between groups. The Statistical Package for Social Sciences (SPSS) tool is used to analyse the items on Table 5 comprising (item 1, item 2, item 3, item 4 and item 5) and also the items on Table 6 comprising (item 6, item 7, item 8, item 9 and item 10).

The second part involves qualitative analysis, and according to Seidman [32] and Folkestad [33] qualitative data analysis does not strictly follow clear-cut principles. Still, it depends on the nature of the data and the procedure it takes. As a result, the theory of Creswell [34] for the qualitative analysis includes data organization, data editing, data coding and commonality or prototypes of the participants. The interview questions prepared by the author were administered to the teachers and students, but probing questions were used in the follow up where it was needed. The participants were met individually, and the time used for the interview was approximately 45 to 50 min on each participant for intact findings. The findings were recorded, coded and written down separately. They are transcribed for analysis and presented in the structured format that presents the qualitative result in themes (for example, Theme 1: "Communicative Tasks in EFL Class by Teachers"). According to Braun and Clark [35], the result of qualitative analysis involves themes to explain participants' views, experiences and perceptions. Thus, we evaluated our qualitative data by applying these criteria. There are two categories of respondents: category (A) for teachers and category and (B) for students and all relevant to the fourth research question of the study. Thus, category (A) comprises ten (10) teachers, and category (B) represents twenty (20) students. The abbreviations or the acronyms (FT) represent the female teacher and (MT) for the male teacher. Furthermore, there are six (6) female teachers representing the following acronyms: FT1, FT2, FT3, FT4, FT5 and FT6. Four (4) male teachers also represented the acronyms of the following: MT1, MT2, MT3 and MT4.

#### **4. Findings**

Research Question 1: What is the competency level of EFL learners on comprehension of pragmatic competence concerning their gender, language group (mother tongues), fields of studies and age?

Table 1 involves students' performance on comprehension of pragmatic competence concerning gender. The pragmatic test results in Table 1 show no statistically significant differences between male students and female students regarding pragmatic competence. Nevertheless, comparing the mean indicates that EFL female students have a slightly higher mean in the comprehension of pragmatic competence (M = 4. 9823; SD = 2.13384) than EFL male students of the lower level (M = 4.7586; SD = 2.33759). The finding indicates at the gender level, the EFL female students perform better in cross-cultural pragmatic comprehension than their male counterparts. For Maccoby [22], gender differences generally show the higher performance of women on verbal tasks and men on the spatial task. Thus, this assumption implies that female EFL interlocutors are more indulged in thorough communication and, as such, are favourably correlated with more affirmative responses relative to the target language in a social context.

**Table 1.** Represents the statistical data concerning students' competency level concerning gender.


Table 2 involves the significance of students' performance comprehension of pragmatic competence test regarding language group (mother tongues). The pragmatic test results in Table 2 show no statistically significant differences among EFL students' language group (mother tongues) on comprehension of pragmatic competence. Nevertheless, a comparison of the means of the three language groups English as a second language (ESL), French and Arabic reveals that students from English speaking countries speaking English as a second language (ESL) have a slightly higher mean in performance of pragmatic competence (M = 5. 3200; SD = 2.15179) than the means of students from Arabic-speaking countries who have medium mean (M = 4.8932; SD = 2.25316) and French-speaking countries of the lower mean (M = 4.4043; SD = 2.17355). The finding indicates that the EFL students of Anglophone background perform better in cross-cultural pragmatic comprehension than their counterparts of Arabic and Francophone backgrounds, who have shortcomings in cross-cultural pragmatic knowledge.

**Table 2.** Represents the statistical data concerning students' competency level in regard to language group (mother tongues).


The pragmatic test results in Table 3 show no statistically significant differences between EFL regarding the field of studies (positive science students and social sciences students) on comprehension of pragmatic competence. However, comparing the mean shows that EFL Social Sciences students in pragmatic competence have a slightly higher mean of performance of (M = 4.9714; SD = 2.24233) than that of EFL positive sciences students of the lower mean (M = 4.8191; SD = 2.19952). The finding indicates that crosscultural pragmatic knowledge is significant in social sciences in classroom circumstances where language abilities are acquired practically to communicate in a social context.


**Table 3.** Represents the statistical data concerning students' competency level with regard to field of studies.

Table 4 involves the significance of students' performance on comprehension of pragmatic competence concerning age. The pragmatic test results in Table 4 show that there are statistically significant differences between the age groups of EFL students on the comprehension of pragmatic competence. Therefore, the comparison of the mean indicates that the competency level of EFL students, especially of the age group between 25 and 34, has higher performance on comprehension of pragmatic competence (M = 5, 3788; SD = 2.31228) than the age group between 17 and 24 (M = 4.6641; SD = 2.16123). The finding indicates that the performance involving students of maturity age (25–34) is slightly higher than the other age groups (17–24; 35–44; 45 and above). Vividly, by the results of this finding, it is important to note that maturity in terms of age is significant in the competency development of language learners.

**Table 4.** Represents statistical data concerning students' competency level with regard to age?


Research Question 2: What are the perceptions of EFL learners in cross-cultural pragmatic competence?

Table 5 comprises items (item 1, item 2, item 3, item 4 and item 5) that are all relevant to research question 2. Hence, in part two of the questionnaire, the five items represent the five questions (1 to 5) in section (B) of the questionnaire found in Appendix A.

**Table 5.** Represents item 1, item 2, item 3, item 4 and item 5 relating to questions (1 to 5) on the questionnaire and relevant to research question 2.


Note SA: Strongly Agree; A: Agree; N: Neutral; D: Disagree; SD: Strongly Disagree.

The result of item 1 shows, on Table 5, that out of 200 respondents, 63 respondents (31.3%) strongly agreed; 83 respondents (41.3%) agreed; 30 respondents (14.9%) disagreed; 6 respondents (4.5%) strongly disagreed; and 15 respondents (7,5%) were neutral in responding to item 1. The participants' mean is M = 3, 81, and the standard deviation is S.Dev = 1.168. The finding reveals an average number of EFL students agreeing to the importance of cross-cultural pragmatics in the EFL context because it serves as a motivational factor that can virtually enhance the competency of EFL learners.

Item 2 illustrates, on Table 5, that out of 200 respondents, 33 respondents (16.4%) strongly agreed; 88 respondents (43.8%) agreed; 46 respondents (22.9%) disagreed; 9 respondents (3.0%) strongly disagreed; and 27 respondents (13.4%) were neutral in responding to item 2. The mean of the participants is M = 3.48, and the standard deviation is S.Dev = 1.107. The finding shows a positive impression of the EFL students agreeing that cross-cultural pragmatic competence is essential in EFL because it provides an opportunity to EFL learners to communicate and share their personal ideas, thoughts and feelings in social contexts.

Item 3 indicates, on Table 5, that out of 200 respondents, 41 respondents (20.4%) strongly agreed; 100 respondents (49.8%) agreed; 29 respondents (14.4%) disagreed; 6 respondents (3.0%) strongly disagreed; and 24 respondents (11.9%) were neutral, either for the benefit of doubt or lack of cultural knowledge. The mean of the participants is M = 3, 71, and the standard deviation is (S.Dev. = 1.046). The finding gives the impression that some EFL students do not know that cross-cultural pragmatic competence is important in improving the communication skills of EFL learners for effective communication in a social context.

Item 4 on, Table 5, indicates that out of 200 respondents, 42 respondents (20.9%) strongly agreed; 92 respondents (45.8%) agreed; 34 respondents (16.9%) disagreed; 4 respondents (2.0%) strongly disagreed; and 28 respondents (13.9%) were neutral in responding for the benefit of doubt or lack of cultural knowledge. The mean of the participants is M = 3, 67, and the standard deviation is S.Dev. = 1.052. The finding depicts that most EFL learners have no doubt in gaining experience and abilities to interpret words and expressions in a social context out of cross-cultural pragmatic knowledge.

Item 5 shows, on Table 5, that out of 200 respondents, 43 respondents (21.4%) strongly agreed; 85 respondents (42.3%) agreed; 36 respondents (17.9%), disagreed; 8 respondents (4.0%) strongly disagreed; and 28 respondents (13.9%) were neutral in responding to item 5, for either the benefit of doubt or lack of cultural knowledge. The mean of the participants is M = 3, 60, and the standard deviation is S.Dev. = 1.130. The finding reveals that an average number of EFL learners are in agreement that "cross-cultural pragmatic competence could expose them to the knowledge of conventions, rules, beliefs and principles of different societies".

Research Question 3: What are the key challenges to EFL learners in cross-cultural communication?

The results of item 6 on Table 6 shows that out of 200 respondents, 37 respondents (18.4%) strongly agreed; 88 respondents (43.8%) agreed; 40 respondents (19.9%) disagreed; 8 respondents (4.0%) strongly disagreed; and 27 respondents (13.4%) were neutral in responding to item 6, either for the benefit of doubt or lack of cultural knowledge. The mean of the participants is M = 3.53, and the standard deviation is S.Dev = 1.125. The majority of the EFL students think that "in cross-cultural pragmatics, the speakers' utterances sometimes become difficult to interpret due to difficult words and implicit expressions". This shows that EFL learners are not endowed with cross-cultural pragmatic knowledge in the classroom irrespective of their linguistic knowledge in syntax and lexis.

**Table 6.** Represents item 6, item 7, item 8, item 9, and item 10 relating to questions (6 to 10) on the questionnaire and relevant to research question 3.


Note SA: Strongly Agree; A: Agree; N: Neutral; D: Disagree SD: Strongly Disagree.

Moreover, item 7 on Table 6 indicates that out of 200 respondents, 31 respondents (15.4%) strongly agreed; 98 respondents (48.8) agreed; 37 respondents (18.4%) disagreed; 4 respondents (2.0%) strongly disagreed; and 29 respondents (14.4%) were neutral in responding to item 7 either for the benefit of the doubt or lack of cultural knowledge. The mean of the participants is M = 3.58, and the standard deviation is S.Dev= 1.065. The finding implies the majority of the EFL learners are involved in situations where "the non-native speakers in cross-cultural interaction face hearing problems due to unfamiliar accents and pronunciations".

Item 8 in Table 6 shows that out of 200 respondents, 36 respondents (17.9%) strongly agreed; 82 respondents (40.8%) agreed; 43 respondents (21.4%) disagreed; 3 respondents (1.5%) strongly disagreed; and 36 respondents (17.9%) were neutral in responding to item 8, either for the benefit of the doubt or lack of cultural knowledge. Moreover, concerning item 8, the mean of the participants is M = 3.58, and the standard deviation is S.Dev = 1.065. The finding reveals that the majority of the EFL speakers are of the view that "crosscultural communicators do face racial challenges, colour and cultural inferiority during an interaction.

Item 9 in Table 6 shows that out of 200 respondents, 53 respondents (26.4%) strongly agreed; 87 respondents (43.3%) agreed; 38 respondents (18.9%) disagreed; zero (0) respondent (0.0%) strongly disagreed; and 22 respondents (10.9%) were neutral in responding to item 9, either for the benefit of doubt or lack of cultural knowledge. The mean of the participants is M = 3.78, and the standard deviation is S.Dev = 1.044. The finding shows that the majority of the EFL speakers proclaim "the cross-cultural communicators experience the problem of attitudinal and language barriers". Thus, they are victims of cultural shocks.

Item 10 in Table 6 shows that out of 200 respondents, 22 respondents (10.9%) strongly agreed; 110 respondents (54.7%) agreed; 30 respondents (14.9%) disagreed; one (1) respondent (0.5%) strongly disagreed; and 37 respondents (18.4%) were neutral in responding to item 10 either for the benefit of doubt or lack of cultural knowledge. The mean of the participants is M = 3.19, and the standard deviation S.Dev = 1.063. The finding indicates that the majority of the EFL learners speculate "the cross-cultural communicators experience emotional interference due to clashes of norms and values of different cultures".

Results of qualitative analysis relevant to research question 4: "What are the personal views and experiences of both EFL teachers and their students on cross-cultural pragmatic in classroom situations?".

The responses are divided into two categories: categories A and B. Category A involves 10 teacher respondents. The teachers include six females and four males. Category B also comprises 20 student participants who responded to the interview guide questions.

Category A: The EFL Teachers' Responses to the four interview-guide questions (Q1, Q2, Q3 and Q4) are found in the Appendix A of the paper.

Category A comprises ten (10) teacher respondents. There are six females and four males. Therefore, the qualitative analysis in this study is based on themes that reference the interview questions found in Appendix A (for example, Theme 1: "Communicative Tasks in EFL Class by Teachers"; Theme 2: "Key Challenges Facing Their Students in Cross-Cultural Communication"; Theme 3: "Learning Strategies of Teachers to Help EFL Overcome Cross-cultural Communication Difficulties"; Theme 4: "Classroom Textbooks as Providing Cross-Cultural Pragmatic Information").

#### *4.1. Themes 1. "Communicative Tasks in EFL Class by Teachers"*

Out of 10 teacher participants, two lecturers (FT2 and MT1) mention class debate as a major activity of communicative task for their EFL students, while three lecturers (FT5, MT3 and MT4) claim that group discussion is the communication task preferable for their students. Moreover, FT3 and FT6 confirm conducting "role-play and pair-work" exercises as communicative tasks for their students in a classroom situation. For example, FT6 explains that "the role-play and pair-work tasks" are useful for enabling their EFL

learners in becoming creative and improving upon their level of expressions in vocabulary and skills of performative utterances.

#### *4.2. Theme 2: "Key Challenges Facing Their Students in Cross-Cultural Communication"*

The finding shows that out of 10 participants, 4 lecturers (FT1, FT3, FT6 and MT2) pinpointed out problems in listening comprehension by students: FT1 explains that hearing problems is one the major problems of her EFL learners because of the series of complaint her students make concerning bad accents and wrong pronunciations. FT6 also speaks about students' difficulty in adjusting to new cultural conventions and practices. MT2 makes a submission on the difficulty to believe in cultural principles of different societies. On the other hand, MT3 and MT4 also touch on problems of racism and cultural inferiority, for example, MT3 proclaims that his students segregate themselves according to their ethnic groups in the classroom.

#### *4.3. Theme 3: "Learning Strategies of Teachers to Help EFL Overcome Cross-Cultural Communication Difficulties"*

The finding indicates that out of 10 participants, 3 lecturers FT4, MT3 and FT5 confirm using original English films and videos as language learning strategies: FT4 affirms making students watch original English films and videos in order to solve cultural communication problems among EFL students. He added that this encouraged them to accept the cultural norms and values of others. Moreover, MT3 claims that making their students watch original films and video tasks helps to create a state of attentiveness, observations and practices for effective pronunciations and intonation on the part of the EFL learners. Additionally, MT2 discloses that he introduces historical books, magazines, newspapers and textbooks to his learners in order to provide them with the needed exposure to the correct pronunciation from the native speakers.

#### *4.4. Theme 4: Classroom Textbooks as Providing Cross-Cultural Pragmatic Information*

The results show that 10 teacher participants (FT1; FT2; FT3, FT4; FT5; FT6; MT1; MT2; MT3; MT4) mentioned one major book titled *Straightforward* that is used for students in EFL classes apart from their personal supplementary textbooks. As a result, FT4 points out that the single textbook in her EFL class provides little information and only on linguistic knowledge such as vocabulary, grammar and pronunciation. In addition, FT5 also explains that the current book for English class provides knowledge of literal or logical meanings of words and not on cross-cultural knowledge. Equally, MT1 says the main textbook *Straightforward* falls short of cross-cultural pragmatic information in EFL class because there are no specific activities designed for enhancing the cross-cultural pragmatic competence of EFL learners in the classroom.

Category B: EFL students' responses to four interview-guide questions (Q1, Q2, Q3 and Q4) which are relevant to research question 4 of the study. The following themes emerged from their responses: Theme 5 "Communication Tasks from Teachers"; Theme 6 "Benefits or Challenges Students Face in Cross-Cultural Communication"; Theme 7 "Language Learning Strategies of EFL Teachers for Students"; and Theme 8 "Textbooks Used For EFL".

#### *4.5. Theme 5: "Communication Tasks of Teachers"*

The finding reveals that out of 20 interviewees, the participants SP2 and SP6 mention group discussion; the participants SP9, SP10 and SP19 mention debate; the participants SP3 and SP20 also lay emphasis on role-playing, and the participants (SP11 and SP17) stress on pair-work activities as communication tasks they experience in the classroom. For example, SP3 states that her teacher provides them with extracts from the school textbook *Straightforward* or from different textbooks for group discussion and dialogue but are unrelated to cross-culture knowledge. SP10 also says that his teacher has them involved in (group debate and video show) as communicative tasks on special topics such as "Hygiene, Marriage and Social Environments" and expected them to describe the characters, the moral lesson in the text and the types of information found in the video shown.

#### *4.6. Theme 6: "Benefits or Challenges Students Face in Cross-Cultural Communication"*

It is noted that, out of 20 participants, the respondents SP4, SP8 and SP12 raise the issue of problems of cultural segregation challenges they face in a classroom situation. At the same time, SP9, SP17, SP19 and SP20 mention a lack of communication due to language barriers, racism and cultural inferiority complex as a problem. For example, SP4 explains that, as they come together to learn EFL, they usually feel comfortable in their small ethnic groups sorted by their cultures and mother tongue rather than generalising EFL groups in the classroom context. Again, SP19 says that, by his observation, African French speakers stay away from their EFL counterparts of Turkish speakers and Arabic speakers who also feel uncomfortable staying with African Anglophone speakers due to language and cultural differences. Furthermore, SP17 claims they have been experiencing cultural trauma due to signs of racism and cultural stigma related to not being accepted in some cultural groups for interactions and group studies.

#### *4.7. Theme 7: " Language Learning Strategies of EFL Teachers for Students"*

The results show that out of 20 participants, SP3, SP4 and SP7 mention "memorization exercises"; SP2, SP11 and SP14 lay emphasis on repetition drills, while SP10, SP15 and SP17 stress on cooperative notetaking as language learning strategies they do experience in the classroom by their teachers. As an example, SP7 claims that their teacher conducts memorization to help them develop the ability of native pronunciation, and SP14 explains that they undergo repetition drills in order to develop the fluency ability. Furthermore, as an example, SP17 explains their teachers conduct cooperative notetaking in order to help them overcome some of their naiveties in cross-cultural interaction.

#### *4.8. Theme 8: "Textbooks Used for EFL"*

All the 20 student participants reference a textbook titled *Straightforward*. For example, SP15 says that their "textbook seems unique in EFL classrooms", but they lack information on cross-cultural awareness. Moreover, SP8 and SP14 point out that the textbook *Straightforward* in EFL class provides them with few information on linguistic knowledge that comprises vocabulary, grammar and pronunciation and not on cross-cultural pragmatic knowledge. Likewise, SP1 and SP6 state that although the textbook *Straightforward* is used daily during contact hours, it does not contain cross-cultural knowledge and activities. For SP17, she observes nothing concerning intercultural awareness when reading *Straightforward*.

#### **5. Discussion**

The findings in this study reveal that the gaps related to these linguistic challenges are due to the inadequate knowledge of cross-cultural pragmatic competence for creating complete success in social communication. The study explores the dynamics of EFL learners, the key challenges facing them and raises concerns on the personal views and experience of teachers and students regarding cross-cultural pragmatic competence. Hence, the prime objective of this paper is related to the enhancement of the cross-cultural pragmatic competence of EFL learners in classroom circumstances.

The findings indicate that the concerns raised by the EFL teachers and students seem significant in this paper and must be addressed in promoting the language proficiency of the EFL learners. The findings reveal the core points of students' performance on the comprehension of cross-cultural pragmatic competence with regard to gender, language groups, the field of studies and age. At the gender level, the EFL female students perform slightly better in cross-cultural pragmatic competence than their male counterparts. Findings reveal on a language group basis that the EFL students of Anglophone background perform better in cross-cultural pragmatic competence than their counterparts with Arabic, Francophone and Turkish backgrounds. Sayahi [36] confirms that the teaching and learning

of EFL in the above mentioned countries is better comparing to the performance level of language competency between Arabic–French and Arabic–English learners in Maghreb countries such as Algeria, Libya, Mauritania, Morocco and Tunisia, all of them with Arabic backgrounds. The findings concerning the field of studies depict that the students of the social sciences perform slightly above their pure sciences counterparts. Explaining the core point on positive and social sciences, Morgan [26] looks at pragmatism as an important domain of philosophy that entails problem-solving experience relative to gaining cultural knowledge from human activities. Furthermore, findings in this paper on age gaps depict the maturity of ages between (25–34), and it is slightly significant in the performance on comprehension of pragmatic competence. By this perception, Polovna [37] asserts that it is important to usher language learners into new platforms regarding age and experience of cultural communication for better interpretation of connotative utterances. Findings imply that an average number of the EFL student respondents agrees that "cross-cultural pragmatics is important in EFL context because it creates the awareness on how the same words can have different meanings in different contexts". In this concept, the two researchers spell out that the prospective fields of misrepresentation and failures in social communication are destined to materialize during the interactive communication among EFL learners in a social context [38].

The findings show a positive impression that cross-cultural pragmatic competence is essential in the EFL context because it provides an opportunity for EFL learners to communicate effectively in social contexts. For Hall et al. [39], the EFL instructors must incorporate pragmatic topics well relative to classroom scheduled book-topics such as topicopening agendas, topic-sustaining matters, topic-maintaining opinions, topic-condemning acts and topic-closing's dialogues that involve expressions such as the following: "What's up now?"; "What else?"; "What's going on?"; "What's wrong?"; "What next?"; "I am overwhelmed"; "We are delighted"; "Enough of that"; "How far?"; "Your Excellency"; and "Your highness".

Nevertheless, the findings concerning the key challenges confronting EFL students on cross-cultural communication in Table 6 shows that the majority of EFL respondents underline communication difficulties with regard to the wrong pronunciation of words, unfamiliar language accents, bias interpretations, attitudinal complex, colour differences and language barriers, use of difficult words, concerns raised on racism, cultural inferiority sentiments and unfamiliar terminologies as parts of cross-cultural pragmatic challenges. Pedagogically, the language philosopher Yule [40] describe these key challenges of mistakes or errors as "bad speech or bad language" because they are emotionally judged, and judgments in reference to language principles and terminologies become a violation of norms and rules in a target language.

The findings on EFL teachers' personal views and experience raise issues on classroom tasks such as dialogue, role-play, debate, repetition drills and presentation that are essential for helping EFL students overcome stage fright relative to cultural differences. For example, in the interview section, the second male teacher respondent (MT2) adds that the communicative tasks allow students to develop their speaking abilities for their everyday interactions with one another. However, from the pedagogical point of view, Tsutagawa [41] proclaims that despite all the classroom language tasks conducted by EFL instructors, role-play becomes the centre of attraction of teaching activities for developing the language skills of EFL students with respect to performing credibly in a cross-cultural pragmatic context.

Moreover, on students' personal views and experience on cross-cultural pragmatic competence, the findings disclose that the complaints of students on cross-cultural challenges in EFL context are due to different cultural backgrounds and beliefs of others that seem difficult to be conversant in within a short time.

Another important issue is related to the single textbook *Straightforward* that is used and seems inadequate for developing pragmatic competence among EFL students in classroom situations. Consequently, Dendenne [42], Matsuoka et al. [43] and Kafi et al. [44]

caution contemporary textbooks and ascertain that textbooks are among the most important instructional materials that have to be considered as the vital backbone for second or foreign language learning. In this situation, the study suggests that cross-cultural pragmatic competence should remain the basic framework for natural interaction with one another, especially in classroom circumstances involving appropriate textbooks and activities such as the reading of authentic materials, cultural-oriented materials and cross-cultural debatable topics. Moreover, this paper draws our attention on the communicative language teaching approach of Wu [27] that seems remarkably significant in enhancing pragmatic competence of EFL learners. The approach is student centred and can permit students' engagement in communicative activities.

Therefore, the findings of this paper arouse our attention to situations concerning cross-cultural pragmatic competence and highlight some suggestions that seem significant in enhancing the pragmatic competence of students regarding foreign language teaching and learning. Furthermore, the literature review related to this paper seems significant since it emphasises the interpretation of words and expressions that consider the norms and values of a target language in social contexts. In view of this, Norton and Toohey [45] confirm that learning grounds, such as in the classroom environment, must be one of the considered centres for acquiring language skills and cultural knowledge.

#### **6. Conclusions**

The findings of the study indicate that EFL students are not furnished with crosscultural pragmatic competence in the classroom and must be provided with classroom tasks to develop both explicit and implicit communicative competence for a sustainable English learning environment. They also lack textbooks comprising pragmatic knowledge. The study further reveals that EFL teachers' pay little attention to cross-cultural pragmatic knowledge and classroom activities in the EFL context. These situations need to be addressed in order to help the EFL learners in enhancing cross-cultural pragmatic competence and to communicate meaningfully in a social context. It is important to state that cross-cultural pragmatic competence can bring to perfection the English language skills of EFL learners.

In the midst of cross-cultural interaction, a communicator speaking a non-native language or L2 may easily create language miscommunication, and this situation becomes the central tendency in day-to-day activities inside and outside of classroom circumstances. For instance, taking "coffee or tea" in a breakfast context may be a "dessert after a meal or before the meal" for one's culture, and it may bring misunderstanding and disappointment to some other communicators when it is defined as "breakfast only" due to some cultural norms and interpretations. As proof of this, the paper tends to raise the tactical awareness of ELF teachers and learners on cross-cultural pragmatic competence. Hence, on the pedagogical point of view, there is a need to consider the communication terms for values and norms by the calibre of words and expressions used amidst target language learners. Based on the results of this study, there is a need to caution EFL teachers and learners to be circumspect when dealing with cross-cultural shocks among foreign students in classroom circumstances, and when things do not happen the way we expected or thought, one must be flexible, adjustable and tolerant of overcoming cultural emotions inside and outside school environments.

#### **The Pedagogical Implication**

The pedagogical implication of this study is to intensify the awareness and the importance of cross-cultural pragmatic competence in a target language, especially in the EFL context. Wu [27] asserts that the basic principle of communicative Language Teaching (CLT) must be virtual "Learner-Centred", involving communicative activities such as group discussion, debate, dialogue, role play, reading practices and presentations. Furthermore, Jie [14] and Kecskes [46] suggest that the development of pragmatic competence must be a motivation for cross-cultural communicators to improve upon their communicative clarity and fluency in EFL or ESL settings. In this manner, EFL teachers are called upon

to develop EFL students' motivation for intercultural relations through cultural related tasks in classroom environments for socialization as well as certification for globalization and partnership. Rumyantseva et al. [47] also highlight that, for shaping sustainable language learning environment, forming multicultural personality through language leaning is crucial.

Consequently, the findings in this study alert us on the urgent need for teachers to abreast themselves with the knowledge of cross-cultural pragmatics and make it teachable, learnable and achievable. Hence, the unpredictable occasions may engage L2 learners in complex scenarios lacking practical information in communication. For this reason, Zakaria et al. [48] and Walsh [49] caution teachers to be circumspect in handling issues pertaining to EFL learners and claim that the high-level compassion of communicative competency is found within the interactional gatherings of cross-cultural communicators on the social platform.

Therefore, because of the findings in this paper and for the tendency of global connections, some positive recommendations on cross-cultural activities must be observed by teachers for the enhancement of EFL learners' pragmatic competency. In this manner, Alemi et al. [50] assume that English textbooks propagate cultural knowledge for L2 learners, and teachers must locate cross-cultural information on internet journal language, culture and society for classroom's achievements. Apart from classroom tasks and other curriculum, this paper suggests some extracurricular activities such as cross-cultural trips for exchange programs, cross-cultural games and festivities, cross-cultural quizzes and debates, cross-cultural theatrical arts, drama or short sketches and cross-cultural expositions. These extracurricular activities seem practically important and must be observed with regard to cross-cultural awareness, sentiment and generalization.

#### **Limitations**

The limitations in this paper are important. The questionnaire at the interview questions has been tested on a pilot basis after consulting three experts in this field of research. Moreover, the pilot study has been a step in the right direction to validate the components of the questionnaire and the interview questions, such as the following: the content, the instructions, sentence structures, and to determine the reliability of the questionnaire and the interview questions. The questionnaire and the interview questions of the study have been validated by administering them on a pilot basis to twenty five (25) students of first-year undergraduates comprising 13 females and 12 males, all of them from CIU (Cyprus International University) who voluntarily responded to the questionnaire. Five (5) teachers and (10) students responded to the interview questions, and approximately 40 to 50 minutes was spent on each respondent. The pilot tests have been conducted on 3 and 4 April 2019. Furthermore, in relation to the study concerning cross-cultural pragmatic competence, the pilot study places into consideration the different continents involving respondents from Africa, Asia, Europe, America and others to establish some facts concerning the validity of the instruments for data collection. Furthermore, the final questionnaire and interview questions were structured based on the reliable responses given by the pilot-test participants. The limitation on the study population of first-year CIU EFL students presents more opportunities for further research that can involve more participants from other sectors of language teaching and learning in Northern Cyprus and elsewhere in regards to cross-cultural pragmatic competence.

#### **Further Research**

Researchers can conduct studies on the following: (1) motivational factors relating to cross-cultural pragmatic competence; (2) specific classroom activities for the enhancement of cross-cultural pragmatic competence in EFL context; (3) interactive strategies for developing cross-cultural pragmatic competence in EFL context.

**Author Contributions:** Conceptualization, B.B. and F.S.; Methodology, B.B. and F.S. and B.M.; software, B.B.; Validation, B.B., F.S. and B.M.; formal analysis, B.B., F.S. and B.M.; investigation, B.B.; resources, B.B. and F.S., and B.M.; data curation, B.B.; writing—original draft preparation, B.B.; writing—review and editing, B.B., F.S., and B.M.; visualization, B.B.; supervision, F.S. and B.M.; project administration, F.S.; funding acquisition, B.B., F.S. and B.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Ethical review and approval were waived for this study, due to reason that it did not give any harm to animals and human beings.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The study did not report any data.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A. A Questionnaire on Cross-Culture Pragmatic Competence**

Please give answers to the following questions for research purposes and all information will be treated in the strictest confidence and do not enter your name.

Your collaboration is cordially important in this research by providing answers to the questions below. Thank you.

PART 1

Demographic Information:

Gender: 1. Male ( ) 2. Female ( )

Language Group: 1. English ( ); 2. French ( ); 3. Arab ( ); 4. Others ( )

Field of Study: 1. Physical Sciences ( ); 2. Social Sciences ( )

Age: A. 17–24 ( ); B. 25–34 ( ); C. 35–44 ( ); D: 45 and above ( )

Continent: A. Africa ( ) B. Asia ( ) C. Europe ( ); D. America ( ); E. Others ( )

PART 2

SECTION A: Relevant to Research Question 1

Students' level of competency on comprehension of pragmatic competence in regard of their gender, language group (mother tongues), fields of studies and age?

Please tick one letter from the options (A, B, C and D) that seems to give the correct answer of each item.

1 = A; 2 = B; 3 = C; 4 = D

1. Jonas is from Ankara. His friend Albert has recently moved to Ankara.

Jonas: "How do you like Ankara so far?" Albert: "I love it."

\*What does Albert possibly mean?

A. He thinks Ankara is a normal city.

B. He is yet to see much of Paris.

C. He thinks the city needs more great changes.

D. He likes Ankara and enjoys living there.

2. Paul and Peter are professors at a college. They are chatting about a student called Ibrahim.

Paul: "How did you like Ibrahim's paper?" Peter: "Well, I thought it was well-typed."

\*What does Peter probably mean?

A. He did not like Ibrahim's essay.

B. He did not really remember Ibrahim's essay.

C. He thought the Ibrahim's topic was interesting.

D. He liked Ibrahim's essay quite a lot.

3. Mary and Margaret are secondary school teachers. Mary is considering teaching

Chemistry, but Margaret has heard it is really challenging.

Margaret: "I don't know, but people say it's really involving".

\*What does Margaret probably mean?

A. She assumes the course may not be very difficult.

B. She thinks Mary can teach that course


literature this semester.

Julie: "How are you doing in French literature?"

Ben: "So, did you watch that basketball game yesterday?"

\*What does Ben probably mean?

A. The content of yesterday's lesson was completely irrelevant to a basketball game.

B. He is doing badly in French literature.

C. French literature is like an easy game for him.


Frank: "The car is almost out of fuel."

\*What does Frank probably mean?

A. He wants Amidu to get fuel for the car.


Felicia: "So, is the rent high?"

Abraham: "is the Pope a Catholic?"

\*What does Abraham probably mean?

A. He does not want to talk about the rent.

B. The rent is high.


Mr. White: "How do you like what you're eating?"

Mr. Brown: "Well, let's just say it's colourful."

\*What does Mr Brown probably mean?

A. He thinks it is important for food to look good.

B. He likes the food.

C. He wants Mr White to try something colourful.


Sheila: "Thanks, you've been terribly helpful."

\*What does Sheila probably mean?

A. Mr Smith helped her a lot.

B. Packing the books was really important as it needed special care.

C. Mr Smith is weak.


Simon: "You know. I've been trying to know if you went out with Paulina?"

David: "Paulina is not really my type."

\*What does David probably mean?

A. He is not sure of his emotions.


D. They did not go out.

10. Mrs. Alibah runs into her colleague teacher Mrs. Ali who recently had a promotion interview.

Mrs Alibah: "By the way, did you get that promotion you applied for?" Mrs Ali: "Good God, I'm so tired of this cold weather."

\*What does Mrs Ali probably mean?

A. She does not want to talk about the promotion interview.

B. She is bored with the promotion interview.

C. She did not understand Mrs Alibah's question.

D. She could not attend the interview because of cold weather.

Answers to comprehension test on pragmatic competence. Questions (11–20)


SECTION B: Relevant to Research Question 2

Perceptions of EFL learners in Cross-cultural Pragmatic Competence.

Note: Definitions of Terms

\*Pragmatics is the study of meanings of words and expressions in a context. It is the study of the use of linguistic signs, words and sentences or expressions in social contexts.

\*Cross-cultural Pragmatics is the study of linguistic signs, words and expressions, accepted by language users from different cultural backgrounds.

\*Cross-cultural Pragmatic Competence is the ability to understand and interpret words and expressions to convey meanings under social context.

Note SA: Strongly Agree; A: Agree; N: Neutral; D: Disagree; SD: Strongly Disagree

Please tick only one letter from the options (A, B, C and D) that seems suitable to each item.

Note: 1 = SA: Strongly Agree; 2 = A: Agree; 3 = N: Neutral; 4 = D: Disagree; 5 = SD: Strongly Disagree.


1 = SA: Strongly Agree; 2 = A: Agree; 3 = N: Neutral; 4 = D: Disagree; 5 = SD: Strongly Disagree


Cross-cultural Pragmatic Challenges


Part 3: Relevant to Research Question 4.

Interview Guide Questions.

Personal views and experience of CIU EFL teachers and students on cross-cultural pragmatic in classroom circumstances.

Interview Guides for Teachers.

Note: Cross-cultural communication is the communication that occurs between people who may have different cultural perspectives of interpretation of words and expressions (Stringer and Cassidy, 2009).

Note: Cross-cultural Pragmatic Competence is the ability to understand and interpret words and expressions to convey meanings in social context.

Interview Questions.


(4). To what extent do the textbooks you use in EFL class provide information on crosscultural pragmatics?

Interview Guides for Students.

Note: Cross-cultural communication is the communication that occurs between people who may have different cultural perspectives (Stringer and Cassiday, 2009).

Note: Cross-cultural Pragmatic Competence: is the ability to understand and interpret words and expressions to convey meanings under social context?

Interview Questions.


#### **References**


### *Article* **Pandemic-Induced Qualitative Changes in the Process of University Studies from the Perspective of University Authorities**

**Vida Navickiene 1, Valentina Dagiene 2,\*, Egle Jasute 2,\*, Rita Butkiene <sup>3</sup> and Daina Gudoniene <sup>3</sup>**


**Abstract:** The pandemic COVID-19 period in education has brought many challenges to all organizations. The activities of the higher educational institutions are being affected and the situation may last for a long time. Under the current circumstances, it is important to shift to distance learning through online processes and improve educational processes at all organizational levels. Institutions have to ensure successful distance or remote learning process by identifying their opportunities, meeting challenges, and establishing the sustainable quality factors for remote or distance learning. This study aimed at identifying the pandemic-induced qualitative changes in studies that have occurred at the levels of university authorities, lecturers, and students. Universities of Lithuania were taken as a case study. The novelty of the research lies in the fact that the focus of analysis is not on the negative effects of the pandemic observed in higher education studies but on finding positive qualitative changes that are also of importance to future studies. Phenomenographic qualitative research strategy was chosen in the research and 15 in-depth semi-structured interviews with experts in university studies were conducted. Seven categories were distinguished during the research representing qualitative changes in studies at three levels—authorities, lecturers, and students. The discussed levels seemed to have a mutual effect on each other. The external motivation of leaders and the support and establishment of work and online study conditions encouraged both external and internal qualitative changes in studies from the perspective of lecturers as well as students.

**Keywords:** higher education; qualitative changes; distance learning; university authorities; lecturers; students

#### **1. Introduction and Background**

Due to the pandemic situation, educational institutions have met many challenges regarding their organization and implementation of distance learning. This has become particularly apparent during the COVID-19 pandemic when higher education institutions were forced to transform their studies and to make them available online. In this situation, the universities have played an important role in supporting stay-at-home society and have been a valuable addition to their productive home environments. Under such rapid changes, the universities should be able to monitor the quality of their studies [1]. The success of studies depends on the decisions of university authorities and lecturers and on their possessed experience and motivation to ensure high quality of transformations in studies. Due to a sudden pandemic and a shift to online learning, many lecturers have not had adequate time to adjust to the new teaching platforms [2,3]. "Educators, learners and institutions have had to adjust to these changes and manage the various increased demands related to workload, new practices and external regulatory authorities" [4]. The

**Citation:** Navickiene, V.; Dagiene, V.; Jasute, E.; Butkiene, R.; Gudoniene, D. Pandemic-Induced Qualitative Changes in the Process of University Studies from the Perspective of University Authorities. *Sustainability* **2021**, *13*, 9887. https://doi.org/ 10.3390/su13179887

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Javier Puente and Leandro Almeida

Received: 9 August 2021 Accepted: 30 August 2021 Published: 2 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

pandemic has stimulated changes and transformations; therefore "a new and transforming attitude is being created, which will change the fundamental essence of universities" [5]. Education is changing its transmission methods, and e-learning will undoubtedly become a vital strategy moving forward [6].

Due to the pandemic, many educational institutions have started to search for effective and alternative ways of learning by using online platforms that universities provided, such as Moodle, Microsoft Teams, Zoom, or others; however, many challenges were encountered, especially in delivery of STEM subjects. Many researchers analyzed various kinds of tools to facilitate learning environments when using online systems, for example, Tencent Meeting System [7,8] or Topic Analysis Instant Feedback System [9]. The number of online course offerings has been increasing significantly, but recent trends have shown a steady progression in the normalization of online studies. There have been periods when higher education was fully transformed and moved online. However, it is not clear whether this transformation produces positive study outcomes.

Over the last years, numerous research studies embracing different areas have been conducted where researchers have analyzed challenges evoked by the pandemic, emerging difficulties or experienced negative effects. Some of them have particularly emphasized and criticized instances where distance learning has prevented students from obtaining practical skills and adapting to the labor market [10–13]. Others have noted that some academics themselves lack knowledge of information technology and online teaching [4,7,14,15] and that teaching and learning resources adapted to pure online education are scarce [1,16,17]. Some others have emphasized problems related to poor mental health, which emerges due to social exclusion and uncertainty of the future situation [18–20]. Adjusting studies to online learning has posed various challenges to educational institutions. The main question that is of interest to many educators and education policymakers is whether online learning is better and more effective than class-based learning [10,21–24]. It can be stated that contact and distance learning will always have their supporters and opponents. However, one or another kind of learning is only a form filled with various tools. Thus, the quality of managing and using such tools depends on willingness to act and the decisions of participants in tertiary education: university authorities, lecturers, and students. On the basis of research results the article attempts to show positive qualitative changes, most of which were created through appropriate decisions of university authorities. The research results can be used in other higher education institutions as good examples of how to improve the quality of the study process.

The novelty of our article is related to its analysis of what qualitative changes were preconditioned by the COVID-19 pandemic. Thus, the article presents the results of research, which disclose the positive consequences of the pandemic. The emerging challenge was accepted with difficulty. Initially, there was a lot of uncertainty, fear, and unawareness of how to act. At that stage, the leadership of university authorities, timely decision-making, and close collaboration with the academic community were of utmost significance. Higher education as an institution is distinguished by slow change compared to other organizations. This is partially due to subordination in decision-making: faculty administration, rectorate, senate, and the university council. However, the most considerable changes during the pandemic have been preconditioned by the determination of the academic community and their internal desire to act. Despite the pandemic situation, which has created many issues for higher education institutions, some positive effects and newly raised opportunities have been recognized. Several innovative approaches and tools for learning online have been developed. Study resources have been revised, restructured, and adapted for students' self-directed learning. Academics and students have been fostered to improve their educational and digital competences. These changes and opportunities have created a space for innovative thinking and innovative solutions [5,12,25–28]. The e-learning quality is the most significant aspect of students' e-learning, constituted by the quality of e-learning tutors, the course material, and the e-learning administration and support services [29]. Although this situation is in line with the vision and mission

of future learning in the era of industrial revolution 4.0 and community 5.0, it still has advantages and disadvantages. However, the current freedom cannot be interpreted as unlimited freedom in learning [30]. Universities must develop innovative ways to deliver teaching without compromising on quality [13].

The pandemic of COVID-19 has forced higher education institutions to rapidly adapt to new conditions and ensure the transition of the study process into distance education. This type of transformation has led to qualitative changes in the study process and opened new opportunities gradually emerging from a stressful and unstable situation. The majority of researchers support the idea of online studies due to a number of reasons, which can be referred to as qualitative factors, e.g., the e-learning systems offer many advantages and compensate for the weaknesses of the traditional learning methods. For example, there are new approaches and tools for capacity development [21,31]; online learning can reduce costs without reducing the quality of learning [32]; and high-quality participation can improve the breadth and depth of student's learning [33]. Therefore, higher education institutions have to adapt their study programs to respond to the needs of the transitional period [34].

Although education has globally transformed and moved online due to the pandemic, it is unknown whether this transformation produces more positive teaching and learning outcomes or negative issues. Therefore, this research problem is formulated as the following question: what qualitative changes in studies were preconditioned by the pandemic from the perspective of the university?

#### **2. Research Design and Methodology**

All the Lithuanian universities terminated studies due to the first wave of COVID-19 pandemic on 16 March 2020. The majority of universities had a two-week break, which was used for preparation to implement distance studies. The biggest university was the only one to continue studies without any preparation. Such a different path to distance studies had a certain different impact on the quality of studies (more of negative character), but the research shows that later these differences disappeared. Thus, the research also included the universities that chose a different strategy of transition from contact to distance studies.

Lithuania, a small country with a population of over 2.5 million, joined the Bologna Process in 1999 and became a member of the EU in 2004. Therefore, its national system of higher education is in line with other countries of the Bologna Process. The higher education system in Lithuania is of binary character and consists of two types of higher education establishments, i.e., universities (study duration—4 years) and colleges (study duration—4 years). The first university was established in Lithuania in 1579 and the foundation of the second one was laid much later, in 1922. The majority of Lithuanian universities started their activities in the middle of the 20th century and, therefore, higher education in Lithuania is rather young. Aiming to increase accessibility of higher education, colleges were established in 2000. However, in contrast to some European countries, the degrees of master and doctor can be obtained only in universities. Thus, these institutions offer broader education not only in terms of content but also in forms. Furthermore, colleges in Lithuania are more focused on implementing regional educational policy, whereas universities respond to national needs. Colleges are established mainly in regions and universities are concentrated in cities.

It should also be mentioned that in 2019 the requirements for external assessment of study programs changed and since then considerable attention has been allocated to systemic improvement of teacher competences and student support. This is also reflected in the research results because experts speak a lot about improvement of teacher competences and centers established in institutions that provide support to them.

There are 11 state universities in Lithuania. They can be divided into three types (as provided for in the national legislation)—firstly, classical universities (2), which implement studies in all the study areas and are the oldest universities in Lithuania; secondly, universities of broad profile (5), which carry out studies in several areas; and thirdly—specialized

universities (4), which target the arts, music, sports, and military sciences [35]. Specialized universities are small in terms of numbers of students and academic staff members as well as the structure of their faculties. Therefore, the volumes of data they can provide for the research are not big. Some study fields offered by them are also available in universities of broad profile.

The research design. The research was carried out from 15 to 30 June 2020. This time period was chosen for the research as it allowed investigation of the challenges posed to the study process by the pandemic, problems encountered by the participants in the study process, and decisions of authorities made in the unstable situation, as well as identification of qualitative changes predetermined by the pandemic.

The research question: what qualitative changes in studies were preconditioned by the pandemic from the perspective of the university? A phenomenographic strategy was chosen to answer the research question. The methodology of phenomenographic research calls for diverse experience of research participants. Therefore, informants with various experiences were included in the research. Conducting the research, criterion sampling was used for selecting informants. The main criteria considered were as follows: (a) informant is from one of the biggest Lithuanian universities (size); (b) from different types of university: classical university and broad profile universities from different towns (type); (c) leaders are available in the university, whose functions are related to the management of studies (taking highest positions in the university, then leaders of lower rank (deans of faculties of humanities, social, technological, engineering sciences, heads of other divisions, who were chosen to make sure that the position of authorities is the same in terms of the range of qualitative changes influenced by the pandemic); (d) informants—experts in their field with at least 7 years of managerial experience related to the analyzed phenomenon (experts, working experience). Therefore, informants in the article are referred to as experts (see Table 1). The triangulation of experts aimed to ensure the validity of the research.



All the experts were master's degree holders, and the majority had a doctor's degree.

The process of studies depended on decisions made by authorities; they managed the whole process of university studies and received information from all the lower divisions. Moreover, vice-rectors saw the situation all over Lithuania because they belong to the Conference of Rectors of Lithuanian Universities, whose members made strategical decisions at the national level during the pandemic. Thus, the experience of such experts is very important for solving the research problem. They were chosen as critical cases, whose attitudes, actions, and activities predetermined further qualitative changes in the study process. Qualitative research was conducted, which included 15 semi-structured in-depth interviews. The verbal consent to take part in the interview was received from all the experts. Conducting the last interviews, the research became saturated with information because experts started repeating information. Thus, it was pointless to take interviews with more experts.

The interviews with experts were done using the Zoom video conferencing platform. The interviews lasted from 45 min to 75 min. The total duration of all the interviews was about 940 min. The received data were decoded and about 130 pages of decoded material was received. To ensure the validity of interviews, they were carried out by two researchers and the received data were processed by three researchers. MAXQDA was used for interview transcription analysis.

The phenomenographic research not only identifies the categories that describe the phenomenon under consideration but also reveals hidden, tacit meanings and presents their interrelationships. The distinguished categories form a hierarchical horizontal order. Data analysis identified descriptive categories and the outcome space of the concept expressed by a network of logically related, hierarchically organized, and systematized categories [36,37]. Authors [38] present the space of results of the investigated concept as a logically structured set of different ways of experiencing a phenomenon. The space of results expresses experiences and discloses internal relations among described categories. The links of horizontal levels that are of the same level are applied in the research [39]. The analysis of research results allowed distinguishing seven categories, which characterize study-related qualitative changes resulted in by the pandemic (see Table 2). The research participants were authorities of universities, who pointed out changes they noticed or even influenced in universities. Conducting the analysis of interviews and applying the methodology of Kinnunen et al. [40] revealed changes at three levels: authorities, lecturers, and students. The table was devised following the methodological approach of Kinnunen et al. [40].

The research data were obtained in line with research ethics and summarized information on respondent affiliation to the city, the university she/he represented, or the current position held.


**Table 2.** The categories characterizing study-related qualitative changes resulted in the pandemic at the levels of university authorities, lecturers, and students (according to Kinnunen et al., 2007 [23]).


**Table 2.** *Cont.*

#### **3. Research Results**

Our research is focused on three key factors identified in education and raised in the pandemic period, i.e., (1) qualitative changes at the level of authorities, (2) qualitative changes at the level of lecturers, and (3) qualitative changes at the level of students.

#### *3.1. Qualitative Changes in Studies at the Level of Authorities*

**Changes in the form of authorities' work.** During the pandemic, the global shift to distance learning also affected the work of university leadership. Due to the pandemic, they were forced to work in other forms, and to chair and hold online meetings of university authorities. *"One of the good experiences of this is that we clearly understood that organizing administrative meetings remotely has numerous advantages. The ministry has finally realized how much time it saves on its meetings [...] after the quarantine ended, this practice remained and now almost half of the meetings, if not more, are organized remotely. The quality of solutions has not really diminished"* (8VR). Meetings and consultations remotely using a video conferencing platform have led to closer communication between the central university management and faculty administration. *"I really appreciate communication in the community because we seem to continue to have more frequent meetings, for example with vice-deans"* (1VR). Such change in the form of work enabled authorities to see that distance work is possible. Since the pandemic delimited everybody and divided the university into many local places, the leaders had to find ways and to assume leadership mobilizing the community for common work.

**Establishment and maintenance of mutual/parity relation-based relationship between the authority and the university community.** Mastering video conferences encouraged communication and collaboration with all of the community. The university leaders did it very actively and at different levels. Quality changes preconditioned by the pandemic period of COVID-19 include the opportunity to cooperate in decision-making and to maintain communication between management and the community. This is emphasized by all the experts and they all list different links, means of communication, or goals.

First, the communication occurred between the different levels of leadership: central administration, faculty deans, vice-deans, and heads of departments: "*Of course, teamwork* *was important [...] Discussing a variety of things, sharing good practices, what works well, what could be done differently*" (2SD). "*I very positively evaluate this communication of community because we are likely to continue having such meetings more often (e.g., with vice-deans.*" (12VR).

Secondly, it can be assumed that communication between the university leadership and lecturers was promoted because of the necessity of managing the situation and explain to them "*why particular decisions have been made*" (1VR). This necessity stimulated positive changes in collaboration. "*There have been a lot of efforts to keep the communication with the lecturers on all issues. They are the people that influence the study process and its quality the most. There has been a great deal of effort to inform them about all sorts of different decisions. Perhaps it was not always done directly, but rather informing through the heads of departments or deans*" (3DD). Communication takes place to not only explain decisions and exchange best practices, but also to ensure a positive psychological climate and maintain team spirit and established traditions. It was also important to stay in constant contact with students, who experienced challenges of distance learning during the pandemic. While communicating with students, it is important to explain not only the decisions that have already been made but also to communicate clearly, explain things that are not clear, and let them know when to expect further information. They were told how "*distance lectures work and how they should behave during them*" (1VR) as well as "*principles of academic integrity*" (13SD). In addition, various recommendations have been drawn up for distance learning. During the pandemic the gaps in communication with students were identified, communication became more consistent and targeted, and new communication channels were opened, which should remain and be maintained in future as well.

It should be emphasized that the biggest change, the transition to working remotely, and the other factors of change listed above were a consequence inspired by the collaboration of the academic community. The experts acknowledge that this is also one of the drivers of change, as everyone felt focused and saw the close focus of the academic community in trying to manage the situation. "*We learned that our lecturers are actually advanced. They started sharing additional tools and instruments and used them creatively. There were certainly those who then voluntarily sent their own materials to other faculties*" (14DS). 2SD: "*The situation showed that the academic community is focused and that even in uncertain and extreme conditions, the university is able to function, to be able to ensure the study process*" (2SD). Thus, challenges encountered during the pandemic not only united the community for the common goal to smoothly move to distance learning, to ensure quality studies, but also encourage collaboration outside the faculty and entering the space of the whole university.

Third, quality studies and their success during the quarantine were determined by the fact that the central or faculty management responded to opinions of lecturers conducting distance studies, rapidly addressing the problems and carrying out prevention. "*After each lecture, lecturers used to write to us, the Directorate of Studies, the vice-rector or the dean, or the Academic Support Centre about the difficulties, what kind of training they wanted or what problems were, or even shared good practices if the lecture succeeded*" (8VR). This provision of teaching experience to the central management ensured the preparation of temporary documents regulating the study process, which covered a wide range of study programs offered by different faculties. It can be stated that the synergy of collaboration emerged due to the pandemic, which introduced positive changes to the further process of studies. The documents regulating studies were not only revised again but also grounded on lecturers' practical experience. Thus, confidence between the leadership and lecturers was created or enhanced.

**Provision of academic and technical support to the community.** The research shows that the leaders provided academic and technical support, which affected the most important quantitative changes—improvement of lecturers' competences. First, the results of the study show that more than half of the experts emphasized the creation of an additional online website/section on the website that publishes all necessary, constantly updated information. "*Methodological and technical information for lecturers, advice for students, the information provided to scientists, administrators and all members of the community. Topical issues*

*are systematically presented, updated so that people can have a single point of access*" (11SD). These qualitative changes in study organization are of long-term character and the community will be able to use them in the future as well.

During the pandemic, two types of training courses were organized: on tools of distance learning (*distance work training for lecturers)* and didactics (*provision of didactic)*, or how to use these instruments in the study process. Most experts stressed the importance of providing lecturers with initial and then continuous support by teaching them how to master distance learning tools. According to the experts, "*The E-learning Technology Centre conducted training weekly, sometimes even twice a week, on how to use tools, also prepared instructions for using Zoom, Teams*" (13SD); "*There have been specific training on how to prepare tests, how to make some recommendations regarding final theses, etc.*" (7DT).

In addition to training and e-mail information, lecturers were given various recommendations to be used when preparing or conducting distance lectures and these recommendations were broad in scope. "*Afterwards those instructions and short videos were very helpful*" (4DT).

Thus, qualitative changes induced by the pandemic had a long-term effect on lecturers' didactic and ITC competences, positively influenced the work of students, and enhanced the lecturers' self-confidence.

Maintenance, creation, or improvement of the technical base also brought a positive qualitative change in studies resulted in by the pandemic. The view could be taken that the university authorities had to reallocate the available resources and to allocate finances to the assurance of high-quality distance studies. Improving/developing a technical base is one of the most important quality factors. First, it was decided to purchase Zoom licenses for videoconferencing enabling lectures longer than 40 min: "*We understood the need and had to buy additional licenses abruptly to make better use of the lecture*" (12VR). Secondly, other necessary technical means were provided, and the need was learned from the faculties: "*Information used to be passed on and the faculty was told to identify the needs, how much of the hardware employees had including cameras, microphones, etc.*" (15DT). Third, some universities made decisions regarding additional rooms to record lectures for the purposes of the faculty: "*We bought and prepared two recording studies: one on the X Street, where we have a building and the other one right here, so it is easier for lecturers to record lectures*" (6DS).

**Supply of new forms of studies**. The pandemic-induced qualitative changes in the studies are also related to the changing structure of studies. The representatives of all universities stated that students have the opportunity to study in a different form of master's degree in the form of blended learning: "*That is why we have made a decision regarding master's studies from the first day of September next year (2021). Totally, 50% of the activities in the program can take place remotely. Two days of learning remotely, two days of regular contact"* (1VR). It should be noted that one university has opened up even wider possibilities by offering distance master's studies in the regions: "*Here we are talking about a regional master's degree, where it has already been planned to take place in a distance way. It was planned to take place in the regions directly, then rescheduled to remote*" (5DS). This form of blended learning provides a greater opportunity for postgraduate study and ensures better attendance at classes. It could be argued that postgraduate studies do not have a lot of laboratory work, but are research-oriented, so the necessary experiments for the final work can be performed on a more flexible schedule.

Secondly, other forms of study and access to them are open to foreign students. For many lecturers, especially in fields of technology and engineering, it is an innovation to use a hybrid form of learning. Prior to the pandemic and the quarantine, hybrid learning was mostly applied when a higher education institution has faculties in different cities. Quarantine has shown the possibility that, with the necessary equipment, it is possible to organize such studies even for students abroad. Foreign students choose this option because they cannot come to Lithuania due to the quarantine: "*We will give lectures to foreigners online, first-year full-time lectures and exercises in foreign English will be given to those students who have arrived in person, and the lecture or exercises will be broadcast remotely in*

*real-time. The lecture is attended online by those students who will not have the opportunity to come. Similar measures will be taken with the postgraduate studies*." (12VR). People in isolation could also take advantage of this opportunity.

Another opportunity for expanding internationalization is virtual /mixed mobility. It is an opportunity for students to acquire/improve their intercultural competences in other ways. "*<* ... *> these concepts will also appear in our new documentation. The European Commission has confirmed that such virtual mobility exists officially. This is the case when students do not go abroad but still study at a foreign university*" (13SD). Virtual mobility creates the possibilities and conditions for halting the decrease of international exchanges and reduces the threat of weakening intercultural competence.

#### *3.2. Qualitative Changes in Studies at the Level of Lecturers*

**Mastering new tools of distance studies.** The research shows that for many lecturers, working remotely was a completely new activity because of a widespread belief that the quality of the study or subject was suffering from it. The pandemic forced everyone to take up teaching activities in a distant way and opened up new opportunities, leading to innovative studies: "*In these two weeks, we have learned more than ever before*" (15DT). The experts say this is one of the things that helped to speed up a lot of other processes: "*Lecturers appreciated that distance learning and distance learning tools are really useful, they can be used with a purpose in mind. It is a great help to lecturers to organize the learning process in more ways and to allow students to learn in more diverse ways. Of course, distance learning should be just one form of learning that complements contact form and diversifies it*" (8VR).

The study has found that the development of high-quality distance learning materials has made a significant contribution to the successful transition from contact learning to distance learning and has resulted in long-time qualitative changes in studies: "*Records, additional tests, additional materials, various external sources improved the study quality*" (15DT). The Moodle environment used by Lithuanian universities has been greatly supplemented by subject material prepared by lecturers and adapted for distance learning: "*At this point, the amount of lecture material in Moodle has been strongly supplemented*" (4DT).

The use of virtual environments has revealed the possibility of improving the quality of learning content: "The study content will improve, since learning material will be uploaded to the virtual learning environment every year and you will be able to update and improve your course every year" (6DS). This has also been noticed by university management that is considering how to motivate lecturers to do so: "It is possible to motivate by offering additional vacation days, financial motivation, it can be encouraged in other ways, but distance courses need to be prepared." (9DS).

Mastery of new tools of distance studies in the future will contribute to more significant qualitative changes in the study process. Most experts emphasize that lecturers who have tried and already have distance learning experience will increasingly want to apply elements of distance learning in traditional studies: "*It is now clear that activities in a virtual learning environment will become more frequent*" (6DS); "*Lecturers plan to move at least part of their subjects to a distance form.*" (13SD). In order to ensure the quality, the lecturers will start preparing for the next semester in advance. It should be emphasized that the lecturers see an opportunity not only to organize the studies more by applying the elements of distance learning but also to use them for organizational work.

**Internal potential of lecturers.** Although the studies during the quarantine period posed many challenges, they also opened up new opportunities and even induced internal qualitative changes. They can be attributed to studies because lecturers' internal potential is directed to student teaching. One of the most important factors is the intrinsic motivation of lecturers. Various opportunities open up when confidence or readiness to deliver distance teaching is gained. This is primarily due to the fact that distance learning was tested and used by all lecturers working during the semester, and not individually as before the quarantine. The internal concept and desire of the lecturer promote the external study processes of the higher school. Thus, massive intrinsic motivation has been gained. Lecturers have gained confidence in their own strengths and in presenting distance learning opportunities for their subjects; trying new ways of teaching and communicating with students; and mastering tools. Such internal changes related to lecturers through mastered tools of distance studies will encourage qualitative changes in the process of studies: "*We will be using a lot more and bolder elements that we have tested and that work well. Lecturers will no doubt feel much more confident in those processes after testing both the study methods in a variety of ways and the assessment methods they saw paying off. Finally, they will definitely feel much more confident in organizing the process itself.*" (2SD). Thus, it can be stated that compulsory distance learning will reduce the gap between contact and distance learning, as it has encouraged the use of certain distance learning tools and even changes in certain levels or forms of study.

The obtained intrinsic motivation encouraged lecturers to exchange their good experiences. This was carried out either on a university-wide basis or within the faculty. Several experts emphasized their importance and positive impact on the studies. "*They organised such workshops, shared experiences or carried out remote training*" (8VR), "*We did internally, at the initiative of our own lecturers who are advanced in the use of technology*" (14DS). It should be noted that if training to work remotely usually took place at the beginning of quarantine on general issues, then the exchange of best practices was usually organized later when lecturers had already acquired experience and self-confidence. A common issue for all lecturers was how to organize student examinations; therefore, "*Many people were very active in sharing ideas with each other on how to organise examination and testing, whether to take tests or use other types of written tasks*" (15DT).

Empowered intrinsic motivation of lecturers provided opportunities both to lecturers and to students (even more to the latter).

#### *3.3. Qualitative Changes in Studies at the Level of Students*

Although there were a lot of discussions regarding the process of managing the situation by transitioning to distance education in the event of a pandemic, thus ensuring the continuity of the quality learning process, such types of learning during the quarantine have led to some changes. Many elements of traditional studies have also changed as a result of the emergency situation and the transition from face-to-face to distance learning. Qualitative changes at the student level are best seen through their inner changes.

**Internal changes in students**. The majority of experts point out that student attendance has improved. Several experts mentioned this positive change, although some of them said that it was true only for a while and others claimed that it was a permanent change. It is entirely possible that this was due to the specifics of the subject and the lecturer's ability to involve students in the active study process. However, almost everyone referred to good attendance: "*Increased attendance is considered to be a success. Attendance increased up to 100%, which was not the case during regular lectures*" (14DS). Only one expert saw a temporary improvement: "*After the study process was restored, the increased attendance lasted for about two weeks. In the third week, attendance fell sharply. Since distance learning was a novelty, everyone wanted to see how things were here. Thinking in the long run that distance learning would lead to better attendance, the answer is no because it was just an effect of innovation*" (12VR).

With regard to profound change, emphasis should also be placed on developing students' self-directed learning. Several experts noticed that the unexpected and even forced transition to distance learning encouraged students to learn more independently and develop learning-to-learn competences: "*Lecturers saw that students are forced to actually study in this way and they have a lot of work to do on their own*" (15DT), "*Students began to realize the importance of independent work. Until then, apparently, no one even thought, say, even in terms of hours, how much of that work should actually be. There was an opinion that everything should be learnt during the lecture*" (14DS).

#### **4. Discussion**

The transformation in tertiary education posed not only challenges but also promoted qualitative changes. New challenges associated with online teaching and learning will create a space for innovative thinking and innovative solutions [5]. The COVID-19 pandemic forced university communities to acquire determination and test distance learning and work. However, the most considerable qualitative changes in the uncertain situation were predetermined by the decisions of university authorities and the new forms of work tested during the pandemic. Decisions arrived at during the pandemic have an impact on the future and, for this reason, technical solutions also have to be considered [15]. Four main quality factors, which require attention, can be distinguished: economic, psychological, social, and environmental ones [32]. Changing the model of management "requires working with existing organisational cultures to ensure the collaborative participation of educators and learners throughout the process" [41]. Communication is one of the most significant factors that preconditions qualitative changes in times of uncertainty [41,42]. It is important to notice that the successful change was due to a focused and collaborative academic community across the university, and not just at the faculty level. Like in the case of face-to-face learning, conscious online learning communities should be established as well [16,43]. Belonging to a community is of utmost importance in distance learning, as is the development of meaningful relationships with one's instructors and classmates and having goals and interests similar to groupmates [44]. Students should be aware of reasons for such changes and how they can accept this [14].

The transfer of the study process to online learning meant that ways of maintaining relations between the lecturers and students, teaching and learning methods, etc. had to undergo changes as well. University leaders were forced to reconsider their actions to make them efficient longer than the transition (pandemic) period and to ensure the quality of this change in the future. The main role here is played by lecturers, who have to create a friendly environment for students through technology-based teaching [17,32,45,46] and maintain the quality of interaction between the lecturer and student [47]. This was one of the difficulties, which forced all the related lecturers to learn themselves. The attention was directed to online pedagogy [1,25]. Namely, here the university authorities made suitable solutions and provided academic and technical support to the community. A flexible and supportive online learning environment was able to fight against social isolation and increase social participation, but lecturers needed constant help to enable them to do this practically [16,23,43,45,48]. Lecturers' ability to work and teach online is one of the success factors and qualitative changes. For this reason, it is necessary to continue and even expand such training courses [34,49]. Students' assessment posed the most serious challenge both to lecturers and students [45]. Various training courses offered on time were very valuable and had residual value because they were attended by a big number of lecturers. Lecturers will need to engage in novel methods to achieve effective teaching outcomes, which may affect the quality of tertiary education [3]. Lecturers' ability to use the newest technologies in online teaching can be a significant factor, which can either encourage or hinder student and professor usage of e-learning. For these reasons, professors are expected to be more facilitators, collaborators, mentors, trainers, directors, and study partners and provide choices and greater accountability for students to learn [32].

However, changes have also brought some positivity as well. It should be mentioned that new opportunities that emerged have been initiated by the increased intrinsic motivation of lecturers. They have gained self-confidence and prepared (learned) to teach remotely. Therefore, deep reasons (intrinsic motivation of lecturers) primarily influenced the implementation and development of distance studies not only during the pandemic period but also after it. Lecturers will want to apply the elements of distance learning in traditional studies, when writing or defending their final project. They have also opened up opportunities and have shown that the work they put in during the pandemic is valuable and sustainable, as the quality of subject content can be improved annually.

Striving for qualitative changes creates a need to strengthen the IT platform by making necessary changes with respect to its continuous availability and uninterrupted services [14]. Users' personal factors have no direct influence on user satisfaction, while platform availability has the greatest influence on user satisfaction [7].

The transformation in tertiary education has encouraged the appearance of new study forms at universities or strengthening of already existing ones. "The move to on-line learning may stimulate an increase in blended and more accessible forms of education and teaching styles have had to change and this may have a lasting effect" [25]. The newly opened opportunity to organize various new courses only partially depends on the encouragement and support of university leadership. Lecturers' intrinsic motivation and acquired competences serve as the most relevant contribution. "Online education and its success lie in the participants and their qualities" [46].

COVID-19 has forced us to attempt to enhance student experiences and learning outcomes via online rather than proximate learning [5]. Students were dissatisfied with many things: accessibility, social, lecturer issues [50], and time costs, physical, and mental work in front of a computer screen [32]. That is why it is necessary to create a support group for members working in different areas to manage the situation in the university [17] and to provide constant support to students [49]. Distance education must be intelligently combined with face-to-face teaching because the student relationship with the professor is essential, and distance education, paradoxically, strengthens it [32]. Faculty and teaching assistants need to provide students with timely feedback, including online video tutoring and email guidance after class. It is necessary to adopt some measures to improve the degree and depth of students' class participation [33].

The main factors influencing user satisfaction with the online teaching platforms were system quality, interaction quality, service quality, and platform availability [7]. The move to distance learning brought innovation and better quality of study [5]. For lecturers, this has involved the experience of mastering the latest methodology and tools of distance learning [51]. Due to online teaching and learning, both students and teaching staff will further develop their online communication and interpersonal skills through regular exposure to online platforms [25]. All the lecturers in the future will have an opportunity to have intensive meetings with students and to co-create learning outcomes via online platforms [5]. The acquired self-confidence in ITC competences enhanced lecturers' intrinsic motivation to further use distance learning platforms. Such learning is acceptable for many students as well and their attendance increased even to 100% [45]. They consider distance learning a good idea and have plans to use it more often during the semester [23]. The e-learning and online students node include one relevant theme, i.e., self-regulation of students [52]. Namely, online learning encouraged students' self-discipline and selfeducation [51]. "Aspects of student characteristics, intrinsic motivation, teacher/lecturer characteristics, infrastructure, system quality, course quality and information, and an online learning environment guarantee existing learning success" [53]. For students to maximally benefit from online learning contexts, online courses must be designed to support students' self-regulation because students no longer have reinforcements commonly found in traditional face-to-face learning contexts [28]. By studying autonomously, students can easily apply a learning approach that aims to self-regulate both their own motivation and desires and the expectations they want to achieve. Such learning leads to the personal life of everyone in viewing learning for himself as having responsibility, as a control for the acquisition of their knowledge [24]. Motivation and self-regulation also played a role in successful online learning. Online students were more predisposed to self-study, self-discipline, and regulate their time management. [54]. Students actively create unique learning experiences by using their own environment and resources [26]. COVID-19 confinement changed students' learning strategies to a more continuous habit, improving their efficiency as well as enhanced their inner responsibility [55].

#### **5. Conclusions**

The conducted qualitative research highlighted three levels (authorities, lecturers, students), where pandemic-induced qualitative changes occurred in the university. The first level of leaders had the most significant influence on qualitative changes related to lecturers and students. In the preparatory phase, shifting from face-to-face learning to online one and later, they were the first to test and become convinced of the reliability of new forms of work. The possibility of working and organizing online meetings regarding studies became one of the qualitative changes not only for the future but also provided conditions for the establishment and maintenance of mutual/parity relation-based relationship between the authorities and the whole university community. It should be acknowledged that communication with heads of departments at different levels, lecturers, and students lacked systematicity in the pre-pandemic period. It was based more on traditional events. During the pandemic, the university authorities assumed the role of leaders to mobilize the community and to encourage successful transfer and continuation of high-quality studies online. Communication was one of the reasons for success. The community members appreciated consideration of their opinion preparing temporary documents, constant relation was ensured, and decisions were explained. The establishment of communication and close relation is the second qualitative change, which can have an influence on further collaboration and nurturance of communication culture. The actions of university authorities providing timely and continuous support to the academic community should be emphasized. Their encouragement and support resulted in systemic learning of new innovative online tools among lecturers so that they are able to maintain a relationship with students. It can be stated that this is a double qualitative change—due to decisions of authorities, training courses of online didactics and ICT were introduced in the university, which enhanced the learning culture of university lecturers and universities became learning organizations. This is important to lecturers as well because they were forced to learn and master new innovative tools of online studies. This knowledge and abilities will be used in the future as well. Moreover, this also contributes to the fourth significant qualitative change in studies—appearance and implementation of new study forms for learners. The development of study forms ensured better accessibility of higher education to different groups of the society.

The new experience acquired by lecturers mastered tools of online teaching and learning promoted innovative studies, preparation of high-quality study material, the possibility for improving the quality of subject content, and application of elements of online learning in the traditional face-to-face studies. This external qualitative change in studies, which was encouraged and supported by the university authorities, empowered and enhanced the intrinsic motivation of lecturers and their self-confidence and opened up new opportunities for innovative studies. Thus, a paradoxical phenomenon can be observed, when pandemic-induced changes forced all the lecturers to learn to pursue the qualitative transformation of studies. However, this obligatory activity did not evoke much resistance from the lecturers. On the contrary, this external act had an influence on intrinsic motivation and encouraged qualitative changes in studies. Intrinsic motivation is the strongest driving power of innovation and change.

The most essential change at the level of students was that they not only embraced the shift to online learning and gained the ability to do this but also improved their attendance and strengthened their ability to learn independently. It can be assumed that having perceived the difficulty of the situation they worked hard to cope with all the challenges. This qualitative study-related change is important to lecturers as well. Reflecting on their experience, they should continue using online tools and other methods that promote students' independence.

Thus, all the discussed levels were not separate aspects of qualitative change in university education, but rather they influenced each other. The external motivation of university leaders, their support, and the creation of conditions for online studies led to external as well as internal qualitative changes in studies for lecturers and students.

The obtained research results have significant empirical importance for planning and maintaining the study process of high quality. It has to be emphasized that the obtained research results are of value for the study process not only during the time of the pandemic. A part of the described processes, such as establishment and maintenance of mutual/parity relation-based relationships between the authority and the university community, as well as timely and continuous academic and technical support to the community, are of paramount importance for a quality study process under varying work conditions and forms. It can be stated that the heightened prevalence of distance learning and work during the pandemic not only will retain its current value but also will expand with a growing supply and availability of such studies. Therefore, not only universities in Lithuania but also communities of other universities can make use of ways and forms of supporting the academic community. For example, motivating and empowering lecturers to work online and to implement the presented content to students in a qualitative way.

There are some limitations of this study. On the basis of attitudes of expertsrepresentatives of university authorities, who participated in the research, the qualitative changes were formulated. Lecturers and students, whose opinions and experience could supplement the acquired research results, did not participate in the research. Only considering the experience of the latter, the experts formulated their attitude. Such research would be one of the priorities of future research. Moreover, only the qualitative changes observed in distance studies during the pandemic are presented in the article. Although the experts were asked about the encountered problems and difficulties learning or working online, these issues were not formulated as part of the goal of this article. Future tasks for researchers could include comparing qualitative changes and encountered problems or challenges as well as suggesting solutions. During the pandemic the whole academic community was forced to work and learn online. Therefore, one more limitation should be pointed out—to what extent the research conclusions could be similar or different under traditional study conditions, when face-to-face learning prevails, and to what extent the distinguished qualitative changes are maintained after returning to the usual form of work when the pandemic is over. Such a research question could be answered by conducting continuous research in the future.

**Author Contributions:** Conceptualization, V.N. and V.D.; methodology, V.N. and R.B.; formal analysis, R.B., E.J. and D.G.; writing—original draft preparation, V.N., V.D. and D.G.; writing—review and editing, R.B. and D.G.; visualization, V.D. and E.J.; project administration, D.G.; funding acquisition, D.G. All authors have read and agreed to the published version of the manuscript.

**Funding:** The research is supported by Lithuanian Research Council financed project "Model of distance working and learning organization and recommendations for extreme and transition period" (EKSTRE) (01-06-2020-31-12-2020). Grant Agreement S-COV-20-20.

**Institutional Review Board Statement:** Ethical review and approval were waived for this study, as this study involves no more than minimal risk to subjects.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to data restriction policy by the grant provider.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **Replacing Work with Study: A Sustainable Development Strategy for Economically or Culturally Disadvantaged Students**

**Yi-Chih Lee**

**Citation:** Lee, Y.-C. Replacing Work with Study: A Sustainable Development Strategy for Economically or Culturally Disadvantaged Students. *Sustainability* **2021**, *13*, 9658. https://doi.org/10.3390/ su13179658

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Javier Puente and Leandro Almeida

Received: 14 August 2021 Accepted: 24 August 2021 Published: 27 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Department of International Business, Chien Hsin University of Science and Technology, Taoyuan City 320312, Taiwan; leeyc@uch.edu.tw

**Abstract:** Education is one of the most effective ways to eradicate and reduce poverty, helping to eliminate it as well as to promote social mobility. Although universities have been extensively established in Taiwan, the country still faces the problem of a large difference in students' academic ability and a widening gap in educational resources among families. This study mainly explored whether the strategy of encouraging learning instead of working for disadvantaged students is helpful to their learning effectiveness. The research samples were collected from the data of college students who participated in the higher education SPROUT project of Taiwan's Ministry of Education. This study enrolled a total of 752 students categorized as disadvantaged. The results found that those students who participated in the project usually devoted more time to study and schoolwork, and so they passed more subjects in the schoolwork with better overall performance in learning. They also obtained the required licenses for employment and more student subsidies to improve their lives. Therefore, the conclusion of this study is that by participating in multiple counseling mechanisms such as schoolwork counseling, license counseling, or career counseling arranged by universities, disadvantaged students can increase their study time, which not only improves their schoolwork learning effects but also enhances their employability.

**Keywords:** disadvantaged students; higher education SPROUT project; learning effects

#### **1. Introduction**

The United Nations (2020) has pointed out that nearly 260 million children worldwide were unable to go to school normally in 2018, with poverty and discrimination as the main causes of education inequality. UNESCO reported that in many countries, children from poor communities, girls, people with disabilities, immigrants, and ethnic minorities are in a severely disadvantaged position in regard to education. In 2018, 258 million children and adolescents worldwide were completely unable to receive education, accounting for about 17% of global school-age children, with poverty as the main reason for their failure to go to school, and most of them live in South Asia, Central Asia, and Southern Africa [1].

The reasons for the term "disadvantaged" may be nationality, ethnic group, economy, culture, language, region, or other factors. However, the factors leading to students being at a disadvantage are diverse and complex, and so one of the main topics often discussed is being economically disadvantaged—that is, the problem of "poverty" [2]. Due to a lack of resources (money being the main source), many people live a life that is lower than the minimum moderate economic resource standard or even below their needs and expectations, which is called poverty [3]. Scholars have argued that poverty affects learning attitudes, learning motivations, and academic achievements [4]. In Australia, economically disadvantaged students also have a poor reading intelligence level [5]. Students from high-income families have more exposure to better resources than students from lowincome families. In addition, parents with high socioeconomic status can provide cultural

capital and social capital to their children, while children from poor families lack such resources [4,6–9].

UNESCO (2017) pointed out that education itself is not only a goal pursued by individuals but also a means of other sustainable development goals. Education is an internal component of sustainable development and is also a key factor to promote the realization of sustainable development [10]. On 25 September 2015, the United Nations General Assembly adopted the *2030* Agenda for Sustainable Development [11], which put forward 17 Sustainable Development Goals, including those related to education. These include "No Poverty", as education is one of the most effective approaches to reducing poverty, and "Quality Education", ensuring inclusive and fair quality education so that all people can enjoy opportunities for lifelong learning to master necessary knowledge and skills and to fully integrate into society. These all represent the importance of education among the Sustainable Development Goals [11]. With the rapid changes in the world, the knowledge update cycle has become shorter, and so the provision of high-quality education and the development of lifelong learning attitudes have become important basic competencies for all students. Many studies have also noted that schools are regarded as an effective intervention in solving the disadvantages of low socio-economic backgrounds [12–14]. People can also reverse poverty through education, which further promotes social strata flow.

According to Effectively Maintained Inequality (EMI) [15] proposed by Lucas (2001), even though the enrollment rate of higher education tends to be high in the United States, the social and economic background of the family still has a great impact on the fairness of higher education opportunities, and both quantitative and qualitative unfairness are factors in the distribution process of higher education. Quantitative unfairness means that the proportion of the population receiving the same level of education in the advantaged groups is higher than that in the disadvantaged groups; qualitative unfairness means that the quality of education received by the advantaged groups is higher than that by the disadvantaged groups. As a result of the expansion, there has been stratification in higher education, and there is a difference in the values of diplomas obtained from different types of educational institutions in the labor market. After the number of people in the advantaged groups receiving higher education reaches saturation, the remaining opportunities can be enjoyed by the disadvantaged groups in reality. However, the educational opportunities that the disadvantaged groups can enjoy are from educational institutions with low diploma value, which causes another unfair phenomenon [16].

In Taiwan, higher education is divided into several types, including "academic" universities that train academic research talents and "technical and vocational" universities that mainly teach applied science and technology courses to train practical professionals [17]. In addition, the ranking of departments in public universities is generally better than that in private universities. Since public universities receive a large number of education subsidies from the government, their tuition fees are relatively low. However, the proportion of students from disadvantaged families in public universities is less than that in private universities. In the labor market, compared with private university graduates, public university graduates are often regarded as a group of people with better human qualities and are generally accepted by business owners [18]. Students with higher average monthly family income have higher admission rates in Taiwan. Therefore, they can enroll at public universities and pay the least tuition fees but enjoy more government education subsidies and better education quality [19,20]. On the contrary, students with poor family backgrounds usually have poor test scores and can only enroll at private universities or technical and vocational universities [21] and pay high tuition fees while receiving fewer government education subsidies and in some cases having to work part-time to pay tuition fees. Furthermore, after their graduation, in addition to having to repay the loans of universities, their employment conditions are not as good as those of ordinary students.

Several authors have presented different projects developed with the aim of helping disadvantaged students, such as: selecting the disadvantaged students in the most vulnerable areas to give financial subsidies for schooling [22], enhancing the provision of library

resources for disadvantaged students [23], providing scholarships [24], providing additional courses [25], and assessing the background factors of schoolchildren and providing scholarships to schoolchildren to increase re-enrollment opportunities [26]. Giving extra grants and providing educational resources have become the main methods for countries to help disadvantaged students. However, when a policymaker allocates resources to schools, they cannot give advice on the best allocation, and the resources given are often limited. In order to maximize resource efficiency, the subsidized units usually adopt different methods of helping students according to the characteristics of the target students. The disadvantaged students often need to spend a lot of time working to earn money for personal expenditure, so they have to sacrifice their study time, which results in low academic performance or even suspension or withdrawal from university. As a result, the poor students with economical disadvantages jointly suffer from educational disadvantages and fall into a vicious circle. In order to break this phenomenon of unequal education, the main motivation of this student aid measure is to encourage students to transfer their limited time from working to earn money to reading to earn money. On the one hand, students are encouraged to read more books to ensure better academic performance. On the other hand, they are also encouraged to spend time learning employment skills. All the time spent participating in learning can be exchanged for grants as living expenses, so that they can earn living expenses while studying, taking into account both study and life. Therefore, this program is aimed at disadvantaged university students. The main contribution of this study is that the studying and the employment skills training were integrated. In addition to providing scholarships for outstanding students, rich curriculum education and additional curriculum training were also provided to train students' employment skills and improve their academic learning effectiveness. Learning autonomy is exercised by students so that the disadvantaged students can choose the most favorable teaching program for their actual needs. Thus, the barriers of educational inequality are expected to be eliminated for socially and economically disadvantaged ethnic groups.

In summary, this study analyzed the following issues.


#### **2. Literature Review**

#### *2.1. Disadvantages*

As pointed out by the United Nations (2021), poverty denotes not only the lack of income and resources that make it difficult to maintain daily life but also hunger, malnutrition, the inability to obtain a full education, social discrimination or exclusion, the failure to enjoy basic public services, reduced access to loans from the capital market [27]. In 2015, more than 736 million people were living below the international poverty line, and more than 10% of the world's population cannot even have their basic needs met for medical care, education, water, and sanitation. Therefore, eradicating all forms of poverty is one of the 17 goals of the 2030 Agenda for Sustainable Development [28]. Taiwan's official definition of low-income households is based on the poverty line as the poverty threshold—that is, the minimum cost of living is employed as the demarcation point of the poverty line, and those people below the level of personal income are the poor [2].

According to United Nations statistics (2020) [29], the global indigenous population only accounts for 5% of the world's total population, but it accounts for as much as 15% of the world's poor population. The main cause is the inequality of education. In addition, the disadvantaged ethnic groups around the world, compared with the local advantaged ethnic groups, usually have lower academic performance, shorter education years, and poorer overall education quality, with an example being the aborigines in Taiwan [30]. Formal education often ignores various traditions of indigenous peoples, with languages, teachers, teaching contents, and methods dominated by mainstream ethnic groups, resulting in systemic inequality in educational opportunities and education quality for indigenous peoples, making them face urban–rural gaps, maladjustment, and living restrictions or other factors, compromising their learning experience and even forcing them to suspend their studies [31]. In Taiwan, during the 2018 academic year the suspension rate of indigenous students in colleges and universities was as high as 8.6% (the same rate was 6.2% for general students), and the dropout rate reached 12.8% (7.1% for general students) [32]. Thus, indigenous students have also become another disadvantaged group in society. Tabi (2016) considered that the challenge faced by many ethnic minority students is the lack of skills that can help them successfully adapt to the university environment. Therefore, Tabi (2016) suggested that supportive strategies such as guidance, counseling, and providing care are beneficial to ethnic minority students [33].

The disadvantages of such students are obviously seen in their studies. The main reasons are that they have received grandparenting, are new residents, or are from economically or culturally disadvantaged families such as having single parents [34], and so their parents are busy with livelihoods and have less or no time to care about their children's education. Therefore, the performance in learning for disadvantaged students is poorer than average in terms of learning motivation and learning self-efficacy [35]. Hung (2001) noted that remedial education programs for disadvantaged students usually fail to achieve the expected results, and so substantial help can only be realized by providing appropriate assistance based on the actual difficulties faced by the students [36].

#### *2.2. Equality of Educational Opportunity*

In the case of limited social resources, how to allocate them, and to whom, a set of principles of fairness and justice is required as the allocation standard. In terms of the argument of the equality of educational opportunity, Coleman [37] pointed out that in different social and economic countries, in pursuit of equality of educational opportunities, the schools play a role in providing specific educational resources and protecting schoolchildren from being exploited by family impact, so that schoolchildren can freely enjoy these resources for their benefits. Volmink (1994) emphasized that the function of schools is to enable students to achieve a performance level of making the best preparations for their future lives. Schools have the responsibility to create fair competition opportunities for each student and not to allow some students to be discriminated against due to their relatively poor academic performance or even deprived of the opportunity to compete fairly with others [38].

The basic argument of educational equality theory is that school education is a preparation field for individuals to participate in the market economy and pursue other social achievements. Meanwhile, the educational achievements of a person have high correlation with other social achievements. Since the fairness of access to educational resources seriously affects the competitiveness of an individual in the labor market, educational equality theorists emphasized the fairness of competition, and they argued that the opportunity for individuals to obtain educational achievements should not be affected by factors beyond their control. In addition, the educational resource allocation in line with the principle of equality should depend on individual talents and efforts, rather than on social class or family background; that is, individual educational opportunities should be guaranteed to be not affected by class or family [39–41].

Undoubtedly, education is a kind of public goods. However, education is also a kind of private goods at the same time, and it is a kind of private goods linked to status, a form of positional goods that can be used to improve work income and enhance social status and prestige for students [42]. As a type of positional goods, education has two forms of benefits. One is the admission benefit of education: A well-educated person finds it easier to enroll at a good follow-up school. The other is the earnings benefit of education: A well-educated person has access to jobs with high salary and high social status [43,44]. The investment in

education can be transformed into an advantage in the labor market and competition for social or political positions. If parents with better resource conditions send their children to private schools or transfer them to star school districts in pursuit of better education quality, when their children gain these advantages, they may make the children who were originally at a disadvantage even more disadvantaged [45]. Therefore, educational equality theorists advocated for the state to enact policies to address the issue of children's learning weakness caused by parents' economic weakness or neglect of children's education or to provide more educational resources as compensatory measures to correct this situation [41].

#### *2.3. Higher Education Sustained Progress and Rise of Universities in Taiwan (SPROUT) Project*

In order to help economically or culturally disadvantaged students to receive a smooth university education, the Taiwan government provides a student aid program for students in colleges and universities whose family income is in the bottom 40%. The project measures include the four items of student subsidies, living subsidies, emergency relief grants, and accommodation discounts, which provide subsidies for students' tuition and fees, living expenses, emergency relief funds, and accommodation expenses. Among them, the living assistance measures for disadvantaged students include exemption or reduction of tuition and miscellaneous fees, assistance with school loans, emergency relief and foreign student loans, as well as grants and scholarships for studying abroad [46].


To accelerate the function of higher education to promote class mobility, the Ministry of Education has also introduced an in-depth study assistance mechanism. It encourages colleges and universities to provide study guidance for disadvantaged students and offer internship opportunities, career planning assistance, society feedback, and student fundraising, so that the economically and culturally disadvantaged students can take care of both schoolwork and daily needs at the same time through the tutoring mechanism of study grants [47].

#### **3. Materials and Methods**

#### *3.1. Classification of Disadvantaged Students*

According to the standards set by the program, the criteria for students to be considered economically disadvantaged or culturally disadvantaged are: (1) indigenous people; (2) low-income households: (a) after the total family income is equally distributed to all family members, the income of each member per month is below the minimum living allowance for the local area; (b) the family property does not exceed the value applicable to low-income households for the local area; (3) low- and middle-income households: (a) after the total family income is equally distributed to all the family members, the income of each member per month is below 1.5 times the minimum living allowance for the local area; (b) the family property does not exceed a certain amount for the local area applicable to lowand middle-income households; (4) children of persons with disabilities; (5) students with disabilities; and (6) children or grandchildren of families in special circumstances: Families in special circumstances refer to those families in which the income of each member does not exceed 2.5 times the minimum living expenses determined the government for the year or 1.5 times the average monthly consumption expenditure per person in Taiwan after the total family income is equally distributed to all family members, the family property does not exceed a certain amount determined by the central competent authority, and one of the following circumstances applies: (a) under 65 years of age, his or her spouse died or disappeared and has not been found for more than six months after the disappearance was reported to the police for search assistance; (b) unable to live together with the spouse due to malicious abandonment or abuse by the spouse, the divorce is confirmed by judgment, or the registration of divorce agreement has been completed; (c) victim of domestic violence; (d) an unmarried pregnant woman of more than three months of pregnancy up to within two months of childbirth; or (e) independently raising a child under the age of 18 due to divorce, widowhood, giving birth out of wedlock, or a grandchild under the age of 18 whose parents are unable to support them and is incapable of working or unable to work due to serious injuries or taking care of a child under the age of six although he or she is capable of working [48,49].

#### *3.2. Research Design*

This student aid program is divided into five counseling mechanisms, namely, academic scholarships, attending license counseling courses, obtaining professional license subsidies, participating in lectures, and participating in off-campus competitions. Students who complete any of these five counseling mechanisms can receive different amounts of scholarships. Students may participate in all or choose their interested items, without a limit in the number of participants. After participation, they can apply for grants from the organizers based on the participation results. The greater number of participants, the more scholarships are obtained.


to enhance the employment opportunities of students after graduation, students who obtain government department licenses or school-designated licenses during their studies are given different amounts of rewards according to different license levels, in addition to license registration fee subsidies.


#### *3.3. Research Samples and Period*

The data of this study were taken from the data of university students who are disadvantaged. The data collection period was from 1 August 2019 to 31 July 2020. The students included all grades of students from freshman to senior, and their study periods were during the daytime, the nighttime, and holidays. Students who had only attended one semester were excluded from the data samples, whereas the rest were included in the database for analysis. In the samples, students who had participated in one counseling mechanism were classified into the experimental group, while those who had not participated in any counseling mechanism were classified into the control group.

#### *3.4. Analysis*

This study used the Statistical Package for the Social Sciences (version 21.0 IBM SPSS Inc, Chicago, IL, U.S.A.) to perform various statistical analyses. Statistical methods such as mean, standard deviation, range, ANOVA, and regression were adopted to perform the overall sample variable analysis for descriptive statistics, and 0.05 was taken as the test level for statistical significance.

#### **4. Results**

This study included a total of 752 students who met the disadvantaged classification and had complete school-year data, including 459 (61.04%) males and 293 (38.96%) females, with an average age of 22.94 years old (STD 5.17). There were 78 students (10.37%) from low- and middle-income households, 144 (19.15%) students from low-income households, 230 (30.59%) students that are children of persons with disabilities, 122 (16.22%) students with disabilities, 149 indigenous students (19.81%), and 29 (3.86%) students from families with special circumstances. Among all samples, 405 students participated in the higher education SPROUT project, including 230 males (56.79%) and 175 females (43.21%), with an average age of 22.35 years old (STD 4.22). There were 49 students (12.09%) from low-income households and 67 students (16.54%) from low- and middle-income households, while 131 are the children of persons with disabilities (32.35%), 73 are students with disabilities (18.02%), 72 (17.78%) are indigenous students, and 13 (3.22%) are students from families with special circumstances, as shown in Table 1.



In the overall sample, students of different types of identity showed statistically significant differences in academic performance (*p* = 0.003), with low- and middle-income students having the highest score of 78.84 (±8.64), children of persons with disabilities having the second highest score of 75.48 (±11.30), and indigenous students obtaining the lowest score of 72.62 (±12.16). In addition, the number of failed academic subjects of the overall sample was also statistically significantly different (*p* = 0.023). The students from low-income households obtained the highest number of failed academic subjects, with an average of 1.81 subjects (range 0–15), followed by 1.80 subjects (range 0–16) for indigenous students, and 0.85 subjects (range 0–9) for students from low- and middleincome households, which was the lowest number of failed subjects. Furthermore, there was also a statistically significant difference in the overall sample for the number of absences in the school year (*p* < 0.001). Indigenous students asked for leave most frequently, with an average of 76.98 lessons (range of 0–335), followed by 56.27 for the students from low-income households (range of 0–403), and the leave requested by the students from low- and middle-income households was the lowest at 41.26 lessons (range of 0–213). In terms of gender, females had better academic performance than males (77.48 vs. 72.77, *p* < 0.001), they failed in fewer subjects (1.13 vs. 1.61, *p* = 0.010), and their number of requests for leave was higher, but there was no statistically significant difference (59.81 vs. 53.05, *p* = 0.164). When the groups were distinguished by participation in the program, those who participated in the program performed better than those who did not, with a higher average score (78.55 vs. 69.81, *p* < 0.001). At the same time, they failed in fewer subjects (0.67 vs. 2.35, *p* < 0.001) and made fewer requests for leave (45.27 vs. 67.79, *p* < 0.001). In addition, the number of licenses required for employment obtained by students participating in the program was much higher than that of students who did not participate in the program, and none of the students who did not participate in the program won prizes in the competitions inside and outside the school, as shown in Table 2.



Mean (SD or range); \* *p* < 0.05.

From the perspective of identity, there were statistical differences in the academic performance of the whole school year (*p* = 0.039). The students from families with special circumstances had the best academic performance after participating in the higher education SPROUT project, with a score of 80.33 points (STD 9.14), while the indigenous students had the worst performance of 76.25 points (STD 8.89). In terms of class attendance, there were differences between different identities (*p* = 0.009); the indigenous students had the worst attendance, with an average number of absences of 66.65 lessons (range of

0–335), and the students in families of special circumstances had the best attendance, with an average number of absences of 30.09 lessons (range of 6–74). However, there was no statistical difference in the use of aid funds and the number of failed subjects.

Further analysis showed no significant difference in academic performance, number of failed subjects, or number of days off for female students of each identity category, but for male students, there was a statistically significant difference in the academic performance of each identity category (*p* = 0.003) and the number of requests for leave (*p* = 0.007). Among them, students from families with special circumstances performed better and obtained 81.38 points (±7.05), and the indigenous students had the worst performance of 72.20 points (±8.90). The indigenous students requested the highest number of days off and obtained 75.94 points (range of 0–335), and the students from low-income households asked for the smallest number of absences, which was 29.39 lessons (range of 0–126). Gender did not cause any difference in the use of grants for students of different identities.

In the counseling program, students mostly chose to participate in academic scholarships (50.0%), followed by attending license counseling courses (17.8%). The top two counseling programs that male students participated in were academic scholarships and license counseling courses, while female students chose to participate in academic scholarships and professional license subsidies. In terms of analysis of the participation of counseling programs, the participation of children with physical and mental disabilities was the highest, as shown in Table 3.



Based on the analysis of the performances of students participating in the program, the higher the amount of the grant received, the better the academic performance (β = 0.184, *p* < 0.001). Among them, female students were more active in participating in the program than male students (TWD 6606 vs. TWD 5682, *p* = 0.016) (Figure 1); therefore, the overall academic performance of female students was better than that of male students (80.24 vs. 77.26, *p* < 0.001), and the number of failed subjects for female students was also smaller (0.50 vs. 0.80, *p* = 0.025). Since students could freely choose to participate in several counseling programs, through the analysis of the number of counseling programs, students who participated in more programs obtained higher total scores in the semester, with a statistically significant difference (*p* < 0.001). In the post-inspection, the students who participated in three or four counseling programs had much better academic achievements than those who chose only one program (*p* = 0.001 vs. *p* = 0.005). However, there was no significant difference in the number of failed subjects and the number of absences, as shown in Table 4.

**Figure 1.** The utilization of grants.

**Table 4.** Learning performance among the number of participating programs.


Mean (SD or range); \* *p* < 0.05.

#### **5. Discussion**

With the change of social patterns, the risks brought by social problems expand, leading to insufficient income and poverty and making the disadvantaged weaker and the poor poorer. People must make a choice when environmental resources are limited. Therefore, the government should integrate its resources with those of the education units, clearly define the disadvantaged who need to be taken care of, and choose to assist those who need the assistance most, rather than offer unlimited help to the disadvantaged students only because they are disadvantaged. The ultimate goal of education is to help the disadvantaged groups in need of assistance to gain a skill and rely on their own strength to achieve a sustainable life, as this is the purpose of maximizing the effectiveness of resources. Therefore, based on educational equity, Taiwan provides subsidies for the participation in different programs for schooling and employment activities to the six disadvantaged groups defined by the education units. It is hoped through participating in various activities that these students can obtain better learning results through sustainable learning. By learning employment skills via taking skill training courses or workshops, they are able to support their daily lives by obtaining subsidies and making a living by studying instead of working, thereby enhancing their future competitiveness. These efforts also echo the OECD's guidance that higher education institutions should provide additional support to students with disadvantaged backgrounds [51] in order to reach the target of equal educational opportunities advocated by the educational equality theory. Meanwhile, for

these additional personnel training measures, different countries should take effective measures to create their own national characteristics.

As found in the research results, the students who participated in the higher education SPROUT project usually devoted more time toward studying. They also passed more subjects, and their overall performance was better. Students willing to participate more in the program activities can obtain higher subsidies, which can indeed improve their overall performance. Students in the experimental group obtained more practice licenses required for future employment than those who did not participate in the experimental group, which showed that the intervention of the program surely helped in the students' future employment competitiveness. The research also showed that female students were more active in participating in various activities of the program, and so their overall learning performance was better than that of male students, and the results also indicated gender differences in the use of subsidies and overall learning performance. Among the five counseling mechanisms, the students participated in the academic scholarships the most, and the students participating in more counseling programs not only obtained more grants but also showed better academic performance. Eventually, the target of the program that let students study more and attain more scholarships was achieved, that is, the target that let students not only reduce the pressure of their lives but also improve their academic and future employment capabilities.

It is worth discussing that the indigenous students, regardless of whether they participated in the improvement of the student aid program or not, had the worst academic performance, the worst attendance in class, and the least participation in the planned curriculum activities. Scholars [52] pointed out in studies that the poor academic performance of indigenous students is largely due to the impact of the poor socioeconomic background of their families. In addition, the ratio of families of indigenous students being low-income households, grandparenting, or single-parent upbringing is also higher than that of ordinary students. Besides the comparatively weaker family financial situations and parenting abilities, indigenous students generally have low interest in learning. Facing the more difficult educational content of universities, they are often relatively uninterested and have low levels of participation, which in turn affects their learning effectiveness. This study also showed similar results to those in the literature.

#### *Limitations*

This study did not provide individualized education courses for students of various disadvantaged identities. Instead, it only provided courses to strengthen the general academic and employability of various disadvantaged students. Therefore, the results showed the varied absorption of the integrated intensified education courses by students of different identities. In particular, it was found that the learning effectiveness of indigenous students in the higher education system was lower than that of ordinary students, and even their willingness to participate in the program was lower. It is suggested that follow-up researchers can provide an individualized education curriculum targeting the features of indigenous students. For example, they should cooperate with specific vocational training units to strengthen the training of employment skills for indigenous students, provide after-school tutoring and career planning guidance for indigenous students, or include the return of indigenous students to their hometown community services as part of the student award. Only by giving individualized considerations to the differences of individual students can the government affectively assist the performance of indigenous students in school. In addition, since the use of program funds was limited to students of Taiwanese nationality, a comparison between students with multiple nationalities cannot be made. Recently, due to the COVID-19 pandemic, students' schooling has been affected in many countries. In Taiwan, schools did not adopt online teaching courses until mid-May 2021 due to the pandemic. The disadvantaged students enjoyed fewer resources than ordinary students, and whether there is a loss of schooling rights caused by COVID-19 and resource restriction is worth discussing.

#### **6. Conclusions**

Poor families have difficulties in life, and their children often do not receive a good education, which leads to their failure in obtaining an appropriate job when they grow up and makes it relatively difficult for them to earn a living. This vicious cycle from generation to generation has caused unfavorable living conditions for future generations and has continued to result in poverty and inequality [53]. As educational units, universities should provide students in need with substantial assistance methods to cultivate students' capability of sustainable learning. If disadvantaged students are forced to leave the educational field due to personal family or economic factors during their higher education, the equality of educational opportunity and the promotion of social mobility will become slogans. It is one of the Sustainable Development Goals of the United Nations to assist the disadvantaged groups to enhance their human capital and provide them with employability through education and training and to ensure that they have fair rights in the acquisition of resources [54]. According to the "asset-based welfare theory", to address the welfare needs of low-income households and special ethnic groups, the government needs to help them in developing human capital, building social capital, and participating in productive employment activities so that these ethnic groups can eventually become a part of the overall social productive economic activities [55,56]. Higher education in Taiwan is an important method for children from middle and low social classes to move upward. By participating in the diversified counseling mechanisms arranged by universities, such as homework counseling, license counseling, and career counseling, the disadvantaged students can reduce their working hours and increase their study time, which can not only achieve better schoolwork performance but also enhance their future employability. Thus, the ultimate target of the equality of educational opportunity in this student aid program will be truly achieved.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data are not publicly available due to privacy issues and to ensure confidentiality of the participants.

**Acknowledgments:** I would like to acknowledge reviewers and editors.

**Conflicts of Interest:** The author declares no conflict of interest.

#### **References**


### *Article* **Exploring Collaborative Problem Solving Behavioral Transition Patterns in Science of Taiwanese Students at Age 15 According to Mastering Levels**

**Cheng-Hsuan Li 1,\*, Pei-Ling Tsai 1, Zhi-Yong Liu 2, Wen-Chieh Huang <sup>1</sup> and Pei-Jyun Hsieh <sup>1</sup>**


**Abstract:** This study analyzed the collaborative problem solving (CPS) behavioral transition patterns of 53,859 Taiwanese students in science at age 15 by using an online Taiwanese CPS assessment that was designed according to the Programme for International Student Assessment 2015 CPS framework. Because of behavioral changes over the testing period, the CPS target skills that corresponded to the assessment items can be viewed as a CPS behavioral sequence. Hence, a lag sequential analysis was applied to explore the significance of the interactions among the CPS skills. The behavioral sequence is coded according to the level of mastery (0, 1, or 2) of items. The CPS transition patterns were analyzed in three gaps, namely the gender gap, the urban–rural gap, and the achievement gap. The findings showed that "Monitoring and repairing the shared understanding" was a crucial CPS skill in science. Moreover, the female students who would follow rules of engagement effectively exhibited higher scores than male students did in monitoring the results of their actions and evaluating their success in solving the problem. No obvious differences were observed in the urban–rural gap, whereas differences were observed in the achievement gap.

**Keywords:** science scenario; collaborative problem solving; behavioral transition patterns; lag sequential analysis

#### **1. Introduction**

According to a review of relevant research and frameworks, key sustainability competencies include systems thinking, interdisciplinarity, anticipatory competence, values and ethics, normative competence, critical thinking and appraisal, interpersonal competence, intrapersonal competence, communication skills, strategic thinking and planning, personal engagement, evaluation skills, and dealing with uncertainty and resilience [1–3]. Moreover, according to Education for Sustainable Development, student-centered methodologies, project or problem-based learning, case study, simulation, and cooperative inquiry are widely used to study sustainability competencies [4]. Collaborative problem solving (CPS) is a crucial competency for students to communicate and contribute to problem solving with team members in school or future workplaces [5–7]. Therefore, education systems need to update curriculum and broaden scope so that students can learn CPS competency for life and employment in the 21st century [7]. Nonetheless, developing a large-scale, standardized CPS assessment that includes scenarios, numbers of team members, collaboration, problem solving, and contexts to understand student performance in CPS is challenging [8].

Smart-classroom development can benefit problem-based learning and collaborative inquiry [4]. In general, two types of CPS are offered in computerized assessments in smart classrooms: human to human and human to agent. Regarding human-to-human assessments, for example, CPS units were developed to assess social skills and cognitive skills for

**Citation:** Li, C.-H.; Tsai, P.-L.; Liu, Z.-Y.; Huang, W.-C.; Hsieh, P.-J. Exploring Collaborative Problem Solving Behavioral Transition Patterns in Science of Taiwanese Students at Age 15 According to Mastering Levels. *Sustainability* **2021**, *13*, 8409. https://doi.org/10.3390/ su13158409

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Javier Puente and Leandro Almeida

Received: 22 June 2021 Accepted: 23 July 2021 Published: 28 July 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

the Assessment and Teaching of 21st Century Skills project. Within these units, individuals collaborate with unknown paired partners online by communicating in a chat box, sharing their understanding, and solving a given problem. Experts grade the individual CPS skills by using rubrics according to the log recorded by the assessment system [5]. However, human-to-human assessment is inappropriate for large-scale assessments because of the pairing process and grading by human experts. In addition, individual scores may be influenced by the unknown paired partner; hence, the human-to-human assessment is not a standardized test.

The Programme for International Student Assessment (PISA) 2015 included a humanto-agent CPS assessment to evaluate the proficiency of 15-year-old students in the 12 CPS skills listed in Table 1 [7]. These 12 skills represent the combination of 4 problem solving skills and 3 collaborative skills. For instance, (B3) "Describe roles and team organization (communication protocol/rules of engagement)" is classified as the combination of problem solving skill (B) "Representing and formulating" and collaborative skill (3) "Establishing and maintaining team organization." According to the draft of PISA 2015 CPS assessment, the scenarios included in the CPS assessment units contain three dimensions, namely task type, settings, and domain content. The task type can be jigsaw, consensus building, or negotiation. The settings can be "private versus public," "technology versus nontechnology," or "school (formal) versus nonschool (informal)." The domain content can be an academic subject such as math, science, reading, environment, community, or politics. Furthermore, the interactions between students and agent(s) are represented as mixed-initiative dialogues in multiple-choice items. Each multiple-choice item in the CPS units is designed to measure one CPS skill proficiency (A1, A2, A3, B1, B2, B3, C1, C2, C3, D1, D2, or D3), and students are assigned a score of 0, 1, or 2 [7]. Because unit items are designed according to a given scenario based on dialogue types, the student's dialogue path when interacting with the agent(s) to solve a problem reflects the CPS behavior of the student. Hence, a lag sequential analysis (LSA) was applied to explore behavioral patterns among students' CPS skills. The present study posed four research questions:



**Table 1.** Matrix of collaborative problem solving skills for PISA 2015 [7].

#### **2. Materials**

#### *2.1. Taiwanese CPS Online Assessment System*

A Taiwanese CPS online assessment system including units in math, science, reading, and social science scenarios was developed for students in Grades 5 to 10 according to the PISA CPS framework shown in Table 1 [9–12]. The science scenario contained two units, the Water Purification unit (Figure 1) and the Slurpy unit (Figure 2). In the Water Purification unit, a task taker (TT) works with two computerized agents to purify dirty water using hands-on materials. One agent is highly collaborative, whereas the other is noncollaborative. The highly collaborative agent always gives positive feedback on how to do the task. However, the noncollaborative agent sometimes gives negative feedback, such as disagreeing with the TT and other computerized agents and making negative comments about the work. The role of the TT is to lead the team and assess the performance of the water filter designs. In the Slurpy unit (Figure 2), the TT communicates and collaborates with a computerized agent to use ice and refrigerants (salt, sugar, monosodium glutamate, and water) to decrease the highest temperature. The team composition (TT and the computerized agent) is asymmetrical. They separately use different proportions of various refrigerants and ice to decrease and record the corresponding temperature. In the end, they collaborate to find the most effective refrigerant and ratio of refrigerant to ice to decrease the temperature. The Water Purification and Slurpy units contain 13 and 17 items, respectively. According to the answering path of each item, students receive a score of 0, 1, or 2 points.

**Figure 2.** Screenshot of the Slurpy unit [9].

Furthermore, according to the answering path in two conversation layers of each item that respects only one skill shown in Table 1, students receive a score of 0, 1, or 2 points based on the following score guide. The score "0 indicates that students respond or provide incorrect information with little relevance to the task. The students contribute minimally to achieving group goals when interacting with team members. The students always work alone. The students do not help the team solve the problem during the mission. The score "1 indicates that the student responds or provides the correct information or actions and fix their understanding when prompted by computer agents. In contrast, "2" indicates that the students are actively involved in the task and select some actions that contribute to the teamwork based on the information provided. The students can communicate with team members, mediate conflicts, and take the initiative to solve the obstacles effectively [7,10,12].

Because the units are designed to be conversational scenarios, the item responses show the proficiency of students mastering the corresponding CPS skills, and the responses of two adjacent items show the behavior pattern between the corresponding CPS skills. For example, the profile of the Water Purification unit is shown in Table 2. The unit contains four tasks and 13 items in total. The second item in Task 2 involves the CPS skill (C2) "Enacting plans." In this item, the TT must implement the plan as discussed with two agents. Hence, in the next item, the TT and two agents must share their understanding of the result after implementation; the task corresponds to (D2) "Monitoring results of actions and evaluating success in solving the problem." This means that the CPS assessment unit mimics a fluent conversation to achieve the common goal of solving the given problem. Therefore, we can analyze specific patterns such as the behavior transition pattern from (C2) to (D2) of students. Specifically, if students are mastering the skill (C2), we can analyze whether they also perform the skill (D2) well.



The overall Taiwanese CPS online assessment system illustrates that the human-toagent approach is feasible for measuring the 12 student CPS skills based on the framework shown in Table 1. Moreover, the internal consistency analysis and multidimensional item response theory model of a large-scale assessment have shown that the CPS scales are reliable and valid, respectively. The results via the CPS online assessment are also consistent with PISA 2012 [12,13].

#### *2.2. LSA*

To address the four research questions, LSA—a common method of identifying behavioral transition patterns—was employed in this study. LSA can be used to determine a given coding (e.g., an activity or behavior "E") followed by another coding (e.g., an activity/a behavior "F"). If the observed frequency of EF in a sequence is significantly higher than the expected frequency (i.e., the corresponding *p* value is less than or equal

to the significance level), then the behavioral transition pattern indicates that E is always followed by F [14–20]. LSA has been applied in various research studies such as clinical interactions, education, social behavior in animals, communication processes, and children's play. [20]. Education researchers Cheng and Hou (2015) applied sequential analysis to explore students' behavioral transition patterns from affective, cognitive, and metacognitive perspectives during online peer assessment [15]. LSA has also been used to analyze students' discussions and interactive behaviors in project-based learning [15–20]. As displayed in Table 2, each item can receive a score of 0, 1, or 2 points to show the student's level of mastery. Hence, using a suitable coding scheme combining the mastery levels and CPS skills, we discussed the behavioral transition patterns of students with varying CPS skill mastery in this study.

In addition, three different gaps were analyzed, namely the gender gap, urban–rural gap, and achievement gap. The gender gap was analyzed by comparing the performance of male and female students. To analyze the urban–rural gap, the students were categorized by school location into three urbanized sectors: commercial areas, emerging and traditional industrial districts, and less developed and remote areas. Then, the participants' behavioral patterns were compared in each area. To measure the achievement gap, the students were divided into a high-score group and a low-score group, and the corresponding CPS behavior patterns were compared. The study also compared the behavioral patterns of male and female participants in each group based on the proposed coding scheme with respect to the response sequence of items.

#### **3. Methods**

The behavioral transition patterns of students with different levels of mastery in CPS skills and the three gaps are discussed in this section; the proposed coding scheme is also introduced. Additionally, this section contains definitions and descriptions of the three urbanized sectors and the participant populations.

#### *3.1. Coding Scheme*

Based on the scenario designs of the Water Purification and Slurpy units, the student CPS skill sequences were

$$\text{A2, A1, B1, C1, C3, D1, C2, D2, D1, C2, D3, D3, D3, D3,}\tag{1}$$

and

$$\text{A1, C1, C2, B2, C1, A3, B3, B1, C2, C3, D2, D1, C1, B1, C2, D2, D3, \dots} \tag{2}$$

respectively. However, the students' level of mastery of these items was different. Hence, the study proposed a coding scheme combining CPS skills with students' level of mastery. For instance, if students earn 0 points on the first item of the Water Purification unit, then the mastery level of these students is below average. Hence, the coding of these students for (A2) "Discovering the type of collaborative interaction to solve the problem, along with goals" is A20. If students earn 1 point, then the mastery level of these students is average. Hence, the corresponding coding is A21. Finally, if students earn 2 points, then the mastery level is proficient, and the corresponding coding is A22. Table 3 lists the proposed coding scheme for each of the 12 CPS skills.


**Table 3.** Proposed coding scheme of 12 CPS skills.

In this study, only 23 behavioral transition patterns (see Table 4) in the CPS skills at the three levels of mastery could be discussed because the analysis of the behavioral transition patterns was limited by the online CPS assessment design. Because the student CPS skill sequences are part of a fixed conversational design in each science scenario, only some behavioral transition patterns could be analyzed using LSA. Table 4 shows which transition patterns could be analyzed by considering both the Water Purification and Slurpy units. The rows indicate the starting behaviors, and the columns contain the subsequent behaviors. In addition, checked cells indicate that the behavioral transition patterns could be discussed in this study. For example, in Sequence 1 of the Water Purification unit, the first two items are associated with the CPS skills A2 and A1. Hence, the behavioral transition pattern from A2 to A1 can be discussed, so the (2,1) cell is checked in Table 4. Specifically, according to the proposed coding scheme, the transition patterns A20 (below average in A2), A21 (average in A2), and A22 (proficient in A2) followed by A10 (below average in A1), A11 (average in A1), and A12 (proficient in A1) can be discussed using LSA. Therefore, the (2,1) cell represents the following nine transition patterns potentially exhibited by students in the science scenarios:

> A20→A10, A21→A10, A22→A10, A20→A11, A21→A11, A22→A11, A20→A12, A21→A12, A22→A12,


**Table 4.** Behavioral transition patterns (indicated from the row to the column) that can be discussed in this study (checks).

Table 4 shows the 23 behavioral transition patterns that are discussed in this study.

Based on the proposed coding scheme, the LSA was applied to match the CPS behavior transition patterns. The numbers of one lag pattern, i.e., the behavior transition pattern, were calculated first in the sequential series data. Then the normalized differences were computed between observed numbers and expected numbers based on the independence assumption. If the corresponding *p*-values are below 0.001, we will obtain the significant CPS behavior transition pattern.

#### *3.2. Participants and Procedures*

The Taiwanese CPS online assessment was administered to 53,859 students in Grades 9 and 10 (approximately 15 years old), of whom 27,656 were male and 26,203 were female. All participants completed both the Water Purification and Slurpy units within the science scenarios. Students who participated in the assessment had been taught the basic concepts of CPS in a 10-minute lesson by their teachers or shown a video recorded by the team of the Teachers' Collaborative Problem Solving Teaching Competency Project in Taiwan [9]. Furthermore, the students completed the Exercise Plan unit to understand how to use the CPS online assessment's interface.

#### *3.3. Three Types of Townships in Taiwan*

Hou et al. [21] divided 358 boroughs and townships of Taiwan into 6 groups according to both the 2000 census and 2004 population statistics for stratification. Among these six groups, statistical tests revealed five significant sociodemographic variables, namely age, education level, industrial structure, occupation, and personal income, which are highly related to levels of development among boroughs and townships. For the present study, the six groups were regrouped into three sectors—commercial areas, emerging and traditional industrial districts, and less developed and remote areas—according to the core population density, percentage of the population with more than a junior-college-level education, the percentage of the population aged 15–64, and percentage of the service population. The participants were classified into these three sectors by their school zip codes. The study included 23,489 participants in commercial industrial areas, 23,196 participants in the emerging and traditional industrial district, and only 7174 participants in the less developed and remote areas. In this study, the behavioral transition patterns of students in each of the three sectors were analyzed to assess the urban–rural gap.

#### *3.4. High-Score and Low-Score Groups*

Based on their total scores in the Water Purification and Slurpy units, participants were divided into two groups: a high-score and low-score group. Students whose total scores were equal to or greater than that of the student in the 33rd percentile were assigned to the high-score group. By contrast, students whose scores were equal to or below that of the student in the 66th percentile were assigned to the low-score group. The high-score and low-score groups contained 20,820 and 16,683 students, respectively. Table 5 shows the total numbers of participants and their proportions in the high-score and low-score groups.

**Table 5.** The number of participants in the high-score and the low-score groups.


#### **4. Results**

This section describes the comparison results of CPS behavioral transition patterns in science scenarios, and those in the gender gap, urban–rural gap, and achievement gap.

#### *4.1. Overall CPS Behavioral Transition Patterns in the Science Scenarios*

Figure 3 shows the behavioral transition patterns from (C2) "Enacting plans" followed by (C3) "Following rules of engagement" with the three mastery levels (below average, average, and proficient). The straight, black arrows indicate that the observed frequency was significantly larger than the expected frequency (i.e., *p* < 0.001) in this study. For example, C22 and C32 indicate that students were proficient in enacting plans and following the rules of engagement, respectively. The sequence of C22 followed by C32 was statistically significant (*p* < 0.001); hence, C22→C32. In other words, if students can enact plans effectively in science, then they can also effectively follow the rules of engagement in science. However, if students were below average or average in enacting plans (C2) in science, no significant evidence suggested how well they would perform in following the rules of engagement (C3) in science.

**Figure 3.** Behavioral transition patterns from C2 to C3.

Accounting for the CPS skills (D1) "Monitoring and repairing the shared understanding" and (C1) "Communicating with team members about the actions to be/being performed," Figure 4 illustrates that if students can effectively monitor and repair shared understanding in science, then they can also communicate effectively with team members about the actions to be/being performed in science.

**Figure 4.** Behavioral transition patterns from D1 to C1.

As illustrated in Figure 5, the behavioral transition pattern from C1 to C2 suggested that students who were proficient in (C1), "Communicating with team members about the actions to be/being performed," were likely to be either average or proficient in (C2) "Enacting plans." In addition, if students could not communicate effectively with team members about actions (C1), then they were unlikely to be proficient in (C2) "Enacting plans."

**Figure 5.** Behavioral transition patterns from C1 to C2.

According to Figures 4 and 5, students who could proficiently monitor and repair shared understanding (D1) were likely to be average or proficient in enacting plans (C2) because D12→C12 (Figure 4), C12→C21 (Figure 5), and C12→C22 (Figure 5). Furthermore, according to Figure 3, some of these students were proficient in following the rules of engagement (C3) because C22→C32. Thus, (D1) "Monitoring and repairing the shared understanding" is a crucial CPS skill in science.

#### *4.2. Comparison of Male and Female Groups*

The comparison of male and female students revealed only one different behavioral transition pattern (from C3 to D2) out of the 23 patterns studied. As shown in Figure 6, no difference was observed in the transition patterns of male and female students who had below average or average mastery of (C3) "Following rules of engagement." However, female students who followed the rules of engagement (C3) were likely to perform better than male students in (D2) "Monitoring results of actions and evaluate success in solving the problem." If female students could not follow the rules of engagement, then they remained likely to monitor the results of their actions and evaluate success in solving the problem either well or very well.

**Figure 6.** Behavioral transition patterns from C3 to D2 in (**a**) male students and (**b**) female students.

#### *4.3. Comparison of Three Urbanized Sectors*

According to the behavioral transition pattern from (C1) "Communicating with team members about the actions to be/being performed" to (C2) "Enacting plans" exhibited by all participants (Figure 7a), no statistically significant pattern from C10 to C22 was observed. That is, overall, students who could not communicate with team members about the actions to be/being performed could not enact plans very well in science. In the comparison of this transition pattern among the three urbanized sectors, students whose schools were located in emerging and traditional industrial districts or less developed and remote areas and

who exhibited low performance in communicating with team members about the actions to be/being performed were unlikely to enact plans in science (Figure 7c,d). However, students whose schools were located in commercial industrial areas could enact plans proficiently in science (Figure 7b).

(**c**) emerging and traditional industrial districts (**d**) less developed and remote areas

**Figure 7.** Behavioral transition patterns from C1 to D2 in three sectors.

#### *4.4. Comparison of High-Score and Low-Score Groups*

Significant differences were revealed between high-score and low-score groups (the achievement gap) because the coding scheme combines CPS skills and mastery levels. Figure 8 illustrates the behavioral transition patterns from A20, A21, and A22 to A10, A11, and A12. According to Figure 8a,b, if students belonged to the high-score group and could discover the type of collaborative interaction to solve the problem, along with goals (A2), then they were likely to be either average or proficient in (A1) "Discovering perspectives and abilities of team members."

**Figure 8.** Behavioral transition patterns from A2 to A1 in the (**a**) high-score and (**b**) low-score groups.

Similar to the behavioral transitional patterns from A2 to A1 (Figure 8), Figure 9 demonstrates that if students in the high-score group were average or proficient in (B2) "Identifying and describing tasks to be completed," then they were likely also average or proficient in (C1) "Communicating with team members about the actions to be/being performed." Moreover, as seen in Figure 10, students in the high-score group who were average or proficient in (B3) "Describe roles and team organization" were likely able to build a shared representation and negotiate the meaning of the problem (B1). Additionally, even if students in the high-score group were below average in B3, they still could achieve an average performance in B1.

**Figure 9.** Behavioral transition patterns from B2 to C1 in the (**a**) high-score and (**b**) low-score groups.

**Figure 10.** Behavioral transition patterns from B3 to B1 in the (**a**) high-score and (**b**) low-score groups.

Regarding the behavioral transition patterns from C2 to C3 (Figure 11), most students in the high-score group could follow rules of engagement (C3) proficiently. However, most students in the low-score group could not enact plans (C2). In addition, if students in the high-score group were average or proficient in (C2) "Enacting plans," then they could follow the rules of engagement (C3) proficiently. Based on the significant transition pattern from (C3) "Following rules of engagement" to (D2) "Monitoring the results of actions and evaluating success in solving the problem," most students in the high-score group (Figure 12a) were proficient in C3 and average or proficient in D2.

**Figure 11.** Behavioral transition patterns from C2 to C3 in the (**a**) high-score and (**b**) low-score groups.

**Figure 12.** Behavioral transition patterns from C3 to D2 in the (**a**) high-score and (**b**) low-score groups.

#### **5. Discussion**

Since 2013, two similar PISA CPS units that require students to make multiple-choice selections and collaborate with team agents to solve problems have been created to assess Taiwanese students' CPS skills in science [10,12,22]. Herborn et al. [23] compared humanto-agent and human-to-human tests in the same scenario from the original PISA 2015 CPS assessment. The authors revealed no significant differences between types of collaboration partners. Moreover, according to the report on CPS provided by PISA, CPS has the highest correlation (0.77) with science; this correlation was greater than that in either mathematics or reading [22]. Hence, the CPS behavioral transition patterns exhibited in Taiwanese students' science assessments were selected for discussion in this study.

Graesser et al. [24] mentioned that if students are proficient in "enacting plans," then they can likely be observed to adaptively respond to and make progress on group goals. From the overall analysis of CPS behavioral transition patterns, we found that if students exhibit high performance in enacting planes, then they also have high performance in following rules of engagement. Additionally, students who could monitor and repair shared understanding were likely to have more conversations and communicate with other team members. By contrast, if a team failed to solve a given problem, then the team members often exhibited less reflective discourse and an inability to transform their discussion into an executable plan to solve the problem [25,26].

PISA results [22] suggested that overall, girls perform significantly higher than boys do in CPS. In addition, past comparisons of girls and boys indicated that girls like to communicate and collaborate with others but boys tend to work independently [27]. However, this phenomenon has evolved over time, and the gender gap has decreased in science, although girls still demonstrate better reception and interpretation than boys

do [10,28–35]. The C3→D2 transition results reveal slight differences between girls and boys, suggesting that if girls can follow the rules of engagement, then they can also monitor the results of actions and evaluate success. However, some boys who could follow the rules of engagement still could not monitor results or evaluate success.

The results of the behavioral transition patterns among the three city sectors suggest that if students whose schools are located in emerging and traditional industrial districts or less developed and remote areas cannot communicate with team members about the actions to be/being performed, they also cannot enact plans in science. However, the school sector was not the most important factor influencing CPS skills. A previous study found that instead, being more physically active or attending more physical education classes per week exhibited a greater influence on CPS [22].

#### **6. Conclusions and Future Work**

In this study, a coding scheme that combines student CPS skills and mastery levels in a human-agent online CPS assessment was proposed to understand the behavioral transition patterns of CPS skills in science by applying LSA. The study provides the following major findings:


This study was conducted in Taiwan; hence, the results cannot be directly expanded to other countries because of differences in curriculum guidelines and commonly used learning models. However, the model, which included a coding scheme and method of applying LSA, may be used in other countries to identify students' behavioral transition patterns. Moreover, the CPS assessment platform has been integrated into a large adaptive learning platform in Taiwan that also includes the corresponding CPS learning materials. Therefore, in the future, teachers may analyze their students' behavioral transition patterns in class by using the assessment units and apply the results to select appropriate activities and teaching materials.

To further enrich the literature related to online CPS assessment, other subjects such as math, reading, and social science can be included to explore overall and individual behavioral transition patterns. The findings also suggest that teachers should design additional activities in class to address their students' weaker transition patterns. In addition, when PISA releases the students' secondary data from the CPS assessment, the proposed coding scheme may be applied to analyze students' behavioral transition patterns within and between regions. Moreover, the cultural differences of each region may influence the behavioral transition patterns when students work together to solve a problem. Therefore, a culture gap between regions or between Asia and the West can be analyzed in the same manner according to PISA secondary data.

**Author Contributions:** Data curation, Z.-Y.L.; Formal analysis, Z.-Y.L. and W.-C.H.; Methodology, C.-H.L.; Software, C.-H.L.; Writing—original draft, W.-C.H.; Writing—review & editing, C.-H.L., P.-L.T. and P.-J.H. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the Ministry of Science and Technology, Taiwan, under Grant MOST 103-2511-S-142-012-MY3, 109-2511-H-142-004-, and 110-2511-H-142-006-MY2.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data are not publicly available but may be made available on request from the corresponding author.

**Acknowledgments:** We thank the participants of this study. We are also grateful to the team of Bor-Chen Kuo for providing the Taiwanese CPS website. This manuscript was edited by Wallace Academic Editing.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **Sustained Use of Virtual Meeting Platforms for Classes in the Post-Coronavirus Era: The Mediating Effects of Technology Readiness and Social Presence**

**Yumi Yi and Rosemary Hyejin Moon \***

Humanities Research Institute, Chung-Ang University, Dongjak-gu, Seoul 06974, Korea; joystu@cau.ac.kr **\*** Correspondence: rosiehyejinmoon@hotmail.com; Tel.: +82-02-813-7354

**Abstract:** In response to the COVID-19 pandemic, educational institutions were forced to turn to online classes that are either recorded or taught live on virtual meeting platforms. Students could, therefore, attend classes from virtually any location using their mobile devices and Internet access. Despite the prolonged pandemic, little attention has been paid to whether offering courses on a virtual meeting platform is sustainable. This study, therefore, explores the antecedents of students' intentions regarding the sustained use of virtual meeting platforms for academic courses. We investigated the relationship between technology readiness (TR) and perceived social presence (SP) within a virtual communication setting with course satisfaction and sustained use intention. Data were collected via a survey from 525 college students in South Korea who had attended classes using a virtual meeting platform. Serial mediation analysis revealed a pathway in which SP and course satisfaction in series fully mediate the positive relationship between technology readiness and sustainability. This study discusses the implications in relation to the sustainability of virtual technology-based courses as a replacement of live classroom-based courses from a user perspective. Further research is needed to understand users' negative experiences of attending courses on virtual meeting platforms.

**Keywords:** virtual meeting platform; technology readiness; social presence; course satisfaction; sustained use intention; serial mediation

#### **1. Introduction**

More than a year has passed since the first case of coronavirus disease (COVID-19) was reported in December 2019 [1]. Since then, the entire world has been experiencing the current pandemic, with many countries implementing stay-at-home regulations to stop the spread of the virus. Education is one of the areas that witnessed the most rapid changes during this period. According to the United Nations Educational, Scientific and Cultural Organization (UNESCO) statistics, 190 countries had closed their educational institutions as of May 2020; these institutions account for approximately 90% of the world's students. Among these countries, 100 have not yet announced the reopening of schools, 65 have partially reopened their schools, and 32 have announced that students will complete their education online [2].

Multiple forms of technologies have been adopted in education in the last decades. However, the COVID-19 pandemic has caused a shift from a universal face-to-face education system based on physical classrooms to a system involving online videos and virtual meeting platforms [3,4]. This transition was not due to the natural progression of technological development; rather, the change was forced upon educational institutions by an external factor—the COVID-19 pandemic [5,6]. Although the necessity for this change occurred abruptly, existing technology enabled a quick transition in response to the external pressure, the pandemic. The application of existing communication technology to facilitate education is more than a temporary fix because of the pandemic; it will be used even after the pandemic, and students will be given a choice of education methods [7].

**Citation:** Yi, Y.; Moon, R.H. Sustained Use of Virtual Meeting Platforms for Classes in the Post-Coronavirus Era: The Mediating Effects of Technology Readiness and Social Presence. *Sustainability* **2021**, *13*, 8203. https://doi.org/10.3390/ su13158203

Academic Editors: Ana B. Bernardo, Enrique-Javier Díez-Gutiérrez, Adrian Castro-Lopez, Javier Puente and Leandro Almeida

Received: 15 June 2021 Accepted: 19 July 2021 Published: 22 July 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

As an example, a study by Pérez-Villalobos et al. found that students in Vietnam and Indonesia preferred distance learning, while students in Poland stated that they would prefer to return to face-to-face learning after the pandemic [8]. These results indicate that the advantages of being liberated from temporal and spatial constraints—an advantage of virtual communication—will still be valid, even when the world is ready to return to physical classroom learning. The value of educational methods involving virtual meeting platforms even after the end of the pandemic requires investigation in terms of its relative effectiveness, as well as students' and instructors' preferences concerning distance learning instead of traditional education methods [9].

To understand the sustained use intentions of educational methods utilizing virtual communication, we must first define the meaning of virtual communication. The term "virtual" indicates a state in which an individual can perform desired actions without physical movement or contact by using new technology media such as a computer [10]. From the communication perspective, virtual communication refers to communication that is carried out from a distance, without requiring physical contact or movement to meet others in person. For example, distance learning in education, which does not require physical contact or movement by students or instructors, is enabled by virtual communication. In its early stages, virtual communication methods included tools such as telephones and chat rooms on the Internet; with technological development came more recent technologies, such as videotelephony and communication systems, that enable virtual meetings. As it advanced, technological development was also applied to education, starting with chat programs used for education in the late 20th century to online community group systems [11].

Among the various virtual communication tools available today, this study focuses on education that utilizes virtual meeting platforms that have been replacing classroom learning since the onset of the COVID-19 pandemic. This system, often referred to synchronous videoconferencing [12] or technology-mediated learning, is regarded as the most appropriate alternative to classroom learning, as the only aspect not present in this educational method is 'being in the same place'. Similar to face-to-face communication between the instructor and the learner, this system also offers immediacy and has the advantage of creating a feeling of teaching presence.

Research on online distance education emphasized the role of social presence as it promotes social interactions in online learning [13,14] and decreases the feeling of isolation that is often reported by online students [15]. While students are currently enrolled in classes that utilize a virtual meeting platform due to the pandemic, they are not voluntary online students; therefore, some are possibly more ready to enjoy the advantages that the newly introduced system brought, while others are more likely to experience difficulty and feel isolated. Considering the above, this study focuses on technology readiness (TR) as a major factor influencing the sustained use intention of education via virtual meeting platforms. It also aims to confirm whether TR influences sustained use intention through social presence (SP) perception and course satisfaction. Unlike studies that focus on the effectiveness of the virtual classroom (cost, safety, equity, feasibility, and efficacy [12,16]), this study is novel as it focuses on the effects of individual tendencies (i.e., TR) and the consequent perception (i.e., SP) of the learners. Furthermore, if learning methods should be analyzed based on effectiveness alone, there may be choices in terms of educational systems from the perspective of instructors in the future. However, confirming the sustained use intentions of education methods relating to learners' preferences has unique value, as it explores the significance of virtual communication education methods from a student's perspective. The current research provided empirical evidence supporting the key mediating roles of SP and course satisfaction, as the results revealed that TR, which is an individual trait, did not directly influence individuals' sustained use intentions of virtual meeting platforms. Rather, it had a positive influence mediated by SP and course satisfaction.

#### **2. Theoretical Background and Hypotheses**

#### *2.1. Technology Readiness, Social Presence, Course Satisfaction, and Sustainability*

Parasuraman defined TR as "people's propensity to embrace and use new technologies for accomplishing goals in home life and at work" (p. 308, [17]). It is an individual trait that can be conceptualized as an overall state of mind or belief that determines an individual's predisposition toward the use of new technology [18]. Based on qualitative research, Parasuraman [17] argued that technology can induce both positive and negative emotions in individuals and that some of these emotions may be dominant in individuals, possibly affecting their degree of technology acceptance. Against this backdrop, their study developed a technology readiness index (TRI), arguing that the accuracy of consumer behavior prediction increases when the consumers' level of TR is considered when a company develops a new technology [17]. The TRI developed by Parasuraman [17] has four dimensions: optimism, innovativeness, insecurity, and discomfort. Optimism refers to a positive view of technology and a general belief in its benefit, including enhanced control, flexibility, and efficiency in people's lives; innovativeness concerns one's tendency to be a technological pioneer. Opposed to these two dimensions are insecurity, which derives from skepticism about the technology's ability to work properly and concerns about its negative consequences, and discomfort, which is the perception of lack of control over technology and feeling overwhelmed [19]. The dimensions of optimism and innovativeness contribute to TR as "motivators", while insecurity and discomfort are considered to be "inhibitors" [19]. Depending on the researchers using the index, TR can be conceptualized as a four-dimensional, two-dimensional, or one-dimensional (overall composite) construct. The TRI showed a positive correlation with technology use tendencies [17], and further studies showed that it had a positive influence on the quality evaluations of online services and online behavior [20]; however, existing findings on the relationship between TR and technology use are inconsistent [21], which calls for further empirical evidence.

As can be seen by the original intent for developing the TRI, studies have used this scale mainly to investigate consumer choice. These types of studies are often conducted to confirm consumer perceptions on matters such as the formats of digital transformation in the market: for example, Internet banking [21–24]. Although relatively scarce, education researchers also emphasized TR in the education context since the success of any system hinges not only on the learners' acceptance but also their state of technology readiness [25,26]. Studies on the relationship between TR and education have investigated the factors that influence students' choice of e-learning [9], the effect of demographics on TR, the qualifications of teachers who integrate technology into their teaching processes [27], and whether TR determines students' readiness for online classes after COVID-19 [28]. El Alfy, Gómez, and Ivanov found that TR also influences instructors' behavioral intentions [29]. As there is currently no empirical evidence supporting the relationship between TR and students' behavioral intentions, we developed the following hypothesis, based on a view of TR as a robust predictor of technology-related behavioral intention [19] and the above findings.

#### **Hypothesis 1.** *TR is positively related to the sustained use intention of education using a virtual meeting platform*.

Presenting the TRI version 2.0, Parasuraman and Colby proposed further exploration of the dynamics involved in technology adoption; they also argued that the TRI may function as a key psychographic variable in decision-oriented research within the context of technology-based innovation [19]. To predict students' intention of sustained technology use, even after the end of the pandemic, we focused on SP and satisfaction as the key mediators in the relationship between TR and sustainability, considering that students' impression of SP is determined by their perception of the use of the medium for education, rather than the medium itself [30], as well as evidence supporting the strong relationship between SP and satisfaction (e.g., see meta-analysis by Richardson et al. [31]). More

specifically, TR not only directly affects sustainability but also leads to different levels of psychological experiences, such as SP, among students; such experiences determine their satisfaction with technology-enabled classrooms, which ultimately leads to sustained use intentions of virtual meeting platforms in learning.

SP was first discussed by Short, Williams, and Christie [32], who defined it as the "degree of salience of the other person in the interaction and the consequent salience of the interpersonal relationship" (p. 59, [15]). According to Gunawardena and Zittle, SP is associated with intimacy and immediacy [33]. Ultimately, this indicates that SP is related to how the user perceives a subject through different media and can thus be defined in diverse ways. Various definitions of SP include the degree to which one feels about the presence of another in communication [34], "the sense of being with others" (p. 456, [35]), the "level of awareness of the co-presence of another human, being or intelligence," and "feeling that one has some level of access or insight into the other's intentional, cognitive or affective states" [36]. Combining these definitions, this discourse not only involves the level upon which one perceives the presence of another in a mediated communication situation but also the degree to which the emotional awareness of others in an online environment is similar to awareness of them in an in-person environment. This is an important indicator for measuring the effects of communication today, at a time when various media environments are unavoidable. As such, several tools to measure SP have been developed [32,37–39], from which we used Kim's [37] work that presented four subfactors of SP: mutual attention and support, affective connectedness, sense of community, and open communication. This scale is particularly suitable, as it was developed to measure SP in the environment of distance education, similar to the setting of this study.

Existing studies on the relationship between TR and SP are scarce; they do, however, include a study on individuals' sharing of knowledge [40]. However, this study did not examine the direct relationship between TR and SP; instead, it examined how each dimension of TR and SP influenced the knowledge sharing intentions of its participants, based on the technology acceptance model (TAM) [41]. The TAM is a psychological theory used to explain individuals' technology acceptance intention or, in other words, behavioral intentions toward technology [41]. As such, there are existing studies that confirm the technology-related characteristics of participants through the relationship between TR and the TAM [42–46]. Furthermore, there are studies that have affirmed the relationship between technology acceptance and SP [47,48], but the context is limited to e-learning or distance learning. These existing studies, which integrate the TAM and TR perspectives, indicate that a positive attitude toward the technology of virtual meeting platforms leads to stronger experiences of ease of use and usefulness, stronger engagement with the course, and a stronger experience of SP during the course. Based on these existing findings, the following hypothesis can be established.

#### **Hypothesis 2.** *TR is positively related to SP*.

In the context of distance learning using virtual meeting systems, the relationship between SP and course satisfaction and that between course satisfaction and sustained use intention are both confirmed by existing studies on e-learning. Stacey observed that SP increased because of messages sent between the students, which led to intimacy between them; this, in turn, strongly induced learning motivation [49]. Furthermore, Gunawardena and Zittle studied the relationship between SP and course satisfaction among college students and found that SP was an important variable predicting satisfaction [30]. Furthermore, Richardson and Swan verified a correlation between SP and instructor satisfaction [39] and considered how such satisfaction was related to learning outcomes. Richardson, Maeda, and Caskurlu also identified factors mediating the relationship between SP and course satisfaction in online learning settings [31]. Based on the existing evidence, we propose the following hypothesis.

**Hypothesis 3.** *Social presence is positively related to course satisfaction*.

Marketing research has examined how satisfaction with a certain product can positively influence the intention to use the product again [50–52]. As such, exploring the factors that affect individuals' degree of satisfaction has been an important research area. Consumers' repurchase behavior is related to service satisfaction, because repurchase is a direct compensatory behavior resulting from satisfaction. In the same vein, students' course satisfaction can predict the "same choice" for future courses. In this regard, the same choice refers to repeated choices regarding several aspects, such as the same instructor, same learning method, same group, or same classmates. However, given that academic achievement is an important yardstick in education, studies on academic achievement have measured whether learning objectives have been met. For example, existing research on problem-solving methods in distance learning includes an investigation on the influence of a method that encourages interaction between small groups of students on academic satisfaction [53] and studies on the influence of technology intimacy or learning environment on distance learning satisfaction [54]. These studies mainly explored factors influencing satisfaction, rather than sustained use intentions toward learning methods, because satisfaction with online learning is expected to influence not only academic outcomes but also sustained use intentions. While there is insufficient research on the effect of academic satisfaction on choosing the same learning methods later, we propose the following hypothesis, based on existing marketing research on the influence of emotional satisfaction on behavior.

**Hypothesis 4.** *Course satisfaction is positively related to the sustained use intention of courses on virtual meeting platforms*.

#### *2.2. Mediating Roles of Social Presence and Course Satisfaction between Technology Readiness and the Sustained Use Intention*

Using cultural comparisons, Rojas-Méndez, Parasuraman, and Papadopoulos [55] discovered that demographic characteristics and cultural differences are associated with the adoption of new technologies. They reported that individual traits associated with technology are related to the individual's use of technology. However, research on technology use intentions in education revolves around the issue of technology acceptance [56,57]. This is because an attitude of not rejecting technology as the medium is one of the most important preliminary factors for technology-enabled education. In confirming the technology use intentions through the TAM model, some studies determined the influence of users' emotional attitudes, such as negative emotions toward technology, on technology adoption [32,56]. However, it is also possible that TR may not directly predict the technology use intentions; this is confirmed by studies that indicate that negative emotions, such as a fear of technology, do not have a negative influence on technology use intentions [58].

Based on existing evidence on the sense of community created during conferences and how it influences how participants consider computer-mediated communication (CMC) as a "social medium" [30], it is reasonable to expect that SP could play a mediating role in an online education environment. Additionally, among two widely accepted mediating mechanisms (TAM and quality-value-satisfaction; QVS) in the relationship between TR and technology use, the QVS includes satisfaction [59]. Blut and Wang [59] argued that satisfaction may function as a key mediator, as people with favorable disposition toward technology are more easily satisfied because they tend to focus on the positive aspects of employing a given technology. Therefore, the hypotheses for mediating effects are proposed as follows.

**Hypothesis 5.** *SP mediates the relationship between TR and sustained use intention of courses on virtual meeting platforms.*

**Hypothesis 6.** *Course satisfaction mediates the relationship between TR and sustained use intention of courses on virtual meeting platforms*.

**Hypothesis 7.** *TR is positively related to sustained use intention of courses on virtual meeting platforms via the chain of SP and course satisfaction*.

#### **3. Materials and Methods**

#### *3.1. Participants and Procedure*

We recruited 552 college students from a South Korean crowdsourcing website similar to MTurk (http://www.embrain.com, accessed on 30 April 2021). Since the onset of the pandemic, the Ministry of Education in Korea recommended universities to turn all classes online, and among several options, the most widely chosen teaching method was real-time online classes using a virtual meeting platform. Therefore, Korean students were considered adequate research sample for the current research aims. The research participants were students who voluntarily participated in surveys on the survey platform as a research participant pool and were paid for participation. We restricted our recruitment to only those who meet the following two criteria: participants must be currently enrolled in a four-year course at a university located in Korea; they must have experience in attending classes administered on a virtual meeting platform, such as Zoom, a popular meeting platform that supports meetings involving up to 1000 participants. Among those who completed the survey, we excluded the data of 13 participants whose ages exceeded 28 years and data of 14 participants whose reaction period was either too short or too long. Of the remaining sample (*N* = 525), 233 were male (44%), and the average age was 21.63 (SD = 2.26). Participants were evenly distributed across grades (1: *n* = 126; 2: *n* = 138; 3: *n* = 114; and 4 or above: *n* = 147). Informed consent was obtained from all participants involved in the study. The institutional review board of Chung-Ang University in Seoul, Korea (1041078-202104-HR-112-01), approved our study and confirmed that no further review is necessary, due to the use of anonymized data.

#### *3.2. Measurements*

#### 3.2.1. Technology Readiness

We measured participants' TR using the TRI 2.0 [17]. The TRI 2.0 consists of 16 items and originated from the 36-item TRI 1.0 scale [17]. The scale is intended to measure "people's propensity to embrace and use cutting-edge technologies." An example item is: "other people come to me for advice on new technology" [19]. Similar to the TRI 1.0, the TRI 2.0 comprises four dimensions—optimism, innovativeness, discomfort, and insecurity—and each dimension includes four items. Parasuraman and Colby [16] reported acceptable results for factor analysis, reliability, and discriminant validity. Two Korean– English bilingual translators independently translated the items from the original English to Korean; when there were disagreements, they reached consensus by editing those items together. Following Parasuraman and Colby [19], we reverse coded the scores on discomfort and insecurity and computed the average score on the four dimensions. The TRI 2.0 uses a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The overall reliability (Cronbach's α) was 0.73, and the reliabilities of each dimension varied from 0.65 to 0.75.

#### 3.2.2. Social Presence

Kim's [37] SP scale was used to measure participants' experience of SP while attending courses using a virtual meeting platform. The scale follows a four-factor model with the following factors: mutual attention and support (6 items), affective connectedness (5 items), sense of community (4 items), and open communication (4 items). As the instrument was originally developed in Korean, translation was not necessary. The SP scale was also rated on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Following Kim [37], we computed a single reliability index (Cronbach's α = 0.90), and the reliabilities of each dimension varied from 0.72 to 0.85.

#### 3.2.3. Course Satisfaction

To measure participants' overall satisfaction with their chosen course, we created a single item: "Please indicate the extent to which you are satisfied with the virtual teaching method." The participants indicated their overall satisfaction using a 5-point scale ranging from 1 (not at all) to 5 (very much).

#### 3.2.4. Sustained Use Intention

Sustainability was also measured using a single item that was developed for the current research objective. The item was: "Imagine that the COVID-19 pandemic came to an end. If you are given a choice between face-to-face lectures and live lectures on a virtual meeting platform, such as Zoom, how likely are you to choose the latter?" The 5-point scale ranged from 1 (very unlikely) to 5 (very likely).

#### 3.2.5. Control Variables

We collected demographic variables to clean the data and utilize as control variables. We asked participants to report their gender (1 = male, 2 = female), student enrollment status (1 = enrolled, 2 = leave of absence), experience in attending courses using virtual meeting platforms (yes or no), age (in years), and grade (based on the number of semesters completed). The following analyses included two control variables: gender and grade.

#### *3.3. Data Analyses*

First, we conducted a confirmatory factor analysis (CFA) to ensure factor structures of the two scales—TRI 2.0 and the SP scale—using AMOS 21.0. Next, SPSS software 21.0 was used to obtain descriptive statistics, reliabilities, and correlations. We then tested our research hypotheses by performing hierarchical regressions and using model 6 of the PROCESS macro [60]. Following the recommendations of Preacher and Hayes [61], we analyzed the serial mediation effect by employing a bootstrapping technique involving 5000 resamples to calculate 95% confidence intervals (CIs).

#### **4. Results**

The second order two-factor model was tested using AMOS 21.0 software, and the CFA indicated a mixed global fit, ranging from acceptable to modest levels: χ2/df = 3.29 (*p* < 0.001), goodness of fit index = 0.82, comparative fit index (CFI) = 0.82, Tucker–Lewis index (TLI) = 0.81, and root mean square error of approximation (RMSEA) = 0.07. Although the results failed to reach the recommended cutoffs [62] for RMSEA (<0.06), TLI (>0.95), and CFI (>0.95), we confirmed that the current second order model outperformed alternative models such as first order one-factor, two-factor, and eight-factor models. Moreover, based on claims that measures with many observed indicators and latent factors often show only a modest to poor model fit [63], we decided to perform further empirical analyses.

Table 1 illustrates the means, standard deviations, and correlation coefficients of the research variables. Gender is correlated with TR (*r* = −0.18, *p* < 0.01), indicating that male participants tend to have higher TRI scores than female participants. Furthermore, participants in higher grades are more likely to rate the sustainability of attending classes on a virtual meeting platform positively (*r* = 0.12, *p* < 0.01). The significant positive correlations between TR and SP (*r* = 0.19, *p* < 0.01) and between SP and course satisfaction (*r* = 0.26, *p* < 0.01) provide initial support for Hypotheses 1 and 2, respectively.


**Table 1.** Means, standard deviations, and correlation coefficients of the research variables.

Notes: Gender (1 = male, 2 = female); Grade (1 = 1 to 2, 2 = 3 to 4, 3 = 5 to 6, 4 = 7 or more semesters completed); *N* = 525; \* *p* < 0.05, \*\* *p* < 0.01. TR: technology readiness, SP: social presence, SD: standard deviation.

This study calculated all path coefficients simultaneously, while controlling for participant gender and grade. As our model involved two mediators—SP and course satisfaction we used model 6 of the PROCESS macro of SPSS [52] to test the serial multiple mediator model with three specific indirect effects (Hypotheses 5–7) and one direct effect. The output of the serial mediator model can be seen in Figure 1, and the model coefficients and other statistics are summarized in Table 2.

**Figure 1.** Statistical diagram of the serial multiple mediator model.

The bootstrap results of the direct effect of TR on sustainability do not support Hypothesis 1, as the bootstrap CIs include zero (BootLLCI = −0.13; BootULCI = 0.36). However, the results of the serial mediation analysis show that the path from TR to SP is positive and significant (*b* = 0.28, *t* = 4.39, *p* < 0.001), supporting Hypothesis 2. Furthermore, the influence of SP on course satisfaction is also positive and significant (*b* = 0.38, *t* = 5.87, *p* < 0.001), and therefore, Hypothesis 3 is supported. Finally, the influence of course satisfaction on sustainability is positive and significant (*b* = 0.72, *t* = 14.31, *p* < 0.001), supporting Hypothesis 4.


**Table 2.** Regression coefficients, standard errors, and model summary information for the serial multiple mediator model depicted in Figure 1.

Notes: SE: standard error, CS: course satisfaction.

Finally, the bootstrap CIs generated by the PROCESS macro for all indirect effects are reported in Table 3. The first indirect effect, labeled "Ind1," is the specific indirect effect of TR on sustainability, through SP perception (X→M1→Y). Unexpectedly, we find a negative and significant indirect effect (lower-level CI = −0.12; upper-level CI = −0.02). Although this indirect effect is statistically significant, we consider this unexpected relationship to be an artifact, due to the possible statistical suppression effect that arises when the overlapping variance between two predictors is large [64]. The following evidence supports our inference on the results for "Ind1": first, the correlation coefficients between TR and SP and between SP and sustainability are 0.19 (*p* < 0.01) and 0.01 (n.s.), respectively; second, conducting a single mediator model (PROCESS macro model 4) did not produce any significant results (Direct effect = 0.11, SE = 0.13, *p* = 0.41 [LLCI = −0.15, ULCI = 0.36]; Indirect effect = 0.01, Boot SE = 0.03 [BootLLCI = −0.04, BootULCI = 0.07]). The second indirect effect, labeled "Ind2," is the indirect effect of TR on sustainability through course satisfaction (X→M2→Y). Unfortunately, the bootstrap result again does not support Hypothesis 6, as the bootstrap CI includes zero (BootLLCI = −0.06, BootULCI = 0.23). Finally, the third indirect effect, labeled "Ind3" (X→M1 → M2 →Y), is the indirect effect of TR on sustainability through perception of SP and course satisfaction in serial. The bootstrap results of the third indirect effect support Hypothesis 7 (BootLLCI = 0.03, BootULCI = 0.13). Therefore, consistent with Hypothesis 7, TR predicts sustainability only through serial mediators. More specifically, TR benefits sustainability only, as it enhances perception of SP when participants attend classes on a virtual meeting platform, and SP increases the likelihood of feeling satisfied with the virtual teaching method.

**Table 3.** Bootstrap confidence intervals for all direct and indirect effects.


Notes: CI: confidence interval, LL: lower limit, UL: upper limit.

#### **5. Discussion**

Virtual classrooms adopted due to the COVID-19 pandemic and the use of virtual meeting platforms that enabled virtual classes brought into focus the advantages of this system especially for those who travel long distances to work or school [65]. However, as distance and time are not the only important variables for purposeful communication at work or in school, it is necessary to carefully examine the variables that lead to intention to continue to use this system, even after the pandemic.

Considering that the virtual meeting platform was implemented as an emergency measure due to the pandemic, this study sought to explore important antecedents that predicted students' intention to continue to use the system even after the pandemic. Drawing from previous research, we developed seven hypotheses explaining the relationship between students' TR and their intentions regarding the sustained use of virtual meeting platforms for academic courses and empirically tested each. The results of our analysis indicated that whereas TR, SP, and course satisfaction were related as hypothesized in Hypotheses 2 to 4, respectively, Hypothesis 1, which claimed the positive relationship between TR and the sustained use intention, was not supported. Furthermore, among Hypotheses 5 to 7, which postulated the mediating effects of SP and course satisfaction, respectively and in series, only Hypothesis 7 was supported. To be more specific, the technology propensity of individuals (i.e., TR) affected their perception of partners (i.e., SP), which, in turn, influenced their satisfaction (i.e., course satisfaction). These factors finally affected their sustained use intention. Regarding failure to support the main effect (Hypothesis 1), the following reasons may provide an alternative explanation on the relationship between TR and sustainability. First, although it was beyond our expectation, our result was consistent with existing studies, which found that negative emotions toward technology do not lead to negative influences on technology use intentions [58]. Second, the result could be affected by the wording of the measurement, as well as the unique characteristics of the current situation. The item used to measure sustained use intention was developed for the current purpose, and it included a hypothetical situation where the students are given an option to choose between in-person or virtual classes after the pandemic. Responses to such a hypothetical situation could yield unexpected influences by confounding variables, such as students' longing to go back to in-person classes or their wish to return to their pre-pandemic lifestyles. This interpretation aligns with Paechter and Maier's [66] work which showed a preference for face-to-face classes, rather than online classes, for interpersonal relationships among students who have not yet experienced face-to-face classes. Similarly, in the current research, we found a negative and statistically significant correlation between grade and sustainability, as shown in Table 1. This could reflect students' longing for situations that they have not yet experienced due to the pandemic, irrespective of their individual attitudes toward technology. This, in part, explains why our results differed from those of previous studies that report an inverse correlation between age and TR [59].

The contributions of the current study are as follows. This study was the first to provide empirical evidence of the relationship between TR and SP. Rather than examining how a specific methodology or a technology device determines SP, this study adopted the perspective of the user or student, keeping in mind that SP experience may change, based on the user's tendencies. Existing studies on CMC showed that, compared with face-toface communication, a technology environment yields more difficulty in achieving social functions, as non-verbal information cannot be easily attained [67]. This indicates that nonverbal information plays an important role in SP [32]. As such, video—which can transmit more non-verbal information—has a higher degree of SP than audio; in this case, SP becomes an indicator of the psychological distance between people who communicate with one another [32]. Under the COVID-19 pandemic, students are exposed to the controlled situation where education involves CMC. However, some students experience a high level of SP in the same context while others do not. We identified TR as an important individual difference and supported that TR is positively associated with students' experience of SP while they attend classes utilizing virtual communication. Second, contradictory to existing studies that showed mixed results on the relationship between TR and technology use [59], this study confirmed that TR was not directly related to sustained use intention but that the relationship was mediated through SP and course satisfaction. These mediating effects confirmed a long history of emphasis on the impact of course satisfaction on various behavioral intentions (e.g., intentions to recommend [68]). Finally, the current approach highlighted the value of investigating the sustainability of the educational environment

from a learners' perspective. Exploring the environmental factors that create subjective experience is not only important for learners who may either choose or change their learning environment but also for policy makers who aim to ensure a quality educational environment despite the limitations that the pandemic produces. Further research is needed to understand how students navigate their experience given the variation in their technology readiness and what can be done to help enhance their experience, quality of learning, and sense of belonging.

As such, this study is meaningful as it adopts a student-centric perspective of the education environment and the use of virtual meeting platforms. However, it has a number of limitations. First, we were unable to determine specific problems that users experience in the communication environment when using virtual meeting platforms. Future studies should therefore examine the limitations regarding the transmission of non-verbal communication, as well as the direct differences between CMC and face-to-face communication. These considerations can supplement the negative factors related to communication and human relationships that are inherent to virtual meeting platforms, which will be used more often in the future. Second, the research was performed cross-sectionally. Pandemicrelated administrative orders and country-wide constraints on the educational system provided special value in investigating students' experience and intentions regarding the methodology that was adopted as an emergency measure. However, as a cross-sectional research design limits causal inference, future research should involve an experimental or longitudinal design to have better control over the environment. Third, in the current study, the second mediator, course satisfaction, was measured with a single item. Several researchers have proposed the multi-dimensionality of course satisfaction. For example, Endres et al. [68] proposed the multidimensional nature of online course satisfaction, including satisfaction with instructors, course materials, learning practices, student-to-student interaction, and online tools. Future research should employ a more elaborate measurement of this mediator to replicate and develop the current research findings. Finally, the virtual learning environment model [69,70] proposed that human dimensions (administrators, students, and instructors) influence design dimensions (organizational, learning, and teaching practices) that affect educational success. As the current study adopted a student-centric perspective, we were unable to include other important predictors of students' evaluation of courses on a virtual meeting platform.

#### **6. Conclusions**

This study aimed to identify factors that influenced sustained use intention of virtual meeting platforms, which were introduced in the education environment abruptly due to the COVID-19 pandemic. The continuous development of technology fostered the employment of various methodologies in education. The introduction of remote learning was not simply to overcome the spatial limitations presented by distance; there has been ongoing research on online learning to secure flexibility in information access and to network with more information. The limitations created by large classrooms have also been presented as a problem. Furthermore, the development of a communication medium that enabled CMC involved attempts at developing a medium that resembles face-to-face communication. The virtual meeting platform developed for this purpose has overcome the limitations of other CMCs in terms of non-verbal information through video systems, as well as the limitations of geographical distance, by using online remote devices. While this system shares features with face-to-face communication, it has some limitations, as communication participants often fail to fully perceive SP as they communicate through media. Furthermore, depending on the users' perception of technology devices, technologymediated communication may disturb their perception of the communication environment being natural, which may influence communication satisfaction.

This study aimed to identify the effects of individuals' tendencies in terms of technology and the perceived SP in the communication environment that utilizes technology on communication satisfaction and sustained technology use intention. The results indicated

that TR, which is an individual trait, did not directly influence individuals' sustained use intentions of virtual meeting platforms. Rather, it had a positive influence mediated by SP and course satisfaction. Ultimately, these findings supported the findings of existing studies, which show that higher communication satisfaction has a positive influence on media use intentions. It also confirmed the influence of TR and SP, which affected satisfaction. Unlike reports of an inverse correlation between TR and age, this study found that students in higher grades had higher sustained use intentions of virtual meeting platforms, compared with students in lower grades. As the students were forced to use technology due to the pandemic, rather than choosing to use it of their own volition, the intention to use virtual meeting platforms after the pandemic reflected the desire for the in-person classrooms experienced in pre-pandemic times and the longing for human relationships on college campuses.

**Author Contributions:** Conceptualization, Y.Y.; Project Administration, Y.Y.; Methodology, R.H.M.; Formal Analysis, R.H.M.; Writing—Original Draft Preparation, Y.Y. and R.H.M.; Writing—Review and Editing, Y.Y. and R.H.M.; Funding Acquisition, Y.Y. and R.H.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2017S1A6A3A01078538).

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) of Chung-Ang University (1041078-202104-HR-112-01, 21 May 2021).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Data available on request.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **The Effects of European Recommendations on the Validation of Lifelong Learning: A Quality Assurance Model for VET in Spain**

**Alfonso Redondo 1,\*, Salvador Castillo 1, Luis Carro <sup>2</sup> and Paulino Martín <sup>3</sup>**


**Abstract:** Validation is an effective procedure for recognising the skills and knowledge acquired by individuals. However, the validation mechanisms in each country are not always easy to understand, especially due to a lack of information/data. The aim of this paper is to design a management system based on processes for the accreditation of professional competences acquired by work experience in Spain, considering European regulations. This is carried out through a contextualization of both regulatory frameworks through a bibliographic review, as well as the analysis of the quantitative and qualitative outcomes of the surveys that groups of experts involved in these tasks/procedures completed in Spain. All this has made it possible to design a model for the validation of learning acquired through professional experience as well as non-formal and informal channels in Europe and Spain, which facilitates the process of accreditation of competences.

**Keywords:** VET; European; validation principles; Spain

#### **1. Introduction**

In the field of Vocational Education and Training (VET), the importance of its professional development and validations are broadly acknowledged at national and international levels. Certification, validation, and identification are widely accepted mechanisms of recognition. Validation means the verification, by the qualified authority, of the outcomes of study obtained by a person in non-formal, formal, and informal education. Validation of the outcomes of non-formal adult education implicates the identification, evaluation and recognition of knowledge and skills a person obtains during life.

Strategic directions of validation of non-formal adult education have shown their significant role in the economy and society in general, allowing professional mobility, social integration, as well as the development of the concept of lifelong education [1]. Validation is an effective procedure to obtain qualifications that allows for recognition of the worker's knowledge; however, the validation mechanisms within each country are not well understood and the corresponding data are not sufficient. VET plays a relevant role in the change towards more knowledge-intensive societies. Indeed, around half of all jobs in 2020 will need a medium level qualification, which will be obtained by some form of VET [2].

In the 1970s, the United States of America raised an educational practice called prior learning assessment (PLA), also known as recognition of prior learning (RPL), assessment of prior learning (APL), prior learning assessment and recognition (PLAR), and validation of prior learning (VPL). This was carried out following research that displayed the viability of equating and assessing adults' prior non-college learning with college-level learning [3,4].

**Citation:** Redondo, A.; Castillo, S.; Carro, L.; Martín, P. The Effects of European Recommendations on the Validation of Lifelong Learning: A Quality Assurance Model for VET in Spain. *Sustainability* **2021**, *13*, 7283. https://doi.org/10.3390/su13137283

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Javier Puente and Leandro Almeida

Received: 10 May 2021 Accepted: 23 June 2021 Published: 29 June 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

The increase of national qualifications frameworks at international levels has generated a supportive legislative and policy architecture for the development of PLA. Over the years, a wide range of practices have been established across the United Kingdom, North America, Scandinavia, New Zealand, Australia, and South Africa. Practices to test prior learning change from large-scale examinations to individual or learner petitioning. Postsecondary institutions have worked out their own approaches to PLA. The portfolio method is often implemented. In this approach, the apprentices display proof of their prior learning and give reasons as to how it relates to specific or general criteria. The access and/or credit are consequently conceded. Portfolios can be complemented using other assessment methods such as demonstrations, assigned essays, interviews, among others. Some PLA need learning from experience to join formal knowledge in a precise manner. Other practices run with the concept of equivalence; that is, determining that learning from experience is usually at the college level.

Terokhina [1] dealt with the issue of recognition of the results of non-formal adult education in the United States of America (USA) in order to be adopted by Ukraine. It was proposed to consider the College Level Examination Program (CLEP) and the credit system of the American Council on Education (ACE), since these are the mechanisms of non-formal adult education validation employed in USA. The process of validation can be assessing, discussion, demonstration of knowledge, training, interviewing, examination, communicative and social skills, critical reflection, expert explanations, simulations, declarative and portfolio methods.

The field of RPL in Quebec is being expanded due to the inclusion of different terminologies and policies established on the reality of contemporary Quebec society. A new stream of research associated to RPL and immigrant settlement and integration is starting to be more distinguished. The work recognized gaps in the research and recommended areas to encourage study, contributing to the field of scholarly endeavours of RPL. The conclusions from the consolidated RPL research should be a profit to the emerging field of RPL research [5].

Two issues in relation to assessment of prior learning were discussed by Aarkrog and Wahlgren [6]. These were the encounter of practical experience and school-based knowledge as well as the validity and reliability of the assessment procedures. The study showed that by mixing various assessment of prior learning methods and comparing the teachers' assessments, these address the issues of validity as well as reliability. In essence, it was discussed that validity and reliability can be boosted if the competencies are well established and the education system is aware of achieving a balance of knowing what, knowing why, and knowing how, given that teachers are appropriately qualified for the assessment procedures.

Paulos [7] studied the qualification of adult educators in Europe with special attention to Portugal. In this country, adult educators are a heterogeneous group. In the last decade, they have mainly worked in vocational training as well as in the recognition of prior learning. As a consequence, the professionalization of adult educators has been incipient, however, according to Guimarães [8], the profession of adult educator has never been organized. The unemployment of a part of the professionals implicated in adult education activities, especially in the recognition of prior learning process, has been due to the political disinvestment in the field of adult education that was carried out in the last few years. It should be relevant to guarantee the continuity of adult education policies to invest in the training of adult educators and secure stable employment situations, among others. These should be basic conditions for providing quality to adult education. Policies based on evidence should also be proposed to improve adult education. In addition, policies should include the valorisation of adult educators, allowing the creation of structured careers and their corresponding professional development.

Di Rienzo [9] carried out a study of empirical research completed in Italy. The interest was derived on the constant rising number of adults who were enrolled in graduate degree programmes. The research method joined qualitative and quantitative approaches, was

composed of a national survey on the basis of a questionnaire sent to the Italian universities, and took methods of qualitative orientation. The survey aided to detect organisational procedures and instruments for the validation of prior vocational learning. Furthermore, it was able to state that it is only from the outlook of lifelong learning that universities were able to tackle the object at issue.

RPL will return to its legitimate position, if it is able to ensure the RPL candidates' accomplishment. As a consequence of this, Snyman & van den Berg [10] set out the relevance of the profile of candidates. Empirical research was addressed, which involved the analysis of RPL candidates' life stories. The conclusion was that this requires an RPL approach that takes into account the significance of the outline at the apprentice, practitioner and institutional levels.

Werquin [11] analysed the recognition of non-formal and informal learning outcomes (RNFILO). This was pointed out as a policy tool and the possible applications and implementations were carried out. A definition of the key terms was put forward as well as a feasible rationale for organising recognition programmes and a schedule of pros and cons to carry them out.

More than 100 high-quality cases were put together on the state of validation of non-formal and informal learning (VNFIL) practices in 27 European countries as part of the Leonardo-OBSERVAL project. These covered specific strategies sustained in its implementation. Different topics were determined from this study and were selected to be analysed in the OBSERVAL-NET project. This project looked to sustain the development of comprehensive, coherent, and flexible models for VNFIL practices and recommendations as a result of a comparative study of samples of practice across European countries [12].

There are multiple experiences related to validation of learning in countries of the European environment, and these have been collected in various reports made by Cedefop under the title "European Inventory for Validation" [13–16]. A research project was completed in order to establish a common evaluation framework for Nordic countries (Iceland, Norway, Sweden, Finland, and Denmark). The project was structured in two phases, the first called "Quality in validation in the Nordic countries," where a context analysis was carried out, and the second part called "Quality Model for Validation in the Nordic countries. A development project 2012-13," in which a series of proposals were accomplished for the definition of a common model under the premise of quality [17]. The corresponding legislation in each country was studied and the references to the quality in these standards, the procedural development approaches from the operational point of view and how the quality could improve the procedure were also examined. The working group used the definition of quality given by the Canadian expert Van Kleef [18], which distinguished different aspects that could show a direct impact in the quality of the validation for these countries. These include:


In 2002, the EU began to tackle VET through a resolution approved by the Council called the "Copenhagen declaration" [19]. It requested the collaboration of the member states to improve VET systems. Two priorities could be highlighted: (a) to develop tools for mutual recognition and validation of competences and qualifications; and (b) to improve quality assurance in VET. In 2004, the contributions carried out two years earlier were confirmed and the established priorities were also developed (Maastricht Communiqué) [20]. Specific duties were entrusted to the member states. These include:


The first specific reference document on validation in Europe was raised in 2004, through a conclusions project called Common European Principles for the determination and validation of non-formal and informal education, in an effort to determine and validate differentiated learning [21]. This document arose in response to the European Council in Lisbon 2000 [22] to set out the challenge of making the European knowledge economy the most dynamic in the world, providing a special relevance to education and training.

The principles and validation elements of learning have been studied due to the accreditation procedure of competences and have had a heterogeneous development both in the European Union (EU) and Spain. This has not provided a response to the implantation of a validation system between the different European countries. The validation data collection from users of non-formal and informal learning is a major challenge and the demand for validation is increasing. Some exceptions have been noted in countries that have validation systems with a long tradition, like France and the Netherlands, where demand seems to have stabilized. This lack of data limits the opportunities for evaluation and control of validation activities.

It is important to understand the regulations that have been done at the European level in terms of validation as well as to the Spanish level in terms of accreditation. In Spain, there are several validation frames that cover various education levels: Qualifications and Vocational Training Act of 2002, Employment Act from 2003, Education Act from 2006, Higher Education Act from 2007, and Royal Decree 1224/2009 [23] on the recognition of professional competences acquired through work experience [14]. There was a joint call for validation in 2011 in order to implement the last decree. Most of the Autonomous Communities, with the exception of one, launched validation calls. The needs for accreditation have not been successfully dealt with, as well as the number of positions and the offer of existing qualifications in the National Catalogue of Professional Qualifications for the Spanish case. The principles and elements that determine the validation in European countries should be identified. The difference between the principles of validation should be presented according to the several international organizations such as UNESCO, European Council, and Cedefop, and the principles and elements that determine accreditation in Spain [24].

There is a shift between countries relative to the knowledge of VET, ranging from 46% to 91% of respondents. This displays a positive image in Europe, in relation to the capacity to make available job positions and prepare people for the workforce. Most respondents coincide that general education shows a more positive image than VET; 75% of Europeans agree with the statement that students with low grades are led to VET. However, the main country differences need further exploration to establish factors influencing the image of VET in different national contexts [25].

According to Cedefop [26], the implementation of validation arrangements is based on several interconnected steps. The main target of these guidelines is that validation is about: (a) how to attribute suitable value to results of non-formal and informal learning; and (b) how to show the results of non-formal and informal learning.

Quality assurance systems are capable of guaranteeing reliable, valid, and credible evaluations. Thus, they should be deepened in the research of this field. The connection between the national qualifications framework and the validation should be established deeper in order to write and to use learning outcomes for validation purposes [27]. As a consequence, the aims of this work are to review the validation framework of learning outcomes in Europe and Spain in order to characterize the validation of the acquired learning by professional experience in non-formal and informal learning, as well as to identify the development of the accreditation programme of competences in Spain.

The outline of the paper is as follows. Section two presents the theoretical framework; section three describes the methodology used; section four analyses the results; finally, section five summarizes the main conclusions.

#### **2. Theoretical Framework**

The adoption and development of qualifications framework based on learning outcomes improve the situation of non-formal and informal learning in the labour market as well as in society. Focusing on the development of validation in the EU, the first official document of the EU that mentions the need of this recognition was the white book of the European Commission related to teaching and learning [28]. The preamble establishes the need to strengthen professional training policies as an element to improve the employability and competitiveness of the companies. Since then, the EU has shown an increasing interest related to the importance of the validation of all types of learning for its economic and social development. In the year 2000, the Lisbon European Council recognized that VET should be a tool that allows social inclusion, cohesion, mobility, employability, and competitiveness [19].

The following premises were used to develop the Directive relative to the recognition of professional qualifications [29]:


These premises allowed for the development of the recognition Directive of professional development [29], which confers the warrant to access the same profession and exercise it in other Member States with the same rights as nationals. It was also established to enhance the procedure to perform the recognition. This process was carried out through the Helsinki Communiqué (European Commission, 2006), which delved into and developed the highlights of the Copenhagen process, especially in terms of transparency, allowing ratification of the Europass and establishing the basis for the European Qualifications Framework (EQF), European Credit System for Vocational Education and Training (ECVET), and European Quality Assurance in Vocational Education and Training (EQAVET). In 2008, the priorities and strategies of the Copenhagen process were revised anew and it was called the Bordeaux Communiqué [30]. This was completed while keeping in mind the new objectives after 2010. In this same year, Cedefop published the European Training Thesaurus [31], which collects all the definitions in the field of vocational training to create a common language in the EU as a tool for the transparency and homologation of the concepts that are being used in each country. In 2009, Council of Ministers of the EU defined a new Framework for education and training. It was called Strategy 2020 [32], which enhanced the Lisbon objectives in 2010 seeking to improve VET, in order to raise employability and cohesion.

Four strategies were defined for which a set of 16 indicators were set up to perform their monitoring and evaluation. These were:


The following landmark in the field of validation was the publication by Cedefop of the Guidelines for the validation of informal and non-formal learning [26,33]. The directives were established, from the point of view of the public policies, in two perspectives. The first perspective establishes the need to coordinate the European principles defined in 2004 to support the development of quality assurance mechanisms, along with the principles of quality assurance and EQAVET Framework, using the tools designed (Europass, ECVET, EQF) to promote validation, comparability, and transparency of VET systems. The second perspective raises the validation process as part of the VET systems, and that the evaluation of the competition presents formative and summative components as well as the need to determine the sustainability of the process through cost-benefit analysis. The document defined the guidelines for validation, taking into account the principles mentioned previously, according to the following points:


In 2010, Bruges Communiqué ratified previous strategic axes in long-term VET [34]. From these perspectives, the following challenges for VET systems were posed:


These challenges are part of the Europe 2020 planning, which sought to set up new competencies among citizens, in order to face new social and economic models. This new European planning can be defined under the following lines of work:


In 2012, the definitive encouragement to validation was produced from two different areas:


Council Recommendation defined the validation as "a process by which an authorized organization confirms that a person has acquired learning outcomes, measured at a relevant level" [36]. The phases of the procedure were also specified:


The first Biennial of Validation of Prior Learning was held at Rotterdam in 2014. International experts reflected on the need to establish alliances throughout this process. Bjørnavold [37] presented an analysis of scope of the validation procedure to European

level. It was explained that advances in policy, methodology, and practice and resources assignation were completed. However, the expectations were not reached according to the estimated initials. The principles that have to support the validation procedure according to Bjørnavold [37] are:


Validation must be a tool to enhance learning throughout the lifetime and the employability of people, especially for those with low qualifications and consequently with greater difficulties in the labour market, boosting the recognition of this tool between companies as a mechanism of professional career and continuous learning. Cedefop presented new guidelines, which were published throughout 2015 [26]. The aim was to specify the Council Recommendation [36] and to ease the implementation of the outcomes analysed by Cedefop in the different European Inventories on the Validation of learning. The new guidelines, based on the different sections of the Council Recommendation [13], defined validation as a procedure that allows for visibility and assesses the results obtained in different learning settings, without including formal education. Following the Recommendation, four phases can be established, such as determination, documentation, evaluation, and certification. However, it is constructed so that the citizen should not use all phases of the procedure. The first two phases can be considered as formative evaluation. Subsequently, it could be a formal evaluation and certification process, such as a summative evaluation. In this manner, the manager's goal is created, which is that the citizens can accredit their competences in a transparent and equitable manner. Unlike the Recommendation, it deals with and deepens the information and orientation phases, which indicates that it should be particularized, coordinated, and near to the citizen. This process is constituted as an essential tool for lifelong learning, which enables for the identification of professional competencies, interests, and personal skills in order to ease decision making and working [38]. Furthermore, it is established that the orientation will be part of the validation procedure, which should include tools such as continuous individual and collective services, specific websites, self-assessment systems, and the adaptation to the personal needs of each user. It is also established that the coordination between the different parties involved in all areas is a fundamental factor. For example, the strategy for the definition of public policies, the tactic for effective management of resources available, and the operation of the homogeneous management of the procedure. It affirms the need to define a legal framework that facilitates this coordination and regulates the procedure by attending to the different

phases independently, determining the management tools by facilitating decision-making based on the needs and expectations of citizens.

The National Qualifications Frameworks should include the mechanisms of nonformal, informal, and formal training, and equivalent in European Credit for Vocational Education and Training (ECVET) [39].

Regarding those responsible for carrying out the procedures, it is important to establish the need to identify each group of interest, and define their functions and participation. Among those responsible, the following groups of interest are identified: (1) Counsellors; (2) Advisors, (3) Evaluators; (4) Managers of the procedure; and (5) Stakeholders: Politicians and Social agents (business associations, trade unions, volunteers, etc.).

The context in which the validation of learning should be developed is also defined. For this purpose, updates to the previous guidelines [33] and Council Recommendation [36] were introduced, aligned with the integration of this procedure in VET systems, and with the use of open resources of education (OER). This process started with a skills audit, including self-assessment questionnaires, and also the development of tools to detect needs of training.

Table 1 shows a comparison of Cedefop principles of 2009 [33] and 2015 [26], with respect to those of UNESCO [35] and Recommendation 2012 [36]. This will allow to determine the principles and validation elements that will be used later to carry out the research.

**Table 1.** Analysis of the validation according to European Recommendations.



**Table 1.** *Cont.*

The major landmarks on the validation procedure are highlighted:


The result of the analysis of the documents described previously is displayed in Table 1. Included are the context in which the recommendation is dictated; the principles that it collects; recommendations that it provides; and the phases that determine the procedure.

VET is associated with education that prepares people for a specific occupation. The opening of the market for training and education, including VET, has raised the relevance of regulation and quality assurance mechanisms to ensure the integrity of qualifications. Misko [40] studied the approaches to the regulation and quality assurance of VET in a number of countries, including New Zealand, Canada (province of Ontario) and two accrediting agencies in USA, as well as selected European member states such as Finland, Sweden, and the United Kingdom. Therefore, the practices imposed overseas can be used to report the development of VET regulatory and quality assurance approaches in Australia.

The openness of the School-to-Work Opportunities Act involved large opportunities for USA, since it had different regional and local models of vocational training. Regarding the changing conditions of a globalised economy, the tech-prep is a suitable example for fitting the traditional school-based vocational education. By matching high school and community college courses, students can meet the growing demand for an extensive knowledge base and problem-solving qualifications [41].

The transfer of VET systems is the subject of international debate, however, there is no sufficient documentation related to the process or study of how such transfers are obtained in practical terms. Pilz [42] studied the potential for devolving Germany's dual vocational training system to German subsidiaries in foreign parts, specifically in Japan, USA, India, and China. The work arranged the range of training strategies established by German subsidiaries. Interviews in more than 40 German subsidiaries abroad showed that local factors in the host country play such an important role that it is not feasible to transfer the German VET system completely to another country. As a result, the findings showed that policy borrowing in the area of VET is presumed to be only partial and will be markedly influenced by the national features of the host country. Furthermore, the vocational and training performance of German corporations at their subsidiaries in China, India, and USA has been dealt with. These three countries present relevant markets for Germany and they are characterised by their varied cultures, employment systems, and VET. The transfer of the German VET system to other countries has also been a subject of study [43].

Switzerland is one of several European countries with a so-called "dual" VET system in which students blend learning in school with learning in workplace settings. In Denmark, Norway, Austria, Switzerland, and Germany, between 30 and 70 percent of students in upper secondary school are involved in such systems. VET is the mainstream upper secondary programme, serving 70 percent of Swiss young people. It teaches a broad set of students including high-tech, health, human service, and traditional trades as well as crafts. One measure of the power of that economy is that Switzerland presents practically full employment, with a youth unemployment ratio that is the lowest among developed countries [44].

VET systems change significantly from country to country due to their present different objectives and because they are embedded within the education and labour market systems of any country. There are typologies that characterise and compare VET systems. However, many of them show weaknesses related to the consistency of their descriptive criteria or, for example, the extent to which the typology can deal with more complex VET systems. Therefore, a new typology that builds on existing approaches from a range of disciplines should be developed. Pilz [45] has put forward a novel typology which allows VET systems in a range of countries to be organized across the different levels involved, in aspects as varied as curriculum design, teaching practices and government regulation. Six countries, Germany, France, India, Japan, China and the USA, are selected as case studies. These have demonstrated sizeable differences but also biased convergences. The typology proposes a framework for explanatory approaches in individual country contexts as well as an opportunity for international comparison of key aspects of VET systems, the possible transfer of VET models from one country to another, and the value enclosed to vocational qualifications.

#### **3. Methodology**

Research presents three clearly identified parts that condition the methodology to be followed in each case [46]. These phases are: Definition of the investigation, Research work or field work, and Analysis of the results.

The scheme used in the first phase is shown in Figure 1. It consists of the need to set up working groups, exchange research objectives, and the management of all documentation that will be used in the process [47].

**Figure 1.** Definition of the research [47]. Green framing is associated to coordinating team; blue (quality groups or improvement); red (key activity); orange (external staff); dark red (quality committee or improvement); and finally, black (all levels).

The entire procedure for the collection of this information must be processed [48] in order to ensure homogeneity in the request, search, and the management of the information (see Figure 2a–c).

For the methodological development of the second phase of the research project, two tools are considered: one of them qualitative and another quantitative [49]. Documentary analysis [50] is used in the qualitative part in order to extract the main aspects of the whole documentation which will be reviewed with the objective to synthesize subsequently the common elements in the scope of the study. The elaboration of a questionnaire is chosen in the quantitative part [51]. This consists of two parts, one closed for data contributions, and another opened to collect the qualification of the respondents on the different phases of accreditation procedure. Once the documentation and information required has been determined, data collection and analysis for both qualitative and quantitative parts proceeds. The range of the regulatory framework should be marked out in a European and Spanish environment. In this phase of research work, the selection and comparison of the different and extensive documentation will be completed [52]. The outcomes will be collected in tables that allow for comparison of the study items with the extracted data from each of them. It will also determine the sample for the execution of the quantitative part [53] based on the analysis of the calls carried out in Spain and taking into account the items extracted from RD 1224/2009 [23], which constitutes the frame of reference for this work. In the analysis of the results, the complexity consists of the integration of quantitative and qualitative results, which are derived from the surveys to the Autonomous Communities regarding the different phases of the procedure. The final aim should be to identify the results of the procedure operation, from the regulatory development in 2009 to the calls held in 2013.

**Figure 2.** (**a**) Information management procedure flowchart. Start. (**b**) Information management procedure flowchart. Medium. (**c**) Information management procedure flowchart. Final.

Finally, qualitative information treatment is carried out using SWOT (Strengths, Weaknesses, Opportunities, and Threats) analysis. This is to determine the relevant points and the improvement areas of the same, as a starting point for the realization of a proposal of a management system model [54] based on processes for the accreditation procedure of competencies in Spain. Some items should be considered to perform an analysis that provides information to assess the aims compliance of the National System of Qualifications and Vocational Training; in particular, the procedure of competencies accreditation. These include:


The goal is to obtain information to analyse the system using systematic procedures, based on quality criteria that allows to perform the final evaluation report and the improved proposal of the system. A specific evaluation of this procedure will be completed, directed to identify the degree of development, adaptation, and homogenization in the process of implementation of the different appropriate Administrations.

Figure 3 summarizes the proposal for a comprehensive management system based on integral, operational, and strategic control processes, for the procedure of recognition, evaluation, and registration of competences.

**Figure 3.** Comprehensive management model for the accreditation procedure.

#### **4. Results**

This section responds to the planning of the research study, which aims to determine the validation principles and elements of the learning in the different regulations. Thus, the validation procedure that has been developed in the EU and Spain should be studied in order to obtain their recommendations and conclusions. This is carried out using the theoretical framework reviewed previously, along with data collected in Table 2.


**Table 2.** Analysis of the validation according to European Recommendations.

From this, the aforementioned principles and elements were extracted, which have been reflected in the models of corresponding records for each geographical area, the European (Table 3) and the Spanish (Table 4).

**Table 3.** Principles and elements of validation according to European recommendations and guidelines.

Principles and elements of the validation of the learnings. Research technique: Comparative analysis. Study parameters: Principles and validation elements. Study units: Recommendations and Guidelines on Validations according to the EU.


**Table 4.** Principles and elements of validation according to Spanish regulations.

Principles and elements of the validation of the learnings. Research technique: Documental analysis. Study parameters: Principles and validation elements. Study units: Competency accreditation regulations in Spain.


#### *4.1. Analysis of the Validation Principles in European Recommendations*

A study has been carried out on the principles and elements included in the different recommendations and guidelines existing in Europe, see Table 2. This is a prior step to the preparation of the results present in Table 5.


**Table 5.** Comparative analysis of the validation principles and elements according to the European recommendations.

Considering the two entries in Table 5, an analysis is carried out. The percentage of each element is collected, see Figure 4. As shown, credibility, legitimacy, reliability, and confidence are presented in all references studied.

**Figure 4.** Percentage of the principles included in the different European recommendations.

Eighty percent of the rules display assurance of individual rights as well as obligations of those responsible. It should highlight that employability and process documentation are presented only in 20% of the references. The identification of the phases shows 40% and indicates the difference between the definition of strategic or political aspects and the operational. On the other hand, analysing from the perspective of the different recommendations, the upward trend can be clearly seen. In fact, the knowledge about the

procedure has evolved, mainly through the Cedefop studies. All the elements described in the normative progression have gradually been introduced. The new directives of 2015 collected 100% of the principles and elements under analysis, which was the highest point of this development, see Figure 5. Finally, Figure 6 shows detail of the elements included in each regulation.

**Figure 5.** Percentage of the principles included in the different European recommendations.

**Figure 6.** Principles and elements collected in each recommendation.

*4.2. Analysis of the Validation Principles in National Regulations*

This subsection is based on Table 4, which includes three aspects: the identification of the normative regulation (Table 6), the identification of reference frames that include the European and the national frames (Table 7), and finally, the principles and elements of the validation in Spanish regulations (Table 8).


**Table 6.** Principles and validation elements in Spanish regulations. Regulation.

**Table 7.** Principles and validation elements in Spanish regulations. Frameworks.



**Table 8.** Principles and validation elements in Spanish regulations. Principles.

Once the reference regulation has been typified, its origin and its traceability is identified. The used reference frames are analysed as well as the attributes that they include, Table 7.

The attributes existence of the European framework can be analysed (see Figure 7); the validation framework for learning is the most referenced (75%); subsequently, mobility, lifelong learning and employability (50%) follow. It should be noted that this last one was only included in 20% of the European recommendations, which provides a relevant indication of the utility that is conceded to the accreditation of competences in Spain.

**Figure 7.** Percentage European reference framework according to Spanish regulations.

The Spanish framework is also analysed. Participation is set up as an essential element (100%), then the requirement to conform the needs of the productive sectors and the centrality in the person (both 50%), and finally, the adaptation to European validation frameworks (25%) collected only in Law 5/2002 on qualifications. The key is that the publication of Royal Decree (RD) 1224/2009 [23] was prior to the guidelines of Cedefop [33]

and Council Recommendation on the validation of learning [36]. As a summary, Figure 8 shows the attributes of each recommendation.

**Figure 8.** Percentage Spanish reference framework according to Spanish regulations.

The analysis shows that two aspects determine the nature of accreditation in Spain: information and integration in national frameworks (100%). These two are the instruments of the national qualifications system and VET regulated by Law 5/2002 on qualifications [55], which sets out a system of information and guidance. On the other hand, the national catalogue of professional qualifications was established as an integrating element of the entire training proposal in VET, as well as for the accreditation of competences. The need for the recognition of competences and the training of human resources responsible for assessment and evaluation presented a significant importance (75%). Figure 9 displays the validation principles collected.

**Figure 9.** Percentage existence of validation principles in each Spanish regulation.

It should be highlighted that the person is not determined as the centre of the process in any regulation as well as the documentation of the processes—items that determine two of the strategic objectives that were established in the model. These include the detection of the citizen needs and the establishment of a documentary system based on processes. The methodology has provided the determination of the validation principles, through the tables and graphs. In this manner, the validation of non-formal and informal learnings acquired by professional experience in Europe as well as Spain has been characterized. As a result, the accreditation of competences procedure in Spain can be carried out through an implementation of a process-based management system. See Tables 9 and 10.


**Table 9.** Cause-effect relationships between objectives (a).

STRATEGY


**Table 10.** Cause-effect relationships between objectives (b).

RESULTS

#### **5. Discussion**

Few countries have established quality codes or guidelines on validation, and the question remains as to whether quality assurance procedures and systems are effectively capable of guaranteeing reliable, valid, and credible evaluations [16]. However, the acceptation and achievement of a qualifications approach established on learning results improves the situation of non-formal and informal learning in the labour market and society. Validation must be an instrument to improve employability and learning throughout life, particularly for people with low qualifications and, therefore, with greater difficulties in the labour market [56]. Thus, national qualifications frameworks are essential tools for the accessibility to the procedure, the transfer and accumulation of learning outcomes, and the application in VET systems [57].

The methodology consisted of three parts: definition of the investigation, research work or field work, and analysis of the results. This requires the development of working groups, exchange of research aims, and the management of all documentation that is used in the method. The entire procedure for the collection of this information has been processed to ensure homogeneity in the request, the management of the information, and research. For the methodological implementation of the second phase of the research, two tools were considered: qualitative and quantitative [58]. The documentary analysis has been used in the qualitative part to derive the main aspects of the whole documentation which will be reviewed with the goal to synthesize the common elements in the scope of the study.

In the study of the outcomes, the complexity consists of the integration of quantitative and qualitative results, which are determined from the surveys of the Autonomous Communities regarding the different phases of the procedure. The final aim was to identify the results of the procedure operation. Qualitative information processing was done using SWOT analysis. This was carried out to establish the applicable points and the improvement areas of the same as a starting point for the realization of a proposal of management system model based on processes for the accreditation procedure of competencies in Spain.

The analysis of the validation principles in European recommendations and national regulations has allowed characterizing the validation of the learning obtained by professional experience and in non-formal and informal settings, allowing to set up the development of the accreditation process of competences in Spain. As a result, these works open the door to the deepening of research in the quality management field of the validation procedure in EU and Spain [59,60].

**Author Contributions:** P.M., A.R. and L.C. conceived of the presented idea and designed the work, P.M. carried out data collection and literature review. A.R. and S.C. proposed the initial structure of the article and planned the methodology. S.C. carried out the presentation of the results after their interpretation, carried out in conjunction with P.M., A.R. and L.C. The final critical review was conducted by S.C., A.R. and L.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** https://incual.educacion.gob.es/documents/35348/80300/Datos\_ Proc\_Acre\_2013\_impre/ad122388-a2b9-4692-97ab-a14907613d2b (accessed on 28 June 2021).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **Active Learning as a Beyond-the-Classroom Strategy to Improve University Students' Career Adaptability**

**Tracy Hui 1, Sam S. S. Lau 2,3,4,\* and Mantak Yuen <sup>5</sup>**


**Abstract:** Geopolitical changes worldwide, together with rapid advances in technology, have created a situation where an individual's working life can present many new challenges. Helping students develop the attitudes and skills necessary to adapt to constant change along a career path has become a priority in education. Developing this career adaptability is becoming increasingly important to the sustainability of democracy, the economy, justice, human values, and equality. The authors of this paper argue that to improve the quality of our education system in universities, active learning should play a more important role to enrich the typical lecturing–learning processes. The aim of the study reported here was to examine the impact from implementing a 3-month active learning program that took university students beyond the classroom to increase their career adaptability and self-esteem and to strengthen their meaning in life. A mixed-method approach was adopted and conducted in two phases with 119 undergraduate students in human resources management at a university in Hong Kong. Findings suggested a significant improvement in career adaptability of the students after participating in the program when compared to a control group. However, data did not indicate any significant change in self-esteem and meaning in life in the training group. The findings support the valuable role of active learning as a strategy to enhance students' career adaptability in a changing but sustainable world of employment.

**Keywords:** active learning; career adaptability; Hong Kong; human resources management; university students; university teaching

#### **1. Introduction**

A review of the literature reveals that in this era of rapid technological and geopolitical change, career adaptability has become an increasingly important research topic in the field of career education [1]. It is also evident that, in the near future, our efforts to adjust to climate change will also affect employment opportunities by creating new jobs and making others redundant. If we are to sustain an acceptable quality of life, the ability of individuals to adapt to constant changes in employment has become a new essential 'life skill'. In addition to career adaptability, self-esteem and meaning in life are important to undergraduates' career and personal development. To our best knowledge, it remains unknown whether active learning contributes to these three outcomes.

#### *1.1. Career Adaptability*

Career Construction Theory defines career adaptability as the readiness to manage predictable and unpredictable career transitions, changes and challenges [2–4]. Career

**Citation:** Hui, T.; Lau, S.S.S.; Yuen, M. Active Learning as a Beyond-the-Classroom Strategy to Improve University Students' Career Adaptability. *Sustainability* **2021**, *13*, 6246. https://doi.org/10.3390/ su13116246

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Javier Puente and Leandro Almeida

Received: 30 April 2021 Accepted: 29 May 2021 Published: 1 June 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

adaptability has been delineated into four self-regulatory strengths: concern, control, curiosity and confidence. Career Concern is the inner urge to plan for a career carefully and establish a career vision. Career Control represents a sense of responsibility and ownership to construct one's career and make career decisions. Career Curiosity refers to the exploration process to synthesize work-related information relative to personal strengths and interests. Career Confidence describes self-efficacy for managing vocational tasks, decisions and transitions [2,5,6]. For undergraduates, these four psychosocial competencies determine how successfully they deal with the school-to-work transition as well as their working life [7].

Career adaptability is an important part of life design [8] and is becoming a popular topic in vocational research and as a subject within career education courses. Two longitudinal studies have found that career courses boost the level of career adaptability of undergraduates in Hong Kong [9,10], and cross-sectional studies suggest that attributes such as personality traits and self-esteem are antecedents of undergraduates' career adaptability [11–14].

#### *1.2. Self-Esteem*

If self-esteem is indeed an influence on older students' ability to acquire career adaptiveness, schools and universities could do more to build students' self-esteem. Self-esteem is reflected in personal beliefs about self-worthiness and value. A person possessing high self-esteem tends to feel accepted, respected and satisfied with who he or she is. That person recognizes, and is comfortable with, their personal strengths and weaknesses [15,16]

Kaveh, Hesampour, Ghahremani and Tabatabaee (2014) [17] found that one way to increase self-esteem of female secondary school students was to have the students engage in active peer-led education. They reported that students who participated in the activities had a significant increase in self-esteem. The research team explained that the increase in self-esteem appeared to stem from the collaboration and mutual support among peers. This type of collaborative learning process acts as a catalyst for behavioral change in terms of belongingness, attachment among peers, self-esteem, and purpose in life [18–20].

#### *1.3. Meaning and Purpose in Life*

Meaning in life is generally described as the feelings an individual has about the significance and purpose of their existence. People who believe their lives are high in meaning tend to develop valuable goals and feel fulfilled [21–23]. An instrument to assess an individual's meaning in life was developed by Steger and Frazier (2005) [23]. They conceived that 'meaning' may be found at two levels: the *presence* of meaning in life and *search* for meaning. The former connects to an already-acquired sense of how meaningful one's life is. The latter refers to the situation where one is still seeking meaning in one's life.

A study found that adolescents with purposeful work goals reported higher meaning in life and in schoolwork [24]. Hence, students who engaged in the program in this study to organize a teambuilding day for other HRM students might perceive their lives to be more purposeful.

#### *1.4. Active Learning*

In university classrooms, 'active learning' is being more widely adopted to enrich typical lecturing–learning processes and improve the quality of our education system. In contrast to the traditional lecturer-centered pedagogy commonly used in universities, active learning is a student-centered method in which students are engaged in the learning process and can benefit from the experiences [25]. Activity-based learning has long been the norm in early childhood education, and through the primary school years, but in universities it had been less common.

The belief that underpins student-centered learning is that it encourages students to become more autonomous by discovering and constructing their own knowledge and using their own initiative. The control is moved from teacher to the students to engage them in meaningful learning and to reflect upon their progress [25,26].

Mounting evidence suggests that active learning surpasses traditional teaching methods in establishing knowledge and skill development [25,27–29]. However, the majority of studies of active learning have only explored its positive impact on teaching and learning effectiveness, so there is a paucity of information on the relationship between active learning and students' personal outcomes (such as career path and career readiness). There is a need to investigate how active learning can be adopted more widely as a strategy to improve older students' career adaptability and personal development [30]. It is hypothesized that a well-designed extracurricular activity-based learning program, implemented beyond the classroom, can be a sustainable approach that adds value to the formal curriculum. It is also hypothesized that the program may have a positive effect on participants' self-esteem, and may strengthen their purpose and meaning in life. An effective learning program should be able to enrich students' professional knowledge, and also strengthen their career resources.

#### *1.5. The Program*

The present study attempted to investigate whether an extracurricular activity with hands-on experiences in taking responsibility as a trainer of others could strengthen students' career adaptability, self-esteem and meaning in life.

The program was built on active learning principles, namely that (i) students learn best by doing, (ii) that activity promotes higher-order thinking, and (iii) that participation offers students practice opportunities that can aid their development [29,31,32]. The university students who participate in the extracurricular program would obtain hands-on experiences in taking responsibility for designing and delivering a team-building program.

The program also drew upon aspects of Career Construction Theory [4,6] and connected three perspectives on vocational behavior—namely, developmental psychology, individual differences psychology, and narrative psychology [33]. Firstly, the developmental psychology perspective focuses on psychosocial adaptation and how individuals manage their own vocational development and difficulties. This 'career adaptability' attribute is the main variable considered in the present study. In particular, the intention was to compare the pre- and post-program changes that occurred for participants in the program (active learning group) and a control group of similar students who did not participate. It was anticipated that, by completion of the learning program, the students might have a clearer plan for pursuing their training and career development (career concern), a greater responsibility of constructing their future profession (career control), a higher curiosity to explore more information for their graduate job (career curiosity), and a higher efficacy to take up an entry-level training position (career confidence). Secondly, the individual difference perspective addresses the differences in vocational personality types and the different occupation interests that people may have [33]. With reference to this aspect, the study also helped to reveal whether the students' self-esteem would be strengthened after participating in the active learning program. Thirdly, the narrative psychology perspective examines the dynamic processes that individuals use to integrate their past life experiences, impose meaning on their present vocational behaviour, and gain inspiration for their future career. Such dynamics help explain why people fit work into their lives differently [33].

Three hypotheses were tested in the study:

**Hypothesis 1.** *Compared to a control group, students who participate in an extracurricular active learning program will show higher career adaptability*.

**Hypothesis 2.** *Compared to a control group, students who participate in an extracurricular active learning program will show higher self-esteem*.

**Hypothesis 3.** *Compared to a control group, students who participate in an extracurricular active learning program will report a higher meaning in life*.

#### **2. Method**

#### *2.1. Design and Delivery of the Program*

The program was offered in two consecutive years to two cohorts of Year 3 Human Resources Management (HRM) undergraduates from a university in Hong Kong. The program took the form of an extracurricular activity with 13 half-day sessions held on 7 Saturdays/Sundays in the summer break (July and August). An invitation was made to the students for them to gain hands-on experience in designing and delivering an experiential team-building program. A professional trainer experienced in adventure programs was appointed as the coach. Under the guidance of the coach, students learned the fundamental adventure-based training concepts such as the value of working as a team, solving problems in a collaborative manner, supporting one another, the role of a facilitator. Students also learned the skills needed to design and deliver in-door experiential games by observing demonstrations by the coach and hands-on participation in some games. Students then spent a few sessions to brainstorm, discuss and refine the training plan for a one-day team-building workshop for which they would be responsible. On the last two half-days, students prepared team-building materials for a trial run.

The teambuilding workshop was to be part of the HRM program orientation held on the Saturday before the new academic year. Each year, around 100 HRM students from different cohorts, plus 15 graduates, usually participate in a team-building workshop as part of orientation for new students. On that day, students and graduates get together and participate in a series of indoor experiential activities and group games to foster relationships, cultivate a sense of group cohesion, and strengthen their team skills such as communication and team problem solving.

After the teambuilding workshop, the professional coach arranged a debriefing session for the student trainers to reflect on and consolidate their learning. The whole program lasted for around 3 months, covering initial training, teambuilding, workshop design to debriefing.

#### *2.2. Research Design and Procedures*

The study used a two-phase longitudinal design to compare the pre- and post-program changes between participants and the control group of classmates who did not participate. All participants were invited to complete an online questionnaire by email and WhatsApp on two occasions. Time 1 (pretest) was conducted before the first session of the program. At Time 2, all participants completed the questionnaire again approximately 3 months later. In addition to the quantitative data from the questionnaire, students of the second-year cohort were invited for a focus group interview in mid-November to collect qualitative responses on the active learning experience and its impact.

#### *2.3. Participants*

The participants in this program were full-time, third-year undergraduates from a Human Resource Management major at a university in Hong Kong. All participants in both training and control groups had completed a credit-bearing course with a fundamental understanding of training and development.

Human Resources Management (HRM) is one of the occupations that increasingly requires a relevant degree qualification [34,35]. In the training of HRM students, academic content and hands-on experiences are equally important to help students appreciate the purposes behind HRM functions and understand best practices [34]. In this respect, traditional pedagogies like lectures and written assignments are often inadequate to develop undergraduates as true professionals.

Previous studies have shown that an active learning approach appears to improve student learning and academic outcomes. For example, MacVaugh and Norton (2012) [36] identified the practical value of active learning in business education, and suggested that future research should investigate the connection between students' active learning experience and their later success on their career path. Similarly, North-Samardzic and De Witt (2019) [37] designed an online simulation program for HRM students to learn and apply theories and acquire conceptual knowledge. Their findings provide meaningful insights into how an online simulation can be an active learning tool that engages students in their learning.

In the first year of the study, 77 students completed the pre-training questionnaire and 57 students participated in the post-training questionnaire. In the second year, 75 students responded to the pre-training survey and 74 students participated in the post-training survey. All participants were invited to provide the last 4 digits of mobile phone number as an identifier. However, as shown in Table 1, only 51 pairs of pre- and post-survey responses were successfully identified by the 4-digit code in the first year in total and 68 pairs in the second year.


**Table 1.** Study participants in the training and control groups.

Independent *t*-test was applied to validate the combination of the two samples from different years, and scores of Year 2017 samples (N = 51) were compared to those of 2018 samples (N = 68) at each time point. There were no significant differences found between two samples for all measures at both pre- and post-training, so the two samples were combined for further analysis. The total sample included 119 students (59.7% female), which consisted of 32 students in the training group and 87 classmates in the control group. The mean age was 21.28 years (range = 20–24; SD = 1.06). Two sessions of follow-up focus group interviews were arranged. Ten students participated, 6 attended the first focus group session and 4 attended the second session.

#### *2.4. Measures*

As all participants were Chinese, the questionnaires were presented in the Chinese language, and the focus group discussions were conducted in Cantonese.

#### 2.4.1. Career Adaptability

The Career Adapt-Abilities Scale—China Form [38] (CAAS-China) was used to measure career adaptability. The CAAS-China is a translation of the English form of the CAAS—International 2.0 [5]. The scale comprises 4 subscales: career concern, control, curiosity, and confidence. Each subscale contains 6 items rated on a 5-point Likert-type scale (5 = Strongest; 4 = Very strong; 3 = Strong; 2 = Somewhat strong; and 1 = Not strong). In the present study, Cronbach alphas were 0.94 (overall), 0.90 (Concern), 0.84 (Control), 0.82 (Curiosity) and 0.88 (Confidence) at Time 1.

#### 2.4.2. Self-Esteem

The *Chinese Rosenberg Self-Esteem Scale* (CRSES) [39] was used to measure the global self-esteem of the participants. The CRSES is a translation of the English form of the *Rosenberg Self-Esteem Scale (RSES)* [16]. The scale consists of 5 positively worded items and 5 negatively worded items on a 4-point Likert-type scale (4 = Strongly agree; 3 = Agree; 2 = Disagree; and 1 = Strongly disagree). Cronbach alpha coefficient in this study was 0.86 before the training program.

#### 2.4.3. Meaning in Life

The *Meaning in Life Questionnaire* [40] was used to assess the meaning in life of participants. The scale is divided into two subscales, with 5 items measuring the presence of meaning in life and 5 items measuring the search for meaning in life. All items are rated on a 7-point scale from 1 (absolutely untrue) to 7 (absolutely true). The reliability of the MLQ

reported in this study was 0.77 at Time 1 and was 0.78 for the Presence subscale and 0.82 for the Search subscale.

For the focus group interview, an established in-depth interview approach with semistructured questions was adopted. The interview protocol consisted of three sections. Section One was a warm-up to invite interviewees to describe their overall experience in the program. In Section Two, students were invited to reflect on how their career adaptability, self-esteem and perceived meaning in life had changed as a result of the training experiences. Some prompts and probes were used during discussions to facilitate students to express their views in more detail. Section Three focused on identifying the most valuable gains, growth and challenges associated with this active learning experience.

#### **3. Results**

This study employed a mixed-methods approach, and the triangulation of data helped achieve more valid research findings. The results reported below indicate that the data collected through quantitative and qualitative methods led to similar findings.

#### *3.1. Pre- and Post-Program Surveys*

Table 2 presents means, standard deviations and internal consistency estimates for all variables at Time 1 and Time 2. The correlation figures at pre- and post-training stages are also shown in Table 2. All variables at Time 1 were found positively and significantly correlated to the score at Time 2.

At Time 1, independent *t*-test found no significant differences between the training and control groups on all measures, indicating that the two samples could be regarded as reasonably matched before the program began.

To compare the scores on career adaptability, self-esteem and meaning in life for each group between Time 1 and Time 2, paired *t*-tests were conducted. For the training group, there was a significant change in career adaptability (*t*(117) = 2.449, *p* = 0.016). In contrast, no significant change was found in self-esteem and meaning in life. For the control group, the paired *t*-test results showed no significant change for all scores from Time 1 to Time 2. This result provided some indication that the changes in HRM students' scores could be due to the effects of the active learning program.

To test the hypotheses, a mixed design of ANOVA tests was conducted. Training participation (training and control) was input as a between-group factor, and time (pre and post-training) as a within-group factor. As illustrated in Table 3, the analysis revealed that there were significant differences at Time 1 and Time 2 between the training in overall career adaptability (*F* (1, 117) = 4.03, *p* = 0.047), career concern (*F* (1, 117) = 6.46, *p* = 0.012), and career confidence (*F* (1, 117) = 4.96, *p* = 0.028). Students who participated in the program showed higher career concern and career confidence, as compared to the control group. These results tend to support Hypothesis 1. However, there was no significant change in the other career adaptability sub-scales (control, curiosity), nor was any change evident in self-esteem, or meaning in life and its two subscales, causing rejection of Hypotheses 2 and 3.



Notes. *N* = 119. Time 1 correlations are shown below the diagonal; Time 2 is shown above in the shaded area. Correlations over time are on the diagonal. \*\*\* *p* < 0.001, two-tailed. \*\* *p* 0.01, two-tailed. \* *p* < 0.05, two-tailed.

<


**Table 3.** Means and Standard Deviations of Training and Control Groups and Results of Mixed Design of ANOVAs.

Note. *N* = 32 for training group; *N* = 87 for control group. Only cases that provide both pretest and posttest responses are included here. Mixed ANOVA = mixed analysis of variance. \*\*\* *p* < 0.001, \*\* *p* < 0.01. \* *p* < 0.05.

#### *3.2. Focus Groups*

The qualitative responses recorded through the interviews were studied by applying content analysis [41]. Main themes and categories emerging from the responses were referred to the respective categories in the survey instrument. For example, in the *Career Adapt-Abilities Scale* (CAAS) [5], a person who exhibits greater 'career concern' is "thinking about what his/her future will be like", and "planning how to achieve future goals". Important statements from the students were then matched to attributes in CAAS. The same procedure was used to evaluate comments linked to career control, curiosity, and confidence. The responses obtained from the focus group interviews are reported below under: (1) overall learning experience; (2) career adaptability; (3) self-esteem; and (4) meaning in life.

*Overall learning experience.* The interviewees consistently agreed that the extracurricular active learning program was a valuable experience. Students described it as a "successful" and "valuable" (even "indescribable") experience that "satisfied" their needs. A student commented that,

"Our classmates who haven't joined [the program] missed a precious learning opportunity that they couldn't obtain again."

*Career adaptability.* The collaboration and hands-on involvement in the planning and executing of a team-building workshop had strengthened students' adaptability competencies. Some of these competencies equate with specific items in the *Career Adapt-Abilities Scale* (CAAS) [5]. These specific items appear in italics in the material below.

For example, two sub-themes related to career concern are identified. First, some students *"become aware of the career choices"*. One student said,

"Among various HRM functions, I like training and development more, but I was not sure whether it would be my career direction. This program makes my career interest certain, and I find I am capable of handling the duties of a trainer."

Another student touched on their *"plan to achieve career goals"*, saying that: "I will take a training and development related internship before graduation in order to prepare for my future career."

*Career control*. A few responses collected in the interviews subtly displayed career control characteristics. Two students shared that they learned to become more proactive and happier, which was similar to *"keeping upbeat"*. Two students also showed the characteristic of *"sticking up for my beliefs"*. One said: "No matter the future job is, and whether I like or dislike it, this [trainer] experience made me realize that a positive attitude is the most important. Every attempt can be beneficial to my future."

*Career curiosity*. All students expressed a view that the active learning program provided *"opportunities to grow as a person"*. They not only learned some training skills and knowledge, they also found teamwork, collaboration and leadership capabilities were enhanced. Two students shared that their reason for enrolling in this program was to try something new, this equates with *"curious about new opportunities"* in the *Career Adapt-abilities Scale*.

*Career confidence*. All six characteristics of career confidence were evident in the focus group conversations. The fact that they achieved the teambuilding day built their confidence by proving that they were able to *"perform tasks efficiently"*. The students were highly responsible and showed that they can *"take care to do things well"*. For example, a student shared: "It is essential to get well prepared and think about how to explain the game rules clearly so that participants find the game easy to understand and attractive." Another opined: "As I have committed to this program, I know I have to be responsible and complete all of the tasks properly."

Moreover, all of the students reported that "learning new skills" was the most valuable takeaway. For instance, a student shared: "I have learned many new skills such as how to lead a team, design team-building activities, debriefing skills and so on."

Indeed, being in charge of team-building day for a hundred participants was a great mission for the students. Throughout the process, they stretched themselves to achieve this mission. A student provided feedback which showed the characteristic of *"working up to my ability"*. He said: "I discovered that training and development is not easy. For instance, trainers are responsible for uncovering people's talents. On the team building day, I tried to facilitate the participants to use their unique abilities in different games bit by bit. But I know I need to sharpen my [facilitation] skills further."

Last but not least, students had demonstrated the confidence and strength to *"overcoming obstacles"* and *"solve problems"*. Some unpredictable happenings occurred before or on the day of team building, but with good teamwork they coped with these. Several students also shared that being a trainer was the very first huge challenge in life. Positive self-talk helped them come out of their comfort zone and enabled them to speak in front of the people and take up the facilitator role. A few students also reminded themselves, "I'm capable!", "Be brave!" and "Just do it!".

*Self-esteem.* Although statistical testing did not reveal significant increases in selfesteem, most of the students reported that they became more confident and optimistic. An interviewee explained: "At that moment on team building day, my self-esteem was high; but after a while, when faced with challenges, I fell back into a person with low self-esteem. I think personal growth takes time. A single exposure can't really build self-esteem."

*Meaning in life.* Again, statistical testing did not provide evidence of increased meaning in life. However, a few students did find that this 3-month program was a meaningful experience that gave them purpose. A student shared: "Before that time, study was the only thing in my life; but this learning experience has widened my horizon and enriched my life. I have learned a lot of new skills which are practical and beyond textbooks. I can use these skills in my future career."

Similarly, another student added: "The most meaningful thing is that I can contribute to the HRM degree program. The team building day fostered the bonding among HRM students of different cohorts."

These two responses somewhat reflect *"presence of meaning"*—but the sense of purpose and fulfilment was confined to the team building event and did not affect students' life as a whole.

One student briefly touched upon *"search for meaning"*, and said: "Everyone pursues meaning in life. I didn't want to be a leader before. In this program I have come out of my comfort zone to be a leader this time. When challenges come in future, this learning experience would remind me how meaningful it is to try, regardless of success or failure."

#### **4. Discussion**

The extracurricular active learning program in this study was designed as part of an initiative for university students that takes learning beyond the classroom. The implications of the study are organized according to four key findings.

First, the HRM students who participated in the study reported a strengthening of three aspects of career adaptability (overall adaptability, career concern and career confidence), whereas no significant change was found among their classmates in the control group. This finding partially supports Hypothesis 1 and is in line with active learning and career construction principles. From the active learning perspective, the program provided a safe collaborative setting for students to plan and implement the team-building day. This undertaking was a work-like mission with a real purpose. The firsthand experience obtained in the program is in many ways a realistic job [42] where students had a taste of the training profession. The students built their work culture and were involved in decision making. As the process evolved, this peer learning community illustrated how active learning effectively helps higher education students' profession development [29].

Second, from the career construction perspective, the student group provided a showcase of how they had to transform themselves to become trainers to lead a one-day teambuilding workshop. To complete this mission, they had to handle a series of tasks and to face many known and unknown challenges. Such transition demanded a high level of adaptability that required students to push beyond their current level to reach a higher capability. Savickas (2005) [33] opined that adaptation is fostered through five stages: orientation, exploration, establishment, management, and disengagement. The students in this study progressed through these five stages. They began the trainer role as an orientation period in which they become aware of the need for acquiring new skills. To adapt to the new role, students explored the domain by seeking information about the functions of a trainer. Then, students gradually established themselves in the new trainer role and learned to manage all new challenges one by one. Finally, they disengaged from the trainer role upon completion of the 3-month program. From the students' responses in the interviews, the experience of being a training facilitator to organize and lead a team on the team-building day greatly enhanced their career concern and confidence attributes. They became more aware of whether training and development is a profession for them to pursue upon graduation.

Third, all participating students found the learning experience valuable, and the changes in their adaptability supports a view that occupation-oriented active learning programs of this type could supplement other university courses to enrich undergraduates' capabilities. The evidence that active learning boosted students' overall career adaptability appears to be a way to prepare graduating students to enter and sustain today's employment market. From West to East, many employers comment that the new generation of graduates are not ready for real-life work situations. For example, Forbes reported that only 13% of adults strongly agreed that college graduates in the US are well-prepared to work [43]. An experiential learning experience such as the program here may serve to narrow the gap between learning at the university and starting a career. In particular, programs such as this could enhance undergraduates' career adaptability required in today's constantly changing world of work [3,44].

Fourth, self-esteem improvement and change in the meaning of life were not produced by this program, possibly because the program was of relatively short duration. Although the students revealed in discussions that they appreciated the program and found themselves with higher self-confidence, the experience did not produce long-lasting change in their feeling of self-worth and meaning in life. The nature of the tasks within the program did not really provide experiences that could have a deep effect on the students' perceived meaning in life. Future versions of the program could endeavor to provide experiences that might address these attributes more directly [45].

#### **5. Limitations**

The interpretive scope of the results is limited by the small sample size, and the 3-month longitudinal nature of the data from a sample of students in one program of a university. Future research should involve a larger number of matched pre- and postsurvey participants in an active training program in other settings and of longer duration. In addition, the present study found that the program did not significantly change students' self-esteem and meaning in life. Yet, in the interview the students did report positive experiences. Future research could confine the scope to evaluate the impact on specific outcomes such as career self-efficacy or meaningfulness of learning. Furthermore, some students said that their positive self-talk helped them overcome challenges encountered in the active learning project. Future research could extend the study and test whether positive self-talk is also relevant to the enhancement of career adaptability.

#### **6. Conclusions**

The study added to the active learning and career adaptability literature in higher education by collecting data using a mixed (quantitative and qualitative) and experimental approach. The findings provide some initial evidence of the potential value of a university active learning program that places students in leadership roles beyond the classroom. It appears that such a program is an example of education which contributes to students' sustainable career planning confidence and adaptability. The results suggest that situations that simulate real-life working conditions and present real-life challenges could add value to undergraduates' career preparation.

**Author Contributions:** Conceptualization, T.H.; methodology, T.H. and S.S.S.L.; data collection and curation, T.H. and S.S.S.L.; writing—original draft preparation, T.H.; writing—review and editing, S.S.S.L., T.H. and M.Y.; All authors have read and agreed to the published version of the manuscript.

**Funding:** The APC was funded by Department of Management, Hong Kong Baptist University.

**Institutional Review Board Statement:** The study was approved by the Research Ethics Committee of Hong Kong Baptist University.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Data are available from the corresponding author upon reasonable request.

**Acknowledgments:** The authors would like to express their appreciation to Mandy Mao and Albert Ng for their technical assistance. Appreciation also goes to the student participants. Valuable comments from the anonymous reviewers were appreciated.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **Unveiling the International Students' Perspective of Service Quality in Chinese Higher Education Institutions**

**Fakhra Yasmin 1, Shengbing Li 1,2,\*, Yan Zhang 3,\*, Petra Poulova <sup>4</sup> and Ahsan Akbar <sup>4</sup>**


**Abstract:** Foreign students' satisfaction with the service quality of Chinese universities is essential for the sustainable internationalization of China's higher education system. The present study employs a survey research method to bring in the foreign students' perspective of the various aspects of service quality in seven key Chinese universities. Accordingly, 618 valid questionnaires were analyzed using descriptive statistics, principal component analysis (PCA), and analysis of variance (ANOVA). The study findings posit that, although foreign students affirm that teachers are supportive and well qualified, they have concerns about the English proficiency of instructors. Likewise, foreign students were not satisfied with the frequency of formal research meetings with their advisers and the assistance with research techniques and relevant literature sources. Overall, female foreign students were less satisfied than their male counterparts. Moreover, foreign students reported higher satisfaction from teaching services and learning resources, moderate satisfaction from advisory services, and meager satisfaction from the administrative and support services of their respective Chinese institutions. Besides, we found significant differences between sample Chinese universities on various constructs of service quality. Likewise, arts and social sciences students were less satisfied with the service quality of the institution as compared to their natural sciences and engineering counterparts. The policy implications of this research for various stakeholders are discussed.

**Keywords:** international students; service quality evaluation; perceived satisfaction; sustainable higher education system; China

#### **1. Introduction**

The United Nations Agenda 2030 for sustainable development goals (SGDs) outlines a vision for a better world that relies on cooperation and interdependence. At the heart of these SDGs is the goal for advancing education. This goal primarily emphasizes universal education and suggests that all countries should ensure the provision of inclusive and equitable education and learning opportunities for all. This goal represents the vision and aspirations of the international educational community for the year 2030 [1].

Higher education institutions play an imperative role in the advancement of sustainability and all the important stakeholders expect them to be sustainable organizations [2]. However, in the context of the internationalization of higher education, sustainability can only be achieved if the students' evaluation is favorable regarding the service quality of their host institutions. China has become a rapidly emerging destination for international students. However, Chinese HEIs are considered to be at the early stage of their sustainable development [3]. Since the reform and opening-up policy was adopted in 1980, the remarkable growth of the Chinese economy has attracted the attention of many

**Citation:** Yasmin, F.; Li, S.; Zhang, Y.; Poulova, P.; Akbar, A. Unveiling the International Students' Perspective of Service Quality in Chinese Higher Education Institutions. *Sustainability* **2021**, *13*, 6008. https://doi.org/ 10.3390/su13116008

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Javier Puente and Leandro Almeida

Received: 5 May 2021 Accepted: 21 May 2021 Published: 26 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

countries in terms of economic and trade as well as educational cooperation with China. The internationalization of education in China evolved through different phases in response to trends in globalization to foster cultural, political, and economic ties between China and the world. As per the recent scenario, the international education plan of the Chinese government for the decade 2010–2020 intends to gradually increase the number of international students, and in 2020, the government intends to accept more than 500,000 foreign students, and among those 150,000 are expected to be research students enrolled in graduate and postgraduate programs. To realize these targets, Chinese universities have extensively expedited the enrollment of scholarship and self-financed students, and in 2018, China accepted 492,185 international students, thus becoming the largest host country in Asia [4]. The country of origin-based statistics of inbound students in China is presented in the table below. Table 1 indicates that the majority of international students in China came from Asian countries.


**Table 1.** Nationwide statistics of international students in China.

Source: Institute of International Education (2019).

After becoming part of the WTO, Chinese HEIs were granted some autonomy to manage their educational affairs to make them competitive with the regional and international seminaries. So, the idea was to establish cooperation with renowned foreign universities to initiate the building of a world-class university with embedded Chinese characteristics. Therefore, Chinese universities adopt the 'going in' and 'going out' approach to facilitate academic mobility, acquire international resources, and promote and facilitate the sustainable development of science and technology in China. However, Chinese institutions are still new in the international higher education environment and somehow lack global visibility and influence, which indicates that the desired outcome of the internationalization process is not yet fully accomplished [5].

Higher education is progressively emerging as a service provider industry that is exerting a higher pressure on management to meet the needs and expectations of their students by providing efficient academic and support services [6]. The concept of service quality was first proposed by Deming and Edwards [7]. Afterward, by developing a service quality model, Parasuraman, Zeithaml, and Berry [8] provided a relatively new and more pronounced view of service quality. This can be elaborated as the gap between anticipation and the actual experience obtained from the consumption of a particular product. Initially, this concept was used in marketing studies. However, keeping in view its importance in all the service-providing organizations, academics introduced this concept in the area of higher education institutions. Hence, the students are viewed as the primary customers of HEIs and as a means to attract and retain these students; universities shall identify and fulfill their needs and expectations.

In the modern world, tremendous challenges are emerging for higher education providers as competition and public demand for higher education have increased exponentially. In this situation, the provision of quality services can serve as a sustainable

competitive advantage for any institution. Despite the importance of this concept, higher educational institutions lack a single standardized definition of overall academic quality [9]. Hence there is no consensus among researchers on a single best way of defining service quality [10]. Students are the recipient of training imparted by the universities, which makes them the prime stakeholders of an institution [11]. Thus the overall educational quality of an HEI depends upon the prevailing management system at a specific HEI [12]. According to Jancey and Burns [13], experiences of students in engaging with various services delivered by the institution during the years of their studentship demonstrate service quality.

In the context of higher educational institutions, the satisfaction of the students is widely measured through service quality offered by the host institution. There are competing arguments about the relationship between service quality and satisfaction. One group of academics contends that service quality is dependent on the level of customer satisfaction, while others postulate that service quality is an antecedent to satisfaction. However, recent literature on this issue is dominated by the researchers who consider that satisfaction is dependent on the level of service quality, see, e.g., [14–16].

Perceived service quality builds a promising image of the institution in the minds of students that afterward leads them to higher satisfaction. International students that are satisfied with their academic and living arrangements in the host country were more likely to share their foreign university with their colleagues and friends in the home country [17]. Hence, an imperative factor for China to become an attractive higher education destination for international students is to ensure that the existing students are reasonably satisfied with various aspects of academic services of their host Chinese university and their overall living experience in China.

This study aspires to answer the following research questions. How do the current foreign students perceive their study and living experience in Chinese universities? Are they satisfied with the various aspects of teaching, learning, support, and living services at their host institution? Will they recommend China as a higher education destination to their colleagues and friends?

The present study makes use of the survey research method to examine the perception of Masters and PhD degree foreign students regarding the service quality of the host Chinese universities. Thus, 618 usable questionnaires were analyzed from the foreign students of seven key state universities of Wuhan, China. To have a reasonable representation from each category, only foreign students enrolled in natural sciences and engineering and arts and social sciences were included of this survey. International students recorded their response on a five-point Likert scale on the dimensions of teaching services, advisory services, administrative support, learning infrastructure, support infrastructure, and their overall evaluation of the satisfaction from pursuing higher education in China. Excel and SPSS software were mainly used to perform item-based analysis, dimension-based analysis, PCA, and ANOVA to comprehensively examine this research topic.

The contribution of this research is two-fold. First, the extant literature in the context of China employs descriptive analysis to evaluate the overall satisfaction of foreign students from their study experience in China and has largely ignored evaluating student's satisfaction from each aspect of service quality. However, in the backdrop of China's rise as a major player in the global higher education market, it is imperative to explore all the important features of service quality to examine the sustainability of China's international higher education. Therefore, the present study fills this void by devising a comprehensive framework to investigate the quality of teaching services, advisory services, administrative support, learning infrastructure, support infrastructure, and overall satisfaction of foreign students. Second, to ensure the empirical rigor of this study we make use of a bottom-up approach to conduct the item-wise, dimension-wise, and variable-level analysis. In so doing, descriptive analysis, reliability analysis, PCA, and ANOVA techniques were employed to uncover each facet of the international student's satisfaction with the service quality delivery of Chinese universities.

#### **2. Theoretical Underpinnings and Literature Review**

#### *2.1. Theoretical Framework*

The satisfaction theory has its roots in the discrepancy theory [18] while over the past few years; scholars have used comparative techniques to model the level of satisfaction or dissatisfaction [19]. A variety of theoretical approaches has been adopted to portray the association between positive disconfirmation or satisfaction and negative disconfirmation or dissatisfaction. Oliver [20] asserted that these approaches could be viewed as a modified version of the consistency theories and primarily focuses on the post-usage evaluation of the customers. Consistency theory hypothesizes that, when the actual performance of a product or service does not meet the expectations of the customers, they will experience a certain degree of dissatisfaction [21]. Thus, to communicate their dissatisfaction, customers will either adjust their expectations about the product or service or the perception about the actual user experience. This theoretical paradigm is in line with the mobility theory of satisfaction proposed by Morris and Winter [22].

Several authors have used comparative techniques to elaborate satisfaction over time. The most significant theoretical perspectives to emerge in contemporary studies include assimilation theory, contrast theory, assimilation-contrast theory, expectancy disconfirmation theory, and negativity theory. Assimilation theory mainly evolved from the theory of cognitive dissonance propagated by Festinger [23]. The dissonance theory claimed that customers draw some sort of cognitive comparison between the expected and the realized performance of a product. Thus dissonance or negative disconfirmation will arise if there is a discrepancy between the expected and the actual realized performance. Such post-usage assessment by the customer was presented as the assimilation theory of satisfaction by Anderson and Fornell [24]. They argued that customers tend to avoid dissatisfaction by making adjustments in their perception of a particular product to bring it closer to their expectations. However, assimilation theory has some inherent weaknesses. Firstly, the theory proposes a relationship between expectation and satisfaction but does not elaborate the mechanism through which disconfirmation of an expectation leads to perceived satisfaction or dissatisfaction. Moreover, the theory claimed that customers adjust their expectations about the performance of a product to avoid dissatisfaction. However, if this phenomenon accurately exists, then dissatisfaction would never be an outcome of a post-usage evaluation.

Expectation disconfirmation theory is the most widely recognized form of the discrepancy theory. This theory posits that customers will feel positively disconfirmed or satisfied if the actual performance experience exceeds the expectations. On the contrary, customers will feel negatively disconfirmed or dissatisfied in case the performance outcome fails to meet their priori expectations. Thus, positive disconfirmation results in increased satisfaction, whereas negative disconfirmation has the exact opposite effect. Moreover, if the actual performance matches with the prior expectations this situation will cause zero disconfirmation with no effect on their satisfaction. Kotler et al. [25] argued that the reason for such phenomenon could be attributed to the fact that customers develop their expectations based on the previous experiences, and the comments made by their colleagues and friends. Oliver [20] stated that the actual outcome can surpass expectations in two ways. The level of actual positive performance is within the normal range, indicating that the product or service was a little better than expected, or the actual performance experience is remarkably good, which means that the customer did not expect that their experience would be so delightful.

The expectancy disconfirmation theory has a perceived advantage as it not only explains satisfaction related to the performance of products but also explains satisfaction from service quality. Parker and Matthews [19] further extended this framework and postulated satisfactions as a discrepancy between the actual and the desired outcomes. This, in some way, relates to the value-percept theory which originated because in some cases customers could be satisfied from service for which expectation never exists. Hence, value-percept theory takes satisfaction as an emotional response that is caused by the

cognitive evaluation process, which is comparing the object or experience with one's values rather than expectations [19]. Therefore, the customer requires that there shall be no disparity between their values, needs, and desires and the actual outcome from experiencing a particular service. Figure 1 presents the pictorial representation of the theoretical framework of this study.

**Figure 1.** Pictorial representation of the theoretical framework based on Expectation Disconfirmation Theory. Source: Parker and Matthews (2001).

#### *2.2. Literature Review and Hypotheses Development*

The students' evaluation of service quality is most frequently investigated in the United Kingdom (UK). The underlying reason may be that the universities in the UK are striving hard to ensure the provision of quality education, support services, and competent lecturers to their students [26–28]. Urban et al. [29] opined that students' views about their experience in higher education are considered very important to examine the quality of education provided by the universities, since students are the main stakeholders and play a vital role in the profitable functioning of higher education institutes.

Williams and Cappuccini-Ansfield [30] argued that students are like customers because they pay tuition fees and universities are the service providers. Thus, students have many expectations from the institution mainly because they pay fees and expect value for their money. Because students are the buyers of the higher education services, their perceived satisfaction is important for the institutions to retain the existing students and to attract new students. So, together with the effective learning processes, increasing students' satisfaction with the institution should be the most preferred goal for every higher educational institution.

The provision of quality services is one of the very factors that influence the satisfaction level of students in a particular higher education institution. It is also considered an imperative dimension to gain a sustainable competitive advantage [31]. Quality aspects and continuous improvement are now among the everyday tasks of higher education management [32]. Because of increasing competition among global educational institutions, they are employing numerous managerial techniques to improve the quality of their services and study programs [33]. Management of academic institutions is of the view that preserving a quality culture and making proactive decisions positively contributes to success [34]. This can be done by introducing a non-formality approach, reducing bureaucracy, eliminating barriers among departments, encouraging flexibility, and appreciating initiative. To preserve a responsive environment and customer-oriented institution is an imperative quality dimension in service providers [35]. Quintal et al. [36] found that, in a sample of Australian universities, a one unit increase in the quality of services resulted on average in a nearly one unit increase in the level of students' satisfaction and trust. The provision of high-quality services in higher educational institutions is essential and imperative as satisfaction is an outcome of quality services provided by an organization [37].

Overall, research on satisfaction suggests that perceived quality is a critical determinant of perceived satisfaction [38,39]. In addition to the provision of quality services, there are few other dimensions to explain the facets of student satisfaction. For instance, students' satisfaction level is also affected by the attitude of admission staff, the admission process, and the information related to admissions that institutions provide to the potential students through telephone, brochures, and websites [40,41]. Elliott and Shin [42] are of the view that focusing on student's satisfaction from service quality allows the management to re-engineer their institution to adjust to the needs and expectations of the students as well as to introduce a mechanism that provides continuous testing of the effectiveness of fulfillment of their needs. They further posit that student satisfaction is an approach to achieve a sustainable competitive advantage in the higher education industry. In addition to this, Khosravi et al. [43] emphasize that meeting the needs and expectations of the students is an imperative task for higher education providers to gain a sustainable competitive edge over their rivals.

Student satisfaction is a complicated idea that consists of several dimensions. Appleton-Knapp and Krentler [44] pointed out two types of factors that influence student satisfaction level: institutional factors; and personal factors. Institutional factors include quality and swiftness of the teacher's feedback, the teaching style of the academicians, clarity of instructor's expectations as well as quality of his instructions, emphasis on research in an institution, and size of its classrooms. Personal factors that are found to influence students' satisfaction levels are age, gender, temperament, employment, students' average grade point, and preferred learning style [45]. Thus, to make sure that students are satisfied, higher educational institutions shall contemplate both personal and institutional factors [46].

Researches suggest a difference in the academic engagement of native and overseas students mainly because of the factors such as unacquainted pedagogic and research practices, cultural and linguistic adjustment problems [47]. Moreover, Sakurai et al. [48] investigated the factors that promote or hinder the academic engagement of international doctoral students and found that supervisory practices were the most prominent factor that shaped students' level of satisfaction. Moreover, the program in which a student is registered also serves as a factor of their satisfaction, as students of the faculty of arts were less satisfied as compared to the students of other hard disciplines. This evidence asserts that assessing the factors that influence the satisfaction levels of international students is not an easy task as these may vary with a change in the nationality or study program.

Alves and Raposo [17] assert that the provision of quality in all academic aspects by the institution establishes a positive image in students' minds which eventually leads to complete satisfaction and loyalty with the institution. Likewise, Yusoff et al. [49] examined the factors that affect student satisfaction in 1200 students from four private Malaysian higher education institutions. Results of the study showed that a suitable size of the class, responsible and competent faculty, and helpful administrative staff were the main influential factors that determine student satisfaction from their institutions.

Likewise, Gruber et al. [28] examined the students' perceived level of satisfaction about their institution by administrating the self-developed questionnaire that covered multiple aspects of the university life of the students. The empirical results indicated that the satisfaction of students was mainly based on a stable relationship between students and the institutional environment. Similarly, Fernandes et al. [50] studied the satisfaction and loyalty of the students enrolled in the United Arab Emirates higher education institutions. A sample of the study consisted of 187 graduates and the required data was collected by administering questionnaires. Their findings indicated that competent instructors that provide quality education were the most significant factor that affects student satisfaction and loyalty to these institutions.

In the context of the UK, Douglas et al. [51] explored the factors explaining the satisfaction and dissatisfaction of the students from the universities. The sample constitutes 350 students from two universities in the United Kingdom. A mixed-method approach was used to analyze the data. Results of the study revealed that quality of teaching and learning plus the learning and administrative support system of the institution directly influence the satisfaction and dissatisfaction of students. Likewise, Kashan [52] observed the factors that significantly affect the satisfaction level of students. The sample of the study was 120 master-level students from a public sector university in Pakistan. The findings of their study revealed that students' satisfaction was mainly influenced by the teaching faculty. Similarly, Butt and Rehman [53] found that the expertise of faculty members was the most influential factor among five analysed factors of student satisfaction.

Reilly et al. [54] observed the experience of American students at the University of Ireland. Data were collected from 150 American students and 149 native students. Results of the study identified that American students had adjustment problems. Furthermore, different levels of academic satisfaction and social support were also found in foreign students enrolled in the long- and short-term programs.

Arambewela and Hall [55] investigated the perceived importance of factors for selecting Australia as a destination for higher education by Asian international students and the level of their satisfaction with these factors. Data were collected through a mail survey and 573 replies were received from international students studying in five different universities in Victoria. Findings of the study reveal that education quality, advanced technology, monetary factors, accommodation expenses, security, status, and prestige of the institution were the significant predictors of international student's satisfaction.

In recent years, China has emerged as a major host destination for international education. However, there is a dearth of empirical literature when it comes to examining the international student's satisfaction with service quality in Chinese universities. Mastoi et al. [56] explored the satisfaction of Chinese students from the service quality of five key Chinese Universities. Their findings posit that, on the whole, Chinese students were satisfied with the teaching and learning environment of their institution. However, they were not satisfied with the quality of administrative support and their interaction with the administrative staff of their respective university.

Ding [57] examined the satisfaction of international students from study and living experience in the universities of Shanghai. Results of their descriptive analysis show that sample students were not satisfied enough with their study and living experience in China which can hamper the sustainable internationalization of China's higher education system. Likewise, Zhong et al. [58] evaluated the expectations of foreign students versus their actual satisfaction from the study in Sino-foreign cooperation institutions. The overall findings conjecture that foreign students were not satisfied enough with their educational experience in China. Besides gender, age, degree level, and tuition fee were the considerable factors that influence the satisfaction of these students.

The synthesis of extant literature shows that educational service quality has been much explored in the context of developed countries. Though the studies in China are mainly based on descriptive evaluation and lacks a comprehensive framework and empirical rigor to investigate each aspect of service quality. Specifically, in the international higher education market the quality of HEIs is measured by the quality of research support offered by the supervisor. However, this aspect of higher education has not been explored in the context of China. Taking expectancy disconfirmation theory as a theoretical foundation of this study, we examine whether the perceived expectations of the students resonate with their actual study and living experience in China. The present research aims to fill this gap by examining the quality of teaching services, advisory services, administrative support, learning infrastructure, support infrastructure, and overall satisfaction of foreign students with their study experience in Chinese universities. Therefore the following hypotheses emerge:

**H1.** *International students are satisfied with the quality of teaching services of Chinese HEIs.*

**H2.** *International students are satisfied with the quality of advisory services offered at Chinese HEIs.*

**H3.** *The quality of administrative services meets the expectations of international students.*

**H4.** *International students are satisfied with the quality of learning and support infrastructure at their host institution.*

**H5.** *On the whole, international students are satisfied with the service quality of Chinese HEIs.*

#### **3. Data and Methodology**

#### *3.1. Sample Characteristics*

The sample of this study consists of the foreign students enrolled in the Master's and PhD programs in 7 Public Universities of Wuhan, China. The rationale behind choosing Wuhan City was that it was considered as an Educational hub of China with the presence of several key state-level universities. Our sample of foreign students included 321 foreign students from Asia, 176 from Africa, and 150 students from Europe & America. This led to a total sample of 647 international students. However, 29 questionnaires were either incomplete or incorrect. Hence, our final sample encompasses 618 valid questionnaires for data analysis. The students' responses were gathered by using a questionnaire from the foreign students of Huazhong University of Science and Technology, Zhongnan University of Economics and Law, Huazhong Agriculture University, Central China Normal University, Wuhan University, Wuhan University of Science and Technology, and the China University of Geosciences. All these universities are part of project 211 which aims at developing 100 key Chinese universities of the 21st century hence receive preferential treatment and financial support from the Ministry of Education, China. All the male or female foreign students enrolled in Master's and PhD programs were made part of the sample. The researcher visited the sample universities to self-administer the questionnaire. The respondents were requested to fill in the questionnaire carefully. The participation of foreign students in the survey was completely anonymous and voluntary. A detailed description of the universities along with the sample size of foreign students is given in Table 2.

**Table 2.** Sample Characteristics.


The survey research method was used to conduct this research. Survey research provides the highest level of generalizability to represent a larger population. Because of possessing a vast number of people who participate in a survey, the collected data has a better comprehension of the relative traits of the general population of the study. Due to the higher representative sample size gathered through the survey research method, it often becomes convenient to report statistically robust results than by using other data collection procedures. The analysis of multiple variables can be effectively performed by the application of survey research. On top of this, for scientific research studies, the survey

method is ideal due to the provision of a standardized stimulus to all participants in the study [59].

Table 3 entails statistics of various demographic attributes of the sample respondents to support a better understanding of the data in this study. The frequency and percentages of the demographic factors, such as gender, age, study major, study year, degree level, and medium of instruction, of the foreign students are presented below.


**Table 3.** Demographic characteristics of respondents.

#### *3.2. Research Instrument*

The questionnaire was organized to investigate various service quality aspects of sample Chinese universities. This section covered foreign student's perspectives on the service quality of their host higher education institute from five key dimensions. The last section examines the overall satisfaction of foreign students by pursuing their higher education from Chinese HEIs. The detail on the construction of each dimension is outlined below.


#### Validity and Reliability of Research Instrument

The principal components analysis technique was used to extract eigenvalues and the variance explained by each component. A component can only qualify its place if the Eigenvalue is greater than 1. Table 4 shows that Eigenvalue is higher than 1 in all cases, therefore, all the service quality dimensions explain a unique aspect of the educational and support services provided by their host Chinese HEI hence confirms the validity of our research instrument. Moreover, the first component explained 31.4% of the variance in service quality. Similarly, the 2nd and 3rd dimensions determined 9.5% and 4.9% of the variation in service quality. While the 5th and 6th components explained 4.2% and 3.1% of the variance in service quality. In totality, all six components explained about 58% of the variance in service quality. Additionally, KMO statistics is a measure of sampling adequacy. The cut-off point for this statistic is 0.50. Thus, a value of 0.94 reflects that our sample is quite adequate.


**Table 4.** Principal components.

Table 5 presents the validity statistics of each research dimension as well as the overall reliability of the research instrument using Cronbach's alpha. The overall reliability alpha for all the service quality dimensions and overall satisfaction is 0.94. This posits that 94% of the variance explained by service quality dimensions is true variance. Thus, it authenticates the reliability and internal consistency of our research instrument.


**Table 5.** Reliability Statistics.

#### *3.3. Empirical Strategy*

For an in-depth understanding, we conduct item-wise, dimension-wise, and variable level analyses of student responses. Considering the nature of data descriptive statistics and figures are used to elaborate the difference between different categories. Principal component analysis (PCA) is a multivariate statistical approach used to reduce the data with several dimensions to few representative variables which provide sufficient information about the entire as was available in the larger data set [63]. Considering that our survey data cover various aspects of service quality, the use of PCA helps us in transforming a bigger set of variables into a smaller set of variables without much loss of information.

Furthermore, an analysis of variance (ANOVA) is employed to observe whether there is any significant difference between various groups of variables. Hence, the use of ANOVA helps us to ascertain whether there are significant differences between service quality dimensions and satisfaction in our sample universities.

#### **4. Data Analysis and Interpretation**

Data were analyzed through Microsoft Excel and SPSS 20 software package. Descriptive statistics, principal component analysis (PCA), and ANOVA were used for analysis to evaluate foreign students' perceived level of satisfaction with the service quality of Chinese universities.

#### *4.1. Statement-Wise Analysis of Service Quality and Perceived Satisfaction*

Table 6 presents the gender-wise questionnaire analysis of service quality and satisfaction statements. Along with the arithmetic mean and standard deviation, a t-statistic is also provided to check whether a significant difference exists between the male and female student's responses on each statement of service quality and perceived satisfaction.

**Table 6.** Questionnaire analysis: difference in the satisfaction level of foreign students by gender.



#### **Table 6.** *Cont.*

The item analysis of teaching services reveals that on the whole female foreign students have a lower satisfaction from teaching services as compare to their male counterparts. Besides, foreign students highly agree with the statement that the teaching staff of their host Chinese university has a supportive and friendly attitude. Moreover, foreign students have a higher level of agreement on the statements that teachers are competent and well qualified and are willing to offer individual attention to the students. However, Figure 2 depicts that the statements that course instructors are easily available for guidance and English speaking skills of teachers are good have the lowest mean score which is in line with reflects that availability and English speaking skills of the sample universities can be improved to enhance the teaching and learning experience of foreign students. These results support H1 and are in line with Wen et al. [31].

**Figure 2.** Mean score of the statements about teaching services. Source: Authors (2021).

The advisory services are of prime importance to the success of a postgraduate program. The mean values presented in Table 6 reveals significant differences in the advisory services between male and female foreign students as male students exhibit a significantly higher satisfaction from advisory services of their host institution. These findings are consistent with Zhong et al. [58] that there exist significant gender-based differences in the satisfaction of international students from the service quality of Chinese HEIs. Figure 3 indicates the mean score of the statements in descending order. On the whole, foreign students strongly agree with the statement that they can contact their supervisor easily and that their supervisor is supportive of their needs. Yet have a moderate mean score on the statements that their supervisor guides them when things go wrong, and that their advisor gives them prompt feedback. Meanwhile, the statements that supervisors monitor the progress of foreign students on a regular basis and that supervisors arrange formal progress meetings, as well as the satisfaction of foreign students with the frequency of these formal meetings, have the lowest mean score which reveals that the overall majority of research advisors are not giving proper time to their foreign students, which can significantly reduce the quality of their research project. Sakurai et al. [48] also propose that fewer meetings with research advisors leads to reduced satisfaction of the students. These results reject our second hypothesis hence sample Chinese universities should reduce workload from the research advisors and shall take such measures which can ensure that foreign students are getting enough guidance and support from their supervisors at all stages of their research project.

**Figure 3.** Mean score of statements about satisfaction from advisory services. Source: Authors (2021).

Overall, the foreign students have a higher mean score on the questions that concern classrooms being well equipped and library and lab resources sufficient to meet their research needs, which shows that the learning infrastructure of Chinese universities accepting foreign students is quite developed as they are getting proper attention and financial support from the government. Yet, students have a lower mean score on the statement that "My school provides enough internship and placement opportunities" and the reason for this is that foreign students usually face language barriers while working with Chinese corporations. Moreover, although foreign students were quite satisfied with accommodation and transportation facilities, they were relatively less satisfied with the medical and canteen facilities, probably due to the differences in the treatment procedures and cuisine between their home country and China. On the whole, these empirical outcomes support the fourth hypothesis of this research, in that foreign students are satisfied with the quality of learning and support infrastructure of their host Chinese HEIs.

As for the overall satisfaction of foreign students, male and female students have a mean value of 3.88 and 3.72 on the statement that "I think I am positively moving towards the fulfillment of my academic goals" revealing that overall, international students are quite satisfied with their higher education from China. Though, female respondents have a significantly lower score on the statement that "I think I have made the right decision to study in China" and the t-value is also statistically significant. Moreover, foreign students have a moderate agreement on the questions that they will recommend china and their university as a study destination to their colleagues and friends. Yet, male students have a significantly higher agreement with the statement that "I will keep in touch with my school after the completion of my degree".

#### *4.2. Category-Wise Analysis of Foreign Students Satisfaction from the Quality of Educational Services Provided by Chinese Universities*

In this section, major and degree level based descriptive analysis is conducted to reveal the aspects of service quality that need further improvement and to allow a comparison between different categories of international students.

Table 7 presents the items with a high percentage of satisfaction of the student responses. 75% of the students majoring in natural sciences and engineering agreed that their course teachers are competent and well qualified yet 65% of the arts and social sciences students favored this statement. Similarly, the majority of the foreign students agreed that the teaching staff has a friendly and favorable attitude towards the foreign students as 72% of the foreign students favored this statement. However, there is a significant difference between the major categories on the statement that students have a clear schedule with their supervisors regarding the frequency of meetings to discuss their research progress. As 68% of the natural sciences and engineering students agreed to this statement, only 52% of arts and social sciences students approved. This shows that natural sciences and engineering are getting better advisory services and support as they can take discuss their research work with their supervisor more frequently and regularly as compared to the art and social sciences majors. These findings resonate with Zhong et al. [58] that demographic factors considerably influence the satisfaction level of foreign students in Chinese HEIs.

On the whole, 71% of the foreign students agreed that the accommodation provided by the institution meets their living requirements. Moreover, 80% of the respondents stated that classrooms provide a favorable learning environment and are well equipped. This evidence suggests that Chinese universities accepting international students have better classroom facilities. Yet, the score on the statements that administrative staff have a positive and supportive attitude towards the foreigners is quite average and shows room for improvement, which negates our third hypothesis. These outcomes are in line with Mastoi et al. [56], in that students are not satisfied enough with their interaction with the administrative and support staff in Chinese universities. Overall, 70% of the international students agreed that the learning environment of the university meets their expectations.


**Table 7.** Study major-wise description of the factors with high level of satisfaction.


**Table 7.** *Cont.*

Table 8 demonstrates the factors with which international students were relatively less satisfied. Merely 47% of the international were in agreement with the statement that the English language skills of the teachers are good. The reason for this is perhaps that the entire education system in China was designed in the native language. Therefore even if the teachers are competent and better qualified in their fields still they may not be able to communicate that effectively as they teach the Chinese students. Hence, the HR office shall make sure that sufficient teachers are available to teach those international students whose medium of instruction is English. As with the advisory services, 64% of the natural sciences and engineering students agreed that their supervisor monitors their progress regularly while only 46% of the arts and social sciences students approved this statement. Similarly, 67% of the students belonging to the natural sciences and engineering category agreed to the statement that their supervisor helped them to develop the research plan yet only 53% of the arts and social sciences students responded in affirmative. This can be attributed to the fact that natural sciences and engineering students have to work in labs, therefore, they have more interaction with their supervisor yet the students of the social sciences have an entirely different type of research work usually have lesser chances to interact with their supervisor.

**Table 8.** Major-wise description of the factors with a low level of satisfaction.


Only 45% of the foreign students affirmed that their school provides enough placement or internship opportunities and the reason is perhaps that companies in China usually use Chinese as the conversation medium, therefore, it is very hard for the foreign students with less Chinese proficiency to find internships. Similarly, only 48% of the respondents stated that they are the menu in the canteen is suitable for them. This makes sense as the Chinese cuisine may be quite different from their native food culture thus it could be very hard for some of the foreigners to adapt to the new food choices. Moreover, 71% of the natural sciences category agreed that they will recommend their university to their colleagues and friend however only 55% of the social sciences category favored this statement. In the nutshell, besides the fact, the number of international students is increasing in Chinese HEIs, but there is still sufficient room for improvement in several areas to maximize the satisfaction level of the international students. In line with the findings of Hill et al. [29] and Sakurai et al. [48], there exist larger differences in the satisfaction level of the students concerning their field of study and the natural sciences and engineering students are relatively more satisfied than their arts and social sciences counterparts.

#### *4.3. Principal Component Analysis (PCA) of Service Quality Dimension and Perceived Satisfaction*

Table 9 reports the PCA of the service quality constructs and the overall satisfaction of foreign students. It is worth mentioning that if a particular item has a factor loading of less than 0.3 it shows that the item does not explain much variance in the dimension to which it belongs. The statement that the teaching staff is willing to provide individual attention to the students has the highest factor loading of 0.58 in the teaching services dimension which shows that Chinese instructors are considerate about the problems of foreign students. Similarly, the statements that offered courses offered to fulfill the needs of the program and sufficient learning material is provided by the instructor having a factor score of 0.57 and 0.55, respectively, which denote that foreign students place significant importance on the courses taught and the provision of learning material. While the statement that the English speaking skills of the instructors are good have the lowest factor loading which indicates the area for sample universities that require considerable improvement. Moreover, the results of Bartlett's test of sphericity suggests that results of the analysis are quite robust.

**Table 9.** Principal component analysis of service quality dimensions and the resulting level of satisfaction.


**Table 9.** *Cont.*


#### *4.4. Dimension-Wise Analysis of Service Quality Constructs and Overall Satisfaction*

In the dimension-based analysis, the service quality constructs are ranked based on their respective mean score. Besides, we provide a university-wise ranking of each service quality dimension to indicate the high performing and low performing institutions based on the foreign student's perception of service quality. Subsequently, ANOVA is performed to examine whether or not a significant difference exists between the service quality aspects of sample Chinese HEIs.

Figure 4 shows the mean score of service quality dimensions in descending order. Teaching services have the highest mean score of 4.14, which shows that foreign students are satisfied with various aspects of teaching services of their host HEIs. Learning infrastructure comes next with a mean score of 3.67 which is attributed that Chinese universities are quite up-to-date in terms of learning infrastructure because of substantial funding from the state government. Moreover, advisory services are ranked 3rd with a mean score of 3.58. This value should be a bit higher and indicate room for improvement. Yet, administrative support and support have the lowest rank with a mean value of 3.54 and 3.40 respectively which implies that sample Chinese HEIs shall take some measures to improve their administrative and support services. The mean score for overall satisfaction is 3.17, which is quite low. This outcome rejects our final hypothesis that on the whole foreign students are satisfied with their study and living experience in China. Thus, an

overall improvement in the service quality variables will ultimately increase the foreign student's level of satisfaction.

**Figure 4.** Ranking of service quality dimensions based on students' responses. Source: Authors (2021).

Analysis of variance indicates whether there are significant differences in a particular attribute between different groups. Table 10 reports the university-wise ANOVA of the service quality constructs. F-statistics and corresponding significance levels posit that there is no significant difference between the quality of teaching services provided by Chinese HEIs. In contrast, there are significant differences between the advisory services and the level of supervisory support provided to the foreign students of different universities. Similarly, there are significant university-wide differences in the level of administrative support provided to foreign students. Moreover, learning infrastructure and support infrastructure dimensions of service quality have the largest F-statistic and are significant at 1%. These results indicate that universities significantly differ in terms of the quality and type of learning and support infrastructure rendered to the foreign students. These empirical outcomes are consistent with Ding [57], in that there are significant differences in the service quality delivery in Chinese HEIs. Yet the empirical results for the satisfaction of the students are only significant at 5% which postulates that there is a considerable difference in the satisfaction level of foreign students with the service quality of their host Chinese HEIs.


**Table 10.** Dimension-wise ANOVA of service quality constructs by taking university as the category variable.

#### **5. Conclusions and Policy Implications**

As a consequence of theoretical and pragmatic investigation of foreign students' evaluation of the service quality of Chinese universities, the study findings lead to several conclusions. These outcomes not only depict the current state of international higher education programs in Chinese universities, but also warrant attention from the respective stakeholders to upgrade the under-developed areas of their service quality, to transform these universities into sustainable knowledge cultivation hubs of international standing.

Against the backdrop of massive financial and policy support from the Chinese government for the sustainable internationalization of China's higher education system, it is imperative that foreign students are satisfied with the service quality offered by their host institution. Thus, we adopt the bottom-up statistical approach to address this research question. As with the academic services of Chinese HEIs, foreign students exhibit high scores on the statements that teachers are competent and qualified, and that they offer a friendly and supportive attitude to foreign students. In contrast, the statement that the English language skills of the lecturers are appropriate showed the lowest score, which must be a matter of concern for the Chinese HEIs if they aim to optimize the teaching and learning experience of these international students. Moreover, foreign students agreed that their research advisors provide them timely feedback and are supportive of their needs. Yet, mixed sentiments were reported on the statement that their advisors arrange enough formal meetings to discuss the progress of their research project. Moreover, female foreign students have a statistically lower score than their male counterparts on such statements as their satisfaction with the frequency of formal research meetings with their supervisor and that they can contact their supervisor easily. Hence, necessitating a concrete effort from Chinese HEIs to not only increase the overall advisory support towards foreign students, but also to address the concerns of female students regarding the access and research advice from their supervisor.

Besides, it is important to examine the perceived satisfaction of international students based on their study major. The description of the statement analysis posits that students belonging to the natural sciences and engineering category have a higher score on various aspects of academic and support services of their institution than arts and social sciences students. These outcomes have policy implications and suggest that, in China, natural sciences and engineering disciplines have progressed well in terms of offering superior services to their students. However, there is considerable room for improvement in the arts and social sciences disciplines to optimize the learning and support infrastructure to maximize the satisfaction of international students.

The PCA analysis highlights that teaching services, advisory services, administrative support, learning infrastructure, and support infrastructure are the key service quality dimensions of Chinese universities. Moreover, the reliability analysis reveals a Cronbach's alpha of 0.94 which supports the validity and reliability of the research instrument used to measure satisfaction. Together, all the six dimensions of service quality explain about 58% of the variance in the satisfaction of international students. It reflects that perhaps there are some personal level motivational and psychological factors that also influence students' satisfaction besides the service quality of their host institution.

Nevertheless, the ranking of service quality constructs based on the response score of students reveals that in general international students are highly satisfied with the teaching services of their Chinese HEIs. Likewise, learning infrastructure was ranked 2nd with a mean score of 3.67. While advisory services were ranked 3rd in terms of students' satisfaction from the supervision and support offered by their research advisor. These results suggest that Chinese universities have taken a step forward to improve the overall academic experience of their international students, yet there is still considerable room for further advancement in teaching and research quality for these students. Administrative support and support infrastructure showed the lowest rank in terms of service quality, which entails that on the whole international students face problems while dealing with the administrative matters with the staff and lack the necessary social support to get adjusted to a new environment. Thus, indicate the areas which deserve special attention from the administration of Chinese universities for sustainable internationalization of their institution. Nevertheless, the score of the overall satisfaction dimension is 3.17 on the scale of 5, which is quite low and could be upgraded only if host institutions take a holistic approach to enhance their service quality rather than focusing on one or two aspects.

Furthermore, the ANOVA based on the university as a category variable resulted in a statistically insignificant difference in the quality of teaching services of sample Chinese universities. However, we found statistically significant differences between the rest of the service quality dimensions such as advisory service, administrative support, learning infrastructure, support infrastructure, and overall satisfaction of international students across the sample universities. These findings posit that, although all these HEIs belong to the 211 project, there exists a significant difference in the quality of academic and support services across Chinese universities. Whereas some universities have leapfrogged in the provision of quality services for foreign students, others lag behind and require concerted policy efforts to upgrade their academic programs for international students. Hence, this research provides a comprehensive empirical perspective on how the Chinese HEIs are moving forward to attain sustainable internationalization of the higher education system. However, our study was limited to international students studying in Wuhan. Future researches in this domain can take a diverse sample from various Chinese cities or provinces to compare the satisfaction of international students. Furthermore, studies can also extend this line of research by conducting a comparative analysis of foreign students' satisfaction studying in Western countries and China. Besides, it will be interesting to account for the impact of student's learning styles in influencing their perceived satisfaction [64,65] and the role of foreign students in promoting Chinese FDI especially in Belt and Road countries [46].

**Author Contributions:** Conceptualization, F.Y. and A.A.; methodology, F.Y. and Y.Z.; formal analysis, F.Y.; data curation, A.A.; writing—original draft preparation, F.Y.; writing—review and editing, P.P. and A.A.; supervision, Y.Z.; project administration and funding acquisition, S.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Primary data obtained through a questionnaire was used in this research. The data is available upon request.

**Acknowledgments:** We acknowledge the support of South China Normal University for financing the open-access of this project. We would also like to thank three anonymous reviewers for their insightful comments to enhance the overall quality of this research.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### *Article* **Technological Platforms for Inclusive Practice at University: A Qualitative Analysis from the Perspective of Spanish Faculty Members**

**Víctor H. Perera 1, Anabel Moriña 1,\*, Nieves Sánchez-Díaz <sup>2</sup> and Yolanda Spinola-Elias <sup>3</sup>**


**Abstract:** Currently, the development of new virtual environments as a complementary tool to faceto-face teaching and the increased presence of students with disabilities at university classrooms are changing the landscape of university teaching. This article analyses the actions of faculty members who carry out inclusive practices in the context of technological platforms. The research was based on the assumptions of the qualitative paradigm, using individual semi-structured interviews with 119 faculty members from 10 Spanish public universities. The results show the reasons for inclusive learning with technological platforms, the use that faculty members make of these platforms in their inclusive educational practices, and the influence of these on the learning of students, especially students with disabilities. The conclusions give a good account of the conditions that determine the pedagogical use that faculty members make of virtual environments to facilitate the inclusion of students.

**Keywords:** higher education; inclusive education; disability; technological platforms; faculty members

#### **1. Introduction**

Inclusive education is a challenge for university systems [1]. Among the Sustainable Development Goals (SDGs) of the 2030 Agenda, Goal 4 "Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all" is promoting a global agenda for the effective recognition of the rights of all people and the construction of a cohesive society that respects the principle of equal opportunities and non-discrimination [2].

Teaching at university today implies assuming that diversity is common, that students learn in different ways, and that their intelligences are multiple. It also involves mastering not only the content of a subject, but also knowing how to teach, adjusting to the needs of students, and making use of different resources, including technological ones [3].

The current educational reality is immersed in a period of adaptation to the digital era [4]. The European Education Area is addressing this challenge by promoting the Digital Education Action Plan [5], which sets out a vision of high-quality, inclusive, and accessible digital education at all levels of education. This new plan includes, among its strategic priorities, the development of a digital education ecosystem by making better use of information and communication technologies to improve the quality of teaching and learning processes. The report "Top IT Issues, 2020. The Drive to Digital Transformation Begins" [6], highlights some of the trends related to the digital transformation of universities. In particular, it focuses on issues related to students and teaching, highlighting the fact that we are moving inexorably toward student-centred systems. In the case of learners with disabilities, many studies confirm that ICTs facilitate their inclusion [7,8].

**Citation:** Perera, V.H.; Moriña, A.; Sánchez-Díaz, N.; Spinola-Elias, Y. Technological Platforms for Inclusive Practice at University: A Qualitative Analysis from the Perspective of Spanish Faculty Members. *Sustainability* **2021**, *13*, 4755. https:// doi.org/10.3390/su13094755

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Javier Puente and Leandro Almeida

Received: 29 March 2021 Accepted: 21 April 2021 Published: 23 April 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

In universities, this process of digital transformation is accelerating as a consequence of a profound change in the demand for higher education [9]. This movement, characterised in part by the presence of a more plural, diverse, and heterogeneous student profile, is playing a crucial role in technological resources [10]. Thus, one of the areas of university activity where the implications of educational technologies are having the greatest relevance is traditional university teaching [11]. In the report "Situation and challenges of Spanish universities in the face of digital transformation", published by the Conference of Social Councils and the Network of University-Business Foundations, it is stated that Spanish universities are quite digitised in terms of new teaching methodologies and new learning and working environments [12]. The predominant educational discourse in the scientific literature on technology and disability focuses on the idea that ICTs are resources that generate real situations of access to inclusive education both in virtual scenarios and in face-to-face teaching modalities supported by technological media. Specifically, training modalities such as e-learning and m-learning can overcome barriers and open up opportunities to facilitate inclusive and equitable education [13].

In the last decade, educational opportunities in higher education institutions have been evolving beyond the traditional classroom environment thanks to existing educational technologies [14]. Moreover, the development of new virtual environments as a complementary tool to face-to-face teaching [15] and the increased presence of students with disabilities in university classrooms [16] are changing the landscape of face-to-face teaching. In Spain, the report "UNIVERSITIC 2017: analysis of ICT in Spanish Universities" highlights that the universities' commitment to technological resources as support for teaching has reached saturation levels, with over 90% of faculty members using institutional platforms for virtual training [17].

The 2017 Horizon Report identified, as a key trend, the combination of traditional face-to-face training with innovative technologies for higher education worldwide [18]. The technology of virtual learning environments, i.e., LMS—Learning Management Systems, is the technological tool that has had the highest rate of acceptance in higher education. In fact, in Spain, all universities have some kind of LMS to support face-to-face training or for the development of blended learning [19]. However, the report "ICT 360◦: Digital Transformation in the University" highlights that the challenges and responsibilities that most universities must face are faculty training and the reformulation of content to adapt to digitisation [20].

This article explores the practices of faculty members who carry out inclusive education with technological platforms. The analysis of their discourses allows us to learn more about how virtual environments can contribute to inclusive learning for all students, with or without disabilities.

#### *1.1. Inclusive Education as a Promoter of a More Democratic and Sustainable Society*

Inclusive education is based on the need for educational institutions to transform their cultures and practices to ensure the learning of all students, promoting their participation and seeking to eliminate the processes that lead to social exclusion [21]. Inclusion is conceived as a process that involves and commits to the transformation of educational institutions in order to provide a response to all students. Inclusion is fundamental for achieving quality education for all students and for the development of a more democratic and sustainable society [22].

When universities are identified with the principles of inclusion, diversity is valued, recognising that there are different ways of learning and that all students bring things of value to the learning environment. In addition, it removes barriers linked to exclusionary practices and works proactively to respond to the needs of all learners [23]. Inclusive practices can enrich the curriculum and the success of all students. Learning-centred approaches and Universal Design for Learning (UDL) have been shown to be effective in inclusive contexts [24].

Research on inclusive education and higher education often highlights faculty members. The majority of studies carried out to date conclude that faculty members are key to the success and sustainability of an inclusive approach [25,26]. When faculty use technological resources in their teaching, including virtual learning environments, inclusive teacher practices can contribute to the retention and success of students with disabilities [27].

#### *1.2. Overcoming Challenges in the Use of Technological Platforms for Diversity*

Currently, most Spanish universities are using LMS as a resource to support face-toface classes in an effort to provide new scenarios and alternative learning solutions [12], especially favourable for students with disabilities [16]. However, as learning with technological resources in the classroom advances as a complement to traditional face-to-face teaching, universities and faculty are facing the challenge of promoting changes in students [28,29], especially related to acceptance and trust in LMSs [30]. On the other hand, studies suggest that students with disabilities are attracted to LMSs due to the ease of access and use as they can set their own pace and intensity of learning according to their interests [31,32].

Virtual environments are improving their functionalities and the quality of materials, making it easier for students to adapt to these new learning environments; however, some studies have revealed that people with disabilities must overcome previous barriers to ensure the appropriate use of LMSs [33,34]. Thus, for example, many of the challenges faced by students with disabilities using LMSs negatively affect their quality of life in terms of stress levels, self-esteem, personal relationships, and time available for other activities [35]. In line with these findings, Maboe [36] confirmed that the time taken by students with disabilities to complete assignments in LMSs is much longer compared to students without disabilities. These studies established that LMSs must be well adapted to ensure efficient and effective use by students with and without disabilities.

The challenges posed by these changes, together with the poor preparation of faculty in the use of LMS, are currently evident [37]. Universities are finding it difficult to convert these virtual environments into learning environments that increase accessibility for persons with disabilities, which makes it necessary to guide faculty members in this transition [38]. Currently, faculty training programmes in technology-supported instruction focus on accessibility issues rather than on understanding the specific learning needs of students with disabilities [39]. In their study, Greer et al. [40] showed that many of the faculty members who teach with the support of technological resources are poorly aware of the fact that LMSs are promising tools for the individualised education of students with and without disabilities. In addition, most faculty members need training to effectively implement individualised and inclusive teaching in a virtual learning environment [27,41].

In the area of difficulties related to faculty use of LMSs, the reviewed studies identified a number of barriers, including fear of using these learning environments, poor trust in the effectiveness of their use in teaching [11], lack of self-confidence to pedagogically use these environments and, in particular, poor preparation for teaching students with disabilities or no experience working directly with students with disabilities in a face-toface or virtual environment [42]. These studies give a clear picture of the existing barriers and a general idea of how faculty training should be geared toward the effective use of this technology. Therefore, it is necessary to design specific training programmes that address the professional development of faculty members in order for them to acquire the technical and pedagogical skills required to integrate and use technological resources appropriately with students with disabilities and, more specifically, to know how to adapt virtual environments and digital materials to make them accessible [43].

#### *1.3. Accessibility in Virtual Environments as a Prelude to Digital Education for All*

From a practical perspective, research on digital accessibility in higher education is in its early stages [44]. With the rise of blended learning, there is an ongoing concern about accessibility, particularly for students with disabilities [45]. In this regard, in Spain, the

Royal Legislative Decree 1/2013, of 29 November [46], which approves the Revised Text of the General Law on the Rights of Persons with Disabilities and their Social Inclusion, and regulates the right to equal opportunities and non-discrimination of these students, establishes that universities must have the necessary support for the inclusion of students with disabilities and mentions the need to include training plans in universal design or design for all people in the different university degrees.

Universal design can help provide greater accessibility in the virtual learning environment, not only for students with disabilities but for all students [47]. Studies have shown that the improvement of student learning increases significantly when the LMS is adapted [48]. However, studies focusing on accessibility do not seem to have consistently considered the design of learning materials in digital format [49], despite the fact that adapting LMSs improves their accessibility when learning objects are adapted [50].

Providing materials in a way that they are accessible to learners, including those with disabilities, is important in the virtual environment. Some works conclude that the usability of LMSs is significantly enhanced when students with disabilities can access the learning materials offered in these environments [3]. For this reason, LMSs should incorporate materials adapted to the personal needs and preferences of all learners, although studies such as those by Brito and Dias [51] conclude that there is no single solution that can meet all individual needs, even when responding to the same type of disability. However, many of the materials that are shared are not adapted, thus the level of accessibility of information in LMSs is often quite low [48].

Adaptations of LMSs and materials are not sufficient if the syllabi are inadequately designed and create additional barriers to participation [52,53]. There are several considerations when designing a syllabus and presenting content that is accessible to all learners, including those with disabilities [40]. One of these considerations is to use the principles of universal design for learning (UDL), which allows "all people to have an equal opportunity to learn" [54]. Designing a training programme based on UDL principles contributes to making education inclusive, improves accessibility without the need for environmental adaptations, and engages participants in their learning [3,55].

In short, based on the use of LMS, and from the perspective of faculty members who carry out inclusive practices, this article analyses the reasons that lead them to promote inclusive learning in virtual environments, the use they make of these in their educational practices, and how technological platforms influence the learning of students with and without disabilities.

#### **2. Materials and Methods**

The design of the present study is framed within the qualitative paradigm. Specifically, in the biographical-narrative research design, focusing in this case on the importance of highlighting the experience of inclusive faculty members. The results of this study are part of a larger research project, funded by the Spanish Ministry of Science and Innovation, entitled "Pedagogía Inclusiva en la Universidad: Narrativas del Profesorado (EDU2016-76587-R)". This study focuses on what inclusive faculty members do and how and why they do it, exclusively analysing the inclusive actions of these faculty members regarding the technological platforms (Moodle and Blackboard), used as a complement to face-to-face teaching.

#### *2.1. Participants*

This study involved 119 faculty members from 10 Spanish public face-to-face universities, belonging to all areas of knowledge. The participants were selected by students with disabilities, through the collaboration of the Disability Support Services of the different universities. The choice of faculty members was based on the following criteria [56], by which they demonstrated that they had carried out inclusive practices: believing in the possibilities of all students; facilitating learning processes; using different methodological teaching strategies; showing concern for their students' learning; motivating students; maintaining close relationships; and favouring interactions between students or allowing students to participate in the class and build knowledge together.

Regarding the profile of the participating faculty members, 24 taught Arts and Humanities (20.2%) (faculty members identified with P1 to P24), 14 taught Engineering and Sciences (11.8%) (P25 to P38), 16 taught Health Sciences (13.4%) (P39 to P54), 25 taught Social and Legal Sciences (21%) (P55 to P79), and 40 taught Educational Sciences (33.6%) (P80 to P119). In terms of gender variables, 69 were male (58.3%) and 50 were female (41.7%). Regarding age, 108 faculty members were between 36–60 years old, 7 were under 35 years of age (7.8%), and 4 of them were over 60 years of age (4.4%). Regarding teaching practice, most of the faculty had more than 10 years of teaching experience (68.4%), with six (6.2%) faculty members having less than 5 years of experience and 24 having between 5–10 years (25.4%).

#### *2.2. The Spanish University Context*

The report "Data on the Spanish University System, 2019–2020" [57] indicates that the Spanish University System is made up of 82 universities, with 50 public universities (49 face-to-face, 1 non-face-to-face) and 32 private universities (27 face-to-face and 5 nonface-to-face). Currently, online university education is almost exclusively dominated by private universities, while public universities provide face-to-face teaching combined with or supported by technological platforms [58].

In Spain, all public universities have support offices for students with disabilities, and 21,435 students with disabilities are currently enrolled in undergraduate courses [59], representing 1.5% of the total student body in Spanish universities. The disability offices ensure that these students have the necessary resources for the development of their learning process and advise faculty members on the reasonable adjustments to be made.

#### *2.3. Data Collection Instrument and Procedure*

The research was conducted on the basis of individual semi-structured interviews in which three questions were explored: (1) the reasons for inclusive learning with technological platforms; (2) the areas in which faculty members make use of LMSs in their inclusive educational practices; and (3) the influence that LMSs have on the learning of students, especially students with disabilities.

The interviews were conducted by members of the research team, who were previously trained for the task. Most of these interviews were conducted face-to-face (89 faculty members). Where face-to-face interviews were not possible, telematic means were used (videoconference with Skype and telephone calls). The interviews were audio-recorded and lasted approximately 90 min.

#### *2.4. Data Analysis*

The information collected from the interviews was transcribed verbatim and processed through qualitative data analysis using an inductive system of categories and codes, which allowed organising and making sense of the information collected [60]. Table 1 below shows the categories and codes used for the development of this study.

#### *2.5. Ethical Issues of the Research*

In terms of the ethical issues of this research, we ensured that the information collected in the interviews was confidential and anonymous. In this regard, the participants and the researchers signed a confidentiality document in which the terms regarding the collection, processing, and publication of information were agreed upon. In addition, the participants were informed of their right to withdraw voluntarily at any time during the research, which would mean that their data would be removed from the study. Therefore, this study complied with the ethical requirements approved by the Spanish Ministry of Science and Innovation.


**Table 1.** System of categories, indicators, and codes used.

#### **3. Results**

This section shows the main results that emerged from the faculty members' discourse on the reasons that led them to inclusive learning with technological platforms, the use they made of these in their inclusive educational practices, and, finally, how technological platforms influenced the learning of students with and without disabilities.

#### *3.1. Why Inclusive Learning with LMS?*

Faculty members who carried out inclusive educational practices incorporated various technological media, especially LMSs, in their syllabi. In general, the faculty had a good concept of the use of these virtual environments in the educational processes, considering them as resources that facilitate educational inclusion.

*I think it's always positive because with these resources you can work online, do a fruitful search. It is a tremendous support. Also, there are resources that help students with disabilities who have a physical limitation (P1).*

Regarding the faculty members' view of the students' relationship with these technologies, some of the participants felt that their use was common among students and that the latter were comparatively better prepared and more technically proficient than the faculty.

*In general the students have no difficulty in adapting and using it because they are better prepared than me. That seems quite natural, and I have had to learn it. (P116).*

Although not all the faculty agreed that the use of these tools was necessary to achieve inclusion, others stated that their interest lied in the potential of these tools to respond to the educational needs of all students.

*Everyone can access and download the curriculum that we have seen in class. For students with disabilities I think it's perfect because they need more resources, but for students who don't have problems, I think they could bring something more to the subject (P37).*

However, for the above to become a reality and for the potential of these tools to be exploited, many faculty members highlighted the need to show a good predisposition toward the use of LMS. They proposed that there should be an improvement in technopedagogical training, which, in the eyes of the students, would help the faculty to be renewed and closer to the way in which learning takes place today.

*I believe that electronic resources should be used for practically everything because students are motivated by them and use them every day. I think that faculty members are the ones who need to modernise a bit (P60).*

#### *3.2. Which Areas of LMS Generate Inclusive Educational Practices?*

The faculty members considered several areas of application of LMSs to support their face-to-face teaching. In this sense, they used LMSs as an environment to support learning processes, as they allowed: (a) accessing this environment free of charge and making direct, easy, and useful use of it; (b) consulting digital study materials and materials of various types that could be shared and distributed in a permanent and organised way, enabling students to follow the lesson in case of being absent from class; (c) submitting assignments for evaluation; (d) consulting the planning and management of the subject (for example, knowing when to propose and submit activities or an exam test, have materials or resources of the lecturer's oral subject available, publishing the notes developed for each subject, etc.); and (e) contacting and communicating directly with classmates and faculty.

*We have the WebCT platform. It is a great invention. I love it, as it is a way of communicating directly with all students and distributing materials (P70).*

A second area of LMSs used by the participants was for pedagogical purposes that facilitated their work, for example: (a) promoting participation in class, using different resources that adapted to different learning styles; (b) facilitating access to certain materials, such as digital books, to work from places other than the classroom; (c) combining and enriching these educational environments with other programmes or applications, in addition to other digital resources that did not exist before; d) providing guidelines for a critical and selective attitude toward the type of digital sources consulted, necessary for the quality of the expected results; (e) familiarising with the digital resource aimed at students who belonged to a generation that was also digital, making them feel more attracted and motivated by this medium; and (f) creating ecological awareness by reducing the use of printed documents, although some faculty members did not renounce the use of the reprographic service to share materials or take written exams.

*I use the virtual classroom a lot. It's a fundamental tool, as it contains the whole course (ppt, videos, even the notes). Apart from the teaching development, the students have the planning of each subject, the contents, what they have to do at each moment, where and when they have to hand in the practical assignments, what exercise they have to do each day... etc. Thus, the virtual classroom is fundamental and is one of the keys to the work in the preparation of the subject, without any doubt (P15).*

The use of the LMS was also focused, as a repository, on the dissemination of essential materials for the development of the subject, with several purposes: (a) to publish documents, texts, notes and presentations for the follow-up of the lessons; (b) to present the syllabus and the teaching project (providing information on the syllabus, assessment calendar, exam regulations, basic and complementary bibliography, etc.); (c) to publish digital files of the course, with the aim of providing information on the subject (e.g., the course content, the assessment calendar, the exam regulations, the basic and complementary bibliography, etc.); (d) to publish the course content and the teaching project; (e) to publish large digital files for downloading and consultation; (f) to link digital resources of interest that complement the information seen in class or even those parts of the syllabus that were not covered in class; and (g) to display documents with the solutions to the problems seen in class. However, some faculty members also used it for other purposes: (a) to create announcements on the board for notices related to the development of the subject; (b) to use social applications such as the blog, forum or wiki to generate group work spaces; (c) to carry out evaluations or self-evaluations of different types (e.g., open-ended, multiple-choice).

*My lessons are based on PowerPoint files, which I then convert to PDF and publish on the platform. Then, the contents of each topic are also published on the platform. If there are interesting videos, we watch them in class and I upload the link to the platform, and I also publish links that we don't have time to watch, but are interesting. These are the resources I need (P28).*

#### *3.3. How do LMSs Influence the Learning of Students with and without Disabilities?*

The valuation that faculty members made of the impact of LMSs on the learning process was generally positive. However, they believed that the impact of LMSs did not depend on whether the student had a disability or not, but rather on whether these media were fundamentally focused on a good didactic use and adjusted to the educational needs of the students. To this end, the faculty alluded to the need of changing traditional teaching methodologies for strategies that integrate these new environments, as well as to the planned use of these virtual spaces in such a way as to produce greater interest in the students.

*More than the platform itself, it is the use of the platform that has an influence, because if you don't use it properly, it has no influence. It is wonderful for any student. I don't know if any technological device would adapt to any specific disability, I suppose it would, but it hasn't been the case. Generally speaking, technological advances have been a revolution. Today we are live and we connect, the Prado Museum, such and such an artist... (P31).*

The faculty members also conferred a distinctive character on the LMSs, by which virtual educational environments had implicit advantages over the training provided in face-to-face learning. Thus, for example, they commented that LMSs facilitated accessibility to reliable information, in an immediate way, thanks to the fact that the media and materials were in digital format. This is why these environments were used to democratise opportunities for participation and give students equal access to a particular subject.

*The fact that there is this tool where you enter and have everything you can think of about the subject, makes the class freer too, doesn't it? The opportunity we have to be together in a space, to dedicate it to other things, to work in groups, to debate... (P20).*

The participants believed that the use of LMSs could influence several areas related to the development of students' learning. Thus, these environments were characterised by: (a) providing workspaces adapted to sharing and building; (b) facilitating the submission of assignments in digital format for supervision and evaluation; (c) providing security by making study materials and support resources available in advance; (d) enabling access to online tutoring sessions for consultation and resolution of academic doubts; (e) promoting didactic interaction with the materials, among the students themselves and between the faculty and the student, allowing open and continuous communication; and (f) favouring the ubiquity of learning in more appropriate contexts and situations.

*Of course, I use the technological platform. Not only to post the material, but also for the submission and evaluation of the activities, and I have sometimes used forum and chat tools to discuss in class some activities that have not been clear. I often use the email to communicate with the students. I also post links to the videos so that they can watch them at home (P91).*

While the faculty felt that LMSs could bring important opportunities and improvements to the learning process for all students, they also highlighted the particular benefits for students with disabilities. For example, some faculty members pointed to the accessibility of these environments as one of the main benefits for students with disabilities who may not be able to attend classes or who simply need more time to understand the material.

*What I see, from the use I make of it, is the possibility for students with disabilities to have the material in their laptops... to have it readily accessible at any time, which does not have to be on paper [...], and they can see it from home, they don't have to travel (P109).*

Another positive aspect deriving from the accessibility of the learning environment from anywhere was related to the possibility of adjusting the learning pace to the available time. This enabled the learner to demonstrate greater creativity and productivity by being able to repeatedly refer to the material, achieve a better understanding of it, and thus reduce their insecurity.

*[...] for people who have a disability in their hand, a pencil and a tablet provide a much better chance of coming to understand and do something that a paintbrush doesn't allow them to do, where they know that going back is complicated if they make a mistake... So the fear issue disappears and creativity flourishes (P17).*

One way in which faculty members facilitated the students' ability to adapt their learning rhythms consisted in videotaping their lessons and sharing them within the LMS. In this way, each student, with or without a disability, and according to their way of learning, was supported by one or another type of study material. Having alternative routes for learning was possible thanks to the diversification of the provided materials (editable Word or PDF files, videos) or the available channels (virtual platform, mobile phone, social networks). This adjustment to the interests, needs, and abilities of each student significantly linked inclusive practices to personalised learning.

*In addition to the fact that I believe it is fundamental to achieve the motivation of students with or without disabilities, I understand that providing the instruments from different perspectives helps all students to feel that at a given moment they are given the instrument that is most appropriate for them to obtain the information (P39).*

However, the participants indicated that, in many cases, these materials did not meet the specific needs of certain disabilities, such as visual impairment, due to the students' own perceptual characteristics, except in the case of one faculty member, who stated that he designed the teaching project with the contents adapted for visually impaired pupils, in a way that they could be seen by any pupil, with or without special educational needs.

*Well, always, when designing the project, when I am already thinking about the contents I am going to develop, in the references, I try, for example, to include books that are accessible to a visually impaired person (P44).*

#### **4. Discussion and Conclusions**

In universities, inclusive education is one of the most complex challenges that faculty members are facing nowadays. In line with the present investigation, other studies have highlighted the importance of the faculty member and, more specifically, their pedagogical practice in the classroom as one of the key elements in the processes of educational inclusion [16,28]. In recent years, national and international reports on the process of digital transformation of education indicate that universities, and specifically Spanish universities, are making significant progress in the field of teaching methodology, by gradually incorporating technological resources that are promoting the transition from traditional teaching to digital, inclusive, and accessible teaching [5,12]. This study has taken good account of this emerging reality by delving, from the perspective of faculty, into the conditions that determine the pedagogical use that faculty members make of virtual environments to facilitate the inclusion of students.

In university classrooms, the presence of technological media, and more specifically the use of virtual environments, is shaping many of the teaching practices that are being considered inclusive. In fact, as is demonstrated in this investigation, numerous studies conclude that LMSs are one of the essential tools in teaching, thanks in part to the fact that their use has practical applications in various areas of learning for students with and without disabilities [48,61]. Along these lines, the main findings of this study emphasise the interest that faculty recognise they have in the possibilities offered by LMSs when used to support face-to-face university teaching in order to develop inclusive practices.

A first conclusion drawn from the obtained results is related to the idea shared by the faculty that the integration of technological resources in their syllabi is a consolidated reality. In this sense, and as is stated in the report on the situation of educational technologies in universities [19], it is worth remembering that all the universities participating in this study had their own LMS for online training, generally Moodle and Blackboard, which they made available to faculty for use as support for face-to-face teaching. From a faculty member's perspective, one of the reasons why this technology has become a key element

in educational practices is due to the consideration that its use facilitates the development of inclusive practices, although it is not the only means to achieve this [61].

With regard to the competence area, several studies have highlighted the difficulties related to the use of LMSs by faculty members [27,41,42,62,63], suggesting the promotion of specific technopedagogical training programmes in addition to improving the predisposition toward the use of these environments, which are aspects that have been widely studied and have been related to the potential of these media to respond to the educational needs of students [37,38]. In a precursor to this work, and despite the fact that universities offer a wide range of courses for training in the use of LMS, Author et al. [43] already pointed out the advisability of attending to professional development with training programmes that enable the use of these technological resources with students with disabilities.

One of the most relevant findings presented in this study relates to the areas in which faculty use LMSs to support their face-to-face teaching. Three areas can be distinguished that facilitate the work of faculty members who carry out inclusive practices and who define the environment for a specific pedagogical purpose. The first area defines the environment as an accessible space that contains the material and tools that support students with and without disabilities, allowing them to regulate the pace and intensity of their learning according to their interests [31,32]. The second area contemplates the environment as an inclusive space for the differentiated performance of face-to-face teaching activities, which facilitates the development of digital educational processes [48,50]. Lastly, the third area considers the environment as an adapted space that is used to store and provide adapted materials, with which learners improve significantly in their studies by being able to interact with learning objects [3].

Numerous studies have reported the challenges that students have had to overcome in order to use LMS as learning environments [30,33,34]. In this study, however, the faculty revealed the wide acceptance and use of these technologies by students, as they provide them with a better preparation and an adequate level of digital competence. This is a favourable situation that does not exempt universities from the difficulties they are facing in making these new learning environments more accessible to people with disabilities [44]. In this regard, overcoming these drawbacks has largely been the responsibility of faculty members, who have highlighted the benefits implicit in the accessibility of environments and materials due to the opportunities and improvements implied in the learning process for students, which lead them to adjust their learning pace and demonstrate greater productivity and creativity.

Finally, regarding the impact of LMS on students' learning, the faculty members valued it positively, highlighting that such impact does not depend on whether the student has a disability or not, but on the didactic approach with which the technology is used to adjust to the educational needs of students [39]. Another opinion is that matching the interests, needs, and abilities of individual students significantly linked inclusive practices to personalised learning. Thus, the faculty members felt that the effectiveness of the technology that mediates teaching depends on the type of disability to which it is able to respond. In this sense, LMSs are not always infallible and it is necessary to use resources adapted to the specific needs of each student with disabilities and their level of technological competence. Therefore, the usefulness of these technological means ultimately depends on the environment [48], its materials [40], and teaching [52,53] being adapted to the students' circumstances in order to optimise their learning conditions. This is only possible if educational policies in universities continue their commitment to offering programmes that ensure faculty training focused on the design of syllabi that are designed from the UDL and contribute to inclusive education being present in virtual environments.

#### **5. Implications for Academic Community**

This study contributes to filling a gap in the scientific literature in light of the ambitious challenge of promoting accessible virtual learning environments. Some practical implications for an inclusive use of LMSs in higher education are presented in this section.

The use of LMS in the classroom has changed the landscape of university education, with important consequences for teaching practice and students' learning experiences. In order for the use of technology to improve the quality of learning and educational opportunities for students with and without disabilities, it is necessary for universities to offer continuous training programmes that allow faculty members to create new scenarios and alternative learning solutions.

In order to effectively implement individualised and inclusive teaching in a virtual learning environment, the environment and material must be adapted to the specific needs of students and their level of digital competence. This requires reflection, dedication and training for faculty members in technopedagogical skills and in the design of specific training programmes based on the principles of universal design for learning that contribute to inclusive education.

#### **6. Limitations and Future Research**

In this study, we identified some limitations related to the methodological design, such as considering the faculty members' discourse from a global perspective, without paying attention to their specific field of knowledge. This means that their view of certain aspects that could be exclusive and defining of the different branches of knowledge analysed was diluted in a discourse without nuances. Likewise, the students' disability could have been taken into account as a second criterion for classifying the faculty members' discourse, which would have helped to identify specific forms of action in their practices according to this variable. Therefore, as a prospective of the study, these limitations should be overcome in order to continue evaluating the educational reality of the practices and challenges faced by faculty members who attend to the needs of students with disabilities. It would also be advisable for future research to further delve into the pedagogical implications for students of the application of the principles of UDL in virtual environments, with digital materials and inclusive educational programmes.

**Author Contributions:** V.H.P., A.M., N.S.-D. and Y.S.-E. have contributed equally to this work and have approved the version presented. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Ministry of Science and Innovation, State Research Agency and FEDER funds European Union, grant numbers EDU2016-76587-R/ Feder Funds.

**Data Availability Statement:** The data can be obtained by contacting the authors of the article directly.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


### *Review* **Teachers' Social–Emotional Competence: History, Concept, Models, Instruments, and Recommendations for Educational Quality**

**Gissela Lozano-Peña 1, Fabiola Sáez-Delgado 2,\*, Yaranay López-Angulo 3,4 and Javier Mella-Norambuena <sup>5</sup>**


**Abstract:** Teachers' social–emotional competencies are essential to educational quality. This study aimed to describe the theoretical background and relevance of teachers' social–emotional competencies. We conducted a systematic review with a critical, theoretical review approach. The results showed that the concept has an increasingly complex history and followed a structured course from 1920 to present. Five main models have been identified: emotional regulation, prosocial classroom, Collaborative Association of Social Emotional Learning, Bar-On emotional intelligence, and emotional intelligence. There are measurement instruments consistent with four of the identified models; however, the model that does not have its own instrument uses different available scales. Specific recommendations are proposed to develop social and emotional competencies in educational public policies, which include school leadership, assessment, and teacher professional training. In conclusion, it is relevant to have clear guidelines that conceive and conceptualize social–emotional competence univocally. These guidelines would allow the design of instruments with a comprehensive and sufficient theoretical base that reflect the multidimensionality of the concept, provide a precise measure to assess the effectiveness of intervention programs, and enlist teachers who seek the development of the different skills that involve social–emotional competencies.

**Keywords:** socioemotional competence; models; emotional intelligence; social intelligence

#### **1. Introduction**

Social and emotional competencies (SEC) have been positioned as a central element in human development because of the high predictive capacity they have towards variables related to the educational context [1–3]. Thus, in recent years, international organizations, such as the European Union, the United Nations, and the OECD have recognized the relevance of SEC. This promotes their inclusion in international conventions and treaties signed with different countries [4]. In the same way, other organizations such as the World Bank, the World Health Organization, and UNICEF have joined efforts to establish a more explicit intention in the development of SECs [5].

The teaching profession is considered one of the most emotionally demanding professions, which can affect mental health and wellbeing [6,7]. It is also associated with episodes of stress and burnout [8]. A study conducted in Mexico [9] with 549 teachers from different educational contexts confirmed that SECs are predictors of burnout, and that teachers, in

**Citation:** Lozano-Peña, G.; Sáez-Delgado, F.; López-Angulo, Y.; Mella-Norambuena, J. Teachers' Social–Emotional Competence: History, Concept, Models, Instruments, and Recommendations for Educational Quality. *Sustainability* **2021**, *13*, 12142. https://doi.org/ 10.3390/su132112142

Academic Editor: Pedro Vega-Marcote

Received: 23 September 2021 Accepted: 26 October 2021 Published: 3 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

general, have low emotional autonomy (M = 3.63; SD = 0.771); therefore, it is important to enhance teachers' personal skills to avoid personal and professional burnout. Another study conducted on 224 elementary school teachers in the United States [10] showed the effects of a program designed to improve teachers' stress by increasing awareness and resilience. The results showed significant decreases in psychological distress, pain-related reductions, physical discomfort, and a significant increase in emotion regulation and some dimensions of consciousness. The authors concluded that teachers who participate in SECrelated programs achieve a positive impact on their own wellbeing. Therefore, teachers' SEC has become relevant to their mental health, as confirmed by a systematic review on social and emotional learning interventions in teachers, which showed positive effects on wellbeing and psychological distress [11].

Teachers' SEC is considered to be a protective factor against stressful situations, in addition to promoting their wellbeing and their sense of self-efficacy in the classroom [12]. They are also relevant since teachers are the ones who execute social–emotional learning programs for students [6]. The literature shows vast evidence to consider SEC as a determining factor for improving educational quality [13–15], as they improve the teacher– student relationship and the classroom climate [3,16]. Teachers with high SEC establish positive relationships, provide support, and model these SEC for their students [17,18]. As a consequence of these positive effects, SEC indirectly improves students' academic performance [19,20].

Although the relevance of SEC of teachers and its contribution to educational quality is recognized, the understanding and delimitation of the SEC concept is an issue under discussion by researchers [21–23]. The multidimensional conformation of the construct that incorporates social, emotional, and other competency-based dimensions has implied an ambiguity regarding which of these three dimensions was the first to develop, to whom its development is attributed, and how they were incorporated into a single construct called SEC. In addition, there are difficulties at the inter- and intra-concept levels. At the interconcept level, SEC has been used interchangeably with other concepts, such as emotional intelligence. At the intraconcept level, inaccuracies are observed regarding the use of competencies, skills and abilities interchangeably, so that efforts are required to specify and clarify the inconsistencies in the literature [24].

Another issue discussed by researchers is related to the models on which the empirical studies related to teachers' SECs are based. These range from the use of theoretical models that consider emotional intelligence as a central concept, to others that incorporate more skills and specify SEC as a central concept [25,26]. This leads to a theoretical confusion in conducting research, due to the use of many variables and instruments when measuring SEC in teachers. With the background presented, having clear guidelines that conceive and conceptualize SEC univocally would allow the creation of instruments with a comprehensive and complex theoretical base that reflects the multidimensionality of SEC, avoiding the creation of subdefinitions in the field with instruments that partially measure SEC.

In the literature there are systematic reviews related to SEC, however, these are focused on students [27–29]; others are related to the field of health [30] and others have focused on constructs related to SEC, such as theoretical reviews on teacher emotional intelligence [31,32]. In definitive, no theoretical or systematic review focused on teachers' SEC has been identified. Considering that the knowledge gaps exposed refer to the theoretical and conceptual imprecision noted by the authors in the research on SEC, a comprehensive review of the literature is required to systematize the knowledge that has been developed on teachers' SEC. Therefore, the purpose of this study was to describe the theoretical background and support the relevance of SEC in educational quality. The study relied on the systematic review approach to answer the following research questions:


This research will contribute to the knowledge by clarifying the theoretical–conceptual and empirical–methodological background regarding the teaching of SEC for future research, and provide new proposals to solve the challenges faced by teachers.

#### **2. Methods**

To answer the research questions of this study, a method that considered two stages was implemented. The first stage consisted of a systematic literature review to identify SEC models and instruments [33] and the second stage considered a theoretical review that included other investigations to account for the historical, conceptual, and theoretical path of the SEC construct [34], and to provide recommendations for the development of teachers' SEC as a way to improve educational quality.

#### *2.1. Systematic Literature Review*

This study was based on the guidelines, standards, and phases used by the available protocol for developing systematic literature reviews of PRISMA [33]. The studies that were identified have been analyzed based on the content-focused evidence informationmanagement technique [35].

#### 2.1.1. Database and Concepts Used to Form the Search Algorithm

To identify the studies on teachers' SEC, an exploration of articles in the Web of Science (WOS), SCOPUS, and ERIC databases was conducted. These three databases have been selected since SCOPUS is the main database for peer-reviewed journals [36], the ERIC database is the main database for exclusive education studies [37], and the WOS database follows rigorous, internationally recognized, research quality standards. The search period was from 2010 to 2021. This time period was decided after an exploration of productivity on SES in the aforementioned databases, which showed an increase in productivity starting in 2010. The latest search and update date was 12 May 2021. The search considered articles where the concepts were incorporated in the title, abstract, and/or keywords of the studies. The concepts used were the different possibilities of the phrases "teacher" and "social–emotional competence." Thus, the search algorithm was formed as follows: ("teacher's social–emotional competence" OR "teachers' social–emotional competence" OR "emotional competencies" OR "emotional training" AND "teacher"). Once the search was conducted in each database, duplicates were eliminated; that is, studies that were in more than one database.

#### 2.1.2. Study Inclusion and Exclusion Criteria

In order to arrive at the most relevant sample of studies to be included in this review, five criteria were defined: (1) in English, Spanish or Portuguese, (2) empirical research, (3) in a school context, (4) focused on teachers (5) complete and accessible manuscript. This research, based on the systematic review method, only included empirical studies to answer question 2, which consists of identifying the models and measurement instruments for teachers' SEC applied over the last 10 years in the research.

#### 2.1.3. Search Process Results

The search process helped identify a total of 95 types of research in the consulted databases (Figure 1). Then, 35 study records were deleted because they were in duplicate or automatically selected by the automation software. As part of the first screening process, of the 60 studies that passed to this phase, two of the authors reviewed the titles and abstracts to ensure that the studies had teachers' SEC as a central theme and met the established inclusion criteria. Disagreements that appeared between the two authors regarding the selection or not of a study, after analyzing its title and abstract, were solved with a discussion involving all the authors. This first screening process resulted in the selection of 44 articles. The second screening process, which consisted of the complete reading of the texts of each study, applied the 4 inclusion and exclusion criteria that were

defined a priori. A total of 29 studies were excluded, and consequently, 15 studies were selected (see Figure 1).

**Figure 1.** PRISMA flow diagram of the search and selection process for articles about teacher's SEC [33], \* records excluded, without teachers' SECs concept in the title, keywords or abstract.

#### 2.1.4. Study Content Analysis Process

The content analysis process of the studies followed two stages. Based on the 15 empirical studies included in this review on teachers' SEC, the first stage of content analysis consisted of reading each text of the studies carefully and meticulously to identify the theoretical models that the authors used to support their research. A matrix was constructed for extraction and systematization of the information (See Appendix A, Table A1). To validate the information extracted from the 15 studies, two of the four researchers extracted the content independently. Only when there were discrepancies, a third researcher extracted the content independently. The final information was agreed upon with all the authors. The second stage of content analysis was to go to the original source of each of the teachers' SEC models identified in empirical studies in the last decade (2001–2021), to describe them, and answer the research questions of the present review.

#### *2.2. Theoretical Review*

The second part of the method was based on a theoretical review with emphasis on the narration of the conceptual and chronological aspects [34], which sought to complete the findings of the systematic literature review to answer research questions 1 and 3 of this study. It consisted of a selection of theoretical articles and key-book chapters that allowed us to establish and describe a historical view of the origin of the SEC construct. To present the conceptual evolution of the construct, empirical studies that have presented a definition of the construct were taken as a reference and allowed us to present its evolution from 1997 to 2020. Likewise, a search process for the SEC concept has been conducted in the SCOPUS, WOS, and ERIC databases without a year limit to analyze the scientific production available for each year until 2021.

#### **3. Results**

Next, the results of this study are presented to answer the established research questions regarding the historical and conceptual path of the SEC construct, its theoretical models and instruments, and some recommendations for developing teachers' SEC as a way to contribute to the quality of education.

#### *3.1. Historical and Conceptual Path of the SEC Construct*

#### 3.1.1. Historical SEC Path

The result of the SEC concept path is presented from two perspectives. The first represents the path from the review of the specific literature of the field and the second from the frequency of productivity of scientific articles evidenced in the WOS, SCOPUS, and ERIC databases.

Regarding the first perspective to account for the historical path, although it is not possible to determine a single temporality of the development of the components that make up the SEC, an emphasis on different periods can be identified over time and four periods can be identified: (1) The beginning of the concept centered on the social component, (2) the beginning of the concept centered on the emotional component, (3) the beginning of the concept with integration of the social and emotional components, and (4) the beginning of the concept that understands the social and emotional component as competence.

In 1920, Thorndike was credited with the concept of social intelligence, and the relevant instrument was developed by Moss and Hunt; however, the measurement results of this instrument were unsatisfactory [38]. These results, together with the rise of behaviorism, diminished research related to social intelligence for a time, as can be seen in Figure 2 [39]. However, the intelligence model of the intellect structure proposed by Guilford and Bandura's theory of social learning appears to establish that there is a combination of social and psychological factors that influence behavior [40].

**Figure 2.** A timeline in the understanding of the evolution of the socioemotional competence concept. Source: [17,21,38,40–42].

Later in 1983, Gardner proposed the theory of multiple intelligences by establishing intrapersonal and interpersonal intelligence [39], which were the basis for defining emotional intelligence proposed by Mayer and Salovey, and were understood as the ability to observe one's feelings and emotions as well as those of others, to distinguish between these

emotions and to use this information to direct action and thought [43], a concept made popular by Goleman [39].

The confluence of the social and emotional components is appreciated for the first time in the social–emotional learning construct developed by nine Collaborative Association of Social Emotional Learning (CASEL) collaborators [41], and is defined as the process through which people acquire and effectively apply the knowledge, attitudes, and skills that are required to understand and manage emotions, establish and achieve positive goals, show empathy for others, establish and maintain positive relationships, and make responsible decisions [7].

Finally, in the literature review, the first definition of SEC which includes competence, was the one established by Elias [41], defined as the ability to understand, manage, and express social and emotional aspects of people for success in the development of tasks, learning, relationships with others, problem solving, and adjustment to the demands of the context.

Regarding the second perspective which accounts for the historical path, Figure 3 presents all the publications per year registered in SCOPUS and WOS with the search for the SEC concept. The purpose of this exploration was to identify the first time the SEC construct appears in the databases; therefore, the search was not limited to a specific time range. In addition, this search allows us to identify the frequency of studies on this topic by year.

**Figure 3.** Number of publications indexed in SCOPUS and WOS databases of SEC concept.

In the ERIC database, productivity is presented by time range; for this reason, it was not included in the graph. In addition, it shows productivity from 2002 onwards since from this date it is part of the new Institute of Sciences of Education [44]. Given these characteristics of the ERIC database, 47 studies were observed ranging from 2002 to 2011, 79 studies between 2012 and 2016, 72 studies ranging from 2017 to 2019, and 40 ranging from 2020 to 2021. The first study published on this basis in 2002 was entitled Showing and telling about emotions: Interrelations between facets of emotional competence and associations with classroom adjustment in Head Start preschoolers, developed by Miller [45].

Regarding the SCOPUS database, the first registered publication was entitled Students' Perceptions of Female Professors, research carried out by Mackie [46], although the SEC concept appears in the research summary, this is not the central theme of the study. After a period of 20 years, a second publication appears, Behavioral Assessment of Coping Strategies in Young Children At-risk, Developmentally Delayed and Typically Developing, developed by Stoiber and Anderson [47]. Like the first publication, it does not develop the SEC concept as the central theme of the study. Between 1996 and 2008, a reduced number of publications, with nine studies, was observed. As of 2010 and until 2018, a constant number of publications is maintained, ranging from three to five records per year. However, in the

last two years 2019-2020, there has been a significant increase in publications related to SEC, with 12 and 13 studies, respectively. Regarding the WOS database, the first publication is developed by Anderson in 1992 [48] with the title Effects of day care on cognitive and socioemotional competence of 13-year-old Swedish schoolchildren, incorporating SEC as one of the central themes of the article. Between 1999 and 2007, only two publications were registered. Between 2008 and 2013, 13 publications were observed. Finally, a greater frequency is observed in scientific productivity on the subject of SEC in recent years from 2015, reaching 30 publications. It can be seen that ultimately, the frequency of productivity per year increases significantly as of 2010.

#### 3.1.2. Definition Path of the SEC Concept

In the available literature, there are various SEC conceptualizations that have been conceived in their first approaches to the concept as the skills, motivations, knowledge, or abilities that a person has to face and master in social and emotional situations with a certain level of efficiency and quality [21,41]; until arriving at more recent conceptualizations such as the effective management of intrapersonal and interpersonal social and emotional experiences, and promoting prosperity and wellbeing, as well as that of others [49]. Considering the diversity of conceptualizations available in the literature, some definitions have been evaluated that allow us to establish an evolution of the concept over the years, as shown in Table 1.


**Table 1.** Evolution of SEC definition.

Source: [10,49–57].

Based on the above-described conceptualizations, it is possible to identify various constructs used to refer to SEC, capabilities, skills, knowledge, attitudes, experiences, abilities, etc. The skills construct has been quite controversial, as it is a polysemic concept since its origin and development can be attributed to multiple disciplines and contexts [58], so there is no single concept. However, there is a consensus in considering ability as having the potential to learn (cognitive, affective, psychomotor); skill refers to knowing how to perform an action, and competence refers to taking actions (performing) with excellence. Capacity, as shown in Figure 4, refers to the basic resources that a person possesses, which have a biological basis [59]. Ability refers to having the potential to learn cognitive, affective, psychomotor skills, etc. Abilities are skills and behavior, that is, they are developed capacities; the skills go one step further, that is, they are the skills used flexibly, correctly, and appropriately in various contexts, that is, to perform actions or perform with excellence [60].

**Figure 4.** Disambiguation of competence concept [60].

A relevant conceptualization suggests that the concept of competence is the combination of the cognitive, motivational, moral, and social skills that a person or a social group possess or can learn, that underlie successful mastery through the understanding and appropriate actions of a variety of demands, tasks, problems, and goals [61]. They are the basic interpersonal, strategic, and execution skills [62]. According to the conceptual analysis and distinction of the concepts of capacity, ability, and competence and assessing the definition proposed by Collie [49], SEC could be defined as the effective deployment of abilities that allows subjects to cope with social and emotional intrapersonal and interpersonal experiences assertively.

#### *3.2. Theoretical SEC Models and Instruments*

#### 3.2.1. Theoretical SEC Models

Five theoretical models that met the inclusion criteria have been selected to be analyzed in the present study (See Appendix A, Table A2): Gross' model of the emotional regulation process in 1998 [63]; Mayer and Salovey's emotional intelligence model in 1997 [25]; Bar-On's emotional intelligence model in 1997 [64]; Jennings and Greenberg's 2009 prosocial classroom model [17]; and the CASEL social–emotional learning model in 2013 [26]. The models are characterized below.

#### Emotion Regulation Process Model

Gross' model is based on emotional regulation, understood as the processes by which people influence the emotions they have, and how they experience and express them. These emotion regulation processes can be automatic or controlled, conscious or unconscious, and their effects can be shown at one or more points in the emotion generation process [65].

This emotion regulation process model facilitates and allows the analysis of types of emotion regulation by establishing five sets of emotional regulatory processes as shown in Figure 5: situation selection, situation modification, attention deployment, cognitive change, and response modulation. This is an elaboration of the two-way distinction be-tween antecedent-centered emotion regulation, a pre-emotion process, and responsecentered emotion regulation, a process that occurs after the emotion is generated [63,66].

**Figure 5.** A process model of emotion regulation [63].

Mayer and Salovey's Emotional Intelligence Model

The Mayer and Salovey ability model, graphically represented in Figure 6, considers emotional intelligence as a concept and conceptualizes it through four basic skills, which are: perceiving and expressing emotions, accessing and/or generating feelings that facilitate thinking; understanding emotions and emotional awareness and regulating emotions promoting emotional and intellectual growth [25]. In this way, these four basic skills are what make this model a skill model, as proposed by Trujillo and Rivas [39] who also classify it as a skill model.

**Figure 6.** Schematic presentation of Mayer and Salovey's 1997 model of emotional intelligence [67].

Bar-On's Emotional Intelligence Model

According to this model, emotional intelligence is a representative set of skills and emotional and social facilitators that interrelate and determine the effectiveness with which a subject understands and expresses himself, understands and relates to others, and efficiently deals with daily demands [64].

Among the dimensions covered by the model as shown in Figure 7 is development: intrapersonal, interpersonal, stress management, adaptability, and general mood. The social components of this model are the interpersonal and adaptability dimensions, and the intrapersonal emotional components are stress management and general mood.

**Figure 7.** Bar-On model of emotional intelligence [64].

#### Prosocial Classroom Model

The Jennings and Greenberg prosocial classroom model has social and emotional skills as its central concept. This skill uses the definition developed by CASEL [26] which involves five main emotional, cognitive, and behavioral skills: self-awareness, social awareness, responsible decision making, self-management, and relationship management.

As can be seen in Figure 8, the prosocial classroom model is structured into five dimensions: social and emotional skill and teacher wellbeing, teacher–student relationships, classroom management, implementation of the social and emotional learning program, and, finally, classroom climate. This model emphasizes the importance of these five dimensions in creating a climate favorable to learning in the classroom and in promoting positive results in student development [17].

**Figure 8.** The prosocial classroom: A model of teachers' social and emotional competence and classroom and student outcomes [17].

The model proposed by Jennings and Greenberg raises social and emotional components. A predominant social component is observed since we find it in its five dimensions. To a lesser extent, it presents an emotional component in two of its dimensions, the teacher's social and emotional competence and wellbeing, and the implementation of the social and emotional learning program respond to the emotional component.

#### CASEL Model

For its part, the CASEL model establishes as a basis the concept of social and emotional learning, defining it as learning that involves processes through which children and adults acquire and develop knowledge, skills, and attitudes that are needed to understand and manage emotions, as well as to achieve positive results, meet goals, demonstrate empathy, maintain positive relationships, and make responsible decisions [26].

Figure 9 represents the first image of the CASEL model, which, despite having been updated, maintains the same skills. This figure raises five interrelated sets of cognitive, affective, and behavioral skills: self-awareness, self-management, responsible decision making, relationship skills, and social awareness. The framework emphasizes the importance of establishing equitable learning environments and coordinating practices in classrooms, schools, families, and school communities to enhance students' social, emotional, and academic learning [67].

**Figure 9.** The five social and emotional learning core competencies [26].

The CASEL model presents social components such as self-management, relationship skills, and social awareness. Likewise, it establishes emotional components such as self-awareness and responsible decision making.

Ultimately, Table 2 synthesizes and represents an extension of the characteristics of the theoretical models on which the research is based on teachers' SEC. The models have been classified in relation to the central concepts and skills that comprise them. Three types of models have been established: emotional intelligence models, among which are the Mayer and Salovey's and Bar-On's models; emotional regulation models, including the Gross' model; and the models of social and emotional development among were identified Jennings and Greenberg's prosocial classroom and CASEL's social–emotional learning models. Although each model presents a different theoretical proposal, all these models consider between four to five integrative skills related to the social and/or emotional field. Regarding the number of citations, the Mayer and Salovey model of emotional intelligence is the most cited in the literature.

#### 3.2.2. SEC Measurement Instruments

Of the five models analyzed in this review, four of them have an instrument consistent with their integrative skills: (1) Gross' model with its emotion regulation questionnaire; (2) the Bar-On's model with its emotional quotient inventory; (3) the Mayer and Salovey's model with its Trait Meta-Mood Scale self-report measure (MSCEIT) performance measure; (4) and the CASEL model with Yoder's Social and Emotional Skills Questionnaire [68,69]. On the other hand, Jennings and Greenberg's model does not have a coherent instrument for assessing its integrated skills. However, the investigations that are based on this model, to empirically measure their integrative skills, use other instruments, thus using more than one, as shown in Table 3.

According to the population of users of the models for conducting research, most of them have been used in adults, children, and adolescents. Meanwhile, Jennings and Greenberg's model has been used mainly in adults.


**Table 2.** Synthesis table of theoretical models on SEC.



**Table 3.** *Cont.*

#### *3.3. Recommendations for Developing Teachers' SEC as a Way to Contribute to Educational Quality*

Understanding SEC from theoretical, conceptual, and empirical aspects contributes to making better decisions in the research and educational field. Therefore, below, some recommendations are proposed for the development of SEC as a contribution to educational quality in (a) the evaluation of SEC at the school and public policy level, (b) teacher training in SEC; and (c) the leadership of educational institutions.

#### 3.3.1. Recommendations for Assessing SEC in Teachers and Their Students

First, it should be noted that SEC can be taught, learned, assessed, and trained [56,70,71]. In this sense, less progress has been made in developing methods to assess social and emotional skills in school [71]. Therefore, it is necessary to establish the SEC assessment in both students and teachers. In relation to the teacher, the use of instruments that allow us to assess their SEC in the classroom should be improved, as well as this, teachers should be trained in the construction of social–emotional learning assessments in order to use them constructively in the assessment of their students [72]. Regarding the student, progress must be made with evaluations specifically designed to measure their knowledge, ability, and social and emotional disposition during interpersonal interactions and their participation in school and community life [73].

Assessing the SEC in teachers and students can also be developed at a standardized level, which would imply consideration of its measurement at the public policy level as an index of educational quality, for example, the incorporation of tests (in students) or certifications (in teachers) at the country level.

#### 3.3.2. Recommendations for Teacher Training in SEC

SECs are malleable compared to IQ; therefore, they can be trained through interventions, including in adulthood [6]. In this sense, if one wants to advance towards a trans-versal development of SEC in education, the incorporation of an explicit approach in teacher training is unavoidable [18]. In this sense, there seems to be a deep disconnect be-tween the skills that teachers require to develop social and emotional learning in schools and what the teacher training universities offer them [74]. This challenge should be raised on three levels: First, in teacher training programs, subjects that cover their entire period of study training, and that allow them to develop and train their SEC, should be incorporated.

Second, there should be SEC development, follow-up, and training of in-service teachers, who probably lack these SECs, and who have probably had to use personal resources to develop social and emotional learning programs with their students. A third level is related to the need for joint work between teacher training entities and school communities to develop and assess SEC in teachers in training and in service.

#### 3.3.3. Recommendations to Strengthen the Leadership of Educational Institutions

Educational leaders have a direct impact on educational quality. The existence of a strong relationship between directive leadership in a school and the achievement of student learning results has been evidenced, in fact, the leader of a school is considered to be the second factor, after the teacher, with the greatest influence on the student academic achievement [75,76]. Given the relevance and impact that a school leader generates, SEC learning or training should be incorporated into the prior and ongoing professional learning of educational leaders [71].

Recently, leadership for social justice has gained notoriety, which aims to address the complexity of highly vulnerable schools [77]. In these contexts, it is essential to exercise leadership that recognizes that SEC, in addition to being a predictor of behaviors or positive results at the school level, acts preventively, as protection against risk factors that can harm students [78], such as problems of violence, delinquency, substance use, and dropout rate [79–81]. Leadership that recognizes that the SEC and wellbeing of the teacher influences the learning context, the implementation of social and emotional learning programs, and the relationships that exist between teachers and students.

#### **4. Discussion**

This review aimed at describing the theoretical background and supporting the relevance of SEC in educational quality by; (1) elucidating historical, conceptual, and theoretical aspects of the SEC construct; (2) identifying models and instruments for measuring SEC; and (3) proposing guidelines for developing SEC as a way to contribute to educational quality. Next, the results are discussed in relation to objectives, educational implications, and limitations, and projections for future research are proposed.

Concerning the historical path of the SEC concept, this study showed an increasingly complex and structured course. Since it is inherent to human intelligence, it becomes more complex with contextual and social development, and poses challenges to decision making and the search for problem solving. Thus, the discussion and reflection of the research regarding intelligence other than the cognitive aspect of the IQ of the human being began with the social intelligence of Thorndike (1920), followed by several other relevant theoretical contributions that contributed to the configuration of the theory of emotional intelligence. This finding is consistent with that indicated by Joseph and Newman [82] who point out that emotional intelligence is the embodiment of the concept of social intelligence.

The study also recognizes that the emotional intelligence theory has been crucial in understanding the SEC construct. This result is consistent with that established by Mikulic and other researchers [83], who warn that advances in emotional intelligence have con-tributed to the delimitation of the SEC construct.

From the results of this study, with regard to the conceptualizations, it can be pointed out that there is a great diversity of SEC definitions. However, it is also possible to observe that these definitions share common elements, favoring the idea that SEC can be observed from a three-dimensional perspective, and configured by a social, emotional, and competence component. Based on this background and framing this concept in the field of school-level teachers, teachers' SECs have been defined as the effective deployment of skills that allows teachers to function in social and emotional, intrapersonal and interpersonal experiences assertively in the educational context.

The results of this study also evidenced the existence of five theoretical models that empirical research uses as a theoretical basis in studies that address SEC with a sample of school-level teachers. Specifically, two of these, corresponding to the Bar-On's and Mayer and Salovey's models, are directly supported by the theory of emotional intelligence, since this is their central concept [25], therefore, they were classified as models of emotional intelligence. In relation to Jennings and Greenberg's prosocial classroom, and that of CASEL social–emotional learning, are more inclusive, since they consider learning and/or social and emotional skills to be central concepts and include social and emotional skills, which are called integrative skills, therefore, these two were classified as social and emotional development models. Finally, Gross' emotional regulation Model was focused on the emotional dimension of SEC.

The emotional intelligence models developed by Bar-On and Mayer and Salovey seem to have been designed from their central concept and skills, as models applicable to any type of context, work, educational or organizational setting, etc., while the models of Jennings and Greenberg and CASEL are models specifically applicable to the educational and/or school environment. In relation to the skills or integral variables of the models, these are quite broad and vary according to authors and models, and this finding is consistent with other studies [23,84]. Although some of these models have been built to contribute specifically to the educational field, they focus on how to generate SEC in students but neglect the development of the application of these models in teachers.

Regarding the measurement instruments, although the four models present coherent and consistent instruments between the skills or variables, it can be observed that the only instrument that explicitly measures SEC is the Yoder social and emotional competence questionnaire, designed and used in the research for the CASEL framework. This instrument allows for the comprehensive measurement of both social and emotional skills, in addition to the purpose that it establishes in its description. Another relevant point is that most of the instruments are able to self-report, which could have the risk of spurious correlations as a consequence of the common bias of the method, with the participants reporting on their social and emotional skills or other results [85]. An exception is an instrument established for Mayer and Salovey's model, which in addition to its self-report measure, the Trait Meta-Mood Scale, presents a performance measure with the MSCEIT. Researchers have warned of the need to continue improving the current ways of measuring teachers' SEC, moving towards the development of instruments to empirically investigate what kind of knowledge and skills teachers should acquire, and the construction of instruments that overcome the barrier of self-reports [86,87].

The limitations of this study consider aspects related to the method. The literature re-view for the first stage of this study considered exploration in only three databases. Another limitation is that only studies published in English, Spanish, and Portuguese have been selected, eliminating those in other languages. Another limitation of the study is related to the sample size of the systematic literature review, with 15 articles resulting from the search in the last 10 years, even though before this period there were some studies that were left out of the analysis.

One of the main strengths of this theoretical study is that it has used a method that complements two techniques, one of systematic literature review and the other of critical review. This has made it possible to account for a historical journey of the SEC concept, identify and analyze the models most used in empirical investigations of teachers, expose the instruments that measure SEC, and specify recommendations for developing SEC as a contribution to educational quality.

Future studies can contribute to the proposal of a solid theoretical model, based on the analysis for developing teachers' SEC. It is also important to consider the development of empirical studies that describe teachers' profiles regarding the SEC levels. This would help identify which variables are related to a high level of SEC at the level of teachers and students. Multilevel interventions should also be developed that evaluate the effectiveness of the training for the improvement of SEC, not only in teachers who are exposed to these trainings, but also with measurement in the student body, to show evidence of how a teacher with high levels of SEC can influence the promotion of these skills in their students and in variables such as the classroom climate, academic performance, and dropout rate. In our current society, social and educational change processes are constantly transforming, conditioning and stressing the teaching profession. Undoubtedly, the COVID-19 pandemic has left an enormous challenge in the educational field, increasing the need to develop SEC in teachers to improve their emotions, relationship with others and to support students emotionally. In this context, future research should consider how to train, coach or improve SEC in both, in-service and preservice teachers.

Including SEC in the educational field has mainly focused on the implementation of social and emotional learning programs aimed at students [5,88]. In this way, it is necessary to advance and incorporate social and emotional learning in the different educational public policies that include school leadership, assessment, and professional teacher training. [5]. This will allow contribution to a new area that supports the improvement of educational quality in schools.

In conclusion, this study contributes to academia and education. In the academic aspect, this study presents to researchers the historical path of the SEC concept and its configuration from different theoretical approaches showing its evolution. It opens the field for the proposal of new theoretical models focused on how to develop the three-dimensionality of the SEC concept, framing it specifically to the research field of teachers. Regarding the contribution to education, this study goes beyond socioemotional learning centered on students, and makes visible the importance of SEC in teacher training. Furthermore, this study makes it possible to rethink education as something that considers teachers as professionals who need to be trained academically and socioemotionally in order to achieve better education and a better society.

**Author Contributions:** Conceptualization, G.L.-P. and F.S.-D.; methodology, F.S.-D. and J.M.-N.; formal analysis, G.L.-P. and F.S.-D.; writing—original draft preparation, G.L.-P., F.S.-D., Y.L.-A. and J.M.-N.; writing—review and editing, J.M.-N.; supervision, F.S.-D.; project administration and funding acquisition, F.S.-D. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work received funding from the National Research and Development Agency of the Chilean Government [ANID, Proyecto FONDECYT Iniciación 11201054] and Scholarship Program/Doctorado Nacional Chile/2020-21202422.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Further inquiries can be directed to the corresponding author/s.

**Conflicts of Interest:** The authors declare no conflict of interest.


1 2


**Table A1.** Extraction Information Matrix (SLR).

4 demands and pressures.

relationship

3


**Table A1.** *Cont.* experience


**Table A1.** *Cont.*


**Table A1.** *Cont.*


**Table A2.**

Extraction information

 matrix: theorical model of SEC.


**Table A2.** *Cont.*

#### **References**


### *Review* **Higher Education Students' Online Instruction Perceptions: A Quality Virtual Learning Environment**

**Kim Hua Tan \*, Poh Phui Chan and Nur-Ehsan Mohd Said**

Centre of Teaching and Learning Innovation, Faculty of Education, Universiti Kebangsaan Malaysia, Bangi 43600, Selangor, Malaysia; p105784@ukm.edu.my (P.P.C.); nurehsan@ukm.edu.my (N.-E.M.S.) **\*** Correspondence: kimmy@ukm.edu.my

**Abstract:** Online instruction has been one of the key delivery methods in the midst of the COVID-19 pandemic due to school closures around the globe. In accordance with the Malaysia Education Blueprint (2013–2025), maximizing the use of information/communication technology has been emphasized to scale up learning quality across Malaysia, including distance and self-paced learning. However, online learning in the country is at its infancy stage with raised issues, causing droppingout and school leaving in higher education. To improve teaching and learning quality, this scoping review aimed to explore higher education students' online instruction perceptions into two main components: research on online instruction perceptions followed by factors influencing online instruction perceptions. Using Arksey and O'Malley (2005)'s methodological framework, 61 articles related to students' online instruction perceptions were identified from Google Scholar, ERIC, and Research Gate databases. In terms of theoretical articles, the results showed that cognitivism, connectivism, and constructivism were the most used theories of online instruction. On the basis of the empirical articles gathered, quantitative research design was the most utilized to collect students' perspectives toward online instruction. As a whole, the findings revealed that motivation and satisfaction were mostly positively perceived by students, whereas, a lack of interaction was highly categorized as an unfavorable online instruction perception. Three main factors were identified: quality instruction, online interaction, and instructional and technical support. Future studies can focus on investigating teachers' online instruction perceptions to achieve quality in higher education.

**Keywords:** online instruction; higher education; students' perceptions; education quality; online learning

#### **1. Introduction**

A controversy over online instruction perceptions has left people in unquestionable doubt. Due to the global pandemic, school closures have been considered an initiative measure in preventing the spread of viruses. Many educational institutions around the world have temporarily closed to cease the spread of the COVID-19 pandemic [1]. Many significant changes have been called upon for adoption, particularly education. With this sudden shift, online learning has been increasingly rising to continue the teaching and learning process. However, the pandemic has caused negative impacts on online education, such as learning loss [2] and exacerbated learning outcomes [3]. Online barriers can be hindrances, including cost [4], teachers' information and communication technology (ICT) skills and their demographic factors [5], poor infrastructure of the university [6,7], lack of online resources [8], classroom management in terms of student participation [9], and teachers' behavioral intentions in adopting online educational technology [10]. To ensure the effectiveness and quality of higher education in online learning, synthesizing existing literature on the significance of online instruction perceptions is critical.

Based on The United Nations' (UN) 17 Sustainable Development Goals (SDGs), one of the goals is to achieve quality education. Quality education not only aims to empower educational opportunities but also to alleviate poverty all around the world by 2030 [11]. According to The Star (2021), urgent action should be called upon in minimizing the digital

**Citation:** Tan, K.H.; Chan, P.P.; Mohd Said, N.-E. Higher Education Students' Online Instruction Perceptions: A Quality Virtual Learning Environment. *Sustainability* **2021**, *13*, 10840. https://doi.org/ 10.3390/su131910840

Academic Editors: Ana B. Bernardo, Adrian Castro-Lopez, Javier Puente and Leandro Almeida

Received: 20 August 2021 Accepted: 26 September 2021 Published: 29 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

divide in Peninsular Malaysia [12]. A number of issues pertaining to online accessibility and basic requirements are found to be addressed promptly, such as Internet connectivity, lack of resources and devices, in order to provide equity and quality in the education field. In line with the Malaysian higher education institution students' learning experiences, immediate action should be taken by universities to avoid losing their students, by considering their dissatisfactions; for instance, virtual learning mode implementation, availability of instructors, and learning performances [13]. Ensuring healthy well-being is imperative to sustainable development. During the pandemic, a study shows that most participants feel apprehensive and helpless [14]. Similarly, another study reveals that excess electronic access can impact tremendously on students' mental health and education in a long term [15]. Hence, quality instruction is essential to reduce stress level. By providing quality education, considering students' perspectives is imperative for integrating proper online instruction.

The quality of online instruction has been given some considerable attention since the rise of online courses offered in education. [16]. Quality online instruction should adhere to the seven principles of instructional practice for an effective teaching and learning process [17–19]. According to Chickering and Gamzon (1989), these seven principles pertain to good instructional practice in undergraduate education: (1) encourage student– faculty contact, (2) encourage cooperation among students, (3) encourage active learning, (4) give prompt feedback, (5) emphasize time on task, (6) communicate high expectations, and (7) respect diverse talents and ways of learning [20]. For student–faculty contact, it is crucial to establish a good rapport, as it is considered a source of student's motivation in determining their engagement and involvement. It can foster students to think about their values and future plans. In developing reciprocity and cooperation, frequent working collaboratively can increase involvement in learning. Sharing ideas and responding to reactions can stimulate thinking skills. Moreover, students are active learners who relate their learning experience and apply it in daily lives through active learning, such as discussions and projects. By providing adequate feedback on performance, it can enable students to assess and improve themselves. They require constructive and immediate suggestions to reflect on what they have learned. Next, effective time management is critical to high performance. Students are required to learn how to manage their time well for effective learning. High expectations stimulate students to perform more and motivate them to be well-prepared. Last but not least, each and every student has their own learning style, therefore, chances should be given to students to show their talents and learn in their way. Hence, this scoping review was guided by two research questions:

RQ1: What are students' online instruction perceptions?

RQ2: What are the factors influencing students' online instruction perceptions?

#### **2. Literature Review**

#### *2.1. Defining Online Instruction*

In general, online instruction is defined as an online learning delivery method between instructors and learners through facilitation and guidance with the use of technology. As shown in Table 1, online instruction can be implemented through asynchronous or synchronous online instruction.


**Table 1.** Types of online instruction.


**Table 1.** *Cont.*

#### *2.2. Significance of Online Instruction*

Students' online instruction perceptions play a critical role in determining learning attitudes, learning outcomes, and personal development. Over the years, many researchers have been endorsing the critical role of online instruction, including quality interaction [41], deep learning experience [42], positive social change [43], creative thinking [44], learning outcomes [45], and discovery learning [46]. Prominently, online instruction is found to impact learners' satisfaction and motivation through good online engagement [38,41,47–49] and constructive feedback [50–53]. In addition, numerous studies have revealed that online instruction is as effective as face-to-face instruction [22,54,55], including learning outcomes. Quality online instruction can promote a good virtual learning environment, resulting in the improvement of learning outcomes. With regard to personal development, students' learning needs and desires can be fulfilled through online instruction. In accordance with self-determination theory (SDT), research findings have reported that online instruction fulfills students' intrinsic needs, thus resulting in high satisfaction levels [56]. Other than that, positive impacts of online instruction on cultural awareness in terms of cultural knowledge and attitudes can be beneficial to some multicultural countries [57]. Instructors' roles have been strongly emphasized to facilitate online instruction, such as connectedness [58] and establishing a good rapport [59]. All in all, quality instruction is significant in creating a conducive virtual learning environment, particularly in higher education.

#### **3. Materials and Methods**

This research utilized a scoping review methodology [60] to examine online instruction and students' perceptions. Regarding the methodology framework, five stages were discussed: (1) identifying research questions, (2) identifying relevant studies, (3) selecting studies, (4) charting data, and (5) summarizing and reporting results.

#### *3.1. Identifying Research Questions*

This study was led by two research questions: What is the scope of research conducted on students' online instruction perceptions? What are the factors influencing students' online instruction perceptions? Articles that concentrate on students' online instruction perceptions or e-learning instruction perceptions were included to be the focus of the study. The target group was tertiary students. The breath of the study was further reviewed.

#### *3.2. Identifying Relevant Studies*

With regard to searching terms, three sets of keywords were used: (1) online instruction or e-learning instruction and (2) perceptions or attitudes. These combination sets of search terms were explored using Boolean operators. That is, the AND operator and OR operator were employed within the sets to discover relevant results. Articles were selected from indexed journals via three electronic databases, namely, Google Scholar, ERIC, and Research Gate, as these databases present a large collection of international education journals.

#### *3.3. Study Selection: Inclusion and Exclusion Criteria*

To determine appropriate criteria, inclusion and exclusion criteria were critical to set the boundaries for the scoping review. During the process of reviewing, areas of inclusion and exclusion criteria were critical to suit the requirements of the scoping review, such as type of publication, year, and language [61]. As delineated in Table 2, it shows inclusion and exclusion criteria of this scoping review.


**Table 2.** Inclusion and exclusion criteria.

Seven inclusion criteria were developed to meet the requirements forreviewing purposes:


During the filtering process, nine exclusion criteria were measured to exclude articles that did not meet the following yardstick.


#### *3.4. Charting Data*

During the selection process, the chosen articles were classified on the basis of the key issues and themes of the review. As presented in Figure 1, it displays a flow diagram for the paper selection process.

**Figure 1.** Flow diagram for the paper selection process.

After screening 724 articles, 61 articles met the criteria. Specifically, articles were then sorted into two former sections known as theoretical and empirical papers. This scoping review further discussed students' online instruction perceptions in terms of positive and negative aspects, followed by factors influencing online instruction perceptions with implications.

#### **4. Results**

The report presents two types of studies, namely, theoretical (*n* = 10) and empirical studies (*n* = 51). It also examines the favorable and unfavorable online instruction perceptions throughout the study, followed by factors influencing online instruction perceptions.

#### *4.1. Theoretical Studies*

As displayed in Table 3, the theories related to online instruction are as follows:

**Table 3.** Learning theories and models related to online instruction.



**Table 3.** *Cont.*

Based on Table 3, ten retrieved articles discussed learning theories and models as fundamental frameworks for online instruction. Cognitivism and constructivism shed some light on the importance of internal processes, such as cognitive development and prior knowledge in online learning. In cognitivism, three types of cognitive loads were identified: intrinsic, extraneous, and germane loads. Online instructional design should reduce cognitive load for online learners to keep instruction clear and concise. On this matter, strong stimuli should be employed in the discovery phase for online learners to make a connection to their prior knowledge [62]. Constructivism implies that collaborative learning is imperative to engage real-life experience learning and enhance metacognitive skills [29,31]. In opposition to cognitivism, behaviorists revolve around overt behavior to be observed and measured through learning outcomes. Notably, the implication of behaviorism is to provide feedback by monitoring students' achievement levels and take corrective action if possible.

Connectivism, heutagogy, and transformative learning theories highlight the benefits of autonomy. Autonomy allows students to be independent and autonomous to explore knowledge. Regarding connectivism, students should optimize the use of technology to substantially expose themselves to information technology and always keep up-to-date for authentic learning [68]. As learners are active participants, heutagogy theory outlines that instructors' roles should be passive as monitors and course designers to promote self-paced learning among learners [62]. Transformative learning theory suggests fostering interactive learning in online learning.

Regarding activity theory, Walker (2020) illustrated underlining online communication and online discussion to facilitate understanding [63]. Gestalt theory expresses prominence of the laws of perception in shaping one's perception in an environment. According to Leftlore (2000), web-based instructional pages should be based on visual design as online learning extremely depends on visual perception [66]. Adequate guidance should be provided to encourage students engaging in online activities. Gray (2019) proposed that literary theory connects online courses as texts to form meaningful interaction and enhance online interaction [35]. Meanwhile, learners can interact with course content at the meaning-making level, increasing one's production and participation.

Case-based methods in instructional design are imperative for self-directed online instruction [67]. Clemons (2005) identified four rudimentary implications of brain-based learning theory, namely memory and retrieval, learning styles, role of emotion, and increasing attentiveness [65]. For instance, students should be given some time to process information after every 10 min of information sharing. A community-based learning

project focuses on quality online design, learner-centered and project-based learning in the alignment to online instruction [50].

As a whole, motivation has been remarkably highlighted in most theories related to online instruction, including intrinsic and extrinsic motivations. Motivation is the core of the learning process to determine one's aspiration and achievement.

#### *4.2. Empirical Studies*

A total of 51 empirical articles were reviewed in examining students' online instruction perceptions. In Table 4, they are outlined as follows:


#### **Table 4.** Students' online instruction perceptions.


**Table 4.** *Cont.*


**Table 4.** *Cont.*


**Table 4.** *Cont.*


**Table 4.** *Cont.*


**Table 4.** *Cont.*

#### 4.2.1. Favorable Online Instruction Perceptions Motivation and Satisfaction

Twelve articles endorsed the significance of satisfaction and motivation as a positive outlook of online instruction. The interactions of students with instructors, fellow students, and contents were considered as sources of high satisfaction and motivation levels. Three types of engagement strategies were recognized: (1) learner-to-learner interaction, (2) learner-to-instructor interaction, and (3) learner-to-content interaction [49]. It proved that the rise of student engagement could enhance satisfaction, motivation, and performance. At the same time, it decreased the sense of isolation in the online learning environment. Bolliger and Halupa (2018) clarified the finding that learners perceived online instruction as positive learning, satisfaction, and progression due to the good practice of online engagement [48]. Learners thought that online discussion could activate their thinking skills to be creative and analytic, thus producing meaningful work by motivation. In accordance with Buckley (2003), learners expressed that online discussion could activate their creative and analytic thinking skills, thereby producing meaningful work by motivation [47]. Satisfaction could be attributed to fulfilling learning needs. The result reported that online instruction fulfilled students' intrinsic needs in terms of SDT, thus resulting in a high satisfaction level [56]. A study showed that learning outcomes and retention showed no significant difference with traditional course [69]. The increase of learners' self-regulated learning and independent learning through online instruction resulted in considerable online learning outcomes with high self-motivation [70]. Providing detailed and useful feedback from instructors is essential to students, as they can be given constructive guidance, thus increasing their satisfaction and involvement [71]. From Wood and Keeler's (2001) finding, auto-email instructions were proven to increase group discussion and participation, resulting in a great sense of online community with high satisfaction level [37]. Online instruction promoted students' motivation, however, flexibility factor should be considered for students' satisfaction [72]. Respondents illustrated that they felt motivated with online instruction in terms of instructor feedback and peer interaction [73]. Based on Wu et al. (2020), students' writing could be improved by using online flipped writing instruction; hence, it enhanced satisfaction among the students [74]. Walker and Kelly (2007) revealed that students showed satisfaction toward work, assessment, and grading [75].

#### Effectiveness

Six examined articles held a positive perception of online instruction, as effective and immediate. These studies showed that online instruction was as effective as face-to-face instruction. No significant difference was found between online and face-to-face instructions regarding learning outcomes [22,54,55]. In the study conducted by Tichavsky et al. (2015), most students held a positive perception toward online instruction in comparison to face-to-face instruction; therefore, more effective online pedagogy should be developed [76]. Moreover, immediate feedback was one of the prominent reasons contributing to the effectiveness of online instruction. Based on the online instruction strategy (SI), students thought online instruction was effective because of timely and specific feedback [52]. A vast majority of students perceived that prompt feedback, especially responses to emails and questions and grade posting, were highly rated [53].

#### Good Engagement

Six examined articles reviewed good engagement, interaction, and participation as positive online instruction perceptions. Tsai, Ku, and Campbell (2021) indicated that engagement and learning outcomes were positively perceived by students who were in highly interactive courses, particularly high student–student and student–instructor interactions [77]. The research presented that both types of interaction played critical roles in online instruction with regard to approaches to online learning engagement [78]. Anonymity was perceived as one of the crucial elements that contributed to online instruction perceptions. Being anonymous in online activities increased students' participation in the discussion [38]. Learners' engagement could be closely related to a few factors, such as learner-centered instruction, learning style, and instructional design [79]. Regarding formative assessment as an online instruction intervention, students held a positive outlook toward engagement in online learning activities [106].

#### Ease of Navigation

Three empirical articles discussed ease of navigation as one of the optimistic perspectives perceived by learners in terms of online instruction. Online instruction was found to be helpful when using e-learning web-based modules by guiding them through navigation easily [80]. An online instruction rubric, QM, was developed to gather the perceptions of students, and the finding discovered that ease of navigation was rated highly among them [81]. In addition, learners found instruction in online navigation to be helpful and effective, as it included introduction to contents, web pages with suggested links, and assignments [82].

#### Comfort and Flexibility

Three empirical article interpreted comfort as an optimistic reaction to online instruction. Receiving instructors' immediate feedback and reading other teams' posts could strengthen learners' reading comprehension, resulting in comfortable and understandable perceptions toward online instruction [51]. According to Sharp (2014), an indication of flexibility should be considered in an online learning environment to enhance student connectedness based on the CoI model [83]. Self-directed learning was perceived as a great option, particularly for female married students, due to societal norms dictating that traditional women are not allowed to participate in school programs. Moreover, some female students felt uncomfortable to take part in late night study groups with mostly male students; hence, self-directed learning could be a better alternative in an online learning environment [107].

#### Positive Online Experience

Two empirical articles stated the importance of positive online learning experience of online instruction. Ivers, Lee, and Carter-Wells (2005) suggested that a positive online learning experience should be created on the basis of students' online instruction perceptions, namely, interaction, instructional support, and prior experience with computers, to spur learning motivation and inclusiveness [84]. Most students favorably perceived that quality academic experience was enhanced through online instruction in comparison to traditional classroom [85].

#### Improvement of Learning Outcome

One empirical article explained that students substantially improved their learning outcome through instructional practice and support based on constructivism theory [86]. Blogging into writing instruction was useful in enhancing writing proficiency, fluency, and awareness of writing for audiences. [87].

#### Acceptance

One empirical article focused on the acceptance level to online instruction. Quality online instruction was favorably perceived by students, as it increased their acceptance level toward online learning [88].

#### 4.2.2. Unfavorable Online Instruction Perceptions

#### Lack of Interaction

Six empirical studies discussed the downside of the perspective toward ambiguous online instruction. Alternatively, empirical articles reflected the issue of online learning, which was lack of interaction, thus forming negative perceptions during online instruction.

As a consequence, it reduced direct involvement, thus becoming less personal. Cole, Shelley, and Swartz (2014) showed that although convenience was the reason that contributed to learners' satisfaction, lack of interaction was the main cause for dissatisfaction due to the online instruction in e-learning [89]. A study revealed that some participants expressed the importance of social presence in terms of open communication and group cohesion [90]. Borstorff and Lowe (2007) noted that little interaction and communication were concerned with instructors and other students, especially young learners who yearned for more interactions with other students than any other groups [91]. Most of the student engagement time dwindled to computer-based instructions and led to a fall in student success [92]. Students perceived online instruction negatively when instructors had very limited interactions with them, especially because it was their first time taking an Internetbased course [93]. Students perceived that the interactions with their instructors and peers had huge impacts on their satisfaction; however, student engagement was rated as the least satisfaction level due to the roles of instructors [94].

#### Ambiguous Instruction

Four empirical studies discussed the downside of the perspective toward ambiguous online instruction. None of the theoretical studies reflected the issue of ambiguous instruction, but four empirical studies explained the barrier faced during online instruction and how it influences students' perceptions. Armstrong (2011) argued that online learning became seemingly superficial without a clear instruction; as a result, it diminished the value of education and academia [95]. The agreement continued when a lack of clear expectations was regarded as one of the hurdles to online learning [3]. While online learning, students might experience complex feelings, including frustration and disappointment due to ambiguous instructions, thus affecting the relationship between instructors and students [96]. When unclear instructions were delivered, online learning became more sophisticated and challenging, especially those who require assistance from their instructors [97]. According to a study conducted by Hancock (2018), no significant differences were observed in students' perceptions of quality online instruction regarding instructor competency and teaching presence to students' intent to persist in school [98].

#### Lack of Technological Skills and Competency

Three empirical articles illustrated that technological skill competency was considered one of the online barriers that formed students' pessimistic perceptions during online learning. Being incompetent of using computers, the role of instructors in delivering effective online instruction was being questioned due to difficulty in communicating in a non-verbal manner and limited time of online learning [99]. Oh and Lim (2005) pointed out that competency in using computer technology could be a factor influencing online instruction perceptions to enjoy online learning [100]. Technology issues were frequently faced by online students, followed by clarity of communication and prompt feedback that would disrupt their online learning experience [25].

#### Lack of Support

Two empirical articles examined the drawback pertaining to types of support about online instruction perceptions. Similarly, the empirical studies identified two types of support: (1) instructional support and (2) technical support, endorsing the pessimistic perceptions toward online instruction. In terms of instructional support, a lack of pedagogical support during the online teaching process was perceived. In addition, instructional support referred to providing information and learning platforms in which it required technical support from online instruction [101]. Zwanch and Cribbs (2021) reported that students' concerns became worse when face-to-face instruction shifted to online instruction during the pandemic; therefore, support from their instructors was critical to scaffold their learning [102].

#### Academic Dishonesty

One empirical article examined the aspect of academic dishonesty as one of the factors to shape students' online instruction perceptions. Students perceived that there could be potential for online cheating throughout the learning process [103].

#### *4.3. Factors Influencing Online Instruction Perceptions*

Three main factors pertaining to online instruction perceptions were identified:

#### 4.3.1. Quality Instruction

Quality instruction must be clear, concise, and understandable to learners. Cognitivism indicates that online instruction should be brief and clean to reduce cognitive burden for online learners. The criteria of quality instruction are andragogical competency, resourcefulness, adequate preparation, effective organization and technological skills, content and currency of knowledge, and instructors' dispositional attributes [104]. Meanwhile, quality instruction is equivalent to clarity and effectiveness. One of the examples of quality instruction is to provide specific, constructive, detailed, and prompt feedback. It is aligned with behaviorism to monitor students' behavioral actions by giving suitable feedback. Based on a study in China, online formative assessment was viewed as effective to enhance student engagement and learning outcomes [105]. Researchers have endorsed the significance of feedback in several aspects, such as instructor's credibility of giving feedback [108], and the outcomes of cognitive and motor skills [109]. Many studies have shown that providing prompt and specific feedback not only strengthens learners' comprehension but also increases their motivation and satisfaction. As a result, it can lower learners' anxiety, thereby escalating the feeling of comfort during online learning.

Meanwhile, ambiguous instruction can cause learners' frustration, disappointment, and other complex feelings. As discussed by brain-based theory, learning is strongly attached to emotion; hence, the change in emotion can affect learning development. It can lead learners to negative online instruction perceptions, thus gradually losing interest and enthusiasm toward online learning. Several researches have drawn attention to the issue of ambiguous instruction in the learning process and its impact on learners [110,111]. Ambiguous instruction can be related to unclear instruction without a clear expectation. Consequently, students perceive online learning as challenging and demanding, whereby it can affect the relationship between instructors and learners and reduce online interaction over time. That is, ambiguous instruction may devalue online education and cause high dropout rate, particularly in the midst of the pandemic.

#### 4.3.2. Social Interaction

Social interaction is critical for effective communication in building good rapport between instructors and learners. Several studies related to the significance of social interaction have been identified across three countries, such as USA, England, and China. Moore (1993) revealed three types of interactions, namely, learner–learner, learner–instructor, and learner–content interactions [112]. Icebreaker discussion was rated the most significant engagement strategy in learner–learner interactions, followed by sending consistent emails as reminders or announcements in learner–instructor interactions and working on real-life scenarios, including presentations and reports, in learner–content interactions [49]. As such, developing a reference for social interaction in online social groups could promote online relationship [113]. Similarly, a study involving 667 students from three private universities in the USA was conducted using the Online Student Engagement Scale to measure student engagement [114]. The result showed that skills, emotions, and performances were positively perceived by students, except for participation [48]. The implication explained that student engagement in the online learning environment might be attributed to the seven best practices of online instruction [20]. To increase student engagement in group participation, a study in the USA found that audio-email instruction can be employed as supplement to text-based instruction to enhance a powerful sense of online community [37]. Other than that, online discussion platforms, such as Facebook, can turn out to be promising tools to enhance online interaction [115]. A study on social interaction indicated that high student–student and high student–instructor interactions were favorably preferred by students toward their perceptions of engagement and academic performance in comparison to two other groups, such as low student–student but high student–instructor interactions and group discussions [77]. In England, a study on engagement approach noted that interaction was critically important to online instruction in terms of student motivation. Positive significant impact of interaction was found to motivate students in virtual learning [116].

Conversely, a lack of social interaction has contributed to the downsides of online instruction perceptions. The teacher role was the most important reason for student motivation, but this situation reduced student interaction due to the large class size [94]. Vygotsky's Zone of Proximal Development, Vygotsky's social constructivism, social learning theory, and activity theory focus on social interaction. Social interaction is critical in building trust, establishing relationships, and establishing a sense of belonging. Students connect their prior knowledge and construct new knowledge through social interaction [22,86]. Constructivism also suggests engaging in cooperative and collaborative learning to increase interactive learning and enhance thinking skills. Various online activities, including blogs and live room chats, should be utilized to enhance online interaction [117].

#### 4.3.3. Instructional and Technological Support

Instructional and technological support are critical to scaffold learning in facilitating learners' cognitive process in online learning. Much current and relevant consensus exists on instructional support [118] and technical support [119,120]. Instructional support can be a source of motivation to promote self-directed learning among learners. Correspondingly, heutagogy outlines the roles of learners as independent and active individuals to promote autonomy in learning. As such, sufficient instructional support is perceived as helpful and effective, making no significant difference between online and face-to-face instructions.

With regard to technological support, ease of navigation is found to be essential in online learning to guide learners to various types of online instruction, mostly web-based instruction, including Learning Management System and Schoology. In accordance with theories, connectivism promotes the use of technology to keep updated and innovative. Masrom (2007) demonstrated that online learning experience can tremendously impact the acceptance toward the use of technology in online learning [121]. Nonetheless, it may be sophisticated for learners who are experiencing some problems regarding technological skills and competency. It can propose numerous downsides that may lead to dissatisfaction. This situation may prohibit them from enjoying the pleasure offered during online learning and disrupt their learning, causing unfavorable online learning experience. Failing to interact with instructors and other learners may reduce their acceptance level toward online education and question the effectiveness of online instruction.

#### **5. Discussion**

In this section, both research questions are addressed to analyze and summarize the findings as follows:

#### RQ1: What are students' online instruction perceptions?

Quality instruction was highly rated for satisfaction and motivation levels, effectiveness, and comfort and acceptance toward online learning. Quality instruction accounted for students' favorable online instruction perceptions [50,51,53,59]. Examples of quality instruction were timely responses, constructive feedback, and immediate grade posting. However, without instructors' presence in giving prompt feedback, students might experience disappointment and dissatisfaction in accomplishing challenging tasks. To some extent, this situation might heighten stress and anxiety levels among students. As a result, heath conditions were deteriorated for excessive use of ICT, particularly mental health. Hence, quality instruction should come into practice in creating a healthy and conducive virtual learning environment [122].

Consequently, online interaction was given the second highest rating for students' positive online instruction perceptions. Good online engagement was critical in online instruction [49,76–78]. The findings suggested that increasing interaction through various online activities could have a great impact on the relationship between instructors and students, including live chat discussions, online quizzes, and online forums. For example, a research endorsed the usefulness of gamification, such as Quizizz and Kahoot!, in promoting better engagement and positive learning experience [123]. "Lack of communication" and "lack of interaction" were the most cited reasons that contributed to unpleasant learning experience. This finding suggested that online learning replied heavily on online instruction to construct knowledge and deliver relevant content in facilitating students' understanding and comprehension toward the online course. Three types of interactions were identified: student–student, instructor–student, and student–content interactions.

Most students evaluated instructional and technical support as significant to their academic performance and constant learning [25,99–102]. Students expressed their frustration and stress in dealing with software and hardware operations due to lack of technology competency. Fundamental ICT knowledge and skills should be acquired by learners to enjoy the pleasure of online learning. Moreover, preferences for learning styles related to technology were considered to increase students' fondness toward online instruction as a medium of communication during online learning. Students equipped with considerable ICT competence and skills showed a great sense of interest and community, thus yielding in satisfying learning outcomes. Instructional and technical support were useful and effective in providing assistance to students in terms of ease of navigation with supportive instruction.

RQ2: What are the factors influencing students' online instruction perceptions?

Several factors influencing students' perceptions toward online instruction were identified. Most students expressed their concerns about the transition from face-to-face instruction to online instruction. In some developing countries, the common circumstances frequently faced by students were Internet accessibility, university facilities, resources, and financial support. As a consequence, these unsolved issues might accumulate students' tension level in facing the exacerbation of their academic performance without instructors' actual teaching presence.

Instructors' online instructional practice was one of the determining factors that could impact students' attitudes related to motivation and autonomy. When designing online instruction, instructors should apply optimal instructional strategies that are compatible with students' ICT competency. Instructional strategies can stimulate students' self-regulation for better personal development. In addition, the design of online instructional activities should integrate with interactive elements, notably group discussion and communication, to enhance the interaction between instructors and students. Through online interaction, students' active participation may increase throughout the learning process.

The nature and format of the work were also influencing factors in assigning tasks after the instruction was given. Assigned work was strongly associated with grading, performance, and feedback. It could provide room for improvement and development among students. Thus, flexibility and convenience elements should be considered in assigning tasks to enable students to complete their work on the basis of their time availability and capability.

#### **6. Conclusions**

Overall, this scoping review examined 61 peer-reviewed studies on higher education students' online instruction perceptions pertaining to types of online instruction and research method. In this scoping review, 14 types of online instruction were identified, such as the seven principles of effective online instruction, web-based instruction, and comprehensive online instruction. With regard to the research method, qualitative research

was the most used for data collection, particularly surveys and questionnaires. Referring to the first research question, quality online instruction is evidently an opportunity to maintain students' enrollment in higher education, empowering educational opportunities and developing a more sustainable education around the globe. Online instruction can change teachers' new form of teaching delivery, improve virtual teaching mode implementation, and build a more student-centered approach in the education field. Upon reviewing and analyzing the articles, the findings revealed that instructors have played a significant role in creating a quality virtual learning environment, notably to enhance students' motivation and satisfaction in learning. Based on the second research question, some existing circumstances can be the barriers of putting quality online instruction in higher education into practice. Improving competency, skills, and knowledge in terms of delivering clear instruction and enhancing interaction can be a challenge for educators. Another hindrance of quality online instruction is the lack of technological skills among educators. It may cause some issues in online instruction delivery to engage students effectively. In achieving quality education as stated in one of UN's SDGs, the implementation of a successful virtual learning should be in line with quality online instruction to facilitate students' learning skills in terms of collaboration, creative thinking, critical thinking, and communication in the 21st century.

#### **7. Implications for Future Research**

From this scoping review, useful and effective curriculum pedagogy is critical to provide a clear educational proposal for future curriculum development. Lack of interactivity with instructors and peers is evidently the main contributor to students' stress and anxiety. To illustrate, students are less likely to engage in online instruction partly due to the failure of receiving constant academic feedback, assessing formative performance, and reaching for instructors' availability. Maintaining the relationship between instructors and students should be considered for quality communication during the learning process. In existing literature, strong evidence reveals that effective online instruction requires students to have self-discipline. That is, the arousal of intrinsic motivation mostly comes from self-regulation to inspire students to reach for higher academic achievement. It can promote self-determination to enhance students' critical and higher-order thinking skills. Thus, an online instructional design should be aligned with viable online activities that suit students' interests and learning styles.

Future research should also consider the importance of instructors' roles in addressing the issues identified in this scoping review. For an effective implementation of online instruction, an online instructional framework and an instructional design should be examined to bring a pedagogical value to the education field. Exposing educators to quality online instruction makes it possible for them to have a comprehensive understanding of how to plan their instructional design for curriculum transformation today. Proper guidelines about online instruction strategies and approaches are crucial to reinforce exceptional virtual learning experience in higher education.

#### **8. Limitations**

This scoping review utilized specific inclusion and exclusion criteria through three main databases, namely, Google Scholar, ERIC, and Research Gate, to retrieve identified articles under the narrow scope. Due to particular inclusion and exclusion criteria, some articles were excluded for reviewing purpose in terms of the selection of articles. Therefore, potential articles in other databases, such as Scopus and SAGE Journals, should be extended to provide comprehensive findings as a whole. This review also examined empirical papers from several countries, such as the USA, Indonesia, Taiwan, Turkey, Korea, Norman, England, and Malaysia. However, more than 60% of the reviewed articles were retrieved from the USA, showing a limited coverage of other countries, such as China and Russia. A limited coverage of studies in Malaysia that focus on higher education students' online instruction perceptions was also observed. Hence, most studies are from foreign countries.

To present a board and inclusive finding, reviewed articles should include other countries, as it may produce slightly different data analysis. In addition, in-service and pre-service teachers were chosen as research participants in most studies exploring online instruction perceptions compared with higher education studies. Many studies also center on online learning, e-learning, online discussion, and online interaction, restricting the number of works that concentrate on students' online instruction perceptions. In future research, all suggestions should be considered to balance practicality with available resources for revealing further evident and relevant findings in the scoping review.

**Author Contributions:** For research articles with several authors, especially conceptualization, P.P.C. and K.H.T.; methodology, K.H.T.; validation, K.H.T.; formal analysis, P.P.C.; investigation, P.P.C.; resources, K.H.T.; data curation, P.P.C. and K.H.T.; writing—original draft preparation, P.P.C.; writing—review and editing, P.P.C., K.H.T. and N.-E.M.S.; supervision, K.H.T.; project administration, P.P.C.; funding acquisition, N.-E.M.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Faculty of Education, Universiti Kebangsaan Malaysia under research grant no. FRGS/1/2018/SS09/UKM/02/1, and the article processing charge was funded by Faculty of Education, Universiti Kebangsaan Malaysia.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in this study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


### **Sustainable Learning Environment by Mobile-Assisted Language Learning Methods on the Improvement of Productive and Receptive Foreign Language Skills: A Comparative Study for Asian Universities**

**Mohsen Mortazavi 1,\*, Mahyuddin K. M. Nasution 2,\*, Foad Abdolahzadeh 3, Mojtaba Behroozi <sup>3</sup> and Afshin Davarpanah 4,\***



**Citation:** Mortazavi, M.; Nasution, M.K.M.; Abdolahzadeh, F.; Behroozi, M.; Davarpanah, A. Sustainable Learning Environment by Mobile-Assisted Language Learning Methods on the Improvement of Productive and Receptive Foreign Language Skills: A Comparative Study for Asian Universities. *Sustainability* **2021**, *13*, 6328. https:// doi.org/10.3390/su13116328

Academic Editors: Ana B. Bernardo Gutiérrez and Adrian Castro López

Received: 24 April 2021 Accepted: 25 May 2021 Published: 2 June 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

**Abstract:** In this comparative study, we investigated different influential factors of mobile-assisted language learning (MALL) for improving productive and receptive language learning skills. To do this, 100 scientific research papers were selected from the top and high-quality journals based on the implications of MALL in higher education language learning. Eight papers were selected according to the specific criteria to categorize the results based on language skills interpretation and technological concepts. Therefore, after thoroughly understanding proposed methodologies and comparing them properly, underlying assumptions about this phenomenon are elaborated, and holistic and sustainable solutions are provided to address this idea. The results of this analysis indicated that Mobile devices are being utilized extensively in developing countries, with vocabulary being the primary language learning area assisted by technology and yielding satisfactory results. According to the findings of this comparative study, LINE and WhatsApp would be of interest among university students to improve receptively (listening and reading) and productivity (writing and speaking), respectively. Moreover, the technology acceptance model (TAM) would be a prominent option for teachers to adapt their current and future educational programs instead of only in-person teaching to improve students' learning quality.

**Keywords:** mobile-assisted language learning; foreign language learning; productive-receptive skills; technology acceptance model; WhatsApp; Line App

### **1. Introduction**

Language education requires efficient foreign language practices due to individuals' urgent need in their travels to other countries. It has enabled them to adapt to fast-changing and competitive societies [1–3]. There is a stunning range of research for academic and educational purposes on learning English. Language education is categorized into four principal steps. These steps are described briefly as communicative competence, linguistic proficiencies, cross-cultural experiences, and multiple literacies [1–6]. Communicative competence is defined as the keen competition among individuals with divergent views and accents to communicate in their relations. It is also described as the fluency of other languages [7,8]. Jeong proposed a qualitative study on the profound impact of multimediaassisted language instruction on improving the communicative competence property of foreign language learners. Multimedia-assisted language learning materials can help

students be more motivated and learn how to enhance their perceptional understanding of foreign language learning. He qualitatively demonstrated the effectiveness of multimediaassisted language learning materials incorporated with instructional strategies to improve their communicative competence in language skills. According to the analyzed results from the provided survey in EFL students' in one of the Korean universities, as the multimediaassisted language learning materials were entertaining and more motivated, it can help them improve their communicative competence of foreign language. Therefore, it is recommended by them as an effective tool to have more practical and efficient ways of learning a foreign language [9]. Valeeva et al. developed a theoretical framework method to study the critical impact of different factors on the foreign language communicative competence. They used theoretical methods contained in publication surveys on the various issues that impacted communicative competence and experience of innovative pedagogical frameworks. It is incorporated with empirical methods that consisted of participant observation, a questionnaire, and teaching staff activity in learning issues. According to their model, it is concluded that personal skills and characteristics would be of essential on the providing of perfect conditions in leaning environments to improve communicative competence [10]. Linguistic proficiencies are considered the individuals' ability to speak skills or perform them [11,12]. Cross-cultural experiences are described as the utilization of culture in terms of language instructions as an outcome [13,14]. It has objectified the classification of knowledge, skills, and the means of process making. Multiple literacies express the cultural or communicative tasks' understandings of the other languages [15,16].

Learning a foreign language is one of life's necessities, primarily used as a lingua franca for peoples' communication with different nationalities in business. Foreign language education (henceforth; FLE) is an area significantly influenced by technological advancements. Nowadays, traditional devices like desktop computers have been replaced with electrical gadgets such as smartphones, equipped with numerous applications. They have developed language learning skills such as grammar, reading, writing, listening, and speaking [16–18]. In this paper, a systematic review was based on application of mobile-assisted language learning impact among higher education students to improve productive and receptive language skills. It is revealed that WhatsApp and LINE would be practical tools for improving productive and receptive skills, respectively.

The pervasive force of using mobile phones to obtain a disseminating source of information quickly has rescued students from monotonous class activities with their colorful applications [18–20]. Therefore, students could generate most of their innovations using **e**-learning devices like mobile phones [21]. For example, students can use different websites to use hypertext, video, mp3 files and find papers and other required information [22]. It can also help individuals be more successful in learning processes like learning a foreign language and preparing themselves for international language exams such as TOEFL, IELTS, and IGCSE [23–25]. Regarding the development of modern technology, traditional societies have been changed to "mobile societies." It means that the changes occurring in these societies are a result of the advent of modern technology that can persuade people to change their lifestyle. Hence, people can utilize mobile technologies in a wide range of life aspects such as teaching in virtual classes and distant learning [26,27].

Integration of mobile devices with educational purposes would be a useful tool to provide language learning facilities via several types of equipment such as YouTube, Instagram, etc. [28,29]. Mobile-Assisted Language Learning (MALL) methods can support learning by offering a unique time and place-dependent language learning experience. MALL can provide convenient, real-time, and contextual learning opportunities for students to be more adaptable to the academic environment [30–32]. It no longer matters if they are present or absent from the classroom [33]. Indeed, one of the most frequently cited benefits of mobile technology includes its potential applicability to produce an authentic environment. It provides the opportunity to be exposed to real objects and daily-life situations [34–37]. Mobile devices are considered useful technologies to spread vocabulary

learning instead of computer-assisted learning tools regarding their potential applicability for learning purposes [38,39].

Seliaman and Al-Turk evaluated mobile devices' advantages in foreign language learning by university students in Saudi Arabia. They concluded that since the students were using mobile technology for accessing the Internet, lecture materials, and sharing information, they are useful tools for learning purposes. According to the ComScore report, comparing the number of desktop computers and mobile device ownership between 2007 and 2015, a strong upward trend was observed in mobile device ownership from 400 to 1900 million. The shift has also led to smartphones' development with more advanced options with appropriate browsers, screens, and applications that provide more compatible platforms for internet use through these devices [40]. Basoglu and Akdemir compared traditional vocabulary learning via mobile phone applications and programs by 60 Turkish undergraduate students. The results indicated that the second approach improved students' vocabulary learning more than the first approach [41]. Chang and Hsu developed a system for using PDAs that involved an instant translation/annotation/multi-user mode in supporting university students in an intensive reading course. According to the results, most of the students agreed that the system was beneficial and easy to use. However, the students who worked in groups had a higher level of comprehension than those who read individually [42].

Recently, Sung et al. conducted a comprehensive meta-analysis to evaluate mobile devices' effectiveness for language learning that included 44 articles written and published over the past 20 years. The results indicated that both adults and school children benefited from the MALL and handheld devices such as smartphones were more effective than laptops. The mobile devices were more functional outdoors rather than in restricted environments (e.g., a classroom) as utilization of mobile devices for vocabulary or mixed language skills is more effective than using them in learning discrete skills such as listening [43].

It is noteworthy that the positive effect of technology on language learning is usually taken for granted. For example, El-Mouelhy et al. found that there is no significant difference between the traditional (i.e., paper-based) and technological (i.e., tablet PCbased) approaches in the English reading class. In their study, the electronic reader system developed to assist elementary school students' EFL reading comprehension was more adequate for high-achieving rather than low-achieving students [44].

In this paper, we aimed to investigate different influential factors of mobile-assisted language learning in foreign language learning among university students. Productive and receptive language learning skills are essential in our systematic review, as they can provide a significant improvement in the students' skills. Moreover, it can increase their motivation, satisfaction, and self-confidence due to distance learning, and it might be an excellent choice to address their weaknesses after classroom teaching.

#### **2. Purpose and Research Questions**

The present study aimed to investigate mobile-assisted Language learning on foreign language learning skills (productive and receptive skills) for higher educational purposes. The following research questions are considered in our paper to be more concise about the analysis discussed mobile-assisted language learning's efficacy:


#### **3. Methodology**

To provide a systematic review based on the selected papers and provide coherency between them, we first chose 100 papers from high-quality journals. The top selected papers from 2012–2020 were chosen to provide a reliable comparative analysis among different crucial factors and valuable tools to improve learning foreign language skills. The selection of these papers was based on the higher education concepts among Asian universities. In this part, we set our first criteria to choose papers based on the mobileassisted language learning (MALL) method for learning foreign languages, especially for productive and receptive language skills. Therefore, 68 papers were selected at this stage.

Next, as review papers did not sufficiently describe the primary purpose of MALL methods for a particular community, 29 papers were rejected. This left 39 papers, and criteria have been considered to choose more relevant papers that can illustrate more efficiently the concept of MALL. These criteria included EFL learners, mobile apps, vocabulary learning, and mobile devices. At this stage, eight papers were selected according to MALL use in Asian universities (see Figure 1). Therefore, foreign language learning methods via various applicable techniques and tools such as electronic gadget devices like mobile devices, mobile applications, etc. were considered. This theory was based on the investigations of Mills et al. to implement a comparative analysis to determine the best method of learning a foreign language [45]. The following criteria would be implemented to select the papers that can be compared together. In this regard, the learning foreign language skills focus was concentrated on higher education students that have dealt with more electronic gadgets and mobile applications through their training.

**Figure 1.** Selection procedure of the papers for comparing in this study.

Selected papers are listed as follows:


#### *Description of the Selected Papers*

Mobile devices are considered efficient tools which can be used for language learning. They can help humans improve their productive and receptive language skills by the use of mobile applications. Muhammed investigated the impact of mobile technology on English language learning among EFL university students. A small group of students (N = 20 EFL students) with different English levels and backgrounds from Sulaimani University in Iraq were selected. To perform the qualitative analysis in this study, a focus group discussion (interview) instead of a questionnaire was implemented to virtually eliminate the required time for analyzing the student's notes. The students appointed to a variety of smartphone applications for English learning which covered the following language areas: English Language Skills (including receptive and productive skills applications), applications that integrated language skills and systems (such as chain of thought), International Test Applications (e.g., TOEFL), and the English language skills applications which were significantly related to reading and listening skills. All the participants used mobile devices to support their learning and understanding in language lectures. All participants reported that smartphones were used as their first priority in foreign language learning issues as some applications have been specifically developed for smartphones. On the other hand, due to the compatibility and portability of smartphones, they can be used as a pedagogical instrument; "mobile learning" or "m-learning." Thus, they encouraged students to learn English even outside the classroom environment. The results showed that 99% of the participants used mobile phones to develop their English skills using PDF books, radio programs, spell checking, and grammar software. They indicated that the significant influence of smartphones on students' English language learning among EFL students at Sulaimani University in Iraq [46].

Agca and Özdemir explored multimedia content integrated into language learning materials using 2D barcode technology for vocabulary learning. The participants were 40 students in the age range of 18–21 majoring in ELT at Gazi University who were randomly assigned to control and experimental groups. A determined printed coursebook containing 84 new words were presented to the students using 2D barcodes placed on the book pages in the experimental group for two weeks. The primary purpose of this course book is to increase the required time needed for learning new vocabularies by reducing the physical distance between Microsoft Tags and the course book. Other students were interviewed about their perception of this new experience, applying this new technology in general conditions, and the Microsoft Tag. The participants in both groups were pre-tested and post-tested on an achievement test to measure their vocabulary understanding level after this test. The results indicated that the mobile-assisted language learning environment led to having significant vocabulary development regarding their experimental group tests. The participants believed that the mobile learning environment was valuable and applicable because they could see the word definition and its visual representation (imagebased vocabularies). It has caused to improve the student's curiosity throughout a funny exercise included Microsoft Tag. It has easy access to information, no need to enter data into the device, and usability in any place where the students reported the advantages reported by the students [23].

Wu explored the effectiveness of smartphones in English vocabulary learning in a study which considered 852 English words accompanied by spelling, pronunciation, graphic representation, Chinese equivalent, and the related syntactic information developed as a JAVA application software (Word Learning). The study was conducted for 50 ESL college participants in the age range of 20–23 in Jiujiang University in China assigned to different experimental and control groups. The students were randomly divided into two groups of 25 students. Although the experimental group used the mentioned application via their smartphones, the control group studied their course book. A pre-test and post-test were administered at the beginning and the end of the study to evaluate students' vocabulary development. Furthermore, they can use Chinese-English dictionaries and consult with the researcher about unknown items too. The test asked students to write down the Chinese equivalents of 100 words selected from the total of 6674 words in the book's glossary. The results showed that the experimental group had better scores compared with the control group. The reason for these high scores corresponded to the following issues. As the experimental group had explicit and direct access to the word learning application in their smartphones, they were enabled to enhance all the features for their vocabulary learning by visual perceptions. Moreover, they can improve their visual understanding with the actual context of vocabulary learning that can be used in their future activity through watching videos and computer games. Learning new and unknown words can give the students the chance to be more curious about finding the meaning of unknown words in sample tests that has caused to have higher scores. Finally, as the students can use the smartphone application for a longer time (through the whole semester), they enabled to administer longer accumulated time to learn these 852 words which was more inconvenient for control group as they can't use course book in every situation. It was concluded that visual media could significantly help students learn a foreign language precisely as far as vocabulary is concerned due to the feasibility and accessibility of smartphones in every situations [47].

Shih et al. adopted a blended teaching approach to examine the influence of incorporating ubiquitous learning into an English spelling course. The participants were 29 Chinese college students who received spelling suggestions via the LINE App on their smartphones for six weeks. These teaching classes focus on improving spelling contexts such as syllables, consonants, short vowels, and distinguishing stress. LINE App was selected because of its focus on vocabulary learning, especially spelling. First, four words were put in the LINE application on Mondays, Wednesdays, and Thursdays as an audio file. Students were required to submit their answers in Chinese through their dialogue window. Then, their teacher checked their pronunciations and gave them marks. The difficulty level of the words was increased each week by the teacher, and the students were notified about their performance and their improper mistakes in their pronunciations. Moreover, the students can use a dictionary to check their pronunciations too. The study had a pre-and post-assessment design. Before and after the intervention, the comparison results showed significant differences in three spelling areas: vowel learning, scramble, and syllable learning, apart from the overall improved performance. The participants' perception about this learning experience was also surveyed, and the results indicated their satisfaction and increased confidence in learning spelling without any fear of new vocabulary learning by listening to vowels, consonants, and syllables. Furthermore, due to their teacher's instruction, they can resubmit or reattempt to check their pronunciations without fear and more confidence. Therefore, it can be concluded that the LINE application would not be restricted to a limited number of places and learning environments; the students can use their smartphones in each situation.

Oz investigated whether prospective ELT teachers (fourth-year student teachers) at a large state university in Ankara (Turkey) own a mobile device, how they currently utilize mobile technology for learning, how they intend to use it in the teaching profession. The main obstacles of this paper corresponded to mobile phones adopting and incorporating in language learning eras. In this regard, an invitation email was sent to 75 students via email to complete the survey, and 98 students were invited to complete a paper-based questionnaire. Finally, 144 (75.7% male and 24.3% female) English teachers (in the age range of 21–29) participated as volunteers in the study. This study had two different parts. First, a survey was conducted to distinguish the age and gender as the characterization of the teachers. Then, 21 self-report questions were administered as a comprehensive questionnaire concentrating on the teachers' perspective about their current ownership, usage, and purchase intent of mobile devices and the obstacles impeding the future use of mobile devices for teaching purposes. Most participants preferred to answer the questions in paper-based questionnaire form, while the others preferred to answer the question online. The findings indicated that most teachers already owned a mobile phone or a tablet with connectivity to the internet. Some of them were also intended to buy other devices such as an iPad and e-book reader shortly. Essential functions of mobile phones including sending SMS, taking photos, making phone calls, recording voice messages and searching on the Internet were the most frequent tasks done by the students and teachers. As far as English language learning was concerned, 88.19% of the participants used online dictionaries, while a lower percentage (63%) used their devices for social networking, and only 22.91% used language learning games. The results also indicated that most student-teachers intended to use mobile devices for teaching and learning activities, including course apps. Using mobile devices (e.g., developing course materials), communicating with their peers and colleagues outside the classroom (68.06%), and developing tests. Additionally, the high diversity of applications and administration factors were the biggest impediments to adopting and using these devices for language learning purposes. These findings have suggested that although almost all the student-teachers commonly own mobile devices and frequently use them, they are not yet well-prepared to use such modern technologies for research and educational purposes and are not aware of the many functions of their mobile devices [49].

Chung et al. investigated using mobile technology use for learning English by 84 tertiary levels EFL university students (68 males and 16 females) majoring in engineering who participated in the research and answered a 20-item questionnaire based on Roger's Innovation Diffusion Theory and Davis' Technology Acceptance Model [53,54]. The primary purpose of this study was to implement mobile learning in developing countries to identify the environmental factors that affect mobile technology acceptance as a useful tool for learning a foreign language. In this questionnaire, the questions were contained four items in each of the mobile phone's convenience, self-efficacy, usefulness, intentional utilization, and compatibility. The authors used a 5-point Likert scale to range their decisions from strongly disagree (1) to strongly agree (5) to indicate and organize their statements. Then, to analyze and provide a descriptive statistical comparison, the statistical software SPSS 17 (IBM Company, New York, NY, USA) was used in their study. According to the findings of this study, mobile technology played a significant role in promoting the students' vocabulary learning and saving more time in learning vocabulary. This corresponds to the impact of mobile devices on the student's behavioral intentions to use them in their learning. Moreover, cultural issues would be a critical point in administering mobile learning acceptance for daily learning activities instead of the actual learning environment. Regression analysis was performed to provide an appropriate relationship between the variables and their effectiveness on each variable. According to this analysis, it was illustrated that perceived helpfulness, self-capability, perceived readiness to use, and compatibility accounted for 71% of the variance explained in behavioral objectives to use mobile technology in English vocabulary learning [50].

WhatsApp is considered an influential and applicable software used for distance learning, especially in higher education training. It can help students and trainers handle various tasks such as graphical tasks, video conferencing, and audio typesetting. Learning foreign language skills by administering this application would be of interest and importance for language institutions and teachers to improve students' levels and assign more different tasks offline. Saritepeci et al. proposed a comprehensive study to utilize WhatsApp to learn foreign language skills and educational contexts remotely, especially for YDS's national academic language exam in Turkey. They conducted their investigations on 29 participants (55.20% male and 44.80% female) with different backgrounds and personal features by the semi-structured interview form. The form consisted of two different sections. In the first part, there are some demographic questions about education level, age, gender, and duration time for using WhatsApp. On the other hand, the second part contained six evaluating questions about the use of WhatsApp in FLA preparations. The age range of the participants is 26–30 (34.5%) and 36–40 (31%). 58.6% of the participants had a master's degree, 37.9% had a bachelor's degree, and 3.4% had a Ph.D. degree. The participants used WhatsApp in various periods (44.80% for less than six months, 41.40% for 6 to 12 months, and 13.80% for 12 months). After collecting the data from questionnaires, a content analysis method was used to have a preliminary coding for the revised statements (punctuation, grammatical errors, and fluency). Then, 38 codes were defined for the preliminary questions to be analyzed. They concluded that WhatsApp's use in learning foreign language skills has indicated that this application could provide instructor-learner interactions, active learning, motivation and satisfaction, and continuous learning. As this exam is structurally based on the reading and grammatical issues, listening and speaking in this application is not hypothesized enough; however, it can be covered by this application for four foreign language learning elements [51].

Zhang and Perez-Paredes investigated the utilization of mobile devices such as Mobile-Assisted Language learning (MALL) on EFL learners for postgraduate Chinese students. They used a mixed-method analysis by combining an online questionnaire (quantitative analysis) and a set of interviews (qualitative analysis) with learning English. The questionnaire consisted of 11 closed-ended and one open-ended (enabled students to propose their viewpoints freely) questions. Moreover, seven close-ended questions based on the demographical information about students were included in the questionnaire. To do this, 95 postgraduate Chinese students (28.4% male and 71.6% female) were chosen from four universities with different backgrounds and learning prospects to participate in a questionnaire. However, eight of these students were preferred to do an interview instead. The percent of students who had passed international language tests is about 5.3%. According to their findings, postgraduate students have indicated that mobile English learning resources (MELR) would be a good option for passing their exams, significantly improving English vocabulary development. These resources contained mobile dictionary applications that are considered the most appropriate supplementary resources. As a result, the duration of using MELR was preferred in a short time (less than one hour or between 1–2 h at least). Therefore, MALL can provide different useful aspects such as short intervals for learning vocabularies. In this regard, vocabulary learning applications and mobile dictionaries would be of interest for postgraduate students and considered the two essentially robust solutions for their learning improvement [52].

#### **4. Results and Discussion**

Here, we consider the following criteria to provide a comparative analysis among different learning foreign language skills. The main reasons why students use smartphones for learning English is mainly reported as strengthening their perception about language experiences and their portability, flexibility, and compatibility. It can persuade them to support their learning during lectures and raise their motivation to develop their language skills, especially listening and reading, to activate their schemata of speaking with others. As it is evident, learning vocabularies by images could exercise a dominant influence on

the vocabulary memorization and cause to bear them in mind, to familiarize individuals with different aspects of an unknown expression without any sense of monotonous for individuals. One of the chief aims of learning English by mobile phones is related to the vast majority of innovative and funny applications that gave individuals the chance to surpass their intelligence; to serve the purpose of being organized in their responsibilities. We divided the comparisons into two different sections to be more concise about the provided analysis.

#### *4.1. Language Skills Interpretation*

First, we consider productive and receptive skills as a separate category to find which methods would be more suitable for each type of skill. Shih has focused on utilizing smartphone Apps (LINE app) to ascertain the perception and effectiveness of college students' performance on English spelling improvement. Smartphone Apps could help enhance listening skills such as syllables, phonemes, and vowels. According to the student's opinions, the LINE app would be an excellent choice to improve receptive skills such as listening and reading. In this application, teachers can introduce the number of selected words daily from simple words in the first sessions, and then it can be more difficult words in the next following days. Therefore, students can read and check the spelling of the provided vocabularies in dictionaries to burn them in mind more concisely [48]. As a result, by practicing daily or on a routine program, students can improve their receptive skills individually by spending less time. In Saritepeci et al.'s paper, WhatsApp is introduced as a useful tool for productive skills such as speaking and writing that can be used in online exam preparation. They concluded that teachers could assign several short writing tasks like short essays for students and simultaneously check their reports. It can help them strengthen their productive skills by using and learning different new vocabularies that provide a large word domain for their future study plans. Thereby, WhatsApp is considered a beneficial tool for improving productive skills as teachers and students can adapt it for international language exam preparation. Moreover, this application can reinforce the conducted activities in classrooms. According to their findings, WhatsApp can provide motivation, satisfactory, and continuous learning for language learners that is another positive point of this application for productive skills enhancement among university students [51]. Consequently, it is concluded that LINE and WhatsApp would be two useful applications for receptive and productive language skills, respectively. This is schematically depicted in Figure 2.

**Figure 2.** The schematic diagram for the comparison of LINE and WhatsApp in receptive and productive language learning skills.

Chung et al. reported similar research describing more advantages of using mobile devices among students, significantly influencing vocabulary learning. Hence, such factors include self-efficacy, behavior intention use, and usefulness, considered as the main steps of this investigation due to the consideration of compatibility, portability, and ubiquity of mobile devices on learning English. Investigating the two groups of males and female teachers in mobile devices enhances learning situations. As clarified in the paper, most teachers used mobile devices and their applications in learning regarding social networking with others and language learning by utilizing game applications to encourage students to improve their perceptions and skills concerning current circumstances [50]. In Chung et al.'s paper, students' intentional behavior on accepting technology models for learning purposes was considered. It can provide a theoretical frame to investigate the usefulness of mobile devices in vocabulary learning. According to the findings of these two studies on the utilization of smartphone applications in foreign language learning, it can be concluded that the technology acceptance model (TAM) would be a prominent option for teachers to adapt their current and future educational program instead of only in-person teaching. This can improve the quality of student's learning as they might not be attended in some sessions or have some understanding problem due to the high number of students in the classroom, and there is no additional time for all of them to participate in group discussions. On the other hand, according to the students' perceptions and their targets for their current and plans, they agreed that technology acceptance models would be vital for them to provide a targeting plan in their studying preparations.

#### *4.2. Technology Concepts*

Mobile English learning resources (e.g., vocabulary learning applications and mobile dictionaries) would significantly improve productive language skills. In Zhang et al.'s paper, MALL is a supplementary method for learning approaches. Therefore, MALL can provide different useful aspects such as short intervals for learning vocabularies. In this regard, vocabulary learning applications and mobile dictionaries would be of interest for postgraduate students and considered the two essentially robust solutions for their learning improvement [52]. On the other hand, Oz investigated prospective English teachers and their insights on mobile devices' purchases for learning purposes. Teachers' intention to use mobile devices to develop material courses and tests for university students is critical for this study. The literature also shows that most researchers tend to address one language skill or evaluate a specific application's influence than overall language proficiency or achievement [42,49,54–56]. It has also confirmed that vocabulary learning is the essential computer-assisted language area. The reason for extensive attention to vocabulary may be that vocabulary is a vital aspect of language learning, to the extent that according to Williams [57]. Concerning the fifth and sixth studies, whereas the former one had a more extensive scope, addressing several research problems (ownership of mobile device, its current and future use for language learning, and identifying the barriers in its use of language learning), the latter one only dealt with the ownership of mobile devices by university students. Because these two studies investigated mobile devices' ownership and mobile technology acceptance, no specific applications were examined. Consequently, it can be concluded that MALL methods would be of interest, especially for academic educational purposes.

The multimedia content integrated into language learning materials to improve vocabulary learning and students' perception using 2D barcode technology has been investigated by Wu as the mixed method. Wu divided participants into two groups, experimental and control groups, to analyze the results more precisely. As can be seen from his results, the experimental group students had memorized more words than the opponents; that is to say that environmentally friendly circumstances played a significant role in learning foreign languages [47]. It can be found that mobile learning environment circumstances (e.g., innovative and funny applications) enhance vocabulary learning. On the other hand, smartphones' and Microsoft Tag's effect on enhancing students' vocabulary acquisition

were considered. It is implied that technology administration in vocabulary learning would be of interest; however, as it is not accessible by all students, they might be ignored in the academic environment [47].

The eight investigational research papers' features are descriptively summarized in Table 1. It concentrates on learning English language aspects by utilizing validated references to facilitate the comparative analysis to provide holistic and sustainable solutions [58]. There are contrary opinions about mobile phones, explained by each article's context, respectively; that is to say that many participants widely mention the importance of mobile devices, especially their powerful applications on nurturing the vocabularies with colorful techniques like images innovative apps. Thereby, it is evident that mobile device apps play a substantial role in fostering a deeper understanding of new experiences and views of new vocabularies and enable users to excel in their tasks and responsibilities more than other participants. The acceptance of mobile technology and ownership of mobile devices as two main factors affecting the integration and use of mobile devices in education is specifically investigated among students to whether they might be useful or not to grasp the importance of each application.

**Table 1.** Analysis of eight divergent papers of learning the English language by mobile devices.



**Table 1.** *Cont.*

#### **5. Conclusions**

Mobile-assisted language learning (MALL) is considered an effective method for improving productive and receptive language skills, especially in higher education users in non-native countries. Thereby, it is necessary to choose homogeneous and relevant papers to discussed more the language skills and technical concepts. According to MALL methods for higher education, the selected papers were chosen from among 100 research papers by focusing on productive and receptive language learning. The findings specifically proved mobile learning effectiveness for vocabulary learning by incorporating audio-visual elements, facilitating students' comprehension and retention. In other words, in line with more communicative approaches to foreign language teaching and learning, more integrative methodologies should be addressed where the language is viewed holistically, and mobile technology is integrated into foreign language learning as a whole, not merely applied to a single language skill. Despite these studies' satisfactory results, the acceptance of mobile technology and developing adequate knowledge of these devices' educational functions appeared to be areas that still require much consideration. The main conclusions of this comparative study are as follows:


**Author Contributions:** M.M.; Writing—original draft, Resources, Formal analysis, Investigation, Methodology. M.K.M.N.; Writing—review & editing. F.A.; Writing—review & editing. M.B.; Writing review & editing. A.D.; Writing—review & editing, Visualization, Supervision. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** All the data are available in the paper.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Abbreviations**


#### **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Sustainability* Editorial Office E-mail: sustainability@mdpi.com www.mdpi.com/journal/sustainability

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18

www.mdpi.com

ISBN 978-3-0365-2445-0