# **Educational Technology's Influence in Higher Education Teaching and Learning**

Edited by Maria Limniou

Printed Edition of the Special Issue Published in *Education Sciences*

www.mdpi.com/journal/education

## **Educational Technology's Influence in Higher Education Teaching and Learning**

## **Educational Technology's Influence in Higher Education Teaching and Learning**

Editor

**Maria Limniou**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editor* Maria Limniou University of Liverpool UK

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *Education Sciences* (ISSN 2227-7102) (available at: https://www.mdpi.com/journal/education/ special issues/education influences).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-6113-4 (Hbk) ISBN 978-3-0365-6114-1 (PDF)**

© 2022 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**


Reprinted from: *Education* **2022**, *12*, 54, doi:10.3390/educsci12010054 ................ **155**

### **Paola S ´aenz-Castro, Dimitrios Vlachopoulos and Sergi F`abregues**


### **Juliana Pattermann, Maria Pammer, Stephan Schl ¨ogl and Laura Gstrein**


## **About the Editor**

### **Maria Limniou**

Maria Limniou (Senior Lecturer in Digital Education and Innovation at the University of Liverpool, UK). Maria has more than twenty years of experience in teaching and learning about the use of Information and Communication Technologies (ICT) for educational purposes gained during her academic degrees and work experience in various UK universities. Her research interests include the influence of educational technology in the teaching and learning process inside and/or outside class, especially on how people learn, how people can be facilitated to learn, and how technology design assists people in learning. Her overall research aim is to explore psychological insights into human interactions with the Internet and digital technologies. In her most recent journal publications, she discusses the curriculum design process and the influence of technology on student engagement. Maria gained her Senior Fellowship accreditation from the UK Advanced Higher Education in 2018, and she has published her work in various journals and with various book publishers. She is currently supervising national and international PhD students in the areas of digital capabilities, student engagement with technology, learning analytics in connection with different learning variables (e.g., motivation, metacognition), constructions of knowledge intelligence in formal online learning environments and informal use of social media, the understanding of innovative learning platforms and university–industry collaboration to enhance student employability skills.

## **Preface to "Educational Technology's Influence in Higher Education Teaching and Learning"**

This book includes the first collection of studies discussing technology-enhanced learning (TEL) and how this has been affected by the influence of the COVID-19 pandemic on the higher education digital transformation. It has been inspired by the efforts that higher education teachers and students from all over the world have made during the pandemic to continue working together, overcoming the constraints of time and location. Potential changes in educational frameworks initially discuss the transformation of universities through the digital learning HeXie ecology model (Li et al., 2022), Industry 4.0 (Christiansen et al., 2022), a digital ecosystem (Timokhova et al., 2022) and digital competencies (Henne et al., 2022). This book then presents case studies regarding the ways that virtual laboratories can support tertiary education in an online environment, offering learners the opportunity to complete their lab tasks without attending physical lab facilities (Hassan et al., 2022) and how virtual laboratories can combine enquiry-based learning and gamification principles to support engineering education in a blended learning environment (Schnieder et al., 2022), while a program of experiments for first-year biology undergraduates allows them to work from home and build confidence in experimental skills, enhancing a sense of community (Rayment et al., 2020). E-portfolios support students in developing their employability, but Gutierrez-Santiuste and her ´ colleagues (2022) discuss the strengths and weaknesses of e-portfolios in preparing social education students to enter professional life. The last part of this book presents examples of how technology is accepted by university teachers in South Africa (Ndebele and Mbodila, 2022), how teacher training could be improved in Colombia considering demographic, socioeconomic and academic characteristics (Saenz-Castro et al., 2021), and how digital device usage and the digital interruptions ´ influence student learning in an Austrian university of applied sciences during webinars and on-campus sessions (Pattermann et al., 2022). The latter part is very important for policymakers as it provides evidence of the current higher education landscape in various countries around the world.

> **Maria Limniou** *Editor*

### *Concept Paper* **Disrupting the Disruption: A Digital Learning HeXie Ecology Model**

### **Na Li 1,2,\*, Henk Huijser 3, Youmin Xi 4, Maria Limniou 1, Xiaojun Zhang <sup>2</sup> and Megan Yih Chyn A. Kek <sup>5</sup>**


**Abstract:** Broad societal disruptions (i.e., the industrial revolution, digitalisation, and globalisation) have created a need for an increasingly adaptive higher education system in recent decades. However, the response to these disruptions by universities has generally been slow. Most recently, online learning environments have had to be leveraged by universities to overcome the difficulties in teaching and learning due to COVID-19 restrictions. Thus, universities have had to explore and adopt all potential digital learning opportunities that are able to keep students and teachers engaged in a short period. This paper proposes a digital learning HeXie ecology model, which conceptualises elements and relationships pertaining to the societal need for a more agile and digitally resilient higher education system that is better placed to confront disruptive events (such as pandemics) and that is able to produce graduates who are well-equipped to deal with disruption and uncertainty more broadly. Specifically, we propose a digital learning ecology that emphasises the role of selfdirected learning and its dynamic interaction between formal, informal, and lifelong learning across a five-level ecosystem: the microsystem, mesosystem, exosystem, macrosystem, and chronosystem. This study contributes to the theoretical literature related to flexible learning ecologies by adopting and incorporating the Chinese HeXie concept into such ecologies.

**Keywords:** digital learning ecology; self-directed learning; learning technology; digital resilience; higher education; HeXie

### **1. Introduction**

Building on the Fourth Industrial Revolution [1], Globalization 4.0 [2] has provided opportunities for industry and education to enhance their connections and collaboration, allowing higher education institutions to reconsider their business models, learning environments, technologies, and pedagogies in the process [3,4]. However, most universities have, until recently, been rather cautious about the continuous disruptions (e.g., new learning technologies, rapidly changing market demands, and political rules) and potential educational transformations [5,6]. This situation changed dramatically in 2020 through the enforced impact of COVID-19. "Across the globe, higher education institutions have been radically reshaping teaching and learning in unprecedented ways, and with rare exceptions, education has moved into the online space at breakneck speed" [7]. It is not that the opportunities have not been there to leverage online environments extensively before 2020, but universities as large organisations tend to be relatively conservative and change-averse. COVID-19 has forced considerable changes and disruptions, such as the determinants of students' perceived learning outcomes and their satisfaction in online

**Citation:** Li, N.; Huijser, H.; Xi, Y.; Limniou, M.; Zhang, X.; Kek, M.Y.C.A. Disrupting the Disruption: A Digital Learning HeXie Ecology Model. *Educ. Sci.* **2022**, *12*, 63. https://doi.org/10.3390/ educsci12020063

Academic Editor: Margus Pedaste

Received: 16 December 2021 Accepted: 14 January 2022 Published: 18 January 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

learning [8] and learner–content interactions [9]. At this stage, it is difficult to predict where these changes will ultimately lead.

*It is almost certain that we will see a significant decrease for at least the next two years in the numbers of students undertaking study abroad and exchange, and it is likely that, during this period, Virtual Exchange will become the new normal [10] (p. 2).*

Slow educational changes, especially when they can be seen to lag behind changes in wider society [4], have consequences for educational outcomes themselves, and the disruption caused by COVID-19 may therefore present somewhat of a silver lining in an educational context [11]. In the end, it may be the disruption needed to cause an educational disruption, through which university education is opened up to a wider learning ecology [12]. The concept of a learning ecology is "consistent with the Gestalt tradition, as part of which the [Bronfenbrenner's] human ecology development model was developed, [whereby] the whole is larger than the sum of its parts" [13] (p. 5). Such an ecology has the potential to promote learner empowerment in terms of self-directing their learning pathways [14], as it would include the formal learning environment of universities (both face-to-face and in the form of formal structures such as learning management systems or virtual learning environments), but it would also connect seamlessly to the plethora of learning opportunities outside of the formal higher education system, including digital learning spaces and platforms on the web [12].

The COVID-19 pandemic has caused a disruption to higher education that may allow for such a learning ecology to emerge. Salmon [3] argues that the digital revolution has created considerable freedom of access to information. In the context of open universities,

*It [the digital revolution] poses challenges but also opens up unprecedented opportunities for democratisation and accessibility. The transformation process has to maintain the referential of the profound incorporation of pedagogical and technological innovation based on research and seek new strategies of organisation and definition of quality, to guarantee its relevance and leadership in the pursuit of the massification of higher education [15] (p. 191).*

As a result of the digital revolution, the knowledge students engage with within universities becomes outdated more quickly due to accelerated innovation and knowledge development rates and is aided by ever-faster digital networks. Next-generation digital learning environments have been proposed by educational technology practitioners to create a transformational shift in how universities design their learning ecosystems for students and teachers to have higher levels of digital resilience [16]. Multiple disruptions imply that our conceptualisation of learning and teaching may need to change accordingly if we are to seize the learning opportunities that contemporary digital environments provide [17–21]. "The agility provided by such an architecture can afford learners and instructors alike the opportunity to 'think outside the box', and reconceptualise their approaches to education" [22]. Society requires a more adaptive learning ecosystem to increase learners' competence in a changing environment, to strengthen universities' resilience in disruptions, and to reshape lifelong and life-wide education with on-demand, tailored, and personalised learning elements.

To address the question of how future universities could develop digital resilience to become more prepared for subsequent disruptions, this paper synthesized a conceptual model based on the Problem-Based Learning (PBL) ecology [13] with an extended layer of the Chinese HeXie concept [23]. The model highlights the role of self-directed learning and digital resilience through formal, informal, and lifelong learning across a five-level ecosystem: the microsystem, mesosystem, exosystem, macrosystem, and chronosystem. It has been suggested that the nature of higher education is "to enable society to make progress through an understanding of itself and its world" [24], which implies that universities are separate from society but are capable of improving society from their enlightened position. This paper discusses the significant but blurred lines of a learning ecology, as it can be seen as a paradox that an inherently conservative higher education system is positioned as

being able to advance society in innovative ways. This study contributes to the literature by emphasizing that higher education has the potential to occupy that position, but only if it is integrated, in agile and reciprocal ways, into the society it is meant to impact and vice versa. In other words, the boundaries and the constraints would need to be significantly blurred and become much more porous so that continuous exchanges and dynamic interactions between universities and their societal contexts become possible.

### **2. Materials and Methods**

This conceptual study employs a theory synthesis method to "achieve conceptual integration across multiple theories or literature streams" and aims to "offer a new or enhanced view of a concept or phenomenon by linking previously unconnected or incompatible pieces in a novel way" [25]. Following Weick's [26] theoretical model development strategy, this paper first reviews the extant literature to identify problems and challenges; second, it summarizes the theoretical model development needs and develops the research question.

### *2.1. Challenges in Formal Learning Environments in the Digital Era*

In the past thirty years, formal learning has been "institutionally sponsored, classroombased, and highly structured" [27]. Universities provide formal learning environments to facilitate institutionalised, chronologically graded, and hierarchically structured formal educational systems [28]. The key assumption in the traditional conceptualisation of formal learning environments has been that learning can be delivered or provided in a discrete, packaged manner, which is timed and clearly demarcated, and symbolised by the walls of the physical classroom and semester timetables [29]. With the development of the World Wide Web (Web 1.0), the emergence of Virtual Learning Environments (VLEs) [30,31] extended formal learning beyond the physical classroom to an online environment with interactive activities, albeit a walled and password-protected one.

McGuire and Gubbins [32] have argued that formal learning has been supplanted by activity-based and technology-based learning, suggesting that activity-based and technologybased learning are not formal learning. However, this is a limited conceptualisation of formal learning in modern digital learning environments. In other words, formal online learning environments, if designed in particular ways, can be characterised by activity-based learning approaches, if often within the walled garden of the VLEs. Thus, the emphasis should not so much be on supplanting formal education but on leveraging informal learning through innovative learning design to make learning overall more effective. Any approaches that could provide students with the systematic knowledge that they require to operate within complex structures [33] beyond their formal studies and that provide sufficient support during knowledge acquisition [34] could be recognised as formal learning.

During the COVID-19 pandemic, students have relied on technology-based formal learning more than ever [35]. In fact, digital disruption had started long before the pandemic caused an additional disruption [36]. Students have long been digitally connected in their everyday digital environments, which, in many ways, could be seen as informal learning spaces. During the pandemic, for example, students have been required to use their mobile devices to attend online lectures from isolated locations (i.e., home, student accommodation) and to engage in online assessment activities [37,38]. These can all be seen as formal learning. However, the same devices give students access to a much wider digital environment that provides potentially endless opportunities for learning beyond the formal learning context, or indeed for deliberately integrating such opportunities into the formal learning environment [39].

A common assumption is that formal intentional learning is more standardized and should be supported by technologies that are designed for educational purposes [40]. This assumption has stopped educational institutions from investigating the possibilities of using or leveraging disruptive technologies to enhance learning and teaching and to even stimulate cutting edge innovation in education [36]. In other words, there is a tendency to categorise digital technologies into particular boxes such as education, communication, or social media, whereas in reality, the lines between them are blurred. Moreover, students will have to use and learn how to use a wide variety of technologies when they graduate and upon entering employment or enterprise environments [41]. This suggests that we need a wider conceptualisation of educational technologies, one that recognises the potential of the overall digital environment as a learning environment, rather than just the formal one. The latest pandemic-induced disruption may serve as a catalyst for that kind of reconceptualization [35].

Centrally supported educational technologies are under institutional control and are characterised by limited uncertainty and high levels of standardization [40]. By promoting the use of these supported centralised technologies, the university can provide institutional support with relatively few resources because their use is predictable and contained [42]. From a business model perspective, there are institutional pressures for high efficiency and limited uncertainty, which explains the attraction of the notion of supported centralised technologies and the resistance to the use of external and potentially disruptive technologies [36]. However, standardisation is sometimes the enemy of creativity and agility, which are some of the key attributes we expect students to graduate with [13,43]. This raises the question of whether a standardised digital environment is capable of preparing students for life beyond their degree studies, which likely involves complex and constantly evolving contexts that require continuous searching for new opportunities and digital tools for business, for creative solutions, and thus for learning.

An example of a learning technology that is instructionally controlled is Virtual Learning Environments (VLEs), which have been adopted by various universities to support teaching and learning [42]. VLEs, as the institutional technologies that define formal learning environments, have largely reproduced, rather than disrupted or transformed, learning and teaching practices [44]. Many studies have revealed that technologies provided by universities for formal learning have not been globally successful in terms of adoption and usage to justify their huge investment [32,45–47]. Teachers and students prefer convenient and easy-to-use technologies, despite many of these technologies not being designed for educational purposes (e.g., Zoom, Microsoft Teams) and despite lacking institutional support [48]. Universities are positioned to remind teachers and students of the coexistence of institutionally supported and non-supported technologies [49]. For example, the ABC learning development framework of the University College London (UCL) has highlighted three types of the learning technologies: UCL supported, provided with limited or no support, and support provided locally in the division/department [50]. The educational transformation of both teachers and students is crucial in the digital learning ecology so that they can feel comfortable using technology for learning and teaching, regardless of whether the university supports it or not [51].

However, the contribution from these non-institutional disruptive technologies is largely unexamined [48]. For example, employers (i.e., industry and small companies) require graduates to work efficiently with many useful technologies and sometimes highly specialised technologies [41,52]. Many of these technologies may not be commonly used in universities for learning and teaching, such as Facebook, Slack, or a whole range of mobile apps [53]. Of course, it is impossible to adopt all possible technologies in a formal learning environment. Yet, it is possible to infuse the curriculum and learning approaches at university with the development of the students' ability to adopt and adapt to new technologies wherever possible and relevant. In this digital era, students should be adaptive to the use of a wide variety of technologies for both their learning and their future careers [54,55].

### *2.2. New Opportunities in Informal and Lifelong Learning Environments in the Digital Era*

The business model of higher education has changed over the years due to marketisation [56,57], which started with mass higher education [58], the introduction of student tuition fees, and the trend of universities selling teaching and research as services with increasing student numbers and reduced budgets [58–61]; the granting of university status to polytechnic colleges [4,59–61]; and the spread of the (UK) Open University model [62]. At the same time, deeper collaborations with industry are sought to reduce the mismatch of the students' employability and the employers' expectations, for example by focusing on entrepreneurial skills [63,64] or through work-integrated learning initiatives [65]. Future education will not be limited to the above models, and a new social contract for education is needed [66].

New models of learning emerged during the COVID-19 pandemic, providing additional opportunities for private contractors/partners to work more closely with universities and enhancing collaborations in innovative learning design that leverage a wider learning ecology [67]. For instance, the University of Illinois at Chicago has started a university– industry partnership in developing hybrid courses based in their VLE [39]. In China, Xi'an Jiaotong-Liverpool University (XJTLU) is piloting several new educational models, such as the "learning mall", with an integration of the physical campus, deep partnership with industry for syntegrative education, and online education [23]. Syntegrative education is a new education model that XJTLU has used to develop globally competitive citizens and to provide opportunities for students to work in the industry alongside their degree, gaining industrial certificates and practical skills during the learning process [23]. This has further opened a door for both universities and private partners to explore the possibilities of crossing the boundaries of informal and formal learning with seamless digital integrations between (and beyond) formal digital learning environments.

### *2.3. The Need for a Reconceptualized Model*

Teacher-centred learning has been largely dominant in modern universities, and perhaps even more so since the massification of higher education [68]. Following the traditional way of teaching, teachers usually act as a "sage on a stage", transmitting knowledge and information to students in a unilateral direction [3,5]. Inspired by constructivist views on learning [69], more student-centred learning environments have emerged to encourage greater participation and collaboration between students who are required to take more responsibility for their formal, informal, and lifelong learning [70–74].

The extant literature has raised questions on how teachers could change teaching approaches by adopting a more student-centrered one, e.g., [11,73]. However, on the one hand, the pandemic-related disruption has led teachers to adopt different teaching approaches [75], while on the other hand, it may have added considerable stress to those teachers lack digital resilience during the COVID-19 pandemic [76].

There is an increasing recognition of the link between student-centred approaches and active learning process that is related to self-directed learning [73]. Since Tough's [77] adult learning research project, the study of self-directed learning (SDL) has taken an adult focus, emphasising learner characteristics [78,79] and the instructional process [80–82]. Self-directed learning readiness has been defined as the degree to which the individual possesses the attitudes, abilities, and personality characteristics necessary for self-directed learning [83].

In summary, the major problems encountered in our literature review were the lack of a comprehensive theoretical model to build agile, responsive, and proactive approaches to developing student self-directed learning competencies across formal, informal, and lifelong learning environments in the digital era [73]. More recently, a growing body of literature has begun to identify the need to address self-directed learning across a lifespan in formal and informal learning environments [84,85]. To develop students' self-directed learning capabilities, teachers need a reconceptualisation of learning environments that would make them not only fit for purpose but that would also "force" teachers to focus more on what fit of purpose is [86]. Regarding the challenges that teachers may face due to a potential disruption, this distinction raises a question: how could future universities gain digital resilience to disrupt the disruption? To answer this question, a digital learning HeXie ecology model has been proposed to build agile, responsive, and proactive approaches to develop students' self-directed learning competence.

### **3. The Digital Learning HeXie Ecology Model**

This paper proposes a digital learning HeXie ecology model to cover the need for agile education with a focus on self-directed learning and digital resilience. The proposed model conceptualises not only the fluidity between formal, informal, and lifelong learning between the teacher and student, but it also supports a dynamic balance of the learning ecology through the HeXie education model (see Figure 1). Further, the proposed model is based on the five levels of Bronfenbrenner's [87] human ecology, which was further developed in Kek and Huijser [13] agile Problem-Based Learning (PBL) ecology for learning. As an active learning approach, PBL "leverage[s] different systems in the agile ecology for learning [and] serve[s] as a curricular and pedagogical vehicle to facilitate the development of a particular way-of-being among students" [43], which includes skills and attributes such as critical reflection and creativity. The proposed model has additionally incorporated the HeXie concept, which supports higher education institutions to adopt an approach to overcome the challenges posed by potential disruptions (i.e., COVID-19) by focusing on the need to continually re-balance.

**Figure 1.** Digital learning HeXie ecology model, adapted from Kek and Huijser's agile PBL learning ecology [13] and Xi's HeXie education model [23].

### *3.1. Self-Directed Learning in the Digital Learning Ecology*

Following Carré [88] research, we define self-directed learning as a dynamic combination of two dimensions: self-determined motivation to learn and self-regulation strategies and abilities in learning. Self-regulation refers to the abilities and strategies of selfregulation in learning, while self-determination refers to self-determined motivation to learn. When confronted with the COVID-19 disruption, many universities rapidly changed from low digital context traditional learning to high digital context online learning [35,89]. Students with higher level digital resilience and greater self-directed learning readiness could adjust themselves [90] in the relatively isolated online learning environment by using digital learning technologies, e.g., an online calendar for time management, online tutorials to seek feedback from teachers, online peer support forums to discuss common issues and share information, and a range of other digital tools and resources that do not form part of the formal learning environment in a strict sense [89].

The digital transformation of higher education may connect students' digital resilience with their self-directed learning readiness. This invites important caveats for it to work: firstly, students need to be digitally connected and capable, which means that they need to be comfortable in navigating the potential that a digital learning ecology offers [13]. Secondly, there can be no assumption that self-directed learning simply happens because

students find themselves in a digital learning ecology. Instead, self-directed learning needs to be deliberately designed into formal learning environments and deliberately taught [73].

As noted, the boundary between formal and informal learning environments is becoming increasingly blurred. However, the distinction between formal and informal learning environments is still rigidly maintained in many ways, as change is resisted and institutions hold on to the ways they have always done things [91]. Yet, the recent COVID-19 disruption may have accelerated the exploration of a more expansive learning ecology that encourages higher level self-directed learning across formal and informal learning environments. For example, in flipped classrooms, students can watch lecture videos or access learning resources provided in the formal learning environment, while well-designed formal learning environments will, at the same time, allow students to explore and draw on their sources for learning in informal learning environments [92].

In this way, they have opportunities to ask questions and formally collaborate with peers in solving problems in a lab or classroom (formal learning in the formal environment), but they can also simultaneously engage with other resources (and other learners) in informal digital learning environments. Indeed, this would be encouraged. The quick development of mobile technologies has enriched the learning opportunities in informal learning environments, as many students have ubiquitous access to digital learning [93]. Watching a 2 min video explaining the epidemic R-nought on a cell phone while taking a bus is a common format of informal learning. Students construct their knowledge both from learning in an informal environment and in a formal environment. Self-directed learning is therefore a competence the student needs to develop urgently as a key stakeholder, for which teachers as the other key stakeholders need to take responsibility.

### *3.2. Digital Resilience in the Digital Learning Ecology*

Formal and centralised technologies may take time to catch up to disruptive situations [94], but in the learning ecology that we discuss in this paper, teachers and students as key stakeholders can use alternative solutions in an agile manner, which creates considerable resilience in the overall learning environment. Although some of these technologies are not specifically designed for educational purposes, they can be used as part of the educational process, which adds authenticity in terms of what students will ultimately need to be able to do when they graduate. The transition from previous education modes to a new educational model in response to disruption is reliant on effective processes for the incorporation of a wide and ever-expanding range of technologies into the learning process. The biggest challenges include the continuous administrative burden of managing user accounts, keeping equal accessibility, providing user training, and support for different technologies.

However, in a disrupted learning environment, this is no longer solely an institutional responsibility, but instead becomes a responsibility of everyone in the learning ecology, including students and teachers as the key stakeholders. In current formal learning environments, teachers and students alike become easily confused if clear instructions on how to use different technologies for different learning and teaching activities are not provided in advance, as the expectation is that institutions provide both the technology and the training. We are suggesting here that this responsibility needs to shift if universities are to become more digitally resilient and to become better positioned to deal with disruptions in the future.

In addition, the richness of digital technology and the use of a wide range of alternative solutions beyond formal learning management systems could increase the university's digital resilience in supporting formal and informal learning and teaching. When disruptions occur, such as the COVID-19 pandemic, universities with limited digital resilience may face different challenges (e.g., lack of solid digital infrastructure to support large group synchronous online learning). For example, given their different levels of technology adoption and the very limited preparation time, some universities hardly have had any centralised technologies at all throughout the COVID-19 pandemic [95–97]. Universities

were thus forced to use whatever technologies were available (e.g., a range of different online conferencing technologies) to facilitate online learning and to address the main problem of a lack of interaction with students.

On the other hand, universities that had already implemented centralised technologies for a long time might have had a strong reliance on rigidified institutionalised practices [98]. It usually takes longer to make changes and upgrade existing technologies that form part of rigidified institutional infrastructures [99]. Thus, when disruption occurs, these institutions are often not agile enough to respond. By contrast, an agile educational ecology includes any potential digital tool that can be leveraged to support flexible learning. However, to actually leverage such digital tools requires astute learning designers to collaborate with academic content experts to develop a responsive, proactive, and agile learning design that is student-centred and that draws on both digital environments and tools that students are already familiar with and ones that they need to become familiar with. In short, such learning design oscillates between the push and pull of a range of ever-changing tools in a hugely dynamic and constantly disruptive digital (learning) environment. The word learning is in parentheses, as a digital environment requires deliberate design to become an effective learning environment.

### *3.3. HeXie Education Model in the Digital Learning Ecology*

In the digital learning ecology, the HeXie education model reflects both oriental and occidental wisdom in education. Figure 2 illustrates the full version of the HeXie education model [23] that our digital learning HeXie ecology model has integrated. The concept of "HeXie" originated from Chinese Confucianism (emphasis on harmony) [100,101] and Daoism (with an emphasis on the Yin and Yang balance) [102]. The HeXie education model was developed based on the HeXie theory [103,104] to couple formal and informal learning based on a lifelong plan with three steps: learning, growth, and conduct. The He principle emphasizes the importance of self-directed learning for innovative and dynamic actions, while the Xie principle focuses on design and planning for digital resilience. The two principles are coupled throughout lifespan through three main steps (learning, growing, and conducting) in a mix of five learning types (inheritance learning, reflective cognition, exploratory integration, interest driven accumulation, and mindset upgraded progress) to achieve the long-term vision and mission of the ability to face a rapidly changing world [23].

**Figure 2.** Adapted from Xi's HeXie education model [23].

The UACC in Figure 2 refers to uncertainty [105], ambiguity [104], changeability [91], and complexity [106]. Whatever disruptions we are facing, the essence of education is to help students understand themselves and to have a vision and mission or life orientation. The digital learning ecology aims to help students learn knowledge and gain the capability to follow their dreams in a practical sense, while the life orientation is the intrinsic motivation for learning. In the original HeXie Management Theory [103,104], the HeXie Theme refers to the key tasks or core business faced by the key stakeholders in a specific period. Key stakeholders will need careful consideration about using the two principles to carry out the task or solve the problem.

In the educational context, the HeXie Theme refers to the key learning tasks a student needs to perform at different learning stages. During their learning journey, students might face various challenges when taking on different learning tasks for certain periods (e.g., for undergraduate students, year 1–2 as a freshman or sophomore, year 3–4 as a senior student). Different HeXie Themes will need different activities to help students implement the plan or to carry out specific learning tasks. Students are encouraged to critically think about the unique features of specific learning tasks and how they could use the two principles (He or Xie or both) and couple them with the HeXie Theme to develop themselves to achieve higher-level life orientation. For example, the Xie principle could better support learning with technologies (e.g., in-class polling or AI grading) to help with prior knowledge and explicit memory-focused learning. By contrast, the He principle could encourage critical thinking for the reflective cognition of the real world, which requires higher learner autonomy. In a flipped classroom setting, the two principles are both required to foster a self-directed, exploratory, constructive, active, experiential, research-led, and syntegrative learning environment.

In higher-level learning, such as self-interest driven accumulation towards ideals, competence development as a global citizen, and interdisciplinary collaboration to address 'wicked problems' collectively (e.g., climate change), students will need to develop a growth mindset [107] that aligns with the HeXie mindset. The HeXie mindset can be developed or nurtured by embracing the ontological and epistemological framework that originated from HeXie Management Theory as a complex problem-solving paradigm. When facing a changing world with the UACC challenges, students will need to clearly understand their life orientation (i.e., vision and mission). Furthermore, they will need to set the core objectives and identify the key learning tasks for each learning stage. Through the dual rationality provided by the He and Xie principles, students can benefit from the Xie principle's systematic support (e.g., institutions, processes and technologies). The He principle can help students make better use of the policies, culture, and emotions to develop a selfdirected learning ability and co-create a humanistic learning environment with teachers and other stakeholders. Through HeXie coupling, students can work towards a vision, optimise, and evolve dynamically based on the HeXie Theme at each stage. Therefore, the HeXie mindset is critical to help students adapt to a future-oriented perspective, while integrating the wisdom of the West and East to find the theme in each stage and to address new trends and issues [23].

### *3.4. The Five Levels of the Digital Learning Ecology*

Bronfenbrenner's five levels consist of the microsystem, the mesosystem, the exosystem, the macrosystem, and the chronosystem. The following section explains the five levels of the digital learning using the HeXie ecology model.

### 3.4.1. Microsystem

The microsystem refers to the formal learning environment where students engage with or are confronted with the curriculum design, physical learning spaces, teachers, and assessment as well as formal digital (or virtual) learning spaces, such as the learning management system, the online enrolment system, and so on. In other words, the microsystem is what we often think of as the university learning environment in a narrow sense. It relates to learning spaces where teachers and students engage with each other directly [108]. It also includes pedagogy, formal learning technologies, and self-directed learning [109], if the latter is indeed designed into the learning environment. This might be influenced by individual factors such as age, emotion, (prior) knowledge, experience, and mindset [99]. Each of these could, in turn, be affected by institutional factors, cultures, and social backgrounds in the mesosystem [43].

### 3.4.2. Mesosystem

The mesosystem level reflects a wider system of connections that include higher education institutions, family, workplaces, social networks, and the wider community [110]. Digital technologies may straddle the boundaries between the microsystem and the mesosystem. For example, university students who have early access to the technologies that are the most commonly used in workplaces might have a greater opportunity to find jobs [54]. However, a university student's socio-economic status, which is connected to family income, may influence their attitude (i.e., self-determination) and ability (i.e., selfregulation) to afford the devices and internet access needed to be able to use technology in formal or informal learning environments (social network and community) [111,112]. University-supported centralised learning technologies could provide students with institutionally licensed services and learning spaces without extra personal cost. These open-source or cheap disrupting learning technologies can serve as alternatives and flexible supplements when centralised technologies encounter disruption. Therefore, when higher education institutions make decisions about technologies and the institutional facilitation of technology-enhanced learning, factors such as access, equality, student employability, and social sustainability need to be considered to reduce the digital divide [39].

### 3.4.3. Exosystem

The exosystem refers to the broader support systems in the learning ecology, both formal and informal, and again, the boundaries between them are often blurred and fluid. This broader support system includes elements such as co-curricular student support (e.g., digital literacy, technology troubleshooting, user guides, instructions from teachers, teacher attitudes, institutional norms, regulations, culture, and cognition), teacher support (e.g., technology troubleshooting, user guides, professional development, student feedback, learning analytics, institutional norms, regulations, culture, and cognition), peer support (e.g., knowledge sharing, peer influence), facilitating conditions (e.g., supported VLE, disruptive technologies, organisational structures, resources), and social influence (e.g., social norms, morality, culture). Leveraging this kind of available support requires initiative and proactive help-seeking where needed; in short, it requires self-directed learning skills, as discussed earlier.

### 3.4.4. Macrosystem

The macrosystem is the wider context in which the learning ecology is situated, for example on a state, national, or global level. Thus, it includes the economy, government, enterprise, non-profit organisations, the natural environment, geographies, religion, culture, health, law, politics, and history. Clearly, during the disruption caused by the COVID-19 pandemic, the macrosystem has become more salient, but it affects all other systems to varying degrees. For example, universities became dependent on government regulations around international travel (e.g., with regard to international students), and they became dependent on government funding (or lack thereof) to cover some of the losses caused by students not being able to travel and come to a physical campus [39]. Again, within the learning ecology thus conceptualised, the notion of self-directed learning becomes very relevant, as it underlies the broader idea of developing lifelong learners who are agile, responsive, and proactive to rapidly changing contexts, including potential disruptions. For example, in a major disruptive event such as COVID-19, self-directed learners would be able to quickly adapt to changing circumstance by developing their digital capabilities by quickly learning new online tools to help them continue their learning in a digital environment [113].

### 3.4.5. Chronosystem

Finally, the chronosystem refers to broader, historical movements, and indeed, disruptions, including, for example, the industrial revolution [1], the massification of higher education on a global scale [56], and digital globalization [114]. For example, the earlier

referred to Globalisation 4.0 [3] would fit into the chronosystem, with a fluid spill-over into the macrosystem. Thus, the chronosystem refers to broad, often generational changes that occur at various points in time, which then have a major (often disruptive) impact. In some cases, they may be seen as paradigm shifts. The emergence of big data over the last decade is one example, and the impact of AI may be another that is still developing [4]. Prior to that, the arrival of the World Wide Web in the 1990s and social media in the first decade of this century constitute other examples [3]. These types of disruptions can be mapped to particular eras, and the responses to them tend to be significant changes in the way higher education is approached. Again, self-directed learning is the central thread that cuts across the different systems as both a way of buffering against disruptions (and hence a form of resilience) and as a way of leveraging the potential that such disruptions may afford.

### *3.5. Balancing the Disruption in the Digital Learning Ecology*

The five systems that make up the learning ecology go through periods of relative calm, even if they are in constant flux. When considering large-scale disruptions, however, another layer could be added to the aforementioned learning ecology, which would be focused on keeping a balance between the situations prior to the disruption and the postdisruption context. The static view examines how a system and its parts behave under a state of equilibrium, while all of the forces affecting it are in a dynamic balance [115]. However, during a disruptive event, each element moves under the influence of forces that push it toward, away from, or between equilibria [116]. The Chinese concept of HeXie could overlay the learning ecology, as it draws attention to how balance can be restored in response to disruption, or more importantly, how a new and ideally more productive and relevant balance may be achieved. The balance here is universal, and other models have explored similar system thinking perspectives, such as Beer's Viable Systems Model [117] and Kaufman's Organizational Elements Model [118].

Overall, the proposed model in Figure 1 illustrates how HeXie education model is focused on the balance between each of the broad elements that relate to student learning in the overall digital learning ecology. The ecology itself is circular, which means that we can start anywhere at any time, and the relationships are dynamic, depending on where we choose to target our analytical focus. In response to disruptions, however, each of the five systems in the learning ecology affects the others to varying degrees, and what this model allows us to do, with the help of the HeXie dimension, is to re-balance after a disruption. Importantly, re-balance refers here to a new equilibrium, which is never the same as the equilibrium that existed prior to the disruption but which may offer new ways of imagining learning and teaching that is both fit for purpose and fit of purpose [86].

### **4. Conclusions**

This paper focuses on the critical role of self-directed learning and digital resilience where both teachers and students are key stakeholders as the co-creators of the digital learning ecology across the microsystem, mesosystem, exosystem, macrosystem, and chronosystem.

### *4.1. Implications*

Our study contributes to the emerging literature on digital learning ecology [43,119] by providing a holistic view that across five ecosystems, while prior studies have made significant contributions in exploring the learning ecology within specific ecosystem. Borge and Mercier developed a micro-ecological framework with a focus on the microanalysis of individual interactions when different cognitive systems interact and modify different learning activities [120]. Further, the University of Illinois has worked on a digital learning ecology where computers are used as mediators in social human connections, as "computers could not simply be applied to education. It had to be (re)designed to align with the social construction that is education" [121]. Van den Beemt and Diepstraten reinforced the importance of the creation of information and communication technology rich social environments in a exo-level learning ecology [122].

Regarding the practical implications, this study has proposed two new constructs: self-directed learning and digital resilience within a digital learning ecology, which may inspire new directions into digital learning analysis, for example, exploratory structure equation modelling through quantitative grounded theory. In terms of learning and teaching practices, the proposed conceptual model might serve as a framework to promote new educational development policy and may encourage innovative pre-sessional and syntegrative programmes for students as well as more effective and agile professional development programmes for teachers.

More importantly, the proposed digital learning HeXie ecology model allows us to conceptualise learning across formal, informal, and lifelong learning in different levels of human ecology, and what is needed in a learning environment in response to and in the aftermath of a major disruption. The main contribution to the current literature is that the proposed model has extended Kek and Huijser [13] PBL ecology for learning by adding another layer in the form of the Chinese HeXie concept. This allows us to find a new equilibrium (or indeed, new equilibria) in relation to student learning.

### *4.2. Limitations and Future Development*

At this stage, this paper has been conceived on a purely conceptual level. Although applying these ideas in practice is more complex, an increasing number of future-oriented universities have made varying degrees of progress [23,39]. Future studies are therefore encouraged to test this model by applying it empirically in different contexts, such as by examining the association between students' self-directed learning ability and their digital resilience in a syntegrative education system based on industry–university partnerships and in the process testing the influence of teacher support for student self-directed learning and digital resilience development in formal and informal learning environments.

**Author Contributions:** Conceptualization, N.L., H.H. and Y.X.; methodology, N.L.; writing—original draft preparation, N.L. and H.H.; writing—review and editing, Y.X., M.L., X.Z. and M.Y.C.A.K.; visualization, N.L. and Y.X.; supervision, X.Z., Y.X. and M.L.; project administration, N.L.; funding acquisition, M.L. and N.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the University of Liverpool, grant number 100699618.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** We thank the editors and the two anonymous reviewers for providing informative and insightful comments to help strengthen the manuscript. We thank the editorial office and the MDPI service team for providing professional and efficient support.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**


### *Article* **A Framework for Developing Educational Industry 4.0 Activities and Study Materials**

**Lasse Christiansen 1,2,\*,†, Tommy Edvardsen Hvidsten 3,†, Jesper Hemdrup Kristensen 2,†, Jonas Gebhardt 4, Kashif Mahmood 5, Tauno Otto 5, Astrid Heidemann Lassen 2, Thomas Ditlev Brunoe 2, Casper Schou <sup>2</sup> and Esben Skov Laursen 1,†**


**Abstract:** The advent of Industry 4.0 is changing the role of human labour towards a more supportive function in the production system, requiring new digital-, technical-, interdisciplinary-, collaborativeand communicative competencies. This challenges educational institutions to develop new teaching activities and materials to address ever emerging needs. To address this, this paper presents an Educational Framework to support educators in developing new teaching activities and study material for Industry 4.0. The model distinguishes itself from other educational design models by combining an iterative approach toward problem-solving, with the concept of authentic task design, as the core elements. Based on 14 pilot cases, it is concluded that educational framework have increased the educational activities in the areas in focus.

**Keywords:** educational design; process model; capabilities; Labour 4.0; teaching

### **1. Introduction**

Information and communication technologies have led to new industry opportunities [1–3], often referred to as the fourth industrial revolution or, in short Industry 4.0. The core concept of Industry 4.0 can be described as machines and products autonomously exchanging information, self-configuring and self-organising, leading to flexible, modular, intelligent, and cyber-physical production environments [1]. This transformation towards smart machinery and new capabilities also set new requirements for work tasks, operators and technicians [4,5].

As a result, the role of human labour in the manufacturing industry is also changing. Human labour is changing towards a more supportive role in the production system rather than directly involved in the production processes, e.g., assembling the products [6–8]. Consequently, employees need new technical-, interdisciplinary-, collaborative- and communicative competencies [6,8]. Traditionally, many educational programs within the technical fields have focused almost exclusively on developing technical skills and much less on social and interpersonal skills [9,10], which is found to be a requirement for Industry 4.0 and is supported by recent studies [11] and reports from the world economic forum [12]. To address the new role of human labour, educational institutions within the technical fields must develop new teaching activities and materials. However, the needs, teaching method and approach for making these changes are unclear, as Industry 4.0 is a broad, alternating, and complex area. Hence, there is a need to support the educators in handling the complexity and uncertainty [13].

**Citation:** Christiansen, L.; Hvidsten, T.E.; Kristensen, J.H.; Gebhardt, J.; Mahmood, K.; Otto, T.; Lassen, A.H.; Brunoe, T.; Schou, C.; Laursen, E.S. A Framework for Developing Educational Industry 4.0 Activities and Study Materials. *Educ. Sci.* **2022**, *12*, 659. https://doi.org/10.3390/ educsci12100659

Academic Editor: Maria Limniou

Received: 24 August 2022 Accepted: 22 September 2022 Published: 28 September 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Since Japan launched its fifth science and technology basic plan in 2016, the term Society 5.0 has emerged as a way of complementing the societal aspects of the Industry 4.0 era, aiming for a prosperous human-centric society [14]. EU's initiative "Industry 5.0" rests upon the Society 5.0 objectives and values, however pointing out the industry's role in the transition. Our understanding is that Industry 5.0 is based on the Industry 4.0 technologies and address the societal and organizational consequences of the Industry 4.0 paradigm as well as needs stemming from sustainability issues and the society through political objectives [15]. Our question is what will the consequences for the learning of professions be as the society moves towards the Society 5.0 "super smart society" [14,16].

The research into this issue is still somewhat meagre, but there are a few aspects that have bearing upon our project. Ref. [17] points out several characteristics of Society 5.0 learning needs: students' ability to think critically, deductively, and inductively. They also state that students must learn faster and to a high extent through practical experience and underline the need for social competences and a high degree of reflexivity. This supports the claim for Industry 4.0 learning needs, and enhance the need for social skills, reflexivity and experiential learning [18,19].

Approaches to these challenges have been identified on the institutional level [13], identification of needed skills [10], and with a focus on learning mechanisms [20]. However, all of these approaches are descriptive in the sense that they describe the optimal learning setting, yet remain unclear on prescribing a path towards it. Several descriptive approaches to teaching the skills required for Industry 4.0 has been presented, most notably the learning factory [21,22] and teaching factory [23,24] approaches. However, Enke et al. criticises these models for their technology-centric approach [25], where the didactic dimension has been underdeveloped compared to mature technical and operational models. Hence, the learning design for technical education still needs improvement.

Traditionally, learning activities have been developed by the educator, by various models, e.g., SMTTE [26] or the Relational model [27]. Newer approaches involve co-creation, which can be used to create a design for the learning activity [28]. However, a prescriptive approach for educators on how to develop learning activities targeting Industry 4.0 does still call for new approaches, due to the complexity and changing requirements within the field.

This research aims to propose an educational framework that educators can use to guide educators in designing learning and teaching materials for a complex and changing environment while considering the didactic challenges. To define the framework, this article answers the following research question: *What should an educational framework address to make it relevant for Industry 4.0, and how can such a framework contribute to better educational targeting of Industry 4.0?*

The article describes the relationship between industrial development and the corresponding learning processes within technical training and education. This is followed by a conceptual discussion of how the current situation of Industry 4.0 affects education. Based on this, the Educational Framework is developed. This is followed by an empirical test of the framework, including a discussion on its contribution and further research.

### **2. The Development of Learning Processes towards Industry 4.0**

Industrial development has historically significantly affected educational programmes' shape, content, pedagogics and didactics [22]. The most notable success criteria for educational programmes is whether graduates from such programmes will be able to fulfil the future needs in the labour market, hence providing the basis for continued growth and prosperity of the society. Understanding the development of needs in the labour market is an essential guide for creating new approaches and methods for developing teaching material and activities [23].

Historically, the Swedish scholar Lennart Nilsson studied the Swedish public system for vocational education and training (VET) through a comprehensive examination of public documents directing and describing the sector [29,30]. The time scope of the study was from the end of the guild-based education in 1846 to 1980. The study resulted in many interesting findings, including the strong relationship between the organisation and activities in vocational schools and the industry they supported. Based on their findings, Nilsson identified the following defining dimensions to describe the relation:


If we accept the significance of the relation between the industrial paradigm and the educational mode in vocational and professional training according to Nilsson and along the dimensions he proposed, we may deduct characterising features of learning activities, objectives, and outcomes for Industry 4.0.

The basic planning unit is the task of manufacturing highly customised products. In training, this translates to a holistic relation to the finalised authentic products and the system(s) needed to manufacture them. Thus, the learning tasks are to create a final product, not tasks adapted for learning a reduced part of a divided learning outcome. The task is also situated in an authentic context of manufacturing systems and machines, such as one can find in the learning factories, which recently have experienced increased popularity [22]. However, authentic can also be understood in a broader sense, as a problem with a real-world context [31], which allows the learner to relate the new knowledge to their existing knowledge [32].

The organisational structure is characterised by highly integrated, flexible, autonomous and automated production facilities, where the machines and systems communicate both independently of humans and with humans. Due to the high flexibility, Industry 4.0 opens for a cost-effective production of small, customised manufacturing runs, while virtual testing and simulations aid the transition from one product (variant) to another. Some of the tasks are performed in virtual workspaces complementing the physical processes. Humans perform tasks where human capabilities are needed and aid the machines' operations [33].

The structure requires a holistic approach to the complete value chain of the operation, which demands interdisciplinary oversight and competence from the workers and professionals in the manufacturing industry. The work mode is task-oriented, highly autonomous and conducted in inter-disciplinary teams and/or networks covering all domains needed to solve the tasks. Summarily, a new educational design tool needs to address the three following areas:


### **3. Methods**

The research has been conducted as an engaged scholarship [34,35], adding to the quality of higher technical education, as well as providing new knowledge within technical education. To achieve this, a three-step approach was applied. This can also be seen graphically in Figure 1.


**Figure 1.** Flow of the applied three-step method.

**Table 1.** An overview of which characteristics education and instruction had during the four industrial paradigms.


These three activities were evaluated in correspondence with the Framework for Evaluation in Design Science Research for research evaluation [36], which describes that problems with high human factors should be evaluated formatively in a natural environment after a few artificial tests:


ment. The training material is published at https://fagskolene.online/courses/tefficpedagogical-framework-for-industry-4-0/?lang=en (accessed on 21 September 2022).

3. The educators who conducted the 14 pilots performed both summative (learning outcome) and formative (reactions to the course) evaluations of the courses [36]. This was done as a combination between the institution's existing evaluation procedures and course-specific activities composed of the educators. After the course, the gains and challenges were reported to the Transforming Educational programs For Future Industry 4.0 Capabilities (TEFFIC) project management. The compiled evaluation can be found at www.teffic.eu or https://www.ucviden.dk/da/projects/ transforming-educational-programmes-for-future-industry-40-capabi (accessed on 21 September 2022).

The evaluations was carried out at the individual institutions according to their quality insurance guidelines. At all institutions, this included both questioner data as well as openended evaluation at the end of the courses. These evaluations were collected by the educators conducting the educational activities, and processed into an evaluation collecting the learning goals, a learning activity description, and the positive and negative quantitative data as well as any qualitative assessments from the local quality insurance system.

To investigate whether the proposed educational framework meets the demands presented in Table 1, the collected evaluations among educators and students are analysed regarding the four Kirkpatrick levels: reactions, learning, transfer, and performance. Trends that do not fall within these four categories will be described [37]. This analysis was conducted as a Gioia analysis, where the qualitative results from the evaluations was grouped first based on the words used by the individual evaluators, and afterwards towards the four Kirkatrick levels [38].

### **4. The Educational Framework**

As described, both the focus on context and tasks and the agility of the design process are vital elements of an educational framework for Industry 4.0. Hence, the framework development presented in this paper started with a focus on these three elements.

Nilsson [29,30,39] argued that professional and vocational skills and competence are consequences of the human's task-related professions. It implies that the tasks dictate the obtained competencies [29,30,39]. This realisation is at the core of the Educational Framework. One of their major discoveries was that reducing the tasks according to the training within industry principles led to decreased learning outcomes and dropout from the study programmes. These findings support the application of authentic holistic tasks at the core of the learning process. This strategy is supported by contemporary research [40,41]. Merrill [42] later developed an instructional theory named 'First principles of instruction', which give explicit guidance on ensuring a higher learning outcome, taking a point of departure in a task-centred approach. A prominent feature of Merrill's theory is that it places the task as the central element in the learning process. It resonates well with Nilsson's work, as he argued that the task is imperative, developing teaching activities and material. Accordingly, a learning process based on the 'First principles of instruction' places the core task at the centre and relates all learning activities. This describes the organisational structure predicted in Table 1, where interdisciplinary groups work on authentic tasks relevant to future employment. Furthermore, the tasks are where the Industry 4.0 context is materialised by using industrial relevant technologies as both case and learning vessel, as described in, e.g., the learning factory literature [22].

The approach embedded in the model is based on agility, as having an agile approach has proven to hold some measurable advantages over traditional educational development methods regarding handling the ambiguous definitions and complexity of the situation [43]. The agile approach allows understanding of the situation to evolve through the process, which is fundamental when the problem at hand is not fully understood from the beginning. This becomes even more relevant when the problem is a dynamic and adaptable authentic

task, as described in Table 1. Hence, the solution and understanding of the problem are co-developed through an iterative process [44].

The model (see Figure 2) aims to combine a holistic approach to developing teaching activities and material (identifying and analysing the needs and boundaries), focusing on the detailed development of the activities/material (the circle). Moreover, the overall agile approach is combined with a task-centred principle, as task-centred models, e.g., authentic task design, have provided significant learning outcomes within technical educational programmes, creating a natural setting [31,45].

The model's core is the iterative approach toward developing the teaching activities, represented as a circle with five steps in the model (Phase 2—see Figure 2). The other two key phases are, (Phase 1), the initial analysis leading to the learning needs and boundaries, and (Phase 3), the execution of the educational activities, including feedback. Below is a short introduction.

**Figure 2.** The Educational Framework.

The initial analysis establishes the learning needs and boundaries. Based on the results of the analysis, activities and materials are developed. Afterwards, the educator executes the activities in collaboration with the students. Finally, the activities and materials developed are evaluated by both the educator, student, and potentially also peers.

### *4.1. Phase 1*

An analysis is conducted to identify any gap between industry needs and educational capabilities to ensure that the general learning goals are supported by practice. The analysis results in a set of learning needs, boundaries and opportunities within the higher education organisation. This initial analysis can serve as the foundation for several educational activities related to an educational programme. The analysis should not be conducted before developing an educational activity but should be updated regularly to ensure its relevance. The analysis framework must be adjusted to the specific situation, educational program, and educational organisation(s). However, the one used in TEFFIC (The Erasmus+ project co-funding the development of the Educational Framework) can be used as inspiration for developing a focused framework [11].

Phase 1 result in a set of learning needs and goals, combined with the institutional capabilities, which are the boundaries for meaningful educational activities targeting the industry.

### *4.2. Phase 2*

The iterative development of education consists of five steps, see Figure 1. In the following, the process of each of the underlying steps is described in greater detail.


The educational activity is ready for use if it meets the learning goals. If not, the activity must be redesigned.

The choices regarding educational technologies, infrastructure, and resources must be considered in this phase. The initial mapping of educational capabilities revealed that many institutions had access to learning management systems, augmented and virtual reality devices, learning factory equipment and simulational software [11]. The learning management systems allow for several didactical approaches, such as blended learning and flipped classroom. These, and other locally available learning technologies must be evaluated iteratively, as suggested in this phase, as it finds potential flaws in the learning design.

Phase 2 results in a learning design which can be taken into the education and be executed among the students. This includes both what, when, where and how to learn, and which infrastructure is needed.

### *4.3. Phase 3*

The execution uses the developed educational activity as a plan, which is the overall reality check for the plan. After the program's execution, several types of feedback can be obtained. This feedback targets the execution, the planned activity, the learning goals, the learning needs and the industry. Several methods can be used to obtain feedback, including interviews, questionnaires, and workshops. The educators' reflections are also crucial in this process, as the educators and the students are participating in the educational activity at different premises. The test approach must be adjusted to the specific situation, educational program, and educational organisation(s).

The feedback acquired in phase 3 comes in several forms, and serves purposes in both phase 1 and 2: Feedback towards students' reactions to the course and learning outcome targets in phase 2. If the students find the learning activity to be poorly organised, if learning goals are missed, or if positive outcomes need to be maintained, this is used in the following design cycle.

Phase 1 is influenced by graduates' feedback on using the acquired skills and how these can aid the choice of learning needs and goals. Likewise, feedback from the industry can be added to new industrial and institutional analyses.

### **5. Learning Activities and Their Characteristics**

The 14 developed learning activities, their contents and results can be seen in Table 2. Note that 5 of the educational activities were second iterations of the same educational activity, and hence received feedback from the first iteration.

These learning activities all provided the desired results learning wise, implying that the educational framework is use-able for planning technical higher education.

**Table 2.** The 14 conducted educational pilots. Two numbers marked pilot with two iterations.



**Table 2.** *Cont.*

### **6. Learning Outcome and Educational Characteristics**

The evaluation of the 14 pilots can be aggregated into five themes; the four Kirkpatrick levels (1) reactions, (2) learning, (3) transfer, (4) performance [37], and given the circumstances of this case (COVID-19), a fifth theme is added; (5) external circumstances.

The majority of students in all pilots reported positive reactions to the organisation and content of the education. The pilots where the participating institutions performed course satisfaction evaluations, the results showed a 75–85% satisfaction with the courses. This is on par with, or above, the European average before the COVID-19 pandemic of 77% [46]. The students were explicit that they appreciated the cases and mini-projects, as well as the video and podcast material. They also reported that the educational material was well made, and that it fits the style and content of the courses. The high degree of blended learning allowed asynchronous watching of video lectures and listening to

podcasts, to which the students reacted positively. All of this contributed to a higher motivation among the students. A few students noted that the open-ended problems were frustrating, which was the only reoccurring complaint about the courseware. This implies that the organisation of the learning, as described in Table 1, provides the students with a satisfying learning environment.

In addition to the positive reactions to the course, the learning outcome was also high. In one pilot, a student noted that everyone seemed to be able to contribute with their cases afterwards and that this aided the discussion. The ability to revisit video and audio material also aided previously failed students, who highlighted this as a significant improvement compared to traditional auditorium lectures. The reflective dimension of most pilots also aided new insights. When reflective questions were part of the preparation for the day, students came better prepared for discussing the topic. The students put in much effort, combined with the authentic tasks they solved: all pilots meet their learning goals. Positive reactions regarding relevancy and motivation can explain this effect. Furthermore, the broader understanding of, e.g., business or digitalisation was also improved during the activities. Some students also reported that abstract content became more accessible and relevant through the authentic cases, helping them understand it better. This implies that the learning outcome from the structure, task type and work mode in Table 1 have fitted for the subjects.

The transfer from the explicit pilots into the general practice of the students was also seen in the evaluations. After some of the pilots, the students were asked to identify further learning requirements and pointed towards different techniques to master or knowledge to obtain. As mentioned, the students could identify further learning goals, both within the application of the learned skills and knowledge within their projects, as well as new skills to acquire. They became more aware of digital tools and solutions, and in the pilots with integrated interdisciplinarity, they also increased their understanding of other related disciplines. This transfer was also seen in some of the semester projects, which tended to be more digitally-minded than previous semesters. This support that the educational framework targets the characteristics of industry 4.0 education, as described in Table 1. The students' performance upon graduation was not evaluated within the project's timeframe, but students reported confidence in job readiness and ability to perform within the topic of the pilots.

While all the above is positive, many students noted adverse effects from COVID-19 (external circumstances), where hardware, internet connections, organisation and social interaction became challenged. All of this might overshadow potential negative feedback, as in general, well-planned and conducted courses suited for blended learning worked well. Hence, the results should be understood in that context, where minor drawbacks of the planned courses could have been eliminated under other circumstances.

With offset in these themes, it can be seen how this approach can enhance industrial learning for targeting the industry in the 4.0 movement, as described in Table 1. As seen from the learning outcome of the 14 pilots, the nature of communication (both digital and analogue, synchronous and asynchronous) enhanced the learning and, at the same time, added the technologically augmented dimension. Along the same lines, the physical and virtual tasks are also presented in this framework and support learning outcomes and transfer. The autonomous and group-oriented tasks also seem to have increased both positive reactions to the pilots and learning outcomes.

### **7. Discussion and Conclusions**

As the industry turns ever more complex, it alters the educational setting that prepares tomorrow's workers for the industry. Hence, methods and tools for targeting this change can aid educational institutions' transition towards educating for Industry 4.0. To guide educational institutions, this paper presents an agile educational framework that combines a process model with analysis and an iterative building phase with the task-centric approach of authentic task design. The task-centric and agile education approaches have prevailed individually within education. By proposing this combination, we presented a tool that can enhance the students' knowledge by keeping the content up to date compared to the complex industry and by providing the student with authentic problem-solving abilities. Nilsson's [29,30,39] analysis of the learning associated with the earlier industrial paradigms suggests that the educational settings affect the amplitude of learning and what is learned. Hence, the learning approach should fit the desired learning outcome in the teaching approach and have a relevant, authentic task. A known catch regarding authentic tasks is that minor details can draw too much attention. Hence, Enke et al. argue that the authentic task should not wander into details, as this leads to many technical-centric learning activities having less than optimal learning design. If tasks focus too heavily on a few technical aspects, they can leave more general perspectives behind [25]. The iterative approach presented in this paper can limit this wandering, as the iterations will identify this. If the technical details are too pronounced, this can be adjusted in the next iteration.

The analysis shows that the industry requests ever-increasing competencies outside the traditional technical domains, the student's learning, personal, and interdisciplinary skills gain further traction. This Educational Framework supports this by suggesting multidisciplinary student groups. As suggested by Prensky, this work targets several fundamental learning areas: creative thinking, problem-solving, system thinking, mindset, innovation, collaboration, and communication (Prensky, 2014). Neither of these competencies are necessarily required to perform a specific industrial task. As they are not core competencies, they aid the overall task solving. Hence, they are contextual competencies [9] that can aid the worker in everyday tasks.

While this educational framework provides tools for designing new educational activities for educators in higher education, specifically for learning Industry 4.0 skills, then the educational framework should be tested in other contexts where multidisiplinarity is key, and learnings can be supported by authentic task design. In addition, future work should specify two things: (1) Deeper case studies, which can provide details on how to use the framework and why it works, along with (2) test in further education, an other important area for industry 4.0 competence development. A limitation to the study is found in its uses in areas with a more strict requirements for what the learning outcome should be, such as law or healthcare, meaning that the everchanging nature of the task-centric educational framework cannot support this material, but the methods and guides created could still be used to improve the teaching methods.

The proposed Educational Framework is a tool for educators to create educational content targeting future industries, which have in this study been tested across various European countries. According to cultural studies [47], this multidisciplinary way of working is culturally fitting these countries, however, exploring how the adoption and use of the educational framework can be used across countries is highly relevant. This aligns well with the call for a Industry 5.0 or Society 5.0 agenda [14].

**Author Contributions:** Conceptualization, L.C., T.E.H. and E.S.L.; Methodology, L.C., E.S.L., A.H.L. and J.H.K.; Validation, All authors on respective institutions; Formal Analysis, J.G., L.C. and T.E.H.; Data Curation, J.G.; Writing—Original Draft Preparation, L.C., E.S.L. and T.E.H.; Writing—Review & Editing, C.S., A.H.L., T.D.B. and T.O.; Visualization, E.S.L.; Supervision, A.H.L. and T.D.B.; Project Administration, E.S.L.; Funding Acquisition, E.S.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Erasmus+ grant number 2018-1-DK01-KA203-047093.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data can be found at on www.teffic.eu (accessed on 21 September 2022).

**Acknowledgments:** The Educational Framework presented in this article is developed as part of the activities in the TEFFIC project (Transforming Educational Programmes For Future Industry 4.0 Capabilities). The project has been co-founded by EU through Erasmus+ programme (Project Reference: 2018-1-DK01-KA203-047093). More information about the TEFFIC project can be found on www.teffic.eu (accessed on 21 September 2022).

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**


### *Article* **Digital Transformation of the University as a Means of Framing Eco-Environment for Creativity and Creative Activities to Attract and Develop Talented and Skilled Persons**

**Galina Timokhova, Yury Kostyukhin, Elena Sidorova \*, Valery Prokudin, Olga Shipkova, Lyudmila Korshunova and Olga Aleshchenko**

> College of Economics & Industrial Management, University of Science and Technology MISiS, 119049 Moscow, Russia

**\*** Correspondence: ejsidorova@yandex.ru

**Abstract:** The purpose of this article is to present the results of the ongoing study of the University digital transformation on the basis of a comprehensive theoretical model. The article describes a conceptual model of the University digital transformation formed based on the comprehensive quantitative and qualitative analysis of potential ecosystem participants in line with the requirements of changing external conditions. The authors list the current results of the transformation and strategic plans considering any achievements adjustment. The National University of Science and Technology "MISiS" (NUST MISiS) has been used as an experimental basis for the research. The key achievement of NUST MISiS such as a digital ecosystem is described in this article. The digital environment of NUST MISiS considers the needs of the University staff and students and contributes to achieving the strategic goals.

**Keywords:** digital transformation; higher education; educational environment; eco-environment

### **1. Introduction**

Any breakthrough in education is difficult to make. Therefore, the four waves of fundamental transformations can be distinguished within the observable period of time:


The strategic approach of Industry 4.0 doctrine includes the establishment of "smart factories" [1]. Introduction of innovative technologies and processes reduces the human labor involvement and boosts production automation. Therefore, consequently it requires qualitatively new human resources: engineering degree and state-of-the-art knowledge [2–4]. Numerous articles of Russian and foreign scientists prove the positive effect of modern technologies and digitalization processes on the economic development of countries and the global economy as a whole, such effects can be classified as synergetic. Digital technologies show most effect on the following three areas: education, science and inventions [5].

Digital revolution, pandemic, globalization and global economy multiple headwinds have resulted in the fact that the existing models of higher education, deemed as achievements of the second half of the 20th century, are currently losing their efficiency [6]. Just as globalization and technology have affected other huge sectors of the economy over the past 20 years, universities will face the need for fundamental changes in the coming 20 years [7,8]. Universities are challenged by digital transformation: profound re-engineering of business processes implying application of a wide range of digital technologies, and development of new educational process models [9–11]. The University digital transformation processes

**Citation:** Timokhova, G.; Kostyukhin, Y.; Sidorova, E.; Prokudin, V.; Shipkova, O.; Korshunova, L.; Aleshchenko, O. Digital Transformation of the University as a Means of Framing Eco-Environment for Creativity and Creative Activities to Attract and Develop Talented and Skilled Persons. *Educ. Sci.* **2022**, *12*, 562. https://doi.org/10.3390/ educsci12080562

Academic Editors: Maria Limniou and João Piedade

Received: 3 March 2022 Accepted: 5 July 2022 Published: 18 August 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

are focused on shifting from formal teaching models to informal models, including through the innovation design of the educational process based on state-of-the-art technological and teaching solutions aimed at education efficiency enhancement, considering the changing demands of the HR market and the vital need to obtain competing advantages in attracting and retaining the students [12–14]. The COVID-19 pandemic period gave rise to a set of models for digital education transformation [15–17]. We think that the task of the university digital transformation is to smoothly and comfortably apply breakthrough technologies to give rise and boost advanced innovations in education [18–20]. It requires not only digitalization of the current processes by partial immersion into the virtual educational environment; it requires complete transformation through development of the University digital ecosystem [21]. The theoretical and methodological basis for the ecosystem development lies in the synthesis of theoretical concepts in the ecosystem economics and indicative coordination and values control, participatory culture of the organization theory, Kolb's cycle in teaching adults [22]. Moreover, requirements of the present-day employers and standards of education are considered. The environment transformation dynamics determines the need for business units self-organization and self-regulation that, in fact, contributes to the transformation of the units themselves. G.B. Kleiner [6] specifies various social and economic ecosystems as key factors in the future economy. The fundamental relations between the ecosystem components is based not on material and information flows exchange, but on the transfer of rights of access to the existential resources of space and time, as well as rights of access to energy resources—activity and intensity. Management of such conglomerates must be based on soft management principles. While the role of the state must be limited to the indicative coordination and values control. The ecosystem creates the environment which makes it possible for all subjects including students to be able to accept and get adapted to a wide range of new technologies (including, digital skills development) [23], to form and show pro-active behavior both in learning, research and development and professional activities [24].

In this respect, the basis for the university ecosystem development includes scientific research results in behavioral economics of education. By combining digitalization opportunities and achievements in behavioral sciences, it is possible to create a conceptually new educational environment, which is flexible not only in terms of external conditions change, but also in terms of needs and abilities of a student. Investigation of behavioral models, in particular, Fogg behavior model [25], and concepts of soft power (nudging) in general [26,27] based on big data in the educational system to create an innovative educational environment, which contributes to the development of a specialist of a different behavior, on the one hand, and allows for saving money due to enhancing the efficiency of educational programs mastering with the maximum possible students retention degree (until the end of their learning period), on the other hand, are deemed as critical under current conditions.

The pandemic, which preconditioned the applicability of blended learning and digital learning environment, encouraged the development and use of innovations in educational technologies. Contemporary challenges evoke the need for investigating the opportunities and efficiency of blended learning models of teaching students during the COVID-19 pandemic. M. Murata et al. [28] are currently studying the opportunities of educational platforms and modern online communications tools for digital learning. R. Miller et al. [29] highlight the problem of inequality in the access to education among different social strata (low accessibility of digital education for low-income family students). Y. Coiado et al. [30] describe the pandemic attributable innovative educational model based on implementation of the problem-based learning approach to a university digital learning environment. The generalized analysis of digital transformation in the higher education system during the pandemic is given in Castro et al. [31].

The purpose of this article is to present the intermediate results of the ongoing investigation on the University digital transformation (as exemplified by NUST MISiS) based on the transformation comprehensive theoretical model, its adjustment as affected by

the changing external environment, technological innovations as well as feedback from the basic participants involved. The article describes the conceptual University digital transformation model formed based on the comprehensive quantitative and qualitative analysis of requests from potential participants of the ecosystem and requirements of the changing external conditions. The article lists the current results of the University digital transformation and strategic plans considering any adjustment of achievements based on processing of quantitative and qualitative indicators of the feedback on the process in progress.

### **2. Materials and Methods**

NUST MISiS is used as an experimental basis for the research. NUST MISiS has transformed from the industry-based institution of higher education not listed on global ratings into the international research university included in top-500 of the institutional global QS rating and top-100 of QS subject ratings and ARWU over the past 10 years (Figure 1).

**Figure 1.** NUST MISiS transformation in 2010–2020.

For establishing an international research university that performs world-rated research and development, NUST MISiS has opened 40 laboratories and research centers. NUST MISiS has set up an Innovations emergence scientific complex due to investments into science infrastructure, and given rise to an enormous number of scientists: 89 of them are internationally experienced researchers with Hirsh index of over 15. Currently, more than 260 young researchers and academic staff under the age of 39 are involved into activities of the university. The university focuses on increasing the number of publications and improving the quality of publication activities by concentrating on breakthrough areas and front most research projects. The university has been structurally and qualitatively transformed due to update of educational programs, increase in computer sciences number and introduction of mandatory English language learning as per the Cambridge University Press curriculum. NUST MISiS has been the first in Russia to implement the 1C-based comprehensive University management system as well as to develop digital services for students, applicants and University staff as part of the University digital transformation. It has dramatically enhanced the efficiency of administrative functions due to formation of the students digital office. NUST MISiS has provided for students profound involvement in scientific research and has become leader in science promotion by introducing the following activities: Young Researchers Contest ScienceSlam (more than 100 thou. views in social networks), Science and Education Forum "Scientists versus myths", well-established

"Public Christmas lectures". NUST MISiS acted as initiator of the largest festival Maker Faire Moscow (50 thou. visitors in 2019), arranged for courses and workshops hosted by the super modern digital fabrication laboratory FabLab (including activities involving schoolchildren and preschoolers) as part of youth policy and young researchers support. The University participated in the Entrepreneurship Students Festival offering workshops and lectures by external experts. More than 15 thou. people visited the Festival in 2020.

The next stage of the University development within the transformed environment pre-determined the need to develop the conceptual digital transformation model of the University to achieve the strategic goals and generate the ecosystem that can contribute to such achievements. A special research was performed to justify and generate the model. The research was carried out according to the mixed methods approach (Figure 2), subtype: triangulation, convergence model. The choice of the research design is determined by specific features of the data collected by the University as well as by the goal which is data-based decision making aimed to improve students collaboration experience with a particular university (single phenomenon).

**Figure 2.** Research methodology.

The consistent mixed method (including collection of both quantitative and qualitative data) was used in the completed part of the research. The same method is applied for the part of the research that is currently under implementation to triangulate the data as well as to obtain specific information from respondents. The quantitative data on the interaction problems within the digital environment, the degree of target indicators achievement as determined by the strategic tasks and objectives of the University, contributed to the basic problems identification and were used to compile quality questionnaires. Use of open questionnaires makes it possible to ensure individualization and specialization of emerging problems and expectations of the ecosystem subjects involved in the process. Sampling of subjects under investigation was purposive i.e., included determination and selection of persons or groups of persons that possessed some special knowledge or experience in terms of phenomena that were of specific interest for the researcher. In this case, information communication technologies integration into the educational and administrative processes is of specific interest. Therefore, the quantitative data collected across the entire multitude of the involved research subjects made it possible to achieve the target indicators and identify the problems and points of growth; while the qualitative research across the limited target samples contributed to the identification of the factors which formed the basis for the technological solutions to be implemented, as well as made it possible to come to some organizational conclusions used to adjust the development trend by accelerating and enhancing the quality of the digital transformation process and its significance for the subjects involved.

More than 20 expert interviews were performed involving the University staff; 6 focus groups of the University students, the total coverage of students interviewing was >500 persons (~21% of the total number of the students studying under some specific major track; academic staff questioning with the total coverage of >50 persons (~17% of the total number of the academic staff for the said major track), more than 20 thou. messages were received to the freshmen telegram-bot and chat. And the University social networking was subjected to analysis. The analyzed information made it possible to identify the most significant and essential issues that contribute to the formation of the interaction experience within the University as well as to select relevant and effective solutions for improvement. It should be noted that collection and processing of the data on all the aspects of the methods applied as given in Figure 2 are continuously ongoing for the quantitative data while data collection and processing is performed as per the established schedule for the qualitative data. Thus, it allows monitoring the development of the pre-determined model of the University digital transformation trend and to timely respond to the identified need for any activities adjustment.

### **3. Results**

The target model of the NUST MISiS digital transformation developed based on the analysis results is focused on the ecosystem framing for creation and creative activities intended to attract and develop talented persons, as well as on the University transformation into the global center for engineering education and science including the following key parameters:


Therefore, the model of the NUST MISiS digital transformation includes the digital culture (of employees, academic staff and students), digital assets (artificial intelligence, e-platforms, and digital avatars), digital business models, and a digital company (Figure 3).

Below are listed the main items included into the culture of the NUST MISiS digital transformation model:


**Figure 3.** The NUST MISiS digital transformation model.

The NUST MISiS digital transformation shall result in establishing an open university, in management changing, personalization and practice-oriented approach of the educational process, enlargement of opportunities on educational formats (practices) that contribute to the formation of "ultra professional" skills, promotion of interaction among all the participants of the educational process at all the stages, formation of the single unified interface, Omni channels and logical consistency of the services that provide for the basic educational process. All these aspects will essentially contribute to the retention rate rise up to 80% as well as increase in the lifetime value of the client (LTV) (Figure 4).

**Figure 4.** Model of transition to the Open University.

A new IT platform has been generated in NUST MISiS (Figure 5) to ensure transformation into the Open University.

**Figure 5.** The new technological (IT) platform.

This integration platform includes the enterprise service bus and various procedures such as management of services, business processes and business rules provided they are continuously monitored, different micro-services and public interface. A large number of various operating packages will be developed based on this IT platform, such as ERP management systems for educational and R & D activities, CRM-systems including trends for personal development of the academic staff, students and employees of NUST MISiS, business analytics and etc.

This system is supplemented with Omni channel communications and virtual assistants that make it possible to completely robotize all the routine business processes of NUST MISiS. Moreover, conditions will be provided to form a sustainable growth of the university's lifetime value for a person throughout the entire life cycle of the latter (Figure 6).

**Figure 6.** Students' through-University journey.

At the first stage of the "University Introduction" life cycle, a potential applicant may take a virtual tour of NUST MISiS, and arrange for educational programs of various higher educational institutions to be compared. It will make it possible for NUST MISiS to be included in the applicant's universities list. And the applicant will be able to apply online (digital infrastructure developed before the pandemic allows for students online application: more than 90% of the applicants made use of the opportunity in 2019, and 100% of applicants made use of the opportunity in 2020) and monitor the entire process in progress in real time: from documents application to enrolment order issue. It takes 20 min for an applicant to apply. The digital trail of a student is generated when an application is filed to the Enrolment Board and is kept until study termination. The rating of applicants' satisfaction with the enrolment system has been 100% for the past several years.

The university activities are focused on increasing the share of public socially essential services available in the e-format up to 95% in accordance with the national development strategy of the Russian Federation "Digital transformation".

Digital solutions contribute to the improvement of interaction with the university at each of the stage of the students through-university journey:


### **4. Discussion**

NUST MISiS performs the digital transformation of processes to ensure the maximum coverage of the needs of applicants, students and academic staff through identification of the most essential issues thus contributing to the generation of the university interaction experience as well as by selecting relevant and efficient solutions for each of them. The basic elements of the formed ecosystem are consistent with the ideas mentioned in scientific studies on the ecosystem approach, indicative coordination and values control, as well as the requirements of the present-day employers and standards of education under the conditions of the dynamic transformation of the environment [6,7,10,13,14].

Generation of a digital ecosystem, which includes the five basic tracks (Figure 7), is deemed as the key achievement of NUST MISiS.

**Figure 7.** The key achievements of the digital transformation policy.

In terms of intending students admission track, the following can be distinguished as the key results:


Digital achievements in terms of the educational process arrangement include the following:


In terms of digitalization of organizational processes, NUST MISiS is the first university in Russia to implement 1C: University to make management decisions and ensure fast and timely interaction and cooperation of subdivisions on a number of issues: starting from compilation, distribution, agreement and approval of academic and extracurricular loads and ending in automated reporting and analytics; NUST MISiS won the international projects contest on management and accounting automation 1C: Project of the year in 2017; NUST MISiS implemented services of E-Credit Book and E-Report Card as part of their policy on abandoning paper versions throughout the documents management.

In terms of online study, the university digital environment is developed—it is a space for testing new technologies and experiments. In case of successful testing results, the innovations are introduced in the existing educational programs. These include creation of new educational digital systems, learning analytics and proctoring, integration of virtual and augmented reality into the educational process and gamification. The University has launched 40 online courses for more than 450 thou. participants and attracted more than 12 higher educational institutions as partners throughout Russia.

According to the current results of the University digital transformation monitoring, the basic trends of the digital transformation at the present moment are: change in students expectations (total experience), technological progress, request for Omni channel environment and external factors effect (Figure 8).

**Figure 8.** External effect.

The Omnichannel environment (Figure 9) is based on the established culture of using a variety of communication channels.

**Figure 9.** The Omnichannel environment.

Teacher-student interaction is executed through the set of channels by their mutual agreement, including LMS Canvas, CHAT, WEB, and e-mail. Interaction of NUST MISiS staff within the university is implemented by various communication means: 1C: UniversityProf, student/University employee personal account, creation of a single account in various systems (LMS Canvas, corporate e-mail, Google G Suite, Office 365 and etc.). The unified system of the documents management DIRECTUM has been implemented and the information portal is used in the University. The management system for scientific activities ISUNTP has been built in the University: the entire process of scientific projects management starting from participants applications preparation, issuing scientific research information control cards and ending in report forms uploading is performed in the system; the Project Management information system, platform visualization system, cloud file storage, videoconference system and etc. are implemented in the University.

By boosting the existing drivers for the digital transformation, the University goes deeper with its comprehensive activities in developing the established ecosystem including through technological solutions integration in the unified electronic information and educational environment LMS Canvas (lms.misis.ru). It is definitely the basic linking element and main working platform for the entire multitude of the University operation processes.

LMS Canvas delivers access to curricula, academic courses and practice working programs, learning and teaching materials and tasks provided by the academic staff. LMS Canvas provides for recording the progress of the educational process as well as interaction between students and teachers.

In addition to LMS Canvas, students and the University academic staff may make use of the University e-library web-site http://elibrary.misis.ru/login.php (accessed on 4 July 2022), this resource grants access to the electronic teaching and learning publications. The e-library is a different specific service which is also used by the University students and staff.

The number of students registered with LMS Canvas is 6361 (parent OO). The number of teachers registered with LMS Canvas is 545 regular teachers, and 31 teachers who are external part-timers.

The digital educational environment is developed based on integration of the electronic teaching and learning management system LMS Canvas (https://lms.misis.ru (accessed on 4 July 2022)) and the automated information system 1C: University Prof in NUST MISiS, special services for the educational process planning and management, webinars arrangement sub-systems based on Adobe Connect Pro Meeting, library and information resources. The teaching and learning platform LMS Canvas of NUST MISiS is a technological basis for the educational process implementation and is widely applied to manage students independent activities and blended learning.

Development of the digital environment of NUST MISiS as a friendly and comfortable ecosystem of the University takes the needs of all the staff and students of the University into account and provides for automated and efficient management of the University key processes.

The NUST MISiS information system creates opportunities to access the personal and reference information posted in students and University staff personal accounts, as well as to receive services in the single-window mode.

Students' personal accounts contain personalized information about the educational process (schedule, announcements, access to educational materials). Students can remotely request for the required statements, control the University subscriptions, and monitor the current events.

Subsequently, implementation of the developed digital transformation of NUST MISiS running based on big data will make it possible to review and change the approach to the educational activities arrangement and implementation, namely to enhance the efficiency of interaction with employers, prepare university graduates of some unique competences, develop the progressive culture available for innovations, set up conditions for continuous professional and personal growth and development (Figure 10).

**Figure 10.** New interaction culture.

Priorities of the digital transformation are shown in Figure 11.

**Figure 11.** Priorities of the NUST MISiS digital transformation.

Therefore, priorities of the NUST MISiS digital transformation include:


The listed priorities are in line with the general trends of the digital transformation identified during the research investigation; please see the review at the beginning of the article [1–31].

Concept-based computations will not work and give appropriate results in achieving the set targets unless they are transformed into specific steps of actions on model implementation in practice. Below is the list of planned activities on digital transformation of a higher educational institution and their brief characteristics.

1. Creation of Personal learning tracks service within the corporate information environment. Implementation of the educational program that allows modeling of its content, evaluation of economic indicators, generation of portfolios of possible educational

tracks based on data analysis for external and internal educational environment of the University.


control, vehicle passage control and visitor entrance control. Availability for students and teachers shall be ensured 24/7.


The listed steps fully implement the conceptual model of the university transformation and are consistent with the results and recommendations of the scientific research results in behavioral economics of education [25–27].

Therefore, by 2030 two vectors of the digital transformation will be implemented: education and clients experience improvement [32].

In terms of transformation of the educational activities, it is planned to achieve the 5 basic vectors:


As part of the clients experience improvement, it is planned to reduce paperwork for teachers, administration staff and students as well as to implement the unified interface, omnichannel and logical consistency of services that provide for the basic educational process, that is absolutely consistent with the general trends identified in research investigations [33–35]. Besides, it is planned to increase the retention rate up to 80% and shift to the open model.

All these actions will make it possible for the university to reach a decent level of competitiveness in the constantly changing external environment and become attractive and appealing for all the interested parties through showing the high performance.

### **5. Conclusions**

Based on the comprehensive quantitative and qualitative analysis of requests from potential participants of the ecosystem as well as based on the analysis of the requirements of external changing conditions, a comprehensive model for the university digital transformation including the digital culture, digital content in the university digital transformation, digital business models and digital company, has been developed and is being implemented in NUST MISiS. Updated state-of-the-art educational management systems provide an opportunity for promoting teachers and students' collaboration to an absolutely new level developing future graduates' competences which can be in great demand at the HR market. The blended learning and e-learning can be successfully implemented provided that following conditions are met: the academic staff shall have appropriate information and communication competences and project-specific expert knowledge; application of electronic educational courses (with advanced functions) shall be encouraged and supported by the university administration; and new science-based pedagogical practices shall be introduced. The increasing number of committed learners is an incentive for updating the content of academic courses programs, for transforming and improving collaboration as part of learning and research activities in compliance with the changing requirements of the labour market and technology opportunities of the contemporary digital economy. The scope of research is limited by the limited number of respondents selected for the investigation—participants of the NUST MISiS ecosystem. However, the experience gained can be of some help for other universities that are just at the outset of their digital transformation. We consider a comprehensive comparative study of the transformation processes of the universities combined by different features, including their size, status, geographic position and etc., as a prospective for our further investigation to identify the characteristic features of the digital transformation processes management and control.

**Author Contributions:** Conceptualization, G.T., Y.K. and E.S.; methodology, Y.K. and G.T.; validation, E.S.; formal analysis, V.P. and Y.K.; investigation, V.P. and E.S.; data curation, O.S.; writing—original draft preparation, O.S., L.K. and E.S.; writing—review and editing, O.A.; visualization, V.P.; supervision, E.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**


### *Article* **Implementing Digital Competencies in University Science Education Seminars Following the DiKoLAN Framework**

**Anna Henne 1, Philipp Möhrke 2, Lars-Jochen Thoms 1,3,\* and Johannes Huwer 1,3,\***

	- philipp.moehrke@uni-konstanz.de

**Abstract:** Prospective teachers must acquire subject-specific digital competencies to design contemporary lessons and to promote digital competencies among students themselves. The DiKoLAN framework (Digital Competencies for Teaching in Science Education) describes basic digital competencies for the teaching profession in the natural sciences precisely for this purpose. In this article, we describe the development, implementation, and evaluation of a university course based on DiKoLAN which promotes the digital competencies of science teachers. As an example, the learning module Data Processing in Science Education is presented, and its effectiveness is investigated. For this purpose, we used a questionnaire developed by the *Working Group Digital Core Competencies* to measure self-efficacy, which can also be used in the future to promote digital competencies among pre-service teachers. The course evaluation showed a positive increase in the students' self-efficacy expectations. Overall, the paper thus contributes to teacher education by using the course as a best-practice example—a blueprint for designing new courses and for implementing a test instrument for a valid evaluation.

**Keywords:** Technological Pedagogical Content Knowledge; science education; student teachers; self-report measure

### **1. Introduction**

More and more schools are equipped with a continuously improving digital infrastructure including school-wide wireless network access, school cloud storage, interactive whiteboards, video projectors, and devices such as computers, laptops, or tablet computers. This opens up a lot of new opportunities but at the same time requires teachers to be trained in new or adapted competencies to fruitfully utilise these digital tools. These competencies are described in various frameworks such as UNESCO's ICT Competency Framework for Teachers [1], the ISTE Standards for Educators [2], or the European Competence Framework for Educators (DigCompEdu) [3], all of which focus on slightly different aspects of the competence needed by teachers for making maximum use of the digital environment. In addition to those generic non-subject-specific frameworks, the DiKoLAN framework (Digital Competencies for Teaching in Science Education) focuses on digital competence for teaching the natural sciences [4,5].

Despite belonging to the generation of so called 'digital natives,' today's young teachers need explicit instruction on how to productively use digital technology in schools [6,7]. Most researchers agree that digital technology needs to be integrated in teacher education curricula, and numerous strategies have been proposed in the literature to facilitate this effort [8]. To address the specific needs of science teachers, the DiKoLAN framework (Figure 1) gives a comprehensive guideline on the topics to be addressed [5]. This guideline has been used to design, teach, and evaluate a course for students in teacher education in the three natural sciences at the University of Konstanz.

**Citation:** Henne, A.; Möhrke, P.; Thoms, L.-J.; Huwer, J. Implementing Digital Competencies in University Science Education Seminars Following the DiKoLAN Framework. *Educ. Sci.* **2022**, *12*, 356. https:// doi.org/10.3390/educsci12050356

Academic Editor: Maria Limniou

Received: 20 March 2022 Accepted: 9 May 2022 Published: 18 May 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

The aim of this research paper is to provide an overview of the current research on the DiKoLAN framework, as well as to present the design and the evaluation of a special pre-service teacher training course tailored to foster the digital competencies described in DiKoLAN. Additionally, the investigation of the effectiveness of the individual learning modules offers a blueprint for future research on the effectiveness of university teacher training on the subject-specific use of ICT in science education.

### **2. Research following the DiKoLAN Framework**

The DiKoLAN framework was first presented in 2020 by the *Working Group Digital Core Competencies* [4]. The framework was first developed for Germany and Austria and later introduced in Switzerland [9]. It was based on initiatives to promote digitisation in schools and to promote the digital competencies of prospective teachers and also based on DigiCompEdu [3], the TPACK framework [10,11], and the DPaCK model [12,13].

The curricular integration of essential digital competencies into the first phase of teacher education requires specific preliminary considerations. To be able to integrate ICT-related elements of future-proof education into the teaching practices of all faculty involved in teacher training at universities, basic digital competencies need to be structured in advance [14].

Based on core elements of the natural sciences, the authors of DiKoLAN propose seven central competency areas [15]: *Documentation*, *Presentation*, *Communication/Collaboration*, *Information Search and Evaluation*, *Data Acquisition*, *Data Processing*, and *Simulation and Modelling* (Figure 1). These seven central competency areas are framed by *Technical Core Competencies* and the *Legal Framework*. The unique feature of DiKoLAN is that the DPaCKrelated competencies are described in great detail and take into account subject-specific, subject-didactic (e.g., [16,17]), and pedagogical perspectives from all three natural sciences (biology, chemistry, and physics).

The framework thus coordinates and structures university curricula [14,15], which has been demonstrated, e.g., for the competency area *Presentation* [18], using the example of low-cost photometry with a smartphone [19], or by means of a project on scientific work [20,21]. Such coordination makes cooperation between different universities, which has been suggested by Zimmermann et al., possible without any significant difficulties [22].

For an overview measurement of DiKoLAN competencies, the self-assessment tool *DiKoLAN-Grid* is available [5], which helps to illustrate respective learning goals in teacher training to pre-service teachers.

Initial empirical studies support the factorial separation of the application areas according to the TPACK and DPaCK frameworks into *Teaching*, *Methods/Digitality*, *Content-specific context*, and *Special technology* [5,18].

### **3. Methods**

In this section, two important methodological aspects are presented: the design of the course and the evaluation of the course using an online self-assessment of digital competencies.

### *3.1. Design of the Master-Course "Science Education III—DiKoLAN"*

The aim of the seminar is to promote digital core competencies in science teaching following the DiKoLAN framework [5]. The students should be made aware of the individual competencies of digital teaching and learning and reflect on their own competencies. Skills that go beyond declarative knowledge are to be acquired through practical phases. Finally, students should reflect on the methods and tools used, and what has been learned should be transferred and related to the school context.

The seminar on didactics was implemented in the summer term of 2021 for advanced student teachers in the natural sciences at the University of Konstanz. Students received 5 ECTS credits for the module, which corresponds to an average weekly workload of 10 h. Two of these hours were spent on synchronous teaching with the entire course, while the remaining time was used for preparation and follow-up, including all exercises. Figure 2 illustrates the phase structure of the 14-week seminar. It starts with a synchronous initial phase, which aims to impart skills. At the beginning, the students get an introduction into learning with and about digital media in science education, including the DiKoLAN competence framework. After the introductory week, one area of competency is highlighted in weekly meetings, which are partly framed by preparatory tasks and further exercises. In the subsequent asynchronous project phase with individual support and advice, the students design a learning scenario, consider the didactic function of the media used, and reflect on the skills required of the teacher and the pupils. In the final examination phase, the designed lesson is presented to the seminar plenum, the learning scenario is implemented in a trial lesson, and a written elaboration is submitted.

**Figure 2.** Phase structure of the seminar.

### 3.1.1. Introductory Module

In the first module, background information is given on the use of ICT in the science classroom, the current situation regarding digital media in schools is examined [23], and initial frameworks such as SAMR [24], ICAP [25], TPACK [10,11,26], and DPaCK [13] are presented and critically questioned. Moreover, the approach to the integration of digital media in the classroom is illuminated, and the didactic functions of digital media in science are explained [27].

### 3.1.2. Workshop Phase: Overview of Modules on Areas of Competencies

In the module on the competency area of *Documentation* (DOC), the data storage processes (from documentation to versioning to archiving) are scrutinised, the documentation of experiments with a digital worksheet is introduced [28], and the documentation of experiments, specifically by students themselves using videos called *EXPlainistry* [29], is presented. As it can be assumed from previous surveys that advanced students already have basic knowledge in the field of documentation [30], the focus in this module is less on the technical aspects and more on the subject-specific context, questions of methods and digitality, and, above all, the integration of documentation techniques into teaching.

The module on the second competency area, *Presentation* (PRE), includes a discussion of the available hardware at schools for presentation and possible scenarios in which digital media are used for presentation. Theoretical principles are presented on multimedia, especially multimodality (which, despite its proven effectiveness, is surprisingly rarely mentioned in physics teacher journals [31]) and multicodality [32,33], as well as cognitive load theory [34]. Recommendations for action on text and image design [33] are presented. Since a certain prior knowledge can also be assumed in this competency area [30], the focus is on presentation forms specific to the natural sciences and methodological aspects.

The third module on *Communication and Collaboration* (COM) revolves around planning collaborative learning settings [35]. Tools for the collaborative editing of texts, mind maps, pin boards, wikis, online whiteboards, and learning management systems are presented and tried out. Finally, different accompanying communication channels between students and the teacher are discussed.

The *Information Search and Evaluation* (ISE) module focuses on the five steps of digital research using the IPS-I model [36]. Various scientific and science didactic databases are presented, and examples of different types of literature are examined. Since it can be assumed that advanced students have a basic background in this area of competency [30], the focus in this module is on methodological issues and integration into lesson planning.

In the module for *Data Acquisition* (DAQ), the possibilities of data acquisition are discussed, especially using a smartphone (e.g., [19–21,37–40]). Various options such as video analysis or taking measurements using an app are tried out. Experimentation in the Remote Lab is also introduced [41]. Furthermore, the necessary steps of teaching with digital data acquisition and the possibilities and challenges of teaching in this manner are discussed.

The penultimate module, *Data Processing* (DAP), presents different coding options for characters and numbers as well as typical problems that arise when importing data, which the students test by using an iPad. The differences between pixel and vector graphics are discussed. The focus is on the structure of the formats, i.e., xml and mp4.

In the last module, digital tools for *Simulation and Modelling* (SIM) are presented along with the competence expectations listed in DiKoLAN and tested in the exercises. Tools are discussed for which empirical findings are available [42–46] or which have already been successfully integrated in other DiKoLAN-oriented teaching concepts [47,48]. The tool types presented are spreadsheet programs, modelling systems, computer simulations, *StopMotion* programs [49], and programs for digital modelling and animation. In addition, Augmented Reality (AR) is discussed as a technique for representing models [50–52].

### 3.1.3. Free-Work Phase: Designing a Lesson Plan

In the free work phase, teams of two students design a lesson on a scenario of their own choosing. In doing so, they are asked to consider what the benefit of using digital media in the learning unit would be for the students and what skills the teaching staff need. During the process, the students write a seminar paper in which they present the scenario and the associated planning and also explain their approach and why they considered

the planning to be didactically appropriate. Throughout the 4 weeks before the exam, the supervisors are available for individual coaching, which is used by students to varying degrees. All materials needed for the lesson are to be created and turned in, even if the lesson is not completely implemented.

### 3.1.4. Presenting the Lesson Plan in a Mock Trial

Finally, the students present their plans at a block meeting. Each participant in the seminar plenum is asked to try out the digital elements of the teaching scenario for themselves as completely as possible. For the supervisors, the following questions play a role in the evaluation:

	- a. Did the students actually create material on their own?
	- b. How much effort was invested in terms of content/time?
	- a. Is there a specific purpose served by the digitalisation?
	- b. Is the media use didactically sensible?

Both the presentations and the written assignments, which have to be handed in before the first presentation, are considered in the evaluation.

### *3.2. Design of the Individual Modules (Using the Example of Data Processing)*

For each workshop, the areas to be covered in the module are selected based on the competency expectations defined in the DiKoLAN framework. When deriving the learning objectives of a module from the orientation framework, three categories were distinguished: Main learning objectives, secondary learning goals, and non-addressed competency expectations (see Figure 3 for an example). Using the area of data processing as an example, the majority of competencies on the level of *Name* and *Describe* are covered in a lecture. For instance, relevant software is introduced and data types common in the context of teaching the natural sciences are shown. Additionally, typical scenarios for the application of digital data processing appropriate to the school curriculum are shown. As an accompaniment to this part, in-lecture and at-home activities are designed to allow for timely application of the topics learned. This includes drawing on an example from data processing, exporting data from digital data acquisition applications, and importing the data into spreadsheet software. There, the data are manipulated by performing various analyses.

To get a first impression of the students' previous experience, the students are asked in the introductory phase to identify which data processing software they have used before and which data manipulations they already know. In the next step, relevant software is introduced, and data types common in the context of teaching and natural sciences are shown. For this purpose, the export and import of data is presented in the first input phase using the example of csv files in the *MeasureAPP* app [53]. In the following phase, common issues related to tablets and data storage locations are addressed. In this context, the difference between csv and Excel files is highlighted. Examples are used to introduce the integer and float number formats. In particular, the coding of characters and numbers is discussed in this context. At the end of the first input phase, the visualisation of data using Excel [54] is demonstrated.


**Figure 3.** Competence expectations defined in the DiKoLAN framework addressed in the respective teaching module *Data Processing* (DAP). Main topics (magenta), side topics (blue).

Using the integrated microphone of the iPad, the students record an audio oscilloscope of a sung vowel sound in individual work during the practice phase using the *phyphox* app [55]. They then export the measurement data as a csv file and then import it into Excel to display the data graphically.

In the second input phase, ways of calculating new data in Excel and using spreadsheets to analyse data are demonstrated, including the aspects of measurement uncertainties, statistics, and regression. The instruction is concluded with an introduction to the

differentiation of formats for images into vector and pixel graphics and to the structure of video formats as containers.

In a final step, the challenges students have encountered so far during the acquiring and processing of measurement data were discussed, and possible solutions were shown.

As a follow-up task, the students recorded a series of measurements of a cooling teacup from which they are to determine the mean decay constant using a spreadsheet program of their choice.

With these initial practical experiences and theoretical foundations from the areas of *Name* and *Describe*, the students then set about working out teaching scenarios in the further course of the seminar to consolidate and extend the skills they have acquired in each module.

### *3.3. Evaluation*

To investigate the effectiveness of the newly designed teaching-learning modules, the change in the participants' self-efficacy expectations is used as a measure of effectiveness and is measured with an online test provided by the *Working Group Digital Core Competencies* [5]. So, the question to be answered is: Is it possible to measure a significant increase in students' self-efficacy expectations in relation to the competences covered in the course? Due to the structure of the seminar, a large effect on students' self-efficacy expectations is assumed for the main learning objectives, a medium effect for the secondary learning goals, and no effects for the areas not addressed.

The measurement of self-efficacy expectation was chosen for two reasons. First, it is precisely self-efficacy expectation that is influenced by experiences during studies and thus ultimately also has an effect on motivational orientation towards the later use of ICT and digital media in one's own teaching [30]. Second, the subject-specific self-efficacy expectation can be assessed much more economically than a specific competency itself [31]. Accordingly, most of the digital competence questionnaires published so far measure self-efficacy expectations, e.g., [5,56–62].

The individual items are based on the competence expectations contained in DiKoLAN and are designed as Likert items. The participants indicate on an eight-point scale their agreement with a statement that describes their ability in the corresponding competence expectation, e.g.,


The items of the questionnaire can each be directly assigned to a single competence expectation. The naming of the items in the data set created in the survey follows the nomenclature in the tables with competence expectations listed in DiKoLAN (Figure 4).

Many competency expectations cover several individual aspects or are described using several examples. In such cases, several items were created, which, taken together, cover the competence expectation as a whole.

The questionnaire was implemented as an online survey with LimeSurvey [63] and made available to the participants of the course in each case as a pre-test in the week before the synchronous seminar session via individual e-mail invitation. Seven days later, the students received the same questionnaire again as a post-test.


**Figure 4.** The nomenclature of competence expectations used in DiKoLAN [4,5]. Adapted with permission from Ref. [4]. © 2020 Joachim Herz Stiftung.

It was hypothesised that the participants would have a higher self-efficacy expectation in the competency areas addressed in the respective modules after the intervention than before. It is also assumed that large effects can be measured for the main learning objectives, whereas at least medium effects can be measured for the secondary learning objectives, the acquisition of which can only be attributed to the brief learning time in the seminar.

### **4. Results**

### *4.1. Sample*

The participants included *N* = 16 pre-service German Gymnasium teachers for science subjects who participated in the newly designed seminar on promoting digital core competencies for teaching in science education according to the DiKoLAN framework. The course is developed for Master's students in the 1st or 2nd semester but is also open for Bachelor's students in the 5th or 6th semester. More than three quarters of the students participated in the voluntary pre- and post-test surveys. However, three participants failed to complete the single surveys. Hence, data from those participants were removed, resulting in a final total of *n* = 13 participants (5 male, 8 female, aged *M* = 23.5 (*SD* = 2.9) years). These 13 participants indicated they studied the following science subjects (multiple answers possible; usually, students must study two subjects): 10 Biology (76.9%), 6 Chemistry (46.2%), 1 Physics (7.7%), and 1 Mathematics (7.7%). They were attending the following semesters at the time of the study: 5th BEd (1; 7.7%), 6th BEd (1; 7.7%), 1st MEd (6; 46.2%), 2nd MEd (4; 30.8%), or 3rd MEd (1; 7.7%).

### *4.2. Statistical Analysis*

The responses were analysed using R statistical software [64]. Means and standard deviations were computed for each item in the pre-tests and post-tests. Wilcoxon signed-rank tests were conducted for each pre-test post-test item pair to test for growth in item means.

The results of the descriptive and inferential statistics are listed in tables in the Appendices A–G. As an example, the results for the competency area Data Processing (DAP) are also presented here.

### 4.2.1. Data Processing (DAP)

Table 1 shows the results for the main learning objectives, and the results for the secondary learning goals are listed in Table 2 (for an overview, the main and secondary learning objectives are marked in the respective table of competence expectations, Figure 3). If several items of the questionnaire can be assigned to a competence expectation listed in DiKoLAN, a mean effect size averaged over the associated Wilcoxon signed-rank tests

(in italics) is given in addition to the effect sizes of the individual Wilcoxon signed-rank tests. For example, the competency expectation DAP.S.N2 ("Name digital tools [ ... ]") is assessed with seven items, DAP.S.N2a-g, which reflect the individual examples mentioned in DiKoLAN (e.g., "Filtering", "Calculation of new variables", . . . ).

**Table 1.** Results of Wilcoxon signed-rank tests for competencies in the area of *Data Processing* (DAP) explicitly addressed as main learning objectives in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


Note: ◦ not tested. \* The average effect size is given for competencies assessed with more than one item.

The results show that there is an increase in self-efficacy expectations in all of the competency expectations addressed as the main learning objectives in the module. All of the tested hypotheses can be accepted.

According to Cohen, the effect sizes determined as correlation coefficient *r* can be roughly interpreted as follows: 0.10 → small effect, 0.3 → medium effect, and 0.50 → large effect [65] (p. 532). However, it must be taken into account that the interpretation of effect sizes should always depend on the context [65]. Since the learning goals addressed in the intervention and the tested self-efficacy expectations were both derived from the competency expectations defined in DiKoLAN and thus correlate very highly, larger overall effects are to be expected than in other studies. Therefore, we raise the thresholds for the classification of the observed effects into small, medium, and large effects for the following evaluations as follows: 0.20 → small effect, 0.40 → medium effect, and 0.60 → large effect.


**Table 2.** Results of Wilcoxon signed-rank tests for competencies in the area of *Data Processing* (DAP) addressed as secondary learning goals in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.

Note: \* the average effect size is given for competencies assessed with more than one item.

The effect sizes of the intervention in this area are always 0.62 or higher if the mean effect size is considered for broken down sub-competencies. Hence, the hypothesised growth in self-efficacy can be observed with large effects of the intervention.

The results of the Wilcoxon signed-rank tests for the secondary learning goals show significant increases in self-efficacy for most of the hypotheses tested. Where single hypotheses must be rejected, only partial aspects of a competence expectation were addressed, as can be expected for a secondary learning objective. The averaged effect sizes mostly show medium effects of the intervention on self-efficacy expectations in these areas, as hypothesised.

For comparison, the mean values of the self-efficacy expectations in sub-competencies not explicitly addressed in the course are listed and examined for differences in mean values (Table 3). As expected, no significant differences are observed between the two test times.

**Table 3.** Results of Wilcoxon signed-rank tests for competencies in the area of *Data Processing* (DAP) NOT explicitly addressed in the respective module and thus NOT hypothesised to change during intervention (for comparison). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


Note: \* assessed with more than one item. ◦ Not tested.

For a better overview, the averaged effect sizes are clearly plotted in Table 4.

**Table 4.** Overview of (average) effect sizes of the effects of the intervention on the competence expectations in the area of *Data Processing* (DAP). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


Note: main learning objectives (**bold magenta**), secondary learning goals (*italic cyan*), and non-addressed competencies (yellow). \* The average effect size is given for competencies assessed with more than one item. ◦ Not tested.

### 4.2.2. Documentation (DOC)

Due to the students' previous experience, which is expected to be well developed (the comparatively high item means in the pre-test support this assumption), the focus in this module is less on the technical aspects and more on the areas of *Teaching*, *Methods/Digitality*, and *Content-specific context* (Figure A1). For the main learning objectives, large effects of the intervention are observed, in line with the expectations (Table A1). As expected, mostly medium (average) effects were measured for the secondary learning objectives (Table A2). The measured effects also show, for example, that within the sub-competency DOC.S.N1, the focus was specifically on versioning management and the possibilities of using corresponding tools, which is why a particularly large effect is measurable for item DOC.S.N1c ("I can name technical options for version management and file archiving (e.g., file naming with sequential numbering, date-based file names, Windows file version history, Apple Time Machine, Subversion, Git, etc.).") but not for DOC.S.N1a ("I can name technical possibilities for digital documentation of e.g., protocols, experiments, data or analysis processes (e.g., using a word processor, a spreadsheet, OneNote, Etherpad).") and DOC.S.N1b ("I can name technical options for permanent data storage and corresponding software offers/archives (e.g., network storage, archiving servers, cloud storage)."). As expected, there were no significant differences in the pre-test and post-test results for the sub-competencies that were not addressed (Table A3).

### 4.2.3. Presentation (PRE)

In the competency area of presentation, as expected, the item mean values in the pre-test are also quite high in some cases, and the students rate their own competencies in this area quite highly. Hence, the main learning objectives are in the areas of *Teaching*, *Methods/Digitality*, and *Content-specific context* (Figure A2). The intervention achieved strong (averaged) effects on the self-efficacy expectations for all main learning objectives (Table A5). Even if not all facets of a sub-competency can always be recorded (PRE.C.N1, PRE.C.D1), a clear increase can still be observed on average. As expected, mostly medium effects are achieved for the secondary learning goals (Table A6). The sub-competencies that were not addressed show no differences except for one (Table A7). Only the item PRE.S.A1c ("I can

set up and use at least one tool/system to represent processes on different time scales.") shows a clear increase in self-efficacy expectations.

### 4.2.4. Communication and Collaboration (COM)

In the module on the competency area of *Communication/Collaboration*, three central topics are placed in the foreground: firstly, the use of digital technologies for joint work on documents (by students as well as among colleagues) and the associated requirements, secondly, the instruction of students to communicate with each other, and thirdly, the exemplary integration into lesson planning. While mainly technical issues and tools are discussed and tested as the main learning objectives, methodological-didactic issues can only be considered on the basis of individual examples. Accordingly, the main learning objectives concentrate on the area of special tools (Figure A3).

The results show no significant improvement in self-efficacy expectations in the learning areas of the main learning objectives (Table A9). For the secondary learning goals, the picture is mixed (Table A10). Although there is a significant effect of the intervention on the assessment of the ability to integrate communication and collaboration into lesson planning (COM.T.A1), it is precisely in the case of the very complex learning objectives (COM.M.N1 and COM.M.D1) that no (or only smaller) effects can be observed in individual sub-aspects. In the competence expectations that were not addressed, no significant differences between the test times can be measured (Table A11).

Overall, it should be noted that the participants already assess their abilities as comparatively high in the pre-test.

### 4.2.5. Information Search and Evaluation (ISE)

The focus of the module *Information Search and Evaluation* is clearly on methodology and lesson planning (Figure A4). The analyses show large effects of the intervention in almost all sub-competencies addressed as the main learning objective (Table A13). As expected, medium effects were observed for the secondary learning objectives (Table A14). In areas that were not addressed, no differences were found between pre-test and post-test (Table A15).

### 4.2.6. Data Acquisition (DAQ)

In the *Data Acquisition* module, a variety of possibilities for the acquisition of measurement data—especially in distance learning—are presented, discussed, and tried out as examples (Figure A5). Accordingly, the contents of the main learning objectives, which all lie in the technical area, can only be briefly touched upon. In individual sub-aspects of the sub-competencies, pronounced effects can be seen, but the average effect strengths are in the range of medium effects (Table A17). Medium effects of the intervention on self-efficacy expectations can also be observed for the secondary learning goals (Table A18). As expected, in the sub-competencies that were not addressed, no differences are registered between the two test times (Table A19).

### 4.2.7. Simulation and Modelling (SIM)

Figure A7 shows the competency expectations addressed in the module *Simulation and Modelling* and distinguishes between main and secondary learning objectives. In the main learning objectives, the intervention results in an increase in self-efficacy expectations with large effect sizes (Table A25). For the secondary learning goals, the intervention had medium to large effects, exceeding expectations (Table A26). For the competence expectations that were not addressed, no significant differences can be determined between the test times (Table A27).

### **5. Discussion**

This section first discusses the effects observed across all modules and the general classification into main and secondary learning objectives. Then, the individual modules are discussed, and implications for improving the teaching-learning modules as well as for designing and developing similar teaching-learning units to promote digital competences are given.

### *5.1. Joint Discussion of the Results of all Modules and the Separation in Main and Secondary Learning Objectives*

### 5.1.1. Effectiveness of the Interventions for the Main Learning Objectives

Overall, the results are largely in line with the expectations. In five of the seven central competency areas (DOC, PRE, ISE, DAP, and SIM), the expected increase in the students' self-efficacy expectation was observed in all main learning objectives with large effects (*r* of 0.60 to 0.91). However, it should be noted that, in some cases, not all aspects of a main learning objective can be addressed, so the effect sizes for individual items may well be lower (*r* of 0.26 to 0.91), even if the averaged effect over all items depicting the competence expectation can nevertheless be considered a large effect.

Only in the competency area *Communication/Collaboration* (COM) does the intervention not lead to a significant increase in self-efficacy expectations in the main learning objectives. It should be noted that the item mean values are already extremely high in the pre-test, which means that the students consider their own abilities in this area to be very high even before the intervention. A similar picture emerges for the secondary learning goals, even though an effect of the intervention can certainly be recognised. Therefore, the competency area Communication/Collaboration (COM) will not be considered in the following observations, and this module will be discussed again afterwards.

### 5.1.2. Effectiveness of the Intervention in the Secondary Learning Objectives

For the secondary learning objectives, the expected picture also emerges for five of the seven central competency areas (DOC, PRE, DAQ, DAP, SIM). For learning objectives that are only tested with one item, the observed effect sizes are in the medium range, as expected (*r* from 0.40 to 0.67). In the module *Information Search and Evaluation* (ISE), contrary to the hypothesis, no significant increase in self-efficacy expectations was observed for the learning objective ISE.C.N2 ("Name several literature databases or search engines [ . . . ]"), although this was clearly the content of the course. However, the students already indicated a comparatively high level of prior knowledge in the pre-test.

In the case of secondary learning objectives, which are regarded as such because only individual selected examples are deepened within the sub-competency areas, the effect sizes to be expected vary accordingly when comparing the items assigned to this learning objective with each other. This observation applies, for example, to DOC.S.N1 ("Name technical approaches [ ... ]") in the competency area of *Documentation*. In the associated module, less emphasis was placed on word processing (DOC.S.N1a) and permanent data storage (DOC.S.N1b), and instead, the possibilities of digital version management (DOC.S.N1c) were discussed in depth, so a significant increase can only be recorded for the third item (DOC.S.N1c) The selection of this sub-aspect was based on the assumption that the students would have less prior knowledge of digital version management than of the other sub-aspects. The pre-test item mean values support this assumption (DOC.S.N1a: 5.46 (1.90), b: 5.69 (1.89), c: 4.00 (2.35)).

### 5.1.3. Differences between the Test Times in Sub-Areas which Were Not Addressed

Differences between the test times belonging to a module (pre-test and post-test) can only be found for one item (PRE.S.A1c: "I can initialise and use at least one tool/system to represent processes on different time scales."). The results from the pre-test (*M* = 4.92, *SD* = 1.71) and post-test (*M* = 6.00, *SD* = 1.91) indicate that the intervention resulted in an improvement in self-efficacy expectation, *V* = 51.5, *p* = 0.014, *r* = 0.71. This is

understandable, since the creation of stop motion videos was specifically practised here, but not all of the presentation forms expected in this sub-competency were covered in the module.

### 5.1.4. Overall Comparison of the Observed Effects

Figure 5 shows boxplots of the observed (averaged) effect sizes *r* for the main learning objectives and secondary learning goals for each competency area.

**Figure 5.** Comparison of the observed (averaged) effect size of the main learning objectives and secondary learning goals. Boxplots visualise the distribution of the (averaged) effect size within a category of learning goals. The green lines show the median of effect sizes within a competency area. An adjusted threshold for large effects 0.60 is chosen (yellow line).

Except for the competence areas of *Communication/Collaboration* and *Data Processing*, there are clear separations between the effect sizes of the main learning objectives and the secondary learning goals, which supports the division into main and secondary learning objectives.

### *5.2. Discussion of the Individual Teaching-Learning Modules*

In the following section, the results of the individual learning modules are examined in more detail separately.

### 5.2.1. Data Processing (DAP)

Out of the 26 sub-competencies in the DAP competency area, 13 were selected as major and 9 as minor learning objectives. Less prior experience was assumed in the areas of *Content-specific context* and *Special tools*, which is why more attention was paid to these areas in the design of the unit. Large effects (*r* = 0.62 ... 0.86) were found between the pre- and post-test for all major learning objectives, as well as medium to large effects for the minor learning objectives (*d* = 0.50 ... 0.63), except for the test items DAP.S.A1b ("I can apply procedures for calculating new quantities in data processing."), DAP.S.A1d ("I can apply procedures for statistical analysis in data processing."), and DAP.S.A1e ("I can apply image/audio and video analysis procedures in data processing."). The structure of the session can be seen well, as the application level played a minor role here and, similarly, for the secondary learning objective test item DAP.T.D1a ("I can describe the didactic prerequisites of using digital data processing in the classroom."). Looking at the

averaged effect sizes in the module (Table 4), it can be confirmed that the areas with greater focus produced stronger effects. Consequently, the focus on the content specific context and the specific tools has proven to be suitable and can be maintained for further courses. In this evaluation, the pre- and post-tests accompanying the synchronous session were considered. However, a significant change in self-assessment in the area of application is expected for the lesson design phase. Therefore, it can be said that, through the module, the competency area DAP can be promoted very well and that this module serves as a basis for further modules for the promotion of digital competences among prospective teachers at other locations.

### 5.2.2. Documentation (DOC)

From the 13 sub-competencies of the competency area of DOC, 8 were selected as the main objectives and 4 as secondary learning goals. Particular attention was paid to the levels of *Name* and *Describe*. In all main learning objectives, a large effect on the growth of the students' self-efficacy expectations (*r* = 0.65 ... 0.88) can be determined by the measuring instrument. As already discussed before, the secondary learning objectives in the area of DOC focused on the students' previous experience, which is why less emphasis was placed on word processing (DOC.S.N1a) and permanent data storage (DOC.S.N1b) and, instead, the possibilities of digital version management (DOC.S.N1c) were discussed in depth, so a significant increase can only be recorded for the third item (DOC.S.N1c). Nevertheless, besides single items with a large effect (DOC.S.N1c), medium effects were found across all competencies of the secondary learning objectives (*r* = 0.53 ... 0.67). As expected, no significant increases in students' self-efficacy ratings were detected in the domains that were not addressed. If the focus is placed on the individual results, it can be seen that high effect sizes were obtainedm especially in the *Teaching* (T) category, reflecting the structure of the session. Therefore, it could be shown that the intervention has a great effect in the areas of the main learning objectives on the students' self-efficacy expectation, which is why this session needs only minor adjustments for further implementations and can be used as a model example for courses at other universities. To be a little more prepared for the session on communication and collaboration (see below), further elaboration could be made in the area of specific technology (DOC.S.N1). Thus, the module fully covers the competency areas taken from the framework.

### 5.2.3. Presentation (PRE)

Out of the 17 sub-competencies that the competency area PRE comprises, only 8 sub-competencies were declared as main and 4 as secondary learning objectives due to the limited time available and based on the assumed prior experience. Particular emphasis was placed on the competencies of the *Name* and *Describe* competency levels and, as described before, mainly in the areas of *Teaching*, *Methods/Digitality*, and *Content-specific context* (Table A5). Out of the 36 test items used to assess the sub-competencies addressed, no significant effect on the students' self-concept was found in 7 cases. In the area of the main learning objectives, these were one item at the naming level and two items at the describing level (see Table A6), each of which is a subitem of a supercategory (PRE.C.N1/D1). Nevertheless, by averaging all of the effect sizes of these supercategories, a large effect (*r* = 0.61 ... 0.90) could also be shown for these two. The same applies to the effect sizes of the superordinate sub-competencies (PRE.S.N1/D1) of the four rejected items from the area of secondary learning objectives (*r* = 0.40 ... 0.54). Thus, based on the results from the evaluation, an area-wide increase in self-efficacy expectations for the addressed competency domains can be determined. The individual results, which show comparatively high effect sizes in all areas of the category *Methods/Digitality* (TPK), reflect, on the one hand, the module structure, since, in this session, the focus was put more on the discussion among the students about the possible effects of the use in the classroom. On the other hand, students estimated their prior experience in the context of *Principles and Criteria for Designing Digital Presentation Media* (PRE.M.N1/D1) to be comparatively

low. Thus, the focus on individual items in the competencies has proven successful, and the unit on presentation can be used as a successful example for the area-wide integration of the promotion of digital competencies in a master's seminar for student teachers.

### 5.2.4. Communication and Collaboration (COM)

For this module, due to time considerations, 4 of the 29 competency expectations were selected as major learning objectives and 11 as minor learning objectives. Thus, only about half of the competencies could be covered. In order to get a better overview of the entire competency area and to better link the different areas of teaching, methods, context, and tools, it would certainly be advisable to extend this module to two sessions for future implementations. Nevertheless, for a first session, the focus on the use of digital technologies for joint work on documents (by students as well as among colleagues) and the associated requirements, as well as the instruction of students to communicate with each other and ultimately the exemplary integration into lesson planning, is considered correct. A Dunning–Kruger effect [66,67] is suspected, indicating that, in the area of the main learning objectives, no major effect on the self-assessment of the students could be achieved, because they overestimated their previous experience. During the course, the students first had to learn that, although they experience themselves as very competent in everyday digital communication, guiding digital collaboration between pupils goes far beyond the skills in everyday life and that completely different tools can be used for corresponding learning activities. Due to this overestimation of their previous experience, mainly technical issues and tools were discussed and tested, whereas methodological-didactic issues could only be considered on the basis of individual examples. If, as described above, some technical tools and tricks are already presented in the *Documentation* module, there is more time for methodology and teaching at this point in the course. The significant effect of the intervention on the assessment of being able to integrate communication and collaboration into lesson planning (COM.T.A1b) particularly shows that this module was able to achieve the goal of strengthening the students' ability to use digital media in the classroom. With the changes described, this unit thus also serves as an adequate starting point for the development of similar modules elsewhere.

### 5.2.5. Information Search and Evaluation (ISE)

The focus of the module *Information Search and Evaluation* is clearly on methodology and lesson planning (Table A13). From the 32 sub-competencies of the competency area ISE, 21 were selected as the main learning goals and 7 as the secondary learning goals. As suspected, the students already rated their self-efficacy expectancy in the areas of *Content-specific Context* and *Special Tools* comparatively high at the Naming level (*M*pre = 5.23 (1.92) ... 6.46 (1.61), which is why only a subordinate urgency was assigned to these areas in the design of the unit. Moderate to strong effects (*r* = 0.60 . . . 0.91) were found between the pre- and post-test for all main learning objectives, as well as moderate effects for minor learning objectives (*r* = 0.42 ... 0.60). As discussed before, the students already indicated a comparatively high level of prior knowledge in the pre-test. As in the previous competency areas, the module structure can also be recognised here with a view to the individual results. Particularly, high effects are visible in the area of *Methods/Digitality*, which also played a major role in the course. Thus, the intervention was found to have a large effect on students' self-efficacy expectations in the areas of the main learning objectives, which is why this session requires only minor adjustments for further implementations and can be used as a model for courses at other universities.

### 5.2.6. Data Acquisition (DAQ)

For this session, only 3 of 16 competencies were chosen as major learning objectives, and another two were chosen as minor learning objectives. As suspected, students' selfefficacy expectations were low in the area of specific technology, particularly on the "apply" level compared to other competency areas, which is why it was emphasised. The guided

application of the tools in the area of data acquisition requires special time in this module, which, however, is necessary because the students come with little previous experience. The guidance on data collection can be considered successful when looking at the results. In order to be able to integrate further competencies into this module, it would be conceivable to outsource the practical phases into a self-study unit so that the synchronous main session can focus even more on the areas of methodology and teaching. Likewise, an expansion to two sessions would be useful so that students can continue to be guided as well. This session is a good example of integrating the competencies from the area of special tools and can be used as a blueprint for such implementations.

### 5.2.7. Simulation and Modelling (SIM)

The finding of a significant effect of the module on the self-concept of the students in 22 of 25 sub-competencies suggests that the students have received a comprehensive overview of the basic competency area of *Simulation and Modelling* with the module according to the addressed competence expectations. The strong average effect of the module on the students' self-efficacy confirms that a targeted promotion of digital competencies from DiKoLAN in university teaching-learning arrangements can in principle be successful. Looking at the individual results, comparatively high effect sizes were obtained in the category *Special Technology*. This is probably due to the weak assessment of prior knowledge by the students compared to the other three categories (Tables A26 and A28). Thus, the effectiveness measurement procedure identified a thematic area with great potential for development in this teaching–learning arrangement. The identified knowledge gap among the students can be explained, since prior knowledge of "special technology" cannot be expected from any of the previous stages of the teacher training program in Konstanz, in comparison to its subject-specific, pedagogical, and subject-didactic overlapping fields. Thus, the intervention was found to have a large effect on students' self-efficacy expectations in the domains of the main learning objectives, which is why this session requires only minor adjustments for further implementations and can be used as a model for courses at other universities.

### *5.3. Final Discussion of the Course Design*

It has been helpful to dedicate a separate week to each competency area, allowing us to cover large areas of the DiKoLAN competency framework in one term, achieving a significant gain in all areas. In addition, it became apparent that some areas (for example, the sessions on *Documentation*—DOC and *Communication*—COM) offer the opportunity to link content across multiple sessions, which can be integrated in future courses. The accompanying tasks create further need for support but also allow for a deepening of the topics addressed in the sessions, for which there would otherwise have been no time. The design of teaching units in particular provides students with initial teaching concepts in which digital media are integrated into lessons.

### *5.4. Final Discussion of the Methodology of Evaluation*

The detailed monitoring of all the modules through separate pre- and post-tests allowed for a very precise observation of the effect of each module on the students' selfefficacy expectations in the different areas. Since a high response rate was achieved despite the voluntary nature of the pre- and post-test, the additional time required of the students is not considered to be too high, but the benefit generated for the further development and confirmation of the course structure is immense. With the help of the test instrument used, we were able to confirm the effectiveness of existing structures and diagnose areas in need of further development.

### **6. Conclusions**

With the help of the test instrument provided by the *Working Group Digital Core Competencies* [5], it was possible to show that the newly designed course aimed at promoting students' digital competencies can specifically promote students' self-efficacy expectations. Accordingly, pre-service teachers feel more self-efficacious after the seminar in large parts of the digital core competencies listed in the DiKoLAN framework. Thus, initial teaching and learning arrangements have been developed and implemented for all seven competency areas relevant to the science teaching profession. Therefore, a repetition and adaptation of such teaching concepts in the university context can be a proven method to fight against the current issues in the use of digital tools in schools. The piloting of the self-efficacy assessment instrument using the developed module as an example shows that it can be used to optimise such teaching concepts: For example, the content of a teaching–learning module could be adapted to the students' prior knowledge and thus made even more effective by means of an anticipated learning level survey in the pre-test. At the same time, the strengths and weaknesses of already-tested modules (as in the presented course) can be revealed so that the modules can be improved and re-tested. Furthermore, this work presents a course that can be used as a best practice example for the development and design of new courses due to its effectiveness demonstrated here. Anyone interested in using and expanding on the material is invited to contact the corresponding author to obtain access to it.

**Author Contributions:** Conceptualisation, A.H., L.-J.T., P.M. and J.H.; methodology, A.H. and L.-J.T.; validation, L.-J.T.; formal analysis, L.-J.T.; investigation, L.-J.T.; data curation, L.-J.T.; writing—original draft preparation, A.H., L.-J.T., P.M. and J.H.; writing—review and editing, A.H., L.-J.T., P.M. and J.H.; visualisation, L.-J.T.; supervision, J.H.; project administration, J.H.; funding acquisition, J.H. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Federal Ministry of Education and Research (project "edu4.0" in the framework of the joint "Qualitätsoffensive Lehrerbildung", grant number 01JA2011) and the German Chemical Industry Association (VCI) (project "Digitale Werkzeuge für den Chemieunterricht"). The APC was funded by the University of Konstanz.

**Institutional Review Board Statement:** All participants were students at a German university. They took part voluntarily and signed an informed consent form. Pseudonymization of participants was guaranteed during the study. Due to all these measures in the implementation of the study, an audit by an ethics committee was waived.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to the ongoing study.

**Acknowledgments:** The authors would especially like to thank our colleagues in the *Digital Core Competencies Working Group* (Sebastian Becker, Till Bruckermann, Alexander Finger, Lena von Kotzebue, Erik Kremser, Monique Meier, and Christoph Thyssen) for their support of this research project, the provision of the test instrument, and for the lively exchange on the research design and the evaluation of the results. We also thank Lukas Müller, who helped us as a student assistant during the seminar. Lastly, we thank the Joachim Herz Foundation, which supports all DiKoLAN projects.

**Conflicts of Interest:** The authors declare no conflict of interest.



**Figure A1.** Competence expectations defined in the DiKoLAN framework addressed in the respective teaching module *Documentation* (DOC). Main topics (magenta), side topics (blue).

**Table A1.** Results of Wilcoxon signed-rank tests for competencies in the area of *Documentation* (DOC) explicitly addressed as main learning objectives in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


Note: ◦ not tested. \* The average effect size is given for competencies assessed with more than one item.


**Table A2.** Results of Wilcoxon signed-rank tests for competencies in the area of *Documentation* (DOC) addressed as secondary learning goals in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.

\* The average effect size is given for competencies assessed with more than one item.

**Table A3.** Results of Wilcoxon signed-rank tests for competencies in the area of *Documentation* (DOC) NOT explicitly addressed in the respective module and thus NOT hypothesised to change during intervention (for comparison). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


\* Assessed with more than one item.

**Table A4.** Overview of the (average) effect sizes of the effects of the intervention on the competence expectations in the area of *Documentation* (DOC). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


Main learning objectives (**bold magenta**), secondary learning goals (*italic cyan*), and non-addressed competencies (yellow). \* The average effect size is given for competencies assessed with more than one item. ◦ Not tested.

### **Appendix B. Presentation (PRE)**


**Figure A2.** Competence expectations defined in the DiKoLAN framework addressed in the respective teaching module *Presentation* (PRE). Main topics (magenta), side topics (blue).


**Table A5.** Results of Wilcoxon signed-rank tests for competencies in the area of *Presentation* (PRE) explicitly addressed as main learning objectives in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.

◦ Not tested. \* The average effect size is given for competencies assessed with more than one item.

**Table A6.** Results of Wilcoxon signed-rank tests for competencies in the area of *Presentation* (PRE) addressed as secondary learning goals in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


\* The average effect size is given for competencies assessed with more than one item.


**Table A7.** Results of Wilcoxon signed-rank tests for competencies in the area of *Presentation* (PRE) NOT explicitly addressed in the respective module and thus NOT hypothesised to change during intervention (for comparison). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.

\* Assessed with more than one item.

**Table A8.** Overview of the (average) effect sizes of the effects of the intervention on the competence expectations in the area of *Presentation* (PRE). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


Main learning objectives (**bold magenta**), secondary learning goals (*italic cyan*), and non-addressed competencies (yellow). \* The average effect size is given for competencies assessed with more than one item. ◦ Not tested.


### **Appendix C. Communication/Collaboration (COM)**

**Figure A3.** Competence expectations defined in the DiKoLAN framework addressed in the respective teaching module Communication/Collaboration (COM). Main topics (magenta), side topics (blue).

**Table A9.** Results of Wilcoxon signed-rank tests for competencies in the area of *Communication/Collaboration* (COM) explicitly addressed as main learning objectives in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


\* The average effect size is given for competencies assessed with more than one item.


**Table A10.** Results of Wilcoxon signed-rank tests for competencies in the area of *Communication/Collaboration* (COM) addressed as secondary learning goals in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.

\* The average effect size is given for competencies assessed with more than one item. ◦ Not tested.


**Table A11.** Results of Wilcoxon signed-rank tests for competencies in the area of *Communication/Collaboration* (COM) NOT explicitly addressed in the respective module and thus NOT hypothesised to change during intervention (for comparison). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.

\* Assessed with more than one item. ◦ Not tested.

**Table A12.** Overview of the (average) effect sizes of the effects of the intervention on the competence expectations in the area of *Communication/Collaboration* (COM). *n* = 13. S: Special Tools, C: Contentspecific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


Main learning objectives (**bold magenta**), secondary learning goals (*italic cyan*), and non-addressed competencies (yellow). \* The average effect size is given for competencies assessed with more than one item. ◦ Not tested.

### **Appendix D. Information Search and Evaluation (ISE)**


**Figure A4.** Competence expectations defined in the DiKoLAN framework addressed in the respective teaching module *Information Search and Evaluation* (ISE). Main topics (magenta), side topics (blue).


**Table A13.** Results of Wilcoxon signed-rank tests for competencies in the area of *Information Search and Evaluation* (ISE) explicitly addressed as main learning objectives in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.

◦ Not tested. \* The average effect size is given for competencies assessed with more than one item.

**Table A14.** Results of Wilcoxon signed-rank tests for competencies in the area of *Information Search and Evaluation* (ISE) addressed as secondary learning goals in the respective module and hypothesised to grow during intervention. *n* = 13.


**Table A15.** Results of Wilcoxon signed-rank tests for competencies in the area of *Information Search and Evaluation* (ISE) NOT explicitly addressed in the respective module and thus NOT hypothesised to change during intervention (for comparison). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


**Table A16.** Overview of the (average) effect sizes of the effects of the intervention on the competence expectations in the area of *Information Search and Evaluation* (ISE). *n* = 13. S: Special Tools, C: Contentspecific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


Main learning objectives (**bold magenta**), secondary learning goals (*italic cyan*), and non-addressed competencies (yellow). \* The average effect size is given for competencies assessed with more than one item.

### **Appendix E. Data Acquisition (DAQ)**


**Figure A5.** Competence expectations defined in the DiKoLAN framework addressed in the respective teaching module *Data Acquisition* (DAQ). Main topics (magenta), side topics (blue).

**Table A17.** Results of Wilcoxon signed-rank tests for competencies in the area of *Data Acquisition* (DAQ) explicitly addressed as main learning objectives in the respective module and hypothesised to grow during intervention. *n* = 10. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


**Table A17.** *Cont.*


\* The average effect size is given for competencies assessed with more than one item.

**Table A18.** Results of Wilcoxon signed-rank tests for competencies in the area of *Data Acquisition* (DAQ) addressed as secondary learning goals in the respective module and hypothesised to grow during intervention. *n* = 10. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


\* The average effect size is given for competencies assessed with more than one item.

**Table A19.** Results of Wilcoxon signed-rank tests for competencies in the area of *Data Acquisition* (DAQ) NOT explicitly addressed in the respective module and thus NOT hypothesised to change during intervention (for comparison). *n* = 10. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


\* Assessed with more than one item. ◦ Not tested.

**Table A20.** Overview of the (average) effect sizes of the effects of the intervention on the competence expectations in the area of *Data Acquisition* (DAQ). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


Main learning objectives (**bold magenta**), secondary learning goals (*italic cyan*), and non-addressed competencies (yellow). \* The average effect size is given for competencies assessed with more than one item. ◦ Not tested.

### **Appendix F. Data Processing (DAP)**


**Figure A6.** Competence expectations defined in the DiKoLAN framework addressed in the respective teaching module *Data Processing* (DAP). Main topics (magenta), side topics (blue). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


**Table A21.** Results of Wilcoxon signed-rank tests for competencies in the area of *Data Processing* (DAP) explicitly addressed as main learning objectives in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.

◦ Not tested. \* The average effect size is given for competencies assessed with more than one item.

**Table A22.** Results of Wilcoxon signed-rank tests for competencies in the area of *Data Processing* (DAP) addressed as secondary learning goals in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.



**Table A22.** *Cont.*

\* The average effect size is given for competencies assessed with more than one item.

**Table A23.** Results of Wilcoxon signed-rank tests for competencies in the area of *Data Processing* (DAP) NOT explicitly addressed in the respective module and thus NOT hypothesised to change during intervention (for comparison). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


\* Assessed with more than one item. ◦ Not tested.

**Table A24.** Overview of (average) effect sizes of the effects of the intervention on the competence expectations in the area of *Data Processing* (DAP). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


Main learning objectives (**bold magenta**), secondary learning goals (*italic cyan*), and non-addressed competencies (yellow). \* The average effect size is given for competencies assessed with more than one item. ◦ Not tested.

### **Appendix G. Simulation and Modelling (SIM)**


**Figure A7.** Competence expectations defined in the DiKoLAN framework addressed in the respective teaching module *Simulation and Modelling* (SIM). Main topics (magenta), side topics (blue).


**Table A25.** Results of Wilcoxon signed-rank tests for competencies in the area of *Simulation and Modelling* (SIM) explicitly addressed as main learning objectives in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.

\* The average effect size is given for competencies assessed with more than one item.

**Table A26.** Results of Wilcoxon signed-rank tests for competencies in the area of *Simulation and Modelling* (SIM) addressed as secondary learning goals in the respective module and hypothesised to grow during intervention. *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


◦ Not tested. \* The average effect size is given for competencies assessed with more than one item.

**Table A27.** Results of Wilcoxon signed-rank tests for competencies in the area of *Simulation and Modelling* (SIM) NOT explicitly addressed in the respective module and thus NOT hypothesised to change during intervention (for comparison). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


**Table A28.** Overview of the (average) effect sizes of the effects of the intervention on the competence expectations in the area of *Simulation and Modelling* (SIM). *n* = 13. S: Special Tools, C: Content-specific Context, M: Methods/Digitality, T: Teaching, N: Name, D: Describe, A: Use/Apply.


Main learning objectives (**bold magenta**), secondary learning goals (*italic cyan*), and non-addressed competencies (yellow). \* The average effect size is given for competencies assessed with more than one item. ◦ Not tested.

### **References**


### *Article* **Virtual Laboratories in Tertiary Education: Case Study Analysis by Learning Theories**

**Jahan Hassan 1,\*, Anamika Devi <sup>2</sup> and Biplob Ray <sup>1</sup>**

<sup>2</sup> College of Design and Social Context, RMIT University, Melbourne, VIC 3000, Australia

**\*** Correspondence: j.hassan@cqu.edu.au

**Abstract:** This paper examines and evaluates Virtual Laboratories (VLabs) in consideration of technology design, educational pedagogy, and outcome in tertiary education context for ICT courses. There is a growing demand for VLabs in tertiary education to support remote, flexible, and equitable learning. Most of the universities in Australia offer distance education to students who do not attend on-campus classes. On-line labs allowing access via an internet connection can offer learners the required infrastructure to complete their lab tasks without attending physical lab facilities. The onset of COVID-19 pandemic in early 2020 has seen further spike in demand for VLabs as accessing online lab facilities to undertake hands on activities from anywhere and anytime was imperative during lockdown periods. Despite their benefits, it is complex to choose an appropriate VLab design or type that ensures effective and improved learning process. This paper presents two case studies using commercial and custom-made VLabs that are analyzed through the lens of learning theories. The outcome of the analysis informs the readers that the teachers' support (human mediator) and VLabs (teaching tool) are interlinked together in a dialectical way which is an important consideration to achieve successful learning outcome. This study will help educators to make an informed decision in choosing an appropriate VLab design for their teaching content to ensure effective learning outcome.

**Keywords:** virtual labs; COVID-19; ubiquitous learning; e-learning; learning theories

### **1. Introduction**

There is a growing demand to support learners in the higher education sector with a diverse requirement of flexibility such as location, time zone, work hours. The onset of COVID-19 pandemic in early 2020 has re-instated this demand [1]. To facilitate such flexibility, ubiquitous learning (U-Learning) which allows learning to take place using any device in a flexible environment of time, place, and pace, needs to be adopted in the higher education sector [2]. U-Learning in the modern era can be supported through the adoption of e-learning which uses the Internet technology to deliver educational solutions to the learners, with the inclusion of networked systems, and a focus on the on-demand learning [3]. Bermejo et al. [4] reported e-laboratory being one of the most interesting solutions for e-learning, which provides students with the opportunity to put their theoretical knowledge to practice by using unlimited internet access to carry out their laboratory exercises on-line, while remote laboratories are physical facilities that are accessed over a network connection and related software [5], Virtual laboratories (VLabs) do not have physical facilities, rather virtual laboratory resources (e.g., hardware, software on the cloud) are accessed using an internet connection [4]. Because of the virtual nature of, and the remote access to, the laboratory resources, many users can access it at the same time, which represents the elasticity of the lab facility beyond what physical labs can offer.

For higher education students in Information and Communication Technology (ICT), the application of theoretical knowledge in practical tasks is essential in gaining the skills

**Citation:** Hassan, J.; Devi, A.; Ray, B. Virtual Laboratories in Tertiary Education: Case Study Analysis by Learning Theories. *Educ. Sci.* **2022**, *12*, 554. https://doi.org/10.3390/ educsci12080554

Academic Editor: Maria Limniou

Received: 28 June 2022 Accepted: 8 August 2022 Published: 15 August 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

<sup>1</sup> School of Engineering and Technology, The Central Queensland University, Rockhampton, QLD 4701, Australia

needed in the industry. For example, multidisciplinary practical subjects like 'Digital Forensics' or 'Internet of Things and Cloud Computing' are best taught using the Problem Based Learning (PBL) pedagogy [6,7]. In 'Digital Forensics' students examine digital data in search of criminal evidence in practical exercises whereas in 'Internet of Things and Cloud Computing', students need distributed networked sensors to learn and apply their skills in practical implementations [8] to achieve their unit learning outcomes (ULOs). Such activities for Digital Forensics require students from all learning modes, i.e., online as well as on campus (Face-to-Face), to access industry grade forensics laboratories (lab) with heterogeneous Operating Systems, licensed software (tools), hardware and systems with administrator level access. Such facilities are beyond the capacity of campus based general purpose computer labs which indicates that there is a need of providing special-purpose labs in campuses which has cost implications. Importantly also, using such (physical) labs to teach does not support the mentioned learning flexibility and equity for all modes of students. This creates inequity for online students who are unable to attend physical facilities. These hurdles can be overcome by using virtual labs in the ICT teaching.

VLabs have been providing practical lab experiences to students during the COVID-19 pandemic since access to physical labs may not exist or be restricted [9,10]. Educators across disciplines have recognized the value of VLabs for their students as VLabs provide flexible learning opportunities, a preparatory environment for physical labs, and collaboration opportunities [9]. The use of technology for accessing and working with VLabs is a bonus in making graduates ready for future employments, with their familiarity of using virtual training that are often used in workplaces [9]. However, the VLab needs to be designed appropriately based on the content and technology associated with the delivery. Furthermore, the VLab designs and case studies are not analyzed and argued using learning theoretical lens.

To help making an informed decision about the choice of VLab types, this paper has presented two VLab types, using case studies: commercial (or off-the-shelf) VLabs, and custom-made VLabs. Based on our use of both types of VLabs in tertiary education, the case studies present first our experience of using MindTap VLab which is a commercial VLab, and then a custom-made, purpose built VLab which we developed using cloud based resources. The case studies are guided by educational pedagogy, technology, design, and outcome of the use of VLabs in tertiary education context for ICT courses. The study has also detailed challenges associated with VLab implementation. This paper can be a great guide for educators to choose an appropriate VLab to support learning philosophy, pedagogy and most importantly learners to achieve best potential outcome.

The rest of the paper is organized as follows. In Section 2, we have provided background of the study where brief overview of VLabs in ICT and non-ICT teaching are detailed in Section 2.1 followed by a discussion on the learning theories in Section 2.2. In Section 3, we have presented case studies of commercial and custom-build VLabs in ICT tertiary education. We provide discussions in Section 4 followed by a conclusion in Section 5.

### **2. Background**

Virtual Labs are a popular teaching tool in Australian universities. Their usage have dramatically increased in the last two years due to COVID-19 pandemic. Our motivation of using virtual lab is grounded by learning theories that helped us to design and utilize it as an effective learning tool. This section has detailed literature on the use of VLabs in various disciplines which is followed by learning theories to evaluate the use of VLabs in tertiary education sector.

### *2.1. The Use of Virtual Labs in the Tertiary Sector*

Although the use of VLabs has seen a renewed interest during COVID-19 pandemic, VLabs have been used in the tertiary sector in various capacities in pre-Covid era too, specially in ICT [2]. With the proliferation of the internet-based applications used by

general public, students from any discipline have familiarity of the online interfaces to use VLabs. Hence, during COVID-19 lockdown, many Australian universities have introduced VLab as a replacement of laboratory facilities, as used in many disciplines [9–12], to allow students complete their laboratory tasks.

Virtual Labs have been used in a range of non-ICT disciplines such as Biochemistry and Molecular Biology [9], Mechanical Engineering [10], Physics [11], Optics [12], and Rehabilitative Sciences [13]. The VLabs are used to complement onsite teaching in Optics [12], train the faculty members on mechanical engineering and student experiments on fluid mechanics [10]. Furthermore, the use of VLabs [10] have enhanced collaborative learning in Biological Sciences higher education [14]. These examples demonstrate VLab's utility even when learners could attend in-person classes (e.g., during pre-Covid era). Virtual Labs are generally used to achieve discipline learning objectives (LOs), however, during the COVID-19 pandemic these have been used as an alternative to physical labs in various discipline offerings that did not use VLabs in the pre-Covid era. For example, Puzziferro et al. [13] presented their experience of delivering VLabs during COVID-19 pandemic for rehabilitation sciences in terms of strategies with instructional cases.

As a teaching tool in ICT discipline, VLabs have been used to achieve various learning objectives in general, as well as a replacement of physical laboratory during COVID-19 pandemic as was for any other discipline. Deng et al. [15] reported the use of a web-based personalized virtual lab environment in the undergraduate teaching of cybersecurity class at Arizona state university. Authors reported that the personalized lab environment enhanced student engagement, better understanding of assessments, and ultimately enhanced learning outcome. At the Central Queensland University Australia (CQUniversity), we have been using VLabs for teaching Computer Forensics, Cloud Computing, and Internet of Things ( IoT) units (subjects) to support our learners and to achieve learning outcomes. To ensure effective and catered learning flexibility and support, we have taken two approaches; custom-built (by the teaching team), and commercially available off-the-shelf VLabs. We have elaborated both approaches, their motivation and design in Section 3.

Well-founded learning theories also support the use of VLabs to achieve more effective learning outcome. For example, according to Siemens et al. [16], learning is a process of developing a learning network and making connections between ideas of human (related with human cognition) which aligns with Vygotsky's constructivism theoretical paradigm, which has emphasized learning being a process instead of product and we need human interaction and symbolic tools to achieve effective learning outcomes and solving critical problems. The VLabs are effective symbolic tools that are designed and supported by modern technology innovations in learning space. Therefore, it is important to discuss learning theories to understand how VLabs can be a teaching tool that is supported by learning pedagogy to ensure effective design and and learning experiences for learners. In next Section 2.2, this paper has detailed learning theories to analyze VLab case studies under the lens of learning pedagogy.

### *2.2. Learning Theories*

This sub-section detailed existing learning theories that have led us to choose effective objects, tools and props to design the VLab. According to contemporary educational research, five major learning theories have been used in higher education classrooms: behaviorism, cognitivism, constructivism, humanism and connectivism [17]. The researchers have found different theories have emerged due to different kind of learning needs of the learners [18] and based on different settings of learning, for example, distance learning, experimental laboratory, school setup and workplace setup, etc. These theoretical lenses help teachers to model their learning strategies and to develop educational technologies to support learning goals. Therefore, our role and interactions around VLab were directed by existing proven practices established by learning theories in the literature. Furthermore, each of these theoretical perspectives play a role in describing how students learn. Each theory is quite different and explains learning in different ways. For example, while

behaviorism, cognitivism, constructivism, and humanism are the core learning theories dominated of instructional environments, a more recently developed learning theory is connectivism which proposes that knowledge is distributed across a network of connections and, consequently, learning is the ability to construct and traverse those networks.

The traditional epistemological paradigms like Vygotsky's cultural-historical theory [19], Bandura's social cognitive theory [20], Bronfenbrenner's ecological system theory [18] and Leontiev's activity theory [21], have emphasized the social, situational, and relational aspects of knowledge and learning. The concept of "mediation" was first used in Vygotsky's cultural-historical theory where higher mental function was viewed as a mediated function [22]. Some of Vygotsky's colleagues and students pioneered and elaborated on their idea. Kozulin et al. [23] states that there are two types of mediation: human mediators and the symbolic (system) mediators. The design of our VLab has embedded both system and human mediation to support students interactions based on reactions collected in the form of systems data.

The importance of Vygotsky's cultural historical emphasized on human interaction to development of higher mental functions (problem solving, logical thinking, attention, abstraction and perception etc.) in a dialectical way, where it has dynamic relations between the external and the internal level instead of linear [24]. Therefore, human interaction to develop higher mental functions is applicable on children's as well as adult's learning. Vygotsky [24] first used the term "psychological (symbolic) tool" to interact with people. According to Vygotsky, symbols can be categorized in two ways


Human intervention is needed to use these symbolic tools purposefully, otherwise, it will not make meaning in the learning process. For instance, the VLab cannot be a technological tool by itself unless the teacher (mediator) is designing it purposefully and guiding students' learning process to achieve their learning goal.

In the 21st century digital era, learning landscapes are network, social and technological based. The constructivist theory of learning emerged prior to the revolutions of information technology (IT), therefore new perspectives of learning theory have emerged which is connectivism. Similar with constructivism theory, Siemens and Dowens [25] connectivism theory emphasized on using of online tools (for example threaded discussion in Moodle, blog posts, second life and synchronous online meetings) to connect with learners. Several researchers found social media platform promoted connectivity, learners' engagement, collaboration and the development of professionalism [26–28], however, there are some challenges like technical problems, privacy issues that the teachers and students faced using this platform [26]. In the design of our VLab, the connectivism is integrated using virtual collaboration/engagement with LMS integration to foster discussion and flexible learning opportunity.

From constructivism theoretical perspective, VLabs can be a great platform for teachers to use as a tool (objective sense) for designing problem-based learning (PBL) based assessments for students to solve (subjective sense) using industry scale technologies. Based on constructivism theoretical perspectives George Siemens (2004) stated connectivism views learners should be developing a learning network and making connections between ideas embedded throughout that network. With the facilities available in the VLab setup, we have been designing PBL based assessments to foster learning connections throughout the VLab resources.

This paper has used a blended approach to integrate connectivism to create opportunity for making connections between ideas with the help of object, prop and mediation (both human and symbolic) as stated in Vygotsky's theory. In the next section, this paper has detailed two different VLabs design, such as commercial, and custom-built, along with their use cases analyzed based on learning pedagogy and outcomes.

### **3. Case Study**

In this section, we have detailed the design, implementation, and outcome of the two VLabs, commercial (off-the-shelf) and custom-built, in Sections 3.1 and 3.2 , respectively.

### *3.1. Teaching Using a Commercial VLab*

At CQUniversity, we have developed and teach computer forensics subjects (units) for both undergraduate and post graduate levels. These subjects integrate digital investigations though a legal lens which is an essential requirement for cybersecurity jobs in the industry. Our ICT students undertake these units in either face-to-face or online learning modes. Further, we do not have any residential schools for the online mode ICT students.

As mentioned in [8], we also realized that 'Digital Forensics' being a multidisciplinary and practical subject, is best taught using the problem-based learning (PBL) pedagogy [6,7] where students learn through undertaking hands-on digital forensics investigations. These investigations would require students to use industry grade, licensed digital forensics software (tools) to examine digital data using industry accepted processes, in search of criminal evidence. According to Vygotsky's theory [29], human development and learning cannot happen in a liner process, rather it must be viewed as a unity of the material world and the individual's internal mental aspect. This unity must be address as the real-life experiences where PBL pedagogy takes place through hands-on activities. To provide our computer forensics students an opportunity to engage in real life problembased learning (PBL), in our initial offerings of the units, we used campus based general purpose computer labs with required forensics software installed on the computers to teach on-campus students. For on-line students, demo versions of the tools were installed on students' personal computers. We faced several challenges below with this setting.


The above mentioned challenges led to a significant hindrance to learning: students missed having a comprehensive experience of working with the feature-rich tools across all OS platforms. The licensing requirement imposed additional barriers too. It was impractical to maintain licenses and updated versions of a plethora of forensic tools in the labs across multiple campuses, but the demo versions lacked functionality which was not a practical choice either. Further, the online and Bring-Your-Own-Device (BYOD) students would have to pay to install these tools on their computers. As a result, students could not fully engage with the practical lab tasks to develop their skills required for solving real-life investigative tasks. As such, the integration of industry grade forensic lab, with unrestricted access to a wide range of licensed forensics software and physical hardware with administrative access, would address hindrances such as (i) practical learning experience, (ii) student engagement, and (iii) equitable accessibility for all students.

We have addressed this by integrating a VLab Environment in the units. The VLab, detailed below, provides our students, of all learning modes, unrestricted access to industry grade forensic tools, hardware, and data to carry out practical exercises of forensics investigations.

### MindTap: Cengage Learning's Virtual Lab

As mentioned previously, without teacher's intervention, tools will be perceived as a simple object rather than a learning tool that will be appropriated for use by the students [23]. For example, a computer will be a simple device or object and cannot be a teaching tool if the teacher is not using it purposefully for the learning process. We approached Cengage Learning [30] for a virtual digital forensic laboratory with widely used forensics tools. Out of a few demonstrated virtual labs, we have chosen MindTap virtual learning environment to customize for our syllabus and implemented it in our teaching. We have been using this VLab environment ever since for all of our forensic units. MindTap provides our students digital accessibility by using virtual forensic tool platforms.

Students get access to a virtual networked lab environment as shown in Figure 1 (source: Cengage Learning [30]), featuring multiple Virtual Machines (VMs) running various Operating Systems (OSs): PLABWIN10 is a standalone workstation running Windows 10, PLABDEFT01 is a Ubuntu Linux workstation, and PLABKSRV01 is a Kali Linux Workstation. Each virtual machine within MindTap VLab environment provides a specific OS interface (see Figure 1), and includes a wide range of forensics tools and data to carry out investigative tasks. This kind of VM setup with multiple OSs allows our students to work with wide range of data sources (data from various OS workstations) from suspects' computers, without requiring access to multiple physical computers running different OSs. The VMs provide a complete access to both digital forensics data and a range of industry-grade digital forensics tools that the students access to undertake investigations to discover forensics evidence. It fosters ubiquitous access to students studying in various enrolment modes, which caters well for online learning during COVID-19 pandemic, which is a recent bonus for our students. MindTap VLab environment, integrated in the LMS (e.g., Moodle), is more than a virtual lab, offering four different weekly modules as shown below that students can explore and learn from.

**Figure 1.** MindTap Virtual Machine (VM) Lab network diagram.


The integration of MindTap VLab into the units' LMS websites allows students' to access it by using their internet connections and a web browser, from anywhere, anytime. This has provided an effective environment to students across learning modes to engage with real-world practice tasks using digital data and hardware located on the VLab environment (no downloading needed), and unrestricted access to a range of industry grade forensic tools. Figure 2 illustrates the Windows 10 virtual machine interface within Mind-Tap environment which hosts a range of forensics tools and data. Students are able to access additional learning resources on the VLab Environment including practice tests through the unit LMS website. As illustrated in Figure 3 (source: Cengage Learning [30]), student can use a specific forensic tool and investigate using data located on the VM. Students can download or upload data to and from the VM and the local computer while a specific tool being used in this environment. As per constructivist epistemology, in this integrated VLab environment, students own the problem and understand/accept learning objectives and they control the problem solving process [32]. The VLabs give them scope to learn from real experiences by using learning resources.

**Figure 2.** A virtual machine workstation within MindTap running Windows Operating System.

**Figure 3.** MindTap VM PLABWIN10 with a specific forensic tool being used.

The customized VLab Environment also provides micro-level, per student task completion details to the teaching team. MindTap's personalizing feature is an essential factor that has helped us to achieve a high-level of student engagement and thereby, enhanced learning outcomes. This was echoed in the literature as stated before, e.g., in [15]. For

example, similar with Cybulski et al. [33], we have found students had opportunities for social interaction to discuss their learning, experiences and the knowledge by using MindTap VLab community forum. The per-student personalised activity features helped us to provide targeted guidance to our students to boost their completion of weekly tasks. The high levels of student engagement has lead to high success rates in these units, while providing students industry level practical skills. A past student recalled the value of the VLab Environment stating that the use of MindTap with its access to the feature-rich tools was invaluable to get the ins and outs of the practical side of the unit which the industry demands.

We have guided our students of all enrollment modes in their forensic investigation tasks through the unrestricted learning environment of the VLab Environment, resulting in an improved student engagement with tasks and satisfaction for all modes of study. The customization feature of the VLab environment was particularly useful to us, as we were able to select weekly topic summary, hands-on exercises, and quizzes that are aligned with our syllabus. Students can access additional learning resources such as the latest industry news automatically via the VLab environment. To support student learning further, we tailored the VLab environment to provide a progressive, online test aligned with the weekly contents. MindTap's readily available quizzes and other learning materials such as topic summary meant that we could customize these activities and focus on guiding students at their personal activity and performance levels, than investing in developing these features ourselves. Our teaching team also enjoyed the features of MindTap including networking, different operating systems, and the needed forensic tools.

We have utilized live analytic of students' engagement and performance of practical tasks collected by VLab environment as presented in Table 1 (data source: [30]). It allowed us to use student-specific, micro level information, e.g., activities accessed (with links to those), time spent and number of logins to the VLab environment to engage in discussions and support/guide students in completing their pending tasks hence improved learning outcome. According to [23], content knowledge will be vague if the teachers do not support the students to use it purposefully in real life experiences. To get better learning outcome, it is important to have psychological/symbolic tools along with traditional way of teaching theoretical content knowledge. To acquire the learning outcome, the teaching tool needs to be use purposefully. For example, if students gather theoretical knowledge of how to use a forensic tool through curriculum-based content in the course but do not have any knowledge how to do it practically in a digital forensic investigation, they will fail to achieve the learning outcomes. For this reason, acquisition of psychological/symbolic tools requires teachers to take facilitator role and deliberately and intentionally use this tool for students' learning purpose. We have found in our case study that teachers were able to provide hand-on activities, quizzes, topic summary and latest industry news using MindTap VLab environment. Our finding also aligned with connectivism theory that knowledge is actuated by learners' participation in a learning community and to connect with others to collaborate and share, they need appropriate tool to create and construct knowledge [34]. We have found MindTap VLab is excellent teaching tool to engage students to complete hand-on activities virtually in a flexible environment considering time, place, and pace.

MindTap VLab has allowed us to access details on each student's status of tasks by clicking on the engagement levels displayed on the VLab environment, to learn about which tasks they are falling behind. Equipped with weekly status of individual students these information have enabled us to offer targeted supports and guide students in completing the tasks while continuing to monitor their personal progress. This has motivated students to enhance their unit task completions, contributing to higher, sustained success rates of the units since the MindTap VLab adoption as illustrated in Figure 4.


**Table 1.** Live analytic of students' engagement with weekly activities.

**Figure 4.** Improved success rates (%) of Digital Forensics units (Empty bars represent the unit not being offered in that Term. T1 refers to Term-1, T2 refers to Term-2).

The use of MindTap VLab has positively impacted the success rates of the units (Figure 4). The unit COIT20267 has maintained a perfect success rate since the last few offerings, with all enrolled students passing the unit. The other unit, COIT12201, has achieved significant improvements in the success rates since the adoption of MindTap. Our observation, therefore, is that students who actually studied the unit (i.e., did not drop out) have all passed the unit since the integration of MindTap (students who withdrew from the unit after the Census date were considered failing the unit which reduced the success rate).

The impact is also reflected in the increased unit satisfaction scores. For COIT12201, the satisfaction score increased from 3.5 in 2019 to 4.5 in 2020, and for COIT20267 it increased from 4.3 in 2019 to 4.7 in 2020 (score out of 5). Student feedback received through the university's unit evaluation surveys echo the quality of the units being outstanding as they got to do activities using many different programs (software tools) in MindTap and then they were able to apply their knowledge to perform forensics investigation of a case study in their assessment task.

In our use of MindTap VLab, it was intentionally setup as a teaching tool together with learning materials by the teacher-mediator in COIT12201 and COIT20267 that gave students the scope to complete the unit successfully, enhancing the success rates of the units (see Figure 4). MindTap was a core part of weekly tutorial activities that the students engaged with, as the teaching staff (teacher-mediator) guided them through the activities. This ensured that students actually used MindTap to engage with their learning, as opposed to providing MindTap as an additional resource for self-learning, which may or may not be used by students on their own. We designed assessments that required students to apply skills developed through the MindTap activities and found that teacher-mediator (subjective sense) had a huge contribution on intentionally using MindTap as a teaching

tool (objective sense) [23] in the MindTap VLab environment for developing cognitive and problem-solving abilities in students.

### *3.2. Custom-Built, Cloud Hosted VLab*

It is challenging to design content to teach emerging technologies like Internet of Things (IoT), cloud computing and Quantum computing due to complex and/or expensive laboratory setup required to deliver hands-on exercises. The matter gets further complex when the content aims to deliver complementary emerging technologies like cloud computing, IoT and bigdata together. In 2016, CQUniversity aimed to deliver two complementary emerging technologies under the unit called "Cloud Computing and Internet of Things for Smarter Applications" to prepare students with skills the industry demanded. The content development of the unit faced two main challenges: (1) finding appropriate content in the form of a textbook that covered complementary aspects of these technologies, and (2) complexity of setting up physical lab equipment to cater for students of multi modes enrolments like online and face-to-face, in multiple campuses. The dedicated physical laboratory could be the easiest solution for the second issue, however, it is infeasible and costly for CQUniversity's teaching delivery, since a dedicated lab would have made it difficult to deliver hands-on activities and realistic assessments for distributed campuses and online students. We note that for a single location unit delivery, a dedicated lab would work fine.

This case study will elaborate our journey to address the second challenge which we have overcome through the utilization of industry partnerships and creative use of technology, to design a virtual lab environment using a hybrid approach. Our hybrid approach is a combination of physical and virtual setups that are enabling students to do all hands-on lab activities to cover practical contents of the technologies taught, without a physical lab set up for each campus.

The journey started by CQUniversity joining IBM's Academic initiative which allowed our students to access IBM cloud for free during their study period of this unit. As illustrated in Figure 5, the IBM Cloud Platform as a Service (PaaS) layer allowed us to setup a Virtual laboratory in the cloud so students can access the system ubiquitously. This VLab has three main components: IBM cloud PaaS, external systems, and students. The IBM cloud PaaS has various development environment like NodeRED and Watson IoT system to connect with external systems like sensors, IoT gateway and social media platforms. The external systems allow students to access physical systems remotely via cloud as illustrated in Figure 5. The external systems connected via middleware tool NodeRED allow students to access hardware signal, their interface and data for development purposes. The student terminal in Figure 5 can be any computing device installed with local NodeRED to connect with external system and IBM Cloud using node based connection via NodeRED. To ensure faster development and deployment via terminal, the student's computing device can use IBM cloud foundry and/or IBM Cloud CLI (Command Line Interface).

The VLab allows students to develop and use cloud-based IoT applications without the need for real IoT hardware. Using the VLab, students can build a virtual sensor network across a city, write an application that monitors the sensor data, and then use the data to solve business problems. Students are able to access their laboratory device and tools from any geographical location and independent of specific software or hardware. This has given the teaching team an opportunity to design Problem Based Learning (PBL) assessments that can be solved by students using industry scale technologies.

The VLab allows CQU to offer the same lab facilities for face to face and online students. In the VLab, without performing any physical or logical installation, students can build sensor networks and smart applications based on sensor data to address a complex business problem. For example, students can deploy a virtual sensor network across a real city using OpenStreet map in the VLab; then write the software to monitor the position of delivery trucks using flow-based development tool called NodeRED for visual programming; then collect system's data for analysis to determine how the business can lower delivery costs. The flexibility of using both open source and cloud based tool to design Internet of Things (IoT) systems have improved students exploring and learning opportunity compare to physical laboratory. The VLab has reduced installation time and cost along with hardware procurement requirements. This has enabled teachers (mediator) to design industry level learning exercises and assessments which are otherwise not feasible. The industry collaboration on this VLab has allowed students to share their work using IBM's readily available cloud server and domain to make it accessible from anywhere in the world and ready for business operation. The readily shareable facilities in the VLab has motivated them to push their learning boundaries and to grow their learning network that have resulted improved learning outcome and success rate.

**Figure 5.** The architecture of the custom-built VLab.

During COVID-19 pandemic, our custom-built VLabs became a blessing for students for using hybrid approach, with a combination of physical and virtual setup. The VLab has allowed us to design authentic assessments where students focus on using technology to solve real business problems. Students are to solve the business problems using the knowledge learned and the available cloud tools in the virtual lab. This makes the virtual lab a core part of their learning activities. Students were challenged in these PBL assessments and, with the guidance and help of the teaching team, these challenges motivated students to actually engage with the activities using the virtual setup to be a lifelong learner as they were bound to think out of the box. As stated in learning theories, the teaching tools have rich educational potential, but these remain inactive if there is no human mediator (teacher as a subjective sense) to facilitate the learning process for learners. The facilitation of learning process by the human mediator can come from both guiding students through the virtual lab activities, as well as designing assessments that require skills developed through the virtual lab activities. This resonates with the way we have used the virtual labs, and therefore, our conjecture is that the VLabs will be an ordinary tool, if teachers do not facilitate learning through guidance and designing PBL exercises and assessments for students to solve using skills obtained from VLab activities, to master reasoning and problem solving skills [23].

Furthermore, the virtual lab caters to all levels of students and allows them to continue their learning beyond the unit content. For example, a more curious student will be able to expand on the unit learning of 'language translation tool' and connect it to 'IBM Watson', which is Artificial Intelligent (AI) service, to create an intelligent language translation tool to solve a business problem. In a virtual lab, students are allowed to expand their knowledge and experience in all categories of cloud services and tools like security, networking, web application, AI and storage, offered by IBM Cloud.

Overall, the custom-built VLab has improved students satisfaction as reflected in students unit evaluations comments and feedback. In Figure 6, the average student satisfaction score of the units, "COIT20260-Cloud Computing and Internet of Things for Smarter Applications" and "COIS13034-Cloud Based Smart Applications Management", are plotted where *X*-axis represents year and *Y*-axis represent success rate in 5-Point Likert Scale with 5 being the best. As illustrated in Figure 6, the student satisfaction score gradually increased from 4 to 4.9 (in average) out of total 5. The student satisfaction score is calculated based on quality teaching, assessments and learning resources. Both of the units have used custombuilt VLab in the unit since 2017. As their were no changes in the unit except introduction of VLab in 2017, we can safely conclude that the VLab has contribution to improve students satisfaction which is also evident in students feedback for the units. Although there were adjustment and struggling periods for first two offerings, in 2017, as detailed in Section 4. These challenges were due to adaptation and evolving changes of technologies around the VLab, however, the advantages overpowered all the challenges.

**Figure 6.** Student satisfaction scores in Cloud Computing units since the adoption of the custombuilt VLab.

### **4. Discussion and Limitations**

From constructivism theoretical perspective, VLabs can be a great platform for teachers to use as a tool (objective sense) for designing problem-based learning (PBL) assessments for students that will be solved (subjective sense) using industry scale emerging technologies. Based on constructivism theoretical perspectives, George Siemens [16] stated that connectivism views that learners should develop a learning network and make connections between ideas embedded throughout that network. In the context of ICT tertiary education, VLabs are connected online platforms for ICT students to solving the real business problems. By getting help and guidance from the teaching team, the paper has found that student satisfaction score is higher in the units that implemented VLab compared to their previous offerings without VLab. The adoption of MindTap and custom-built VLabs have provided the students equitable access to an industry-relevant learning environment to prepare work-ready graduates skilled in emerging technologies. This has also allowed us to support students with targeted guidance, and enhanced student engagement and completion of the practical tasks, leading to higher success rates. This study has revealed that using virtual labs to replace or complement the physical lab facilities supports the ubiquitous learning from any place, time and pace in a flexible environment.

VLab is a technological tool [24] which gives facility to students to contribute in Problem Based Learning (PBL) and synthesize ideas, developing learning network and solving any real business problem using theoretical knowledge by making connections between ideas embedded throughout the network [25]. In Vygotsky's time the psychological/symbolic tools were identified as language, sign, letters, mathematical codes etc., however, we have found in our current study that in the 21st century, technological tools can be addressed as teaching tool where it meditated as object to organize individual cognitive and learning functions in different contexts.

Our observations from these case studies have indicated that even though the custombuilt and commercial VLabs are allowing students to solve real business problems using cloud hosted tools to fulfill assessment criteria, they need human facilitators to give guidance and help to overcome mentioned challenges. Therefore, our conjecture is that custom-built and commercial VLabs both can be teaching tools to use for solving any practical business problems by students, but without human mediation (teachers' support) the VLabs will be identified as another content item (teaching material), rather than a tool as a learning material. We observed that using VLabs with support from the teaching team, students had the opportunity to solve real business problems. This provided them a sense of satisfaction with their learning journey which students expressed through their feedback comments in the end of term unit evaluation surveys run by the university. This experience may have significantly contributed to the enhanced satisfaction scores of the units. Figure 7 illustrates that our result shows teachers' support (human mediator) and VLabs (teaching tool) are interlinked together in a dialectical way which is important to consider to achieve successful learning outcomes.

Despite the overall success, the VLab has raised unique challenges in its initial offerings due to some students' traditional expectation to perform lab exercises using physical lab setups. This was successfully overcome using support of the entire teaching team (human mediator), training documents and video instructions (teaching tool) for students. Furthermore, the evolving nature of emerging technologies like IBM Cloud and IoT technologies also added an extra layer of complexity which was addressed by continuous testing of entire virtual lab setup throughout the teaching term.

It is important to note that the VLabs' availability is highly dependent on the steady internet connection. While this generally is not an issue in Australia, occasionally some students face internet connection issues which can cause dissatisfaction among them. However, students are able to continue their work and complete it at a later time when they get their internet connection back. So, with a bit of patience to persevere during the occasional internet connection breakages, VLabs can offer a lot of benefit to the institutes and their learners.

**Figure 7.** Dialectical relationship of teachers' support and VLabs for achieving unit learning outcomes.

In this case study, we have reflected on our experience from teaching specific ICT tertiary subjects using the mentioned approaches to VLabs, hardware setups, and software tools, over a number of years. The learning theories that we have discussed in this paper aligned with our practice of how we used the VLabs in our teaching, which may not be seen as a generalization for VLab usage. We also note that our findings are not based on a research study, rather from a case study, however, our observation is on-going and the practice is supporting our students' learning experience positively. Nevertheless, our experience and conjectures will be helpful to educators when deciding on the adoption of VLabs in their teaching.

### **5. Conclusions**

In this paper, we have reviewed the use of Virtual Laboratories (VLabs) and their utility in the tertiary education sector through the lens of learning theories, to understand their features and benefits, and most importantly, how VLabs should be used. Accessing online lab facilities to undertake hands-on activities from anywhere and anytime is imperative not only during COVID-19 lockdown periods, but in general at anytime as most of the universities are offering online mode teaching. VLabs, accessed via an internet connection, can offer learners the crucial infrastructure required to complete their lab tasks without attending physical lab facilities which is particularly helpful to distance education students. On-campus students can also re-emphasize their learning using the on-line labs outside their class times. Despite their benefits, choosing the right type of VLab is not a simple task since it must be accompanied by an effective and improved learning process. The presented case studies and analyses have revealed that teachers' support (human mediator) and VLabs (teaching tool) are interlinked together in a dialectical way which is important to consider to achieve successful learning outcomes.

**Author Contributions:** Conceptualization, J.H. and B.R.; methodology, J.H. and A.D.; validation, A.D.; formal analysis, J.H., A.D.; investigation, J.H. and B.R.; resources, J.H.; writing—original draft preparation, J.H., A.D. and B.R.; writing —review and editing, J.H., A.D. and B.R.; visualization, J.H., A.D.; project administration, J.H.; funding acquisition, J.H. and B.R. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by CQUniversity Learning and Teaching Awards LR6321, LR6356.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**


### *Article* **Comparison of In-Person and Virtual Labs/Tutorials for Engineering Students Using Blended Learning Principles**

**Maren Schnieder \*, Sheryl Williams and Sourav Ghosh**

Wolfson School of Mechanical, Electrical and Manufacturing Engineering, Loughborough University, Loughborough LE11 3TU, UK; s.r.williams@lboro.ac.uk (S.W.); s.ghosh2@lboro.ac.uk (S.G.) **\*** Correspondence: m.schnieder@lboro.ac.uk

**Abstract:** The paper compares the effectiveness of in-person and virtual engineering laboratory sessions. The in-person and virtual laboratory sessions reported here comprise six experiments combined with short tutorials. The virtual lab combined enquiry-based learning and gamification principles. The integration of the virtual labs with in-person teaching created a blended learning environment. The effectiveness of this approach was assessed based on (i) the student feedback (i.e., a questionnaire with open-ended questions and Likert scale feedback), (ii) the students' engagement with the virtual lab, and (iii) the impact on the academic performance (i.e., class test results). The students reported greater confidence in the understanding of theory in the virtual lab than the in-person lab. This is interesting given that the instruction for the virtual lab and the in-person lab of one experiment is identical (i.e., same instructor, same enquiry-based learning techniques, and same explanations). The students also appreciated the ability to complete the virtual lab anytime, anywhere, for as long as they needed, and highlighted the benefits of the interactivity. The median class test scores of the students who completed some or all the virtual lab experiments was higher than those who did not (83–89% vs. 67%).

**Keywords:** gamification; flipped classroom; virtual labs; remote lab; virtual lab; enquiry-based learning; inquisitive learning; interactive learning

### **1. Introduction**

The requirement for online learning during the pandemic [1,2] offered valuable opportunities to create and test virtual versions of in-person labs as well as to combine online and in-person laboratories in a blended learning environments. Students receive face-to-face instructions as well as complete activities at home using various technological resources in a blended learning environment [3].

A specific form of blended learning is the flipped classroom methodology [4]. It is an active learning methodology [5] and is regarded as one of the most innovative pedagogical approaches [6]. In a traditional classroom, students learn the content in the lecture and practise on their own at home through homework [6]. The learning activities are rearranged in a flipped classroom setting [4]: the students learn at home using virtual resources, such as videos or texts, before attending the lecture. By doing so, the students are better prepared for the face-to-face session to practise and apply what they have learned beforehand [7]. The interest in flipped classrooms is on the rise [8] due to the increasing availability of emerging technologies [4] such as Web 2.0 tools for use as personal learning environments [9]. This allows students to learn 'just-in-time' and provides them with 'at-your-fingertips' learning opportunities [9]. By personalising the learning process and changing it into a learnerdriven rather than a tutor-driven approach, the students are empowered to control their own educational process [4] and adapt their own learning environment to their personal preference [9].

**Citation:** Schnieder, M.; Williams, S.; Ghosh, S. Comparison of In-Person and Virtual Labs/Tutorials for Engineering Students Using Blended Learning Principles. *Educ. Sci.* **2022**, *12*, 153. https://doi.org/10.3390/ educsci12030153

Academic Editor: Maria Limniou

Received: 31 January 2022 Accepted: 20 February 2022 Published: 23 February 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Another promising learning technique is gamification [10,11]. The aim of gamification is to increase the student's motivation, engagement and productivity [12]. Gamification usually does not require actual games to be included into the learning environment. The focus should rather be on integrating game design elements into the learning environment [12] in order to transmit similar experiences [11]. Gamification can be used to increase the learning performance, commitment, and satisfaction of the students, which are important elements in educational environments [13].

The contribution of this paper is an evaluation of a virtual lab for first-year mechanical engineering students. The virtual lab was integrated into the module as a flipped classroom lab (i.e., integrated with the in-person lab) and as a remote lab (i.e., to replace the inperson lab) depending on the student's circumstances. It included gamification elements to increase student motivation, comprehension, and engagement with the lab.

At first, the students' engagement with the virtual lab was analysed using the data gathered by the online platform of the virtual lab. Then, the feedback from the students about the virtual lab was evaluated. Finally, the exam scores of the students completing only the virtual lab experiments (V21), only the in-person lab experiments (P21), a mixture of both in-person lab and virtual lab experiments (Mix21), and both virtual and in-person versions of all the experiments (VP21) were assessed and compared along with student scores in the previous two years (P19, P20).

This paper builds on a shorter survey about the same virtual lab with fewer participants published in [14]. Due to the overwhelmingly positive feedback from the first short survey, we created a more in-depth survey to gain valuable insights into the students' opinion. By doing so, we not only evaluated the opinion of the students about the virtual lab but also their measurable performance.

### **2. Literature Review**

### *2.1. Flipped Classroom*

Most of the research studies on flipped classroom principles focused on higher education [13]. Akçayır et al. [13] conducted a systematic literature review of 71 studies about flipped classrooms. They concluded that 52.1% of all studies reported improvement in student performance based on their GPAs, standardized test scores, and course grades. 18.3% and 14.0% of the studies reviewed in [13] reported increases in student satisfaction and level of engagement, respectively. Additionally, Akçayır et al. [13] listed the following as common benefits of flipped classrooms: motivation (9.9% of the studies), increased knowledge (9.9%), improved critical thinking skills (8.5%), and confidence (7.0%). A plausible reason for this could be the greater interaction in the face-to-face learning part of the flipped classroom methodology and the increased responsibility of the students to complete the preparatory work for the face-to-face learning part [3]. The most commonly mentioned pedagogical contributions were enabling flexible learning (22.5% of the studies), individualized learning (11.3%), enhancing enjoyment (11.3%), better preparation before class (8.5%), fostering autonomy (8.5%), offering collaboration opportunities (5.6%), and more feedback (5.6%). The benefits of the increased focus on active learning in the face-toface lectures is frequently mentioned in the literature [3]. Additionally, a flipped classroom allows for a more efficient class time (12.7%) and more time for practice (7.0%) [13].

On the other hand, some researchers concluded that creating videos for flipped classroom models might not be worth the effort [13]. They suggested focusing on selecting appropriate in-classroom activities instead [13]. In fact, 14.0% of the studies reviewed by Akçayır et al. [13] reported an increased time consumption due to flipped classroom, and higher workload (7.0% of the studies [13]), especially for creating quizzes for interactive learning [15]. Other researchers reported that flipped classroom could be a cost-effective option when the student numbers were increasing or funding for staff was reducing [3]. Limiting factors reported for flipped classroom were a lack of student preparation before class (12.7% of the studies), students needing guidelines at home (9.9%), and being unable to seek help while out-of-class (9.9%) [13]. A total of 11.3% and 9.9% of the studies reported

that students found the flipped classroom too time consuming and that their workload increased, respectively [13]. A total of 8.5% of the studies reported that students did not prefer flipped classroom [13], possibly because they were used to passive learning from traditional lectures, which required less engagement and proactive effort from them [16]. A total of 5.6% of the studies reported that students had adoption problems and felt anxious about the flipped classroom [13]. The flipped classroom experience can also be negatively influenced by the video quality (12.7% of the studies), inequality of technology accessibility (8.5%), and the need for technology competency (7.0%) [13]. A potential way to overcome these problems could be providing support staff to help lecturers create pre-class activities [3]. Further, O'Flaherty [3] argues that students nowadays expect technology to be used in their learning environment.

According to Akçayır et al. [13], the most common out-of-class activities are videos (78.9%), reading (49.3%), and quizzes (42.3%). Özbay et al. [7] reported that most studies used online lectures (45.8%), online videos (37.5%), textbook (29.2%), online quizzes (20.8%), and power point lectures (20.8%). This is in line with O'Flaherty et al. [3], who found that most flipped classrooms used pre-recorded lectures, annotated notes, automated tutoring systems, and interactive videos. According to O'Flaherty [3], most pre-class activities used in the studies they reviewed were taken from pre-existing resources, such as videos from the Khan Academy (https://www.khanacademy.org/, accessed on 29 January 2022). Most studies regarded quizzes as beneficial as they allowed measurement of student learning [13], provided a reason [17], and encouraged the students to complete the out-ofclass activities [18].

### *2.2. Gamification*

The research on gamification in education is continuously increasing since 2013 [12]. While gamification was first applied in the marketing and business sector [11], it is nowadays most commonly used in computing subject areas [12]. Gamification is most frequently used in universities and companies as an in-house training strategy [11]. Gamification is aimed at increasing the intrinsic motivation of students [19]. Based on the systematic literature review by Subhash et al. [12], most studies reported improved attitude, engagement, motivation, and student performance. The most frequently used game elements are badges, leader boards, levels, feedback, and points [12]. Alhammad et al. [11] provided a comprehensive overview of literature reviews on gamification.

### *2.3. Surveys*

The most common method to evaluate flipped classrooms are surveys with Likert scales and open-ended questions [3]. Several studies such as [5,18,20] evaluated the effectiveness of the flipped classroom methodology based on, for example, surveys to measure student satisfaction and comparison of the exam results to measure the performance. According to Özbay et al. [7], the most common study form was a pre-test and post-test measurement of a single group (i.e., flipped classroom taught group) as well as pre-test and post-test measurements of two groups (i.e., traditionally taught group vs. flipped classroom taught group).

### **3. Methodology**

### *3.1. Overview*

The way students engage with laboratory sessions had to be adjusted in the academic year 2020–2021 due to the restriction of in-person teaching caused by the pandemic. Hence, it was necessary to create a virtual version of the in-person lab. The lab is part of the Statics & Dynamics module (first year, second semester) for mechanical engineering students at Loughborough University, UK. Between 150–170 students enrolled each year between 2019 and 2021.

### *3.2. Description of the Experiments*

The lab includes the following six experiments: Experiment 1: Epicyclic Gear Train Experiment 2: Rolling Down an Inclined Plane Experiment 3: Toppling vs. Sliding on Inclined Plane Experiment 4: Three Bar Linkage–Crank Connecting Mechanism Experiment 5: Four Bar Linkage Experiment 6: Energy Methods A detailed explanation of the labs can be found in [14].

### *3.3. Description of the In-Person Labs*

Before the COVID-19 pandemic, 12 groups of around 3 students worked on the 6 experiments (two sets of equipment per experiment). The groups were facilitated by 3 instructors in addition to a supervisor in-charge of overseeing all of the lab operations. The Epicyclic Gear Train experiment (experiment 1) requires the most instructions due to the complexity of the gear changes. Hence, one instructor supervised both groups of students who worked on experiment 1. Another instructor supervised the four groups working on experiment 2 and 3. The third instructor supervised the remaining students. The student-to-instructor ratio was between 6 students (experiment 1) and 18 students (experiment 4–6) per instructor. The students had 40 minutes to complete each experiment.

### *3.4. Description of the COVID-19 Compliant In-Person Labs*

To reduce the risk of spreading COVID-19, a few changes had to be made to the lab. The students had to show the results of a recent COVID-19 test on their phone before they entered the lab and had to clean the equipment after they finished the experiment. We allowed the students to arrive and enter the labs during a specified timeframe to reduce queuing. We reduced the time per experiment from 40 min to 30 min to give the students sufficient opportunity to clean the experimental rig. The students worked alone and only rarely in pairs. To compensate for this, we increased the number of instructors to 4 instructors, plus a supervisor in charge of overseeing the lab operations (note: three of the instructors in 2021 were also instructors in 2020), therefore resulting in a student-instructor ratio of between two to a maximum of eight students per instructor (usually 2–4 students per instructor). The student-to-instructor ratio for the Epicyclic Gear Train experiment was 2 (maximum 4) students per instructor. Lower student-teacher ratios allowed the instructor to guide the students to the correct answer by applying inquisitive learning strategies. The instructor for the Epicyclic Gear Train experiment created the virtual labs. Hence, the instructor knew the questions used in the virtual lab to guide the students to the correct answer. Therefore, the students were guided to the correct answer in the virtual lab and in the in-person lab in the exact same way. This is not the case for all other experiments.

### *3.5. Description of the Virtual Experiments*

The virtual lab was implemented on Loughborough University's web-based virtual learning platform, LEARN. Hence, the students were familiar with the learning platform. Following the definition by Heradio et al. [21], the platform was classed as a remote accesssimulated resource. Web-based applications have the advantage that they do not require a specific operating system and are therefore more portable as well as not requiring access to the user's hard disk [21].

The goal was to create the virtual version of the in-person lab to be as interactive as possible but without developing any new software. The students collected data in the virtual lab as they would do in the in-person lab. For example, they measured the time required for a cylinder to roll down an inclined plane using a stopwatch in experiment 2, or measured angles using a real protractor in experiment 4 and 5. Using real/physical equipment during the virtual lab allowed students to experience the hands-on feeling similar to in-person experiments.

Multi-cam editing and slow-motion cameras were used to enable students to gain a better understanding of the mechanics of the experiment. For example, they saw the Energy Methods experiment (experiment 6) from three angles in slow motion to enable the students to see the relationship between the tension in the spring, the extension of the spring and the movement of the weights in the dynamic experiment. Slow-motion footage allowed the students to determine the exact video frame when the cylinder starts toppling. Hence, the measurement precision of the virtual experiments was usually more precise compared to the in-person experiments.

Active learning methodologies such as inquisitive learning and inquiry-based learning techniques have been used to guide students to the correct answer in the interactive tutorials instead of telling them the answer. Inquiry-based learning encourages students to discover information themselves instead of the teacher stating facts [22]. It is regarded as one of the most important teaching models and enhances the self-learning skills as well as the problem solving skills of the students [22,23]. These learning techniques have been used to guide students to the correct answer in the interactive tutorials as opposed to telling them the answer.

To create an interactive learning environment, a variety of interactive question types have been used instead of multiple-choice questions. For example, the students used building blocks to construct equations and to draw free body diagrams etc. The students also used physical measuring equipment (e.g., stopwatch, protractor) to measure the data in the experiments. This level of interactivity ensures that the students must at least try to complete the virtual experiment and try to work through the theory before receiving any feedback. The instructor can check how the students interact with the virtual lab through the web-based virtual learning platform. Hence, it is not enough for the student to simply log into the web based virtual learning platform and then just do other things. In order to obtain the points or progress through the virtual lab, they have to physically interact with the virtual lab.

It should be noted that the virtual labs were created by the instructor who supported the first experiment (i.e., Epicyclic Gear Train) in the in-person lab. Hence, both the virtual version and the in-person version of the Epicyclic Gear Train experiment use the exact same script of questions to guide the students to the correct answer. In fact, the scripts were so similar that the instructor did not need to finish a question before the student gave the correct answer as they recognised the question from the virtual lab. This has the advantage that the virtual version of the first experiment can be seen as an exact copy of the in-person lab. Differences in the results are therefore caused by the delivery mode (virtual vs. in-person) and not due to the explanation given.

The students spent as much time as they wished on the virtual lab, worked at any time, from anywhere, and repeated the lab as many times as they wish.

Sometimes it was easy to predict which mistakes the students would make, and therefore the students received feedback with hints to specific mistakes they made. The performance of the students in the virtual lab was also monitored to identify opportunities to improve the virtual lab for future cohorts. The possibility to monitor the students' performance while doing their out-of-class work is a unique advantage of flipped classroom and blended learning [24].

### *3.6. Evaluation Methods for the Effectiveness of the Virtual Lab and the Blended Learning Experience*

The effectiveness of the virtual lab and the blended learning experience has been evaluated based on three aspects: first, the engagement of the students with the virtual lab was analysed, as the virtual lab was not a mandatory part of the module and could be done either before, after, or in between the in-person lab sessions. Second, a post-course survey has been used to gain insight into the students' satisfaction and recommendations. Third, the academic performance has been evaluated by comparing class test results of various groups of students based on a Kruskal–Wallis test.

All students who completed all virtual labs regardless of whether they performed the in-person lab were asked to complete the survey (Appendix A). The survey included open ended questions and Likert scale feedback. The survey was approved by the Loughborough University Ethics Review Sub-Committee (2021-5123-3838). A total of 25 students completed the survey between 13 May 2021 and 4 June 2021.

In addition, at the end of the semester, all students took part in a class test, which was conducted online using LEARN which provided insights on the improvements in learning outcomes as a result of the virtual lab.

### *3.7. Description of the Study Groups*

Three different year groups have been compared. The 2018/2019-year group attended the in-person lab as normal. While most of the students in the 2019/2020-year group attended the in-person labs as normal, some students could only attend three of the experiments in-person and watched videos of the other three experiments due to the COVID-19 lockdown. Note: these videos were not the same as the videos used in the virtual labs. We accommodated seven different groups of students the 2019/2020-year: (1) those who only performed the virtual labs as they could not return to campus, (2) those who only performed three of the experiments as virtual labs and the other three experiments as in-person labs, as they returned too late on campus to complete all labs in-person, (3) those who completed all six experiments virtually before the in-person labs, (4) those who completed the six experiments in-person before doing them virtually, (5) those who performed all labs in-person and completed the virtual labs in between the two in-person session, (6) those who completed all labs in-person and no virtual labs, and (7) those who have done neither the in-person lab nor the virtual lab.

We are comparing the following six groups:

P19: Completed the in-person labs in the year 2019.

P20: Almost all students completed the in-person labs in the year 2020 (only a few students were unable to attend the second half of the labs because of the COVID-19 lockdown).

The groups for the year in 2021 are combined as follows:

P21: Completed all COVID-19 approved in-person labs in the year 2021 but no virtual labs. V21: Completed all virtual labs in the year 2021 but none of the in-person labs. VP21: Completed all virtual labs and COVID-19 approved in-person labs in the year 2021. Mix21: Worked on a mixture of virtual labs and COVID-19 approved in-person labs in the year 2021.

### *3.8. Limitations*

The students were not allocated into specific groups and chose the group that fits best to their living situation (e.g., cannot return to campus). While this approach was appreciated by the students, the disadvantage is that this study is not a randomised control study. Hence, the results might be biased given that the students in group VP21 are probably the students who are rather keen and might therefore be better performing students. On the other hand, the effort students put in also varies in a properly randomised study and therefore might affect the results especially when the number of students is low. The authors refrain from conducting statistical test in most cases given that the number of students who took part in the survey was low.

### **4. Results**

### *4.1. Engagement with Virtual Lab*

Given that the virtual lab was not mandatory, each experiment had only been attempted or completed by 101 to 118 of the 166 students who signed up for the module.

Table 1 illustrates the students' interaction with the virtual lab; 100% represents the total number of students who had at least attempted a virtual experiment. Around 50% of the students who attempted a specific experiment completed it at least once. Interestingly, the experiments have been completed multiple times by around 9% of the students. For example, they could have completed it as preparation for the in-person lab session and later as preparation for the class test.

**Table 1.** Attempts and completions of the virtual lab (100% is the number of students who attempted or completed a specific virtual lab).


A possible reason for one third of the students not attempting any virtual lab could be that the virtual lab was too time consuming (Figure 1). The median duration the students take to complete a virtual experiment are 35 min, 31 min, 26 min, 20 min, 16 min, and 14.5 min. While the number of questions/tasks varies between the virtual labs (i.e., 17, 10, 13, 9, 13, and 15), the main reason for the reduction in the duration is most likely the increasing familiarity with conducting experiments virtually.

**Figure 1.** Distribution of the time-scale for each student to complete one of the six experiments in the virtual lab (using only the highest aggregated grade attempt for each student) (dotted line = median).

Figure 2 shows the score of the students for each of six experiments in the virtual lab (i.e., the highest score achieved for students who attempted the lab multiple times). The scores (i.e., points for correctly answered questions divided by the maximum number of points) of the Epicyclic Gear Train (experiment 1) are the worst, which could be caused by the students not being familiar with doing a lab virtually, but also because it is the most difficult experiment. Apart from the first two experiments, the average score is always larger than 70%.

**Figure 2.** Distribution of scores achieved by each student in one of the six experiments in the virtual lab (only the highest score of each student is illustrated here) (dotted line = median).

### *4.2. Student Feedback and Satisfaction*

### 4.2.1. Self-Perceived Level of Difficulty of the Virtual Lab

A post-course survey was conducted with students who completed the virtual labs with 25 responses. The students were asked whether the level of difficulty of each experiment in the virtual labs was appropriate (Figure 3). Most students were happy with the level of difficulty of the virtual lab. Apart from the 5th experiment, only one or two students selected disagree and no student selected strongly disagree. Experiments 2 and 3 were the easiest, and experiment 5 was the most difficult. This is in-line with the results from an earlier, shorter survey published in [14]. While the tasks in experiments 4 and 5 were the same, the students used a Three Bar Linkage in experiment 4 and a Four Bar Linkage in experiment 5. With the only difference being that the students received fewer hints in experiment 5 compared to experiment 4. Hence, it is understandable that the students struggled more with experiment 5 compared to 4.

**Figure 3.** Level of difficulty of the virtual lab is appropriate (values in percent).

4.2.2. Confidence of Understanding the Theory of the In-Person and Virtual Lab

The students were asked whether they felt confident that they understood the theory after they completed either the in-person and virtual lab. Figure 4 only includes the students who completed all virtual and in-person labs. Figure 4 shows that more students understood the theory after doing the virtual lab than the in-person lab. Apart from the last two experiments (Exp5 and Exp6), the number of students who were confident that they understood the theory after doing the virtual lab was between 23 pp and 33 pp higher than the in-person lab. Between 81% and 90% of all students rated the virtual lab better or the same as the in-person lab. Apart from the first experiment (Exp1), between 38% and 48% rated the virtual lab better than the in-person lab. A possible reason for this is that Exp1 was the only experiment where active learning techniques were used in both the in-person and the virtual labs.

**Figure 4.** Confidence of understanding the theory after completing either an in-person or virtual lab (only students who completed both, the in-person, and virtual labs) (*N* = 21) (values in percent).

The authors refrain from conducting a statistical test to determine whether there is significance due to the low number of participants.

It was interesting to note that 24 pp more students agreed or strongly agreed that they understood the theory after doing the virtual lab compared to the in-person lab of the Epicyclic Gear Train (Exp 1) given that both were presented by the same instructor using the same script/questions. A possible explanation is that the practical part of this experiment takes at least 20 min, leaving only 10 min for the theory. Hence, the students might feel rushed to give an answer without having enough time to think properly. In contrast, the students spent significantly longer on the theory in the virtual lab. This might be caused by students being hesitant to guess answers in the virtual lab given that their answers will be recorded and visible to module tutors and module leaders. (Note: the virtual lab is voluntary and not graded). Due to this, the students spent more time thinking about an answer which could have a positive effect on their understanding. In the survey, 86% of students agreed or strongly agreed that the virtual lab gave them more time to understand the theory than the in-person lab and 33% of the students struggled to finish the in-person lab within the allowed time frame.

From the above observations, it can be reasonably concluded that the virtual lab was better in teaching the theory than an in-person lab. This implies that inquisitive learning techniques cannot only be used for in-person teaching but also be used successfully for interactive online tools, like the virtual lab.

In Figure 5, all 25 students were included (21 students who performed both the inperson and virtual labs and four students who only performed the virtual lab). Three of these four students always agreed or strongly agreed that they understood the theory of the virtual lab. If all 25 students are included, then 72–92% of the students agreed or strongly agreed that they understood the theory after doing the virtual lab.

**Figure 5.** Confidence of understanding the theory of the virtual lab (all students) (*N* = 25) (values in percent).

4.2.3. General Questions and Satisfaction of the Students with the Virtual Lab

As can be seen in Figure 6, some students seem to struggle to finish the in-person lab in the allocated time (agree + strongly agree: 36%), and the majority of students agreed (59%) or strongly agreed (27%) that the virtual lab gave them more time to understand the theory.

While 64% of the students agreed that the theory is easier to understand in the virtual lab (18% disagreed), only 18% of the students agreed or strongly agreed that it was easier to perform the experiment online (56% disagreed or strongly disagreed). The majority of students seems to prefer to be taught the theory through the virtual lab but want to complete the experiment in-person. With only 9% of the students agreeing or strongly agreeing that the virtual lab should replace the in-person lab, it can clearly be seen that the students see the benefits of both and would not want to miss either. In fact, there was a clear consensus that both the virtual and the in-person lab should be offered (agree or strongly agree: 91%, nobody disagreed).

All students agreed (35%) or strongly agreed (61%) that the opportunity to complete the virtual lab at any time is beneficial.

Given that more students did not feel that the virtual lab was more engaging than the in-person lab, it seems that the virtual lab needs improvements. However, it is possible that the students were simply had enough of online learning, as only online learning was allowed for most parts of their first year and lectures were still online at the time of the lab. The students might have simply enjoyed mixing with their friends in person in the lab. (Note: Household mixing indoors was still illegal at the time of the lab and most students were not allowed to return to campus for the previous few months). Nevertheless, 83% agreed or strongly agreed that the virtual lab improved their learning experience and 43% of the students agreed or strongly agreed that they encouraged others to complete the virtual lab as well. This compares to only 13% of the students who disagreed with that

statement. Additionally, 55% agreed or strongly agreed that they felt more motivated to complete the theory in the virtual lab than in the in-person lab. Only 32% disagreed or strongly disagreed with that statement.

**Figure 6.** General Questions (sum is not 100% given that students could choose not to answer a question if they, e.g., have not done the in-person lab) (values in percent).

### 4.2.4. Open-Ended Feedback Questions and Student's Opinion

In the last part, of the survey, the students were asked to write down what they liked and did not like about the virtual lab.

A few major themes were prominently in the responses: (i) the ability to complete the lab on any day at anytime from anywhere, (ii) interactivity, and (iii) increased confidence and understanding. Seven students commented that they appreciated that they were able to complete the virtual lab at any time. One student stated for example: "Being able to access the material as a student who couldn't return to Loughborough this term was very useful". Several students mentioned that they appreciated the interactivity of the virtual lab. They benefitted from testing their understanding and receiving immediate feedback. The different question styles increased the engagement with the virtual lab. Three students mentioned that the virtual lab increased their confidence in the in-person lab. One student stated, e.g., "Every virtual lab increased my confidence and understanding of the experiment." Seven students commented positively about the explanations, hints, and videos. One student stated, "In the planetary gear experiment the explanation was

really easy to follow, and it gradually became less spoon-fed which was good for practice." Another student stated, "the videos were helpful in explaining the theory and were concise too". However, one student felt that the explanations were not detailed enough. We did not intend for the students to complete the virtual lab while they performed the in-person lab. However, one student completed the virtual lab while doing the in-person lab at the same time: "It followed the lab sheets with the same order so it was easy to complete both at the same time. There was an opportunity at every stage to check whether you've made a mistake." This option of conducting the virtual labs and the in-person labs could be worth investigating further. Three students mentioned that it was difficult to motivate themselves to complete the labs and it is very time consuming: "I also found myself getting bored a little bit quicker than I might in the normal lab [ ... ]". Only two students mentioned that they were missing the face-to-face interaction of the in-person lab.

Overall, the students seem to be happy with the virtual lab and it increased their confidence in understanding the in-person lab. Especially the interactivity and convenience improved their learning experience.

### *4.3. Student Performance: Test Results*

After removing all students who scored 0 (i.e., did not take part in the in-class test), the number of students in each group were 152 (P19), 169 (P20), 67 (P21), 42 (Mix21), 9 (V21), and 27 (VP21). The median scores were identical in 2019 and 2020 (i.e., P19: 79%, P20: 79%). This result might seem unexpected. However, only the last in-person lab session had to be cancelled due to lockdown. Hence, only a small group of students were affected by this. Note: The class test was online in 2019, 2020 and 2021. A clear difference can be seen in the scores of the students in 2021 (Figure 7). The median score of the students who only attended the in-person lab (P21) is lower than all other groups (72%). This is most likely caused by the change to COVID-19 secure labs, which negatively affected the learning experience compared to the normal lab. The median score of the students who completed a mixture of virtual and in-person labs (Mix21) was 83%, who completed only virtual labs (V21) was 89% and those who completed both (VP21) was 83%.

**Figure 7.** Class test scores of various groups of students (P19 and P20 as well as Mix21 and VP21 have the same median score).

A Kruskal–Wallis test and a t-test was conducted on two groups at a time to determine whether the scores are statistically different. Kruskal–Wallis test compares ranks and is a nonparametric test. This test is more suitable for this dataset due to the large variation of the number of students per group [25]. A significance level of *p* < 0.05 was chosen. As can be seen in Figure 8, there is no statistical significance between most groups.

**Figure 8.** Statistical significance (*p*-value) for each combination (i.e., pairs) of the groups. Kruskal– Wallis test (**a**), *t*-test (**b**) (white: no statistical significance).

The group P21 and P20 are different from the group V21 and VP21. The p-values range from *p* = 0.014, to *p* = 0.023. Hence, the students who completed all virtual labs performed better than the students who did not. Note, the groups P19 and P20 are significantly different in both tests even though the median is the same. This can be explained by the fact that the Kruskal—Wallis test tests ranks and not the median.

### **5. Conclusions**

The student feedback from the survey highlights the students' appreciation for the virtual lab. While the students rejected the suggestion to use the virtual lab as a replacement for the in-person lab, most students preferred to complete both the in-person and the virtual lab. This result is interesting given that this option apparently doubles the workload of the students. The main advantages of the virtual lab mentioned by the students was the ability to complete the virtual lab anytime from anywhere. In addition, the interactivity of the virtual lab was appreciated by the students. Based on a Kruskal–Wallis test, it can be concluded that the test scores in a class test are significantly different between students who completed all and those who completed none of the virtual labs. The median score for the students who completed all of the virtual labs was higher than for the students who completed only the COVID-19 secure in-person lab. This indicates that students not only preferred having a virtual version of the in-person lab, but it also improved their learning outcomes. Even though the virtual lab was not mandatory, each of the six experiments was fully completed at least once by half of the students who attempted it. Around 9% of the students completed the experiments multiple times.

The results of this study indicate that it is best to offer both virtual and in-person learning environments to maximise student satisfaction, learning outcomes, and class test performance.

### **6. Future Work**

In future work, students will be randomly allocated in groups instead of allowing them to choose their preferred group and study method. In this study, this was impossible due to the pandemic affecting when the students returned to campus, and therefore students were left to choose their preferred study method (i.e., groups) depending on when they were on campus. We hope to collect more data (class test results and survey responses) to increase the statistical significance of future evaluations. In addition, an investigation on whether the virtual lab should be completed before, during, or after the in-person lab will be conducted.

**Author Contributions:** Conceptualization, M.S.; methodology, M.S.; software, M.S.; validation, M.S.; formal analysis, M.S.; investigation, M.S.; resources, M.S. and S.W.; data curation, M.S.; writing original draft preparation, M.S.; writing—review and editing, M.S., S.W., and S.G.; visualization, M.S.; supervision, S.W. and S.G.; project administration, S.W., S.G. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The survey has been reviewed and deemed appropriate by the Ethics Review Sub-Committee at Loughborough University (2021-5123-3838).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **Appendix A**

The survey, which was conducted using Microsoft Forms.





### **References**


### *Article* **Investigating Student Engagement and Making Science Real during a Pandemic: Bioskills at Home, a Case Study**

**Sarah Rayment \*, Karin Garrie, Ishwinder Kaur, Gareth McVicker, Emma Storey, Jody Winter, Luigi A. De Girolamo, Callum Rimmer, David Negus, Carl Nelson, Jonathan Thomas, Michael Loughlin and Jess Dale**

> School of Science and Technology, Nottingham Trent University, Clifton Lane, Nottingham NG11 8NS, UK; karin.garrie@ntu.ac.uk (K.G.); ishwinder.kaur@ntu.ac.uk (I.K.); gareth.mcvicker@ntu.ac.uk (G.M.); emma.storey@ntu.ac.uk (E.S.); jody.winter@ntu.ac.uk (J.W.); luigi.de-girolamo@ntu.ac.uk (L.A.D.G.); callum.rimmer@ntu.ac.uk (C.R.); david.negus@ntu.ac.uk (D.N.); carl.nelson@ntu.ac.uk (C.N.); jonathan.thomas@ntu.ac.uk (J.T.); michael.loughlin@ntu.ac.uk (M.L.); jess.dale@ntu.ac.uk (J.D.) **\*** Correspondence: sarah.rayment@ntu.ac.uk

**Abstract:** Development of key practical skills is fundamental to bioscience courses in higher education. With limitations on access to laboratory time due to the COVID-19 pandemic, a "Bioskills at home" kit was developed to create opportunities for first year undergraduate students to develop these skills using online support resources to guide their activities and build communities of learning. Equipment and activities in this kit enabled students to practice key skills such as pipetting, data handling, experimental design and microscopy, as well as build an online peer learning community through the use of discussion boards and microscopy competitions that encouraged students to explore their local environment. Students who engaged with these activities reported increased confidence in key practical skills. Practical assessment of skills showed that that there was no reduction in the proportion of students who succeeded in achieving the pipetting learning objective compared to previous years, despite a significantly reduced on-campus provision. Although the celebration event to choose the microscopy competition winners was well attended, there was limited use of the discussion boards by students to build a community of learning during the term. Refinement of this initiative will focus on providing greater scaffolding to encourage greater engagement with activities and enhance community building.

**Keywords:** bioscience; home labs; COVID-19; practical skills development; learning communities

### **1. Introduction**

*1.1. Practical Science and Meaningful Learning*

Practical classes are a key aspect of teaching in science as they provide the opportunity for students to experience theoretical concepts in a real-world environment [1]. Such experiences are beneficial in fostering interest in the subject and motivating students, both of which have long been recognised as important factors in student learning in science. Studies in secondary schools highlighted that even sixth form (post-16 education) students (who had been considered to have demonstrated an interest in science through their choice of study) reported that laboratory classes increased their enjoyment and interest in their subject [2]. A recent study further highlighted that university students found that student motivation was increased further if they were able to conduct laboratory-based research (open-ended enquiry) as part of their course [3].

Learning theories, such as Novak's model of meaningful learning [4], support the idea that practical work would enable students to integrate their experiences to create a deeper learning experience. This theory, which evolved from the concepts described in Ausubel's assimilation theory for cognitive learning [5], described that for individuals to form a network of interconnected long-term memories (rather than rote learning) requires

**Citation:** Rayment, S.; Garrie, K.; Kaur, I.; McVicker, G.; Storey, E.; Winter, J.; De Girolamo, L.A.; Rimmer, C.; Negus, D.; Nelson, C.; et al. Investigating Student Engagement and Making Science Real during a Pandemic: Bioskills at Home, a Case Study. *Educ. Sci.* **2022**, *12*, 106. https://doi.org/10.3390/ educsci12020106

Academic Editor: Maria Limniou

Received: 12 January 2022 Accepted: 28 January 2022 Published: 3 February 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

integration of new ideas or information with their pre-existing knowledge. According to Novak's model, to achieve this required both cognitive (reasoning) and affective functions (such as student interest/motivation). More recently, it has been recognised that meaningful learning in a laboratory setting requires not only these domains but also psychomotor functions such as precision [6]. The model of meaningful learning has been examined through the development of the Meaningful Learning in the Laboratory Instrument (MLLI) which has shown the importance of all three domains in student learning [7,8].

It is widely accepted that enquiry-based learning has positive benefits in terms of fostering deeper levels of learning and building students' transferrable skills [9,10]. Studies at a secondary school level show that enquiry-based learning in laboratories develops student metacognitive skills as they have control over problem solving and subsequently build awareness of their own physical and cognitive skills [11]. In this context, development of metacognitive skills would not only lead to more meaningful learning but also enable students to apply these problem-solving skills to other areas of their academic and personal life.

A review of literature relating to university laboratory provision also described how using open-ended enquiry-based laboratories improved learning outcomes as well as being more enjoyable [12]. Examples of this can be seen throughout bioscience disciplines such as the RNA interference experiments described by Kudell [13], contextualisation of investigations into disease-causing proteins using authentic medical case files [14] and development of a programme for students to work together in small groups on exercise physiology projects [15].

However, a report based on a discussion between bioscience academics recognised student interest was only one of several potential benefits of practical classes; they also highlighted the importance of practical classes in developing practical skills, confidence building, awareness of lab health and safety, better understanding of how scientists approach addressing research questions, as well as helping to foster a sense of professional identity [16]. Similar observations about the importance of building practical competencies have been explored in chemistry [17,18].

### *1.2. Institutional Context*

The onset of the current COVID-19 pandemic has presented a significant challenge when planning for delivery of the twelve Royal Society of Biology accredited practical skills-based biosciences courses at our institution. Our undergraduate degree programs usually span three years, with all full-time courses offering the option to additionally carry out an industrial placement ("sandwich") year if students aspire to gain work experience in a workplace setting. Each degree program is comprised of equally weighted modules, some of which are taken by more than one degree program. For example, students on all degree programmes will undertake the "Practical Techniques in Biology" module.

Plans for the 2020–2021 academic year required a balance between our professional body and associated accreditation requirements, the subject benchmark standards as described by the Quality Assurance Agency [19], and occupancy restrictions that were imposed to create a flexible COVID-secure learning and teaching environment. All of this took place amidst changing government guidelines due to emerging scientific discoveries about the nature and transmissibility of the virus [20].

All our bioscience courses have a central ethos of providing quality, work-place-like practical classes linked to course disciplines in our state-of-the-art laboratories [21]. In the 2014 audit of provision of "wet" laboratory opportunities for first year and second year bioscience students across the UK higher education sector, it was reported that students received an average of 98 h (± 39 h) of laboratory time [22]. At that time, our bioscience courses provided above average provision of wet laboratory classes (108 h). However, it should be noted that this has reduced in the intervening years with the increased availability and implementation of dry laboratory options. The COVID-19 pandemic required a compromise in the offering and reduction in the number of opportunities to practice lab skills due to the reduced laboratory occupancy levels mentioned above.

To facilitate the rationalisation of practical session provision, laboratory classes were mapped to learning outcomes and skill sets that students in each course were able to acquire at each level, using a skills tracker that has been developed within the department (see Supplementary Materials, Table S1). A key consideration in this was the expected graduate attributes that were set out by the Royal Society of Biology degree accreditation programme [23]. We also considered which level of the course students were at during the 2020–2021 academic year and which skill sets they already had acquired earlier on in the course. As such, we identified our first-year students, who had already been affected at college or school in 2019–2020 before joining us, as a year group that may benefit most from additional opportunities to gain confidence and familiarity with scientific equipment and procedures while being at home as well as benefitting from more opportunities for community building and peer-to-peer interactions. This level also presented the simplest labs to mimic safely in the home setting.

### *1.3. Supporting Practical Science Outside the Laboratory*

As technology has advanced, the range of methods that academics have available to develop students' understanding of the links between their theoretical and practical work has expanded. In a similar way to practical laboratory experiments, simulated laboratories can have a significant impact on student understanding of processes and real-world phenomena [9]. In addition, creating virtual learning environments (VLE), such as that described by Sotomayor-Moriano et al. [24], allows students to explore aspects of their subject which might otherwise be inaccessible to them. In the scenario described by Sotomayor-Moriano et al., this refers to access to equipment but can also refer to health and safety or ethical concerns that would make the experiment prohibitive (examples are described by Lewis [25]). These virtual enquiry-based laboratory experiences have been described as being as effective as using protocol-driven practical laboratories in terms of student learning, at least as measured by exam performance [26].

A further consideration in the use of resources and simulations is that they can be used alongside laboratory classes (rather than replacing them) to scaffold student learning or skill development. Studies have shown that familiarising undergraduates with material related to their class before the class itself (e.g., using web-based activities and quizzes) increased lab learning gains as measured by the proportion of students who could achieve one of the lab learning outcomes on their first attempt (plating bacteria for single colonies; [27]), as well as having the potential to increase student confidence [28,29] and feelings of preparedness [30].

In response to the COVID-19 pandemic, bioscience academics from numerous universities have created a community of practice called DryLabsRealScience designed to provide support in developing alternatives to wet lab classes and capstone projects [31]. In addition to sharing practice, this group also curates resources that are hosted on the lectuREmotely webpage: this website was part of an initiative by colleagues at De Montfort University to support staff in developing teaching strategies and accessing resources during the pandemic [32].

However, whilst these technologies can have significant benefits, it is important to recognise their limitations: simulations and other resources are limited by their inability to provide hands-on training in the use of individual techniques or pieces of equipment, which is an important aspect of student learning in bioscience disciplines [25]. In the context of this study, for example, virtual resources could be used to support familiarisation with correct pipetting, but it would not prepare students for the experience of performing the pipetting action and identifying the change in resistance at the stopping points. Additionally, the data generated using these models are limited by the assumptions used to create them, which means that in most cases students do not experience atypical data, as can be the case in laboratory experiments. Arguably, students benefit from these experiences as it enhances their problem-solving skills, confidence and resilience in dealing with failed experiments and, potentially, their understanding of the theoretical concepts underpinning

the techniques they are using. In particular, purposefully using a productive failure design as that used by Lam [33] can allow students to learn more deeply than if they initially succeed. This cognitive processing used questions and generation of ideas and explanations to address experimental observations, although, in this study, this learning was a collaborative process that brought students to consensus through constructive argument which could be later consolidated by staff.

### *1.4. Community Building*

Another aspect that has been disproportionally affected by the reduction in practical sessions is the social aspect of the laboratory class. Feeling a sense of connection to their course and peers may be particularly problematic for this year group as they have not made the choice to learn remotely and have not already built a network of peers through previous years of university-level study.

Research from as far back as 1963 has shown that students appreciate the opportunity to learn in an environment in which there is social interaction between themselves, their peers and educators [2]. Indeed, the idea that peers can influence student learning is a concept that has its roots in Vygotsky's social constructivism [34]. In addition, working in small groups in the laboratory has also been shown to help students develop a sense of social cohesion, which leads to a more collaborative and effective learning environment [9].

Learners who are studying remotely often feel a sense of isolation and loneliness, and this negatively impacts both on their learning and retention [35–37]. A key aim of building learning communities is to help these first-year students who have yet to develop a network of peers to feel connected to their course and their peers. These interactions, which can be peer–peer or peer–academic, help to develop students' understanding and sense of belonging, which will support their learning, attainment and progression.

### *1.5. Aim*

The aim of this research was to develop a program of experiments for first year undergraduates to be able to undertake in their home environment to build confidence in key experimental skills and facilitate interaction between students. For first year students who had not had the opportunity to form social networks prior to the COVID-19 pandemic and were going to now receive a blended approach to learning with limited opportunities for face-to-face learning, the development of forum discussion around the activities was considered an important aspect. It was hoped that this initiative would foster a sense of community amongst students as well as build student confidence and competence in key academic skills.

### **2. Materials and Methods**

### *2.1. Planning the Bioskills at Home Experiments*

All bioscience students are required to evidence competency in key laboratory skills in their first year, namely the accurate use of micropipettes and ability to light microscopy for cell counting. This is an on-campus assessment that takes places at the end of this first term after the laboratory sessions have been completed.

In term 1, the number of on-campus laboratory sessions was reduced from six to two so when considering how to align the home experiments, ensuring the students had opportunities to practice assessed skills was a key consideration. A review of the six laboratory sessions that would normally take place in term 1 allowed identification of some that could be easily and safely adapted to be carried out by potential novices at home whilst remaining aligned with the teaching content and learning outcomes (LO). The skills tracker (as outlined in the Supplementary Materials) was used to identify what opportunities for skill development would be "lost" due to a number of first year lab classes not taking place in the 2020–2021 academic year, and these were mapped to ensure there were alternative opportunities available in other years of the course. Where possible, we aimed to further support the development of these skills by their inclusion in the home lab experiments.

In addition to ensuring that home labs were aligned with learning outcomes for the term 1 modules and that they supported development of practical skills, when compiling the list of kit contents, it was considered imperative to minimise the risk of barriers to participation. For this reason, students were provided with everything required, with the exception of the food substances used in the yeasts' growth medium. The pack contained an introductory leaflet which explained what the pack was for, to take care and only use the kit as directed (for safety reasons) and where to access activity instructions; these instructions were hosted through the institution's VLE. Staff were able to monitor engagement with the activities through interaction with these resources or the discussion boards.

### *2.2. Pipetting*

Accurate pipetting is a core bioscience skill that underpins a wide range of techniques and assays that are used extensively throughout undergraduate courses and one that our institution assesses in person at the end of the first term. This assessment usually takes place once students have completed their six term one practical classes, which offer multiple opportunities to practice the skill. However, in 2020/2021, the number of oncampus laboratory opportunities to familiarise themselves with the equipment and process of pipetting prior to assessment were reduced to two due to social distancing. Providing students with opportunities to build confidence and reproducibility in this technique through bespoke exercises using the Bioskills at home kits was therefore identified as a priority. These activities were specifically designed for this initiative, as the development of pipetting skills is integrated into the range of practical classes undertaken by students throughout term 1, rather than being taught separately. Each student was provided with the three micropipettes most commonly encountered in undergraduate practical classes: 2–20, 20–200 and 100–1000 μL (see Figure 1).

**Figure 1.** Essential equipment and consumables that formed the basis of the Bioskills at home kit including micropipettes of three assorted sizes (centre) and a digital microscope.

The activities designed for this section of the Bioskills at home initiative were aimed at supporting the students with handling the micropipettes, guiding them with loading and emptying. Practicing the pipetting action was considered a low-risk activity as it did not always require the use of liquid and for most activities the students could practice using water. A suite of resources was created in the university VLE which incorporated video resources tailored to each activity, written guidance and pre-lab simulations from Learning Science (Bristol, UK) for using different types of pipettes.

The pipetting exercises using 2–20, 20–200 and 100–1000 μL micropipettes were based on the 2,3,5 exercises described by Professor Wolf [38]. The dilution exercise was also developed to aid students in preparing 1 in 2, 1 in 5 and 1 in 10 dilutions using all three volume ranges of micropipettes mentioned above.

Further exercises supported students in developing the ability to check the accuracy and precision of their pipetting skills, as well as troubleshooting issues with pipetting. These experiments gave the students measurable results that they could analyse to selfassess the reproducibility of their pipetting and therefore continue to improve their skills. To accomplish this required the students to use a coloured liquid that could be pipetted onto filter paper and zones of spread measured. It was important to minimise the safety risk and so the coloured solution supplied in the Bioskills kit was a domestic food safe colourant (a blackcurrant cordial concentrate). The final iteration of resources for this activity consisted of introductory material and bespoke videos introducing students to the activities, as well as a targeted video for each of the 4 tasks.

### *2.3. Microbial Growth*

In the first term, students would usually construct a yeast growth curve for *Saccharomyces cerevisiae var. carlsbergensis* in the laboratory by measuring changes in optical density (resulting from yeast growth) of growth media using a spectrophotometer. The aims of this experiment are to aid their understanding of associated lecture content about the rate of microbial growth compared to higher organisms, gain experience in plotting graphs (using a semi-log scale) and develop skills in data analysis. This experiment also supported concepts explored in assessed coursework undertaken as part of the microbiology module (a module taken by many students in their second term); students could use data from their home experiment for this coursework if they chose to.

Due to the need to rationalise the provision of laboratories, students did not complete this experiment in class. However, given that the learning objectives associated with this practical were integrated both within and between modules, this experiment was identified as one that would have value in the Bioskills at home initiative.

There were a number of logistical and safety issues that needed to be overcome to ensure the learning objectives could be met in a home environment. Perhaps the most obvious of these was that the students would not have access to a spectrophotometer. As an alternative to this, students were provided with a fading greyscale number scale (as shown in Figure 2) and were asked to record the highest number that they could observe through their experimental tube. Importantly, working with the scale rather than the spectrophotometer allowed the experiment to be re-designed to incorporate an additional learning objective and was a good way to prepare students who may use McFarland standards (which is widely used in antibiotic susceptibility testing; [39]) later in their course. This learning objective was that students should understand the importance of including controls in the experimental design. In the Bioskills at home version of this experiment, students set up a non-inoculated control to show that the media they had created at home was not contaminated.

Another logistical issue was that it was not possible to provide students with individually prepared tubes of pre-sterilised media for their experiments so it was necessary to consider how students could create a suitable media in a home environment. An important consideration was to ensure that the uninoculated media was clear so that the scale could be read throughout. To achieve this, several different products were tested but Marmite (Unilever, London, UK) mixed into hot water was selected as the basic growth media (see Figure 2). In the development process, it was observed that there was a difference in the growth of the yeast in the Marmite media with or without granulated sugar (freely available in shops) dissolved into the Marmite media (1 heaped teaspoon for each 240 mL of media). Comparison of microbial growth in the presence or absence of sugar formed the primary experiment that students conducted. This experiment, in itself, allowed students to consider what factors could influence the growth of micro-organisms and why this was the case. Students were encouraged to extend this to consider the impact of other substances found in their home environment on microbial growth. For example, how was growth affected by products with potential for anti-microbial properties such as mouthwash?

**Figure 2.** Image (**A**) shows examples of growth (24 h) in control and inoculated samples, in a range of different media created to test which would be the most suitable for student use. From left to right, these were: apple juice (inoculated with yeast); Marmite/sugar media (uninoculated control); Marmite/sugar media (inoculated with yeast); Marmite/sugar media (inoculated with soil). Note the difference in opacity between the middle two samples, indicated by the visibility of a metal skewer held behind them. This concept evolved into a number scale used in the final version (which is shown in image (**B**)).

There were a number of health and safety issues that needed to be addressed for this experiment to be suitable for the home environment. These are outlined in Table 1, alongside the steps taken to address them.

**Table 1.** Summary of the health and safety considerations for the Bioskills at home microbial growth experiment.


### *2.4. Microscopy*

Under normal circumstances, students would have multiple opportunities to use a binocular compound microscope to visualise a range of samples, as well as a practical-based seminar that is dedicated to microscope alignment and focussing. Students' opportunities to practice with the compound microscope were limited to one session (instead of three) due to the availability of lab time during the COVID-19 pandemic. The light microscopes used in the teaching laboratories can reach a magnification of 1000× (using the 100× oil immersion lens), which is suitable for observing bacteria. However, they are large, heavy and expensive, making them unsuitable for a home lab experience.

Knowing that it would not be possible to replicate the microscope experience that students would have had in the laboratory, the aim of including this type of experiment in the Bioskills at home pack was to focus on the other ways in which this type of skill could be used. An important consideration for selecting an appropriate microscope for the students to use was its ability to capture images and short videos. The intention for the first term microscopy activities was to stimulate the students' scientific interest in the world around them, as well as to build engagement with the university's VLE (and each other) through a series of microscopy competitions. This was a valuable opportunity for community building, which was otherwise likely to be limited because of restrictions in place due to the pandemic. Competitions were run for the first two months of the first term with students asked to submit their best images and videos to the following competition categories:


Students were given the opportunity to vote for competition prize winners (awarded a certificate and prize) during an online celebration event at the end of the first term. This included the mystery images competition where students made their guesses of what the magnified entries were during the session. Each student who contributed a mystery image received a certificate and a prize, as did the person who correctly guessed what the image was.

For students to be able to participate, the microscope that they were provided with needed to be capable of sufficient magnification to be able to view cells, protists and other biologically interesting specimens (e.g., tardigrades). They also needed to be affordable, compact, lightweight and compatible with a range of devices, including smartphones and laptops so that all students would have an opportunity to participate, irrespective of whether they had regular access to a computer. Academic staff tested the small Rotek-EU digital USB/WIFI microscopes (Shenzhen, China; as seen in Figure 1) at home during the first pandemic lockdown in spring/summer 2020 and were able to view a range of different specimens of appropriate size as can be seen in Figure 3.

Students were provided with a written guide on the VLE to ensure that they were supported in their use of the equipment, with additional opportunities for discussion of any issues encountered via the discussion forum. Protocols for microscopy were risk assessed for the home environment prior to assembling the student kits.

### *2.5. Delivering the Bioskills at Home Kit to Students*

Based on the experiments that the academic team had designed, a list of kit components was drawn up. These elements all needed to be costed, sourced and delivered within the timescale and within the financial constraints of the project. A bespoke bag was designed and sourced to provide a way for all these components to be safely packaged and easily transported home by students. The estimated cost of the kit only was GBP 100 per student.

As can be seen in Figure 4, the components of the kit included: a range of plasticware (including multiple sizes of pipette tips, tubes of diverse sizes and weighing boats) that was available within the university laboratories but needed to be individually packaged, reagents which needed to be aliquoted and labelled, individually printed guidance on safe use of the kit and a printed scale for the microbiology experiment and health and safety equipment. Overall, 1000 Petri dishes, 2000 microscope slides, 2000 coverslips, 4000 50 mL tubes, 5000 plastic Pasteur pipettes, 6000 gloves, 11,000 microcentrifuge tubes and 40,000 pipette tips had to be counted and bagged.

**Figure 3.** Sample of microscopy images produced by academic staff using the Rotek digital microscope. Images show onion skin stained with ink with visible cell walls and dark blue staining of nuclei (**A**), nematode worm and Closterium (desmid, freshwater algae) (**B**), chironomid (non-biting midge larva (**C**), a human hair (**D**) and a C-chip haemocytometer (**E**).

**Figure 4.** Example of a deconstructed term 1 Bioskills at Home kit including kit bag, protocol, consumables, pipettes and microscope (**left**) and the completed collection of bags prior to distribution to students (**right**). [Images have been obscured with white boxes for anonymization purposes].

There were several significant challenges to overcome in the successful delivery of the Bioskills at home kits. First, the number of students in the 2020–2021 intake was significantly larger than had been predicted (an increase of 20% on the previous year), meaning that the required number of pieces of equipment rose significantly and that the time taken to put the kits together also increased substantially. It took approximately 200 h (spread across the technical team) to put the kits together, time which needed to be balanced against the existing commitments of preparing for a new academic year and the increased time-pressure resulting from adjustments necessary to ensure that teaching labs were COVID-secure. The impact of the pandemic itself in complicating this process cannot be underestimated, as much of the kit preparation was conducted by team members working from home, with the requisite requirement for transporting kit contents between campus and home. This presented significant logistical challenges and required a high degree of co-ordination and planning between team members to effectively deliver the project.

### *2.6. Logistics of Distributing Bioskills Packs to Students*

In the first week of term, students had the opportunity to meet with their personal tutors and fellow students in their tutor groups. Being one of only a few sessions that students would have face-to-face in the first term, this was an ideal opportunity to distribute the Bioskills bags. A room was secured to act as the base for storing the completed Bioskills bags and tutors were asked to collect bags for their groups from this base (as shown in Figure 4). This was thought to be the most efficient and COVID-secure way to deliver the bags as tutorial sessions took place at different times across campus. However, it was soon recognised that many students were not able to collect their bags as they were either ill with COVID-19, self-isolating or unable to come to campus due to either personal reasons or travel restrictions.

Students who were later able to attend campus for lab classes were able to collect the kits at the end of their lab sessions or from campus-based drop-in sessions run by the technical team. Unfortunately, not all students were able to collect kits in person, so further support from the technical team and the school's administrative team was necessary to ensure delivery of the Bioskills kits to these students. The first step was to ensure that all kit components could be safely delivered through the postal system, including whether they could be sent internationally. This required talking to multiple couriers as destinations ranged across several continents, including UK and mainland Europe, South America, Africa and Asia. Although students received their kits at different times, the timing of the activities was flexible with the only deadline being for the microscopy competitions, which closed at the end of November.

Given the logistical challenge of decontaminating equipment and the benefit of easy access to equipment, such as micropipettes, throughout their degree, Bioskills bags were not returned at the end of the year.

### **3. Results**

The Bioskills at home initiative packs were successfully delivered to all first-year students in term 1 of the 2020–2021 academic year either during tutorial sessions, oncampus drop-in sessions or via delivery to their home address (where necessary). Of the prospective intake of 507 students, a total of 450 students were enrolled into the term 1 modules. Engagement with individual activities is described below but student reflection on the experience of using the kit is highlighted in the quote below:

*"I personally found the home kits to be a challenging and engaging way to improve my practical skills in a way that positively impacted my performance in lab assessments".*

### *3.1. Pipetting Skills*

An example of the expected outcome for the 4th task set for the students can be seen in Figure 5. Whilst discussion about the pipetting activities was very limited on the discussion boards, the proportion of students who accessed resources for the various activities on the VLE was much higher.

Based on the VLE analytics, 43.8% of students engaged with the introduction to the learning pack and 22.9% of students engaged with the pipette home exercise guidance online. The videos that were created to support the pipetting activities were watched by 12.7–19.6% of students with the video for the first exercise (how to operate the pipettes to dispense liquid) the most watched. Learning science resources were engaged with by 7.1–10.0% of students. With the exception of the introductory material, these analytics showed a lower proportion of student engagement than other formative activities such as pre-recorded lectures, seminars and laboratory protocols (31.4–90.3%) and quizzes (26.8–37.4%).

**Figure 5.** Example of the type of results expected in a filter paper experiment designed to enable students to test the reproducibility of their pipetting skills by measuring the size of spots produced from pipetting multiple replicates of specific volumes.

Of the students who were able to attend the practical skills assessment session at the end of the first term, 98.2% successfully completed the pipetting learning outcome at the first opportunity. This was similar to the percentage of students who had attended the assessment in previous years: 95.4% of students successfully passed the pipetting part of the assessment in 2018–2019 and the pass rate in 2019–2020 was 99.4%.

A reflection on undertaking the pipetting tasks in the Bioskills at home kit provided by one of the first-year students is shown in the quote below. In this, the student refers to the practical skills assessment that they undertake in one of their term one modules (the practical techniques module). In this context, the student described that undertaking the exercise prior to assessment helped them to improve the consistency of their pipetting, which aided them in completing their assessment.

*"I decided to do this activity right before my Practical Techniques assessment. In a previous lab, I made some errors in dilutions which affected my results. I was not as used to making dilutions in small volumes, so this activity gave me a lot of practice. Particularly, in the 20 μL to 200 μL range. The activity also helped my pipetting become more consistent which was important to my assessment as I had to aspirate and dispense the same volume many times. As a result, I was able to complete my assessment with ease."*

### *3.2. Microbial Growth*

Staff prepared a growth curve for the yeast experiment as described in the student instructions to test the outcomes that students would expect to achieve (see Figure 6): presentation of experimental data in graphical form clearly enabled students to observe the effect of adding sugar to the growth media.

**Figure 6.** (**A**) Example of the microbial growth experiment (23 h after inoculation) conducted by staff as per the student protocol and (**B**) data generated from this experiment comparing growth of yeast with or without sugar added to the media over a 12-h period with appropriate experimental controls. The lines for the Marmite and sugar control are superimposed over the Marmite only control as they had the same value at each time point.

Based on comments on the discussion board, only three students confirmed that they had completed the microbial growth experiment or shared images and/or experimental data. However, the analytics from the university VLE showed that 52.4% of students engaged with the resources online.

Two students who had undertaken the microbial growth experiment reflected on the experience. For the first student, the value they ascribed to the activity was in building their confidence in pipetting, which they linked, as can be seen in the quote below, to one of the two face-to-face laboratory sessions that the students undertook. The phosphate assay laboratory class was retained as it gave students both a valuable opportunity to practice and assess the reproducibility of their pipetting, as well as being the first time they worked with small volumes in a microtiter plate (which is an assay format that many students will utilise repeatedly during their degree).

*"I found doing the microbial growth curve experiment before the phosphate assay lab really useful as I felt confident with pipetting."*

The quotation from the second student shows how this aspect of the Bioskills kit helped them to think more deeply about how their experiment linked to the underlying theory and also about experimental design.

*"I did this activity after I had a lecture on bacterial growth requirements. It made me think a lot about what conditions I needed to control whilst doing this experiment. For example, I tried to control temperature fluctuations by placing my tubes in a water bath near a radiator. This was to allow any heat to be distributed evenly. I also ensured that all tubes had the same amount of nutrients. As I was recording my results, it made me think about what processes were occurring inside the tube and how this affected the growth rate. I was able to use the results from this for my coursework as well. Overall, it was a very fun and useful activity to do."*

### *3.3. Microscopy*

The microscopy competitions ran from the start of term 1 (early October 2020) until near the end of November 2020. Within that time, students were able to submit entries to the four competitions:


**Figure 7.** Image of a spider's web on the underside of a leaf taken using the Rotek digital microscope and entered into the best biological image competition.

Overall, a total of 12 competition entries were made. Whilst the number of entries was low, at least 20% of students engaged with the online resources to support setting up the microscope (26.0%), guidance on how to use a haemocytometer (20.4%) and information on the different competitions (25.5%).

A similar level of engagement was seen at the celebration event, where approximately 100 students (22.2% of the cohort) attended, with at least half of these actively voting on competition entries.

Reflecting back on the microscopy awards ceremony (which was held online due to pandemic restrictions), one of the students highlighted that, for them, this helped to bring students together. Furthermore, the competitions themselves had helped to motivate them to explore their environment, as well as positively impacting upon their subject interest (see quote below).

*"* ... *I really enjoyed it and I thought it was a fun conclusion to the competition which brought people in biosciences together for some fun in a difficult year. The competition was also a great addition as it allowed us to be creative and remember that you can learn a lot from your own immediate environment. It also encouraged me to use any equipment I have to experiment with nature around me and allow this to further my scientific understanding as well as passion."*

### **4. Discussion**

### *4.1. Evaluation of Bioskills at Home Initiative*

It is clear that those students who engaged with the various activities in the Bioskills at home experiments derived benefit from doing so. The quotes from students highlighted that the tasks they were set improved their confidence (particularly in terms of pipetting) and maintained or enhanced their motivation and interest in the subject. Each of these are valuable aspects of the affective domain that can have a positive impact on meaningful learning, as described by Novak [4], and which has been acknowledged by academics as an important reason for first year undergraduates to undertake practical work [16]. Our institution is not the only one to assess key practical competencies. Seery et al. [40] have used "digital badging" as a micro-accreditation to show when students had reached a certain level of skill or demonstrated competency in a particular technique. Using a digital format allows students to use these via professional networking sites which has the potential to enhance employability and may inherently motivate students to want to "collect" more badges in a similar way to computer game achievements/badges.

### 4.1.1. Pipetting Task

Building student confidence in pipetting was a significant goal, as it is a skill that most students will use throughout their course and beyond, which is why it forms part of the skills competencies that are tested in the laboratory assessment at the end of the first term. Despite reduced opportunities to practice lab skills on campus (which could be further reduced if students were self-isolating or subject to travel restrictions from their home country), the fact that the proportion of students that successfully completed the pipetting assessment is similar to that seen in the previous two years suggests that the initiative was successful in ensuring students' pipetting competencies. Pipetting was a skill that formed part of both retained laboratory classes; therefore, it is possible that students who were able to attend at least one of the laboratory classes were able to pass this assessment without the use of the online pipetting resources. This may be an explanation why the guidance on "how to operate the pipettes to dispense liquid" was the most used resource (19.6%), as this would be the key resource for students unable to attend laboratory classes as well as students wanting to remind themselves of what they learnt in the class and practice independently. This could explain the apparent disparity between the proportion of students using the resources and the pipetting assessment success rate. For those students who made use of the pipetting experiments (up to one fifth of the student cohort), performing the pipetting experiments would have provided potential benefits in being able to improve their pipetting reproducibility and troubleshoot issues in pipetting in addition to being able to accurately use a micropipette.

### 4.1.2. Bacterial Growth Curve

In the student quote from the microbial growth experiment, there is reference to how they had considered a range of factors that were involved in the experiment and how to ensure that they controlled the conditions that they were using. This description indicates that the student was using higher order skills than would normally be expected at this level, although this had formed part of the overall aim of the task and is in keeping with the observed benefits of enquiry-based learning [9,10]. In the laboratory framework proposed by Seery et al. [41] in chemistry, the first year of laboratory classes have been described as important in forming a foundation in the key procedures and skill competencies that students would need to be familiar with in their discipline; a point supported by bioscience academics [16]. According to Seery et al. [41], knowledge of these procedures would be built on in subsequent years to enable students to work toward designing their own experiments, initially based on the key procedure they have learnt about but expanding on this to be able to perform open-ended experiments. Seery's proposed framework situates these higher-level skills as appropriate for the third year of Scottish undergraduate degree programmes (equivalent to a second-year undergraduate programme in England). Since the Bioskills activities described were undertaken by students in the first term of their first year at university, this suggests that engaging with the activities starts the process of developing higher order skills as an earlier stage than otherwise might be expected.

One area that none of the quoted students mentioned but which had been factored into the design of activities were the analytical aspects of the tasks. It is difficult to assess whether this is because the students did not perform this part of the task or whether they considered other aspects more valuable and therefore noteworthy in their reflections. This aspect was included in the study design as it is well established that undergraduate students suffer with maths anxiety and so building up students' confidence with the types of analyses that they would use on their course was considered a valuable opportunity by the academic team. An example of the extent to which science undergraduates experience maths anxiety can be seen in a study of 1153 university students in the UK. Using the maths anxiety scale (survey), it was found that maths anxiety in science disciplines (excluding computing and engineering) was the second highest amongst undergraduates with only arts, media and design students scoring more highly [42]. In particular, bioscience undergraduate students have been observed to have difficulty working with contextualised questions [43]. This is problematic as these types of questions have relevance in the context of their degree programme.

Of all the experiments designed for the Bioskills at home kit, this experiment received the most attention from students in terms of accessing the associated resources: twice as many students accessed the microbial growth curve resources (52.4%) compared to either of the other experiments (maximum of 26.0% for microscopy resources, 22.9% for pipetting resources). Whilst it is not possible to comment on what proportion of students who accessed the material actually performed the experiment, completion of the experiment would have given experience of data analysis, graph plotting that would have complimented lecture content as well as valuable practice at experimental design (including the use of relevant controls).

### 4.1.3. Microscopy

The most significant issue that was noted in the Bioskills at home initiative was a lack of engagement with some of the activities. As can be seen in the findings, the discussion boards had relatively few contributions from students and even the most well supported areas of the boards, such as the microscopy competition, showed a relatively low number of entries (a total of 12 entries across the 4 competitions) considering that the cohort was approximately 450 students.

The original aim of the microscopy experiments in the Bioskills kit was to engage student's scientific interest and build a sense of community through the discussion boards. As described with both the pipetting and microbial growth curve, students appeared reluctant to use the boards despite the possibility of posting anonymously. However, the celebration event where students voted for winners in the microscopy competition was well supported and students attending this (which accounted for approximately a quarter of the cohort) engaged well with the session. This is supported by the student quote which highlighted that it had been a good way to bring people together.

### *4.2. Technical Challenges*

The enormity of the task presented by sourcing, packing and distributing the Bioskills at home packs to the anticipated cohort of 500 first year students cannot be underestimated, particularly when set against the challenges presented by delivering on-campus practical classes alongside supporting lab classes that had moved online (e.g., by recording experiments and photographing experimental results for students to use in their assessments). Furthermore, working practice for technical and administrative staff had been significantly impacted by the pandemic with social distancing in place in the labs as well as staff working in "bubbles" to mitigate against infection risk and to ensure that technical support could be sustained even if staff were required to self-isolate. The preparation of the Bioskills bags, particularly considering the increase in anticipated intake (a 20% increase compared to expected numbers

of students), put increasing pressure on staff to meet short deadlines. However, a combination of meticulous planning, interpersonal support and teamwork enabled staff to meet these challenges and successfully deliver the target resources to students.

### *4.3. Future Development of the Bioskills at Home Initiative*

With the arrangements for the forthcoming academic year being put into place, there is an opportunity to reflect on the successes and limitations of this initiative and how this can be developed further to meet its original objectives. Even if students are largely able to return to campus teaching and lab provision is increased, there is still value to be gained from retaining and improving the Bioskills at home project. Apparent lack of engagement with some activities was the most significant factor which limited the success of this initiative. Whilst we are in the process of collecting data to better understand the students' perspective, experience within the university overall suggests that navigating a year at university during a pandemic has proven to be a significant challenge to students and that they can easily become overwhelmed. The flexibility of the Bioskills activities was designed to allow students to work at a pace that suited them; however, it seems likely that the unstructured nature of the initiative contributed to the lack of engagement as students lacked defined deadlines for the most part. Further, it seems likely that the fact that these activities were not mandatory or assessed meant that students were more likely to engage with the activities if they already felt some degree of engagement with their course, whereas those who struggled to engage with the course may not have been inspired to participate in the initiatives without a clear incentive. With this in mind, going forward, the activities retained will be actively embedded into face-to-face tutor group sessions (typically a group of 10–12 students who are enrolled in the same course) with additional contact points with tutors and student mentors [44] used to guide the activities. It is hoped that by embedding activities into the first tutorial session that students have with their personal tutors, it will help to break the ice and start to build community between students and foster engagement with the initiative that can be sustained and developed throughout their first year.

In addition to embedding the activities in a more structured way, the activities that students are asked to perform will be streamlined to retain those considered to have the highest value and therefore being most cost effective. There was certainly value in the pipetting activities both in terms of student confidence and successful completion of the practical skills assessment, so these will be retained. However, whilst the competition celebration was valuable in terms of generating interest and community building, the microscopes were an expensive component given the level of engagement with the competitions themselves. As such, the microscopy activity will not be included in the next iteration of the Bioskills kits and instead the competition and celebration will be refocussed on other activities. In addition, as students return to campus, they will have increased opportunity to use microscopes in the laboratory so there is less incentive for students to perform this with alternative equipment. The microbial growth experiment will also be retained alongside the pipetting experiment as it is both cost effective and the resources on the VLE showed the highest level of engagement of any of the activities (52.4%) even though this has not translated to experimental data being shared to the discussion boards. Additionally, as this experiment links not only to term 1 modules but has links to other modules in first year and beyond, this was seen as a high value activity. Raising student awareness of how these activities link to other modules, including the final year project module, through discussion with staff and student mentors could raise engagement especially as some mentors will have used this year's kit and so would be able to advocate for its benefits.

### **5. Conclusions**

Whilst logistically challenging, providing first year undergraduate students with a range of practical activities to complete at home can provide benefits to their learning in a pandemic and beyond by enhancing confidence in using these techniques in lab or assessment settings, as well as potentially enhancing higher order skills such as experimental

design. To increase engagement and community building with home lab kits requires both flexibility but also scaffolding to achieve maximum impact.

**Supplementary Materials:** The following supporting information can be downloaded at: https: //www.mdpi.com/article/10.3390/educsci12020106/s1, Table S1: Skills tracker. Excerpt from the skills tracker which is being used to map the skills gained from different modules for each course for each year of study. The table below shows the modules taken by first year undergraduate students studying on the biochemistry and microbiology course. The term 1 modules (Practical Techniques in Biology and Living systems) were those used to assess what skills would be lost due to social distancing and hence one of the important considerations in deciding on the activities in the home lab kit. Skills attained from these modules are listed under different category headings: Personal, professional and reflective; generic scientific skills; subject specific skills and graduate attributes. The source of these skills (T for taught as provided by lectures, seminars and lab classes; A where these are part of an assessment), which modules have these skills and the number of opportunities that there are across the different modules are shown in the skills tracker. The excerpt shown below contains examples (but not all skills) for each of the categories. The generic skills text highlighted in bold text directly link to aspects of the home labs kit.

**Author Contributions:** Conceptualization, J.W. and K.G.; methodology, K.G., S.R., I.K., G.M, E.S., C.R., C.N, J.T. and D.N.; validation, J.W., G.M. and S.R.; formal analysis, S.R.; investigation, J.W., K.G., S.R., I.K., L.A.D.G., E.S., C.R., C.N., M.L., J.D., J.T. and D.N.; resources, E.S.; data curation, J.D. and S.R.; writing—original draft preparation, S.R., K.G. and E.S.; writing—review and editing, J.W., K.G., S.R., M.L., C.N. and G.M.; visualization, S.R., J.W., G.M., K.G., I.K., L.A.D.G. and E.S.; project administration, J.W., K.G. and E.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Student microscopy images and reflections reproduced with permission.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The authors would like to thank the following members of the Nottingham Trent University School of Science and Technology technical and administration teams for their hard work and dedication in ensuring that the bioskills at home initiative was successfully de-livered: Ray Workman, Emma Rubbi, Laura King, Rebecca Coxhill, Ava McMullin, Shaff Hussain, James Kelly, David Green, David Wilkinson, Jayne Spence, Georgina Bradshaw, Emma Storey, Tom Brookes, Sam Poole, Macsen Fryer, Helen Wallis, Gareth Williams, Jessica Fountain, Jill McElvaney, James Kelly, Caroline Simiyu, Sowmya Suresh Menon, Luke Russell, Abby Whiley, Sam Coggin, Lisa Appleby, Tom Haffenden and Iain Walker.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**


### *Article* **Higher Education Students' Perception of the E-Portfolio as a Tool for Improving Their Employability: Weaknesses and Strengths**

**Elba Gutiérrez-Santiuste 1, Sonia García-Segura 2,\*, María Ángeles Olivares-García <sup>2</sup> and Elena González-Alfaya <sup>2</sup>**


**Abstract:** This study analyzes the strengths and weaknesses of the professional e-portfolio as a tool for preparing students in higher education to enter the labor market. It also examines students' level of planning to enter professional employment, and the help that they receive with this task from the university. The research is quantitative and observes the students' opinions before and after they create their own professional e-portfolio, as a case study. We used the analysis of means to determine the trend in the aspects analyzed over time, and the Student's *t*-test and Cohen's d to determine the effect size. We also performed correlation analysis between the different categories and subcategories proposed. The results show that the e-portfolio is a tool with strengths for labor market entry, while also revealing the weaknesses that students find in it. At the very least, the e-portfolio was useful to the students in planning their entry into the workforce. The correlations show high levels among the strengths but not among the weaknesses analyzed.

**Keywords:** career planning; electronic learning; employability; higher education

### **1. Introduction**

Various studies analyze the relationship between the technological tool of the eportfolio (EP) and the entry into workforce, providing diverse perspectives and relevant information on the topic. EPs have also been implemented in various areas for professional improvement—for example, in healthcare, engineering, and education. At its most basic, the EP in the professional life seems to be, as Chang (2006) [1] notes, a personal homepage or electronic curriculum vitae (CV). At its most complex, it can become a person's digital identity.

In general, studies follow lines of research based on developing professional frameworks [2–7] and on the professional development of the employed [8,9]. In both cases, most studies focus on developing a reflection on practices before and during employment. The EP is also used, however, on a continuum from education to employment in the workforce—that is, from anticipating professional practice while still a student to improving professionalization among the employed [10,11]. Whichever the case, reflecting on various aspects of professional tasks becomes the way to achieve the results desired—the anticipation of professional practice or improvement of professionalization. As Alam et al. (2015) [12] indicate, however, most EPs currently used in education are a hybrid spanning the functions of (a) development: to show advances in skills over a period of time; (b) assessment: to evaluate students' performance; and (c) exhibition: to demonstrate the student's skills and samples of their work available to potential employers.

Furthermore, various research studies also point to another function of the EP which is more related to professional development and employability [13–16]. Through the EP, not only do future workers get in contact with their potential employers, but they also

**Citation:** Gutiérrez-Santiuste, E.; García-Segura, S.; Olivares-García, M.Á.; González-Alfaya, E. Higher Education Students' Perception of the E-Portfolio as a Tool for Improving Their Employability: Weaknesses and Strengths. *Educ. Sci.* **2022**, *12*, 321. https://doi.org/ 10.3390/educsci12050321

Academic Editor: Maria Limniou

Received: 18 March 2022 Accepted: 30 April 2022 Published: 3 May 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

develop and demonstrate different skills and competencies related to specific jobs, taking responsibility for their employability. However, Bennett (2018) [17] (p. 39) indicates that 'employability development is not limited to discipline skills, knowledge and practices. Rather, it develops students' abilities to conceptualize their future lives and work by learning the practice of the discipline and developing their metacognition'. Therefore, the development of employability is also conditioned by those practices and experiences related to the students' interests.

### *1.1. The Role of EP in Preparing for Professional Practice*

The first use of the EP, in preparing for professional practice, is discussed in the study by Carl and Strydom (2017) [3], whose goal is to determine whether the theoretical foundations and expectations of the EP align with the current practices and attributes of students' training during their practice. For these authors, EPs are increasingly considered in preservice teacher education programs to enable Education students to reflect during and on their practice in a structured way. Such reflection enables students to demonstrate growth and development as professionals. This study also analyzes the daily reflections and regular online interactions online with classmates and members of the project. Institutions play an important role in implementing theoretical foundations and teacher training, as well as in the reconceptualization and understanding of what is really valued in professional practice. Whether the EP is considered as a professionalization tool or as a tool for representation of effective practice, creating the EP involves a series of phases: orientation, novice, advanced beginner, approaching competence, and graduation/entry into the profession (Clarke and Boud, 2016) [18]. This study concludes that feedback is important to achieving the learning outcomes. Similarly, a study by Jorre and Oliver (2018) [19] based on students' perceptions indicates that increasing their capability for employability requires receiving advice from employers, professionals, and recent graduates.

Along similar lines, the study by Faulkner et al. (2013) [4] on the fields of Engineering and Law analyzes whether the EP would train students to articulate their achievements and understand professional frameworks. The low level of reflection and training for personal development in the study's results is not surprising, since individuals automatically orient themselves to their comfort zones—that is, students prefer to maintain their current perspective instead of looking for ways to develop. While EPs help to capture evidence of development, their value is limited to students who appreciate personal and professional development, as in the study by Beckers et al. (2016) [20]. On the other hand, including formative comments by mentors, classmates, and others encourages and supports the process of transforming students into professionals [4,13]. The study by Pool et al. (2018) [21] illuminates how advisors interpret the student's professional training in their portfolio (traditional, not electronic) and concludes that different mental models for evaluating performance influence judgments by evaluators, potentially affecting the feedback and thus credibility of their decisions.

Hallam and Creagh (2010) [22] report that anticipating professional practice increases understanding of the need for interoperability among the different areas of education and employment significantly. The report argues that the EP should not be ignored if higher education wishes to fulfill its function of producing qualified professionals who will play an important role in the success of the community and economy. Higher education should thus transform its policies so that future workers can develop the skills, competencies, values, and behaviors that improve their employability, as indicated by the research by Okolie et al. (2020) [23], Olivares-García et al (2020) [24], and Reddy (2019) [25]. In this sense, several research studies suggest that educational programs must be evaluated and adapted to incorporate strategies for the development of self-managed employability by university students so that they are able to respond to the demands of employers [26,27]. Not only can the EP become a tool to improve learning skills and competencies, but it can also contribute to improving employability and business skills by bringing them closer to the current business landscape [28]. This process is what Yan et al. (2016) [29] call

productive learning, which is characterized by a motivation for learning conditioned by intrinsic interests, self-management, and self-reflection skills, as well as collaborative work.

Using the decomposed theory of planned behavior (DTPB) model, Ahmed and Ward (2016) [2] found statistical support for three factors that influence the acceptance of the EP for personal, academic, and professional development. These factors are Attitude toward the Behavior (e.g., ease of use), Subjective Norm (e.g., peer influence), and Perceived Behavioral Control (e.g., facilitating conditions and self-efficacy).

A series of studies analyze the professional EP from the perspective of groups of students, graduates, professors, institutions, and employers. A literature review by Kinash et al. (2016) [6] identifies 12 strategies related empirically to improvement in employability among graduates by asking the group different questions. The key findings indicate discrepancies between the strategies indicated in the literature and those indicated in the surveys, as well as discrepancies among the groups relative to the strategies. The most important job search strategies for all groups analyzed were:


Along similar lines, Ritzhaupt et al. (2008) [30] focused on understanding the student's perspective of EPs and their use. Their research incorporates four domains, including employment, and connects these domains to the four groups involved: students, administrators, professors, and employers. The results indicate that students' perspectives on the EP are multidimensional, with three different and internally consistent underlying constructs: learning, evaluation, and visibility. In this study, only 19% of those surveyed believed that their EPs were beneficial for securing employment.

Haffling et al. (2010) [31], in contrast, analyzed how EPs can be used as tools for evaluating professional competence in a clinical setting. They examined specifically whether the students' reflections include categories of professional competence and satisfaction with use. The findings put emphasis on affective questions, particularly self-awareness of feelings, attitudes, and concerns, as well as ethical problems, clinical reasoning strategies, and future training in communication abilities. The students were satisfied with the EP, as it gave them opportunities to reflect on professional questions, but they needed better instructions. Another study of a clinical environment, by Schneider et al. (2016) [32], focused on continuous professional development to encourage the individual to pursue lifelong learning as a way of maintaining professional competence. The students found that preparing the EP was challenging (40%) but also reported that the EP was effective for autonomous learning (54%). Similarly, Beckers et al. (2016) [20] conclude that the EP facilitates the development of self-directed learning skills.

Kabilan (2016) [5] concluded that Facebook—used as an EP—contributes significantly to the professional development of future teachers in five ways: community of practice, professional learning and identity, relevant skills, resources, and trust. Among other aspects, this includes collaboration, sharing of experience, building the EP, and creating networks. The students' experiences led to richer ideas that facilitated the reconstruction and reconfiguration of significant personal knowledge and increased their learning and professional development. According to the author, although the participants were motivated by grades at the start of the project, they gradually began to identify socialization processes as modes of autonomous learning and self-knowledge. In the same line, Machado and Urbanetz (2019) [33] indicate that the EP may also become an opportunity to foster creativity and self-sufficiency, as well as reflection and self-awareness.

As for prior research, we can identify a series of strengths of the EP viewed as an educational tool for entering the workforce. These include the fact that the EP helps students to become reflective and conscious of their personal and professional strengths and weaknesses, while also making their existing developing abilities more explicit [3,22]. It also helps them to demonstrate professional development [4] and self-consciousness of emotions, attitudes, and concerns. The EP was further effective in autonomous learning and reflection on ethical problems, clinical reasoning strategies, and future training in communication skills [31]. Slepcevic–Zach and Stock (2018) [7] indicate that students in the final stretch of their studies are interested in job search and more generally in the professional direction they wish to pursue. In this study, 64.5% of students claimed that the EP supports orientation to their professional careers.

On the other hand, studies indicate a series of weaknesses, among them, as indicated by Ross et al. (2009) [34], the need for clear information, the need for support in presenting the EP, and anxiety, perhaps due to the challenging character of building the EP [13,32]. Carl and Strydom (2017) [3] also indicate the need to provide students with sufficient training, continuous technical support, and the design of innovative opportunities for sustainable student learning. Faulkner et al. (2013) [4] analyzed the challenge for professors of the relatively low level of reflection and planning of personal development and agree with Hallam and Creagh (2010) [22] regarding the need for interoperability among the different areas of education and employment.

### *1.2. The EP for Professional Improvement*

In the second case presented in the Introduction—professional improvement—prior research provides valuable information on the use of the EP by working professionals. The diverse results obtained revolve around the utility of active reflection and planning [8,35], interest in adopting the EP as a tool for permanent learning [36], creation of professional communities [9], and demonstration of achievements related to professional development [1,35,37].

Among the strengths found for professional improvement, Andre (2010) [35] indicated the capability to store and recover information, and the provision of tools to support the structuring and preparation of reports in order to develop and communicate professional achievements. Chang (2006) [1] described the EP as a transparent tool for diversity and equity, noting its advantage for first adopters in a competitive labor market. Among the tool's weaknesses, Andre (2010) [35] notes that compiling and managing an EP can be a slow and irrelevant process if not implemented correctly. Hampe and Lewis (2013) [8] also view as a challenge the engagement of the participation of personnel in making the practical reflections. Chang (2006) [1], in turn, indicates some potential barriers, such as cost, acceptance, privacy, propriety, the inertia of the process, and consistency.

We sought to answer the following questions in relation to labor market entry:


### **2. Materials and Methods**

### *2.1. Instructional Design*

The instructional design consisted of the delivery of the subject guidance, professional training, and socio-labor integration through theoretical and practical classes held in parallel during two semesters to achieve the learning outcomes. The theoretical sessions focused on becoming familiar with and critically analyzing the current state of the labor market, with in-depth knowledge of the status of the most vulnerable groups.

The practical classes consisted of building a professional EP, a task required to pass the subject. The first session, prior to the construction of the EP, included a self-knowledge workshop, in which students could anticipate and reflect on their own capabilities, interests, and professional goals. Two educational seminars were subsequently held at intervals of a month to analyze the content of the professional EP (What is an EP? how can we use an EP in the job search, how to build it, where to present it). In the other practical classes, each student had to complete their EP as an independent project, receiving advice from professors with expertise in finding jobs and EPs. Different active job search techniques were used, as well as an analysis of jobs that facilitate students' making well-grounded decisions and specifying professional goals for the short, medium, and long term. A forum was also planned, with three threads: technical problems, how to make a professional EP, and aspects of content about which the students could consult and ask questions.

The data collection was conducted in accordance with the ethical protocols of the Declaration of Helsinki and with the consent of the participants, guaranteeing their anonymity.

### *2.2. Sample*

The sample consisted of a group of 54 students in the third year of Social Education out of the total 65 enrolled students, as a case study [38,39]. Social Education is a degree program (260 enrolled students in total) oriented to providing scientific and experiential training in the fields of non-formal education: adult education, social integration of maladjusted people and those with disabilities, and socio-educational action (Faculty of Education, n.d.) [40] taught at a Spanish University. The sample was selected by convenience sampling.

The study was performed during two semesters of the academic year 2017–2018. Sample distribution by sex was 81.5% women and 18.5% men, a percentage that reflects the feminization of education in Spain. The participants' ages ranged from 19 to 27 (σ = 2.05; M = 21.57). The K-S normality test showed that the sample distribution was normal (K-S = 0.894, *p* > 0.05).

### *2.3. Research Methodology*

The quantitative analyses were performed using two questionnaires—pre-test and post-test—to determine the students' opinion of the strengths and weaknesses of the EP as a tool for the job search and of the various aspects of preparation for employment (planning and training). A pool of 87 items was constructed based on the prior literature and on the research team's knowledge. After analyzing redundancy, appropriateness, ambiguity, and length following the recommendations by DeVellis (2017) [41], the questionnaire resulted in 41 items (version 1). The questionnaires consisted of three categories: strengths, weaknesses, and preparation for the workforce.

The content was validated by five experts in labor market entry and educational technology, producing the final version of the questionnaire, composed of 31 items. The consensus technique was used since it provided the reviewers' view of each item qualitatively. The items were distributed as follows: three items on sociodemographic data; 30 on the research goals (with the exception of one item, which contained 9 Likert-type sub-responses and one open-response question). The structure of the questionnaire is detailed below:

Strengths contain the subcategories:



Preparation for the workforce contains the subcategories:


The scales included [41] one 5-point Likert scale (1 = completely disagree; 2 = partially disagree; 3 = partially agree; 4 = completely agree) and one semantic differential (1 = none; 2 = basic; 3 = user; 4 = Expert). Both scales also included the option 0 = don't know/no answer.

The Alpha (α = 0.97) did not vary significantly: if an element was eliminated all values were greater than 0.96. Nor did we find items that had to be eliminated because they reduced the internal consistency of the instrument.

The minimum and maximum ranged from 0 to 4 in all categories, indicating that all categories were used—the full response scale. Since kurtosis had two negative coefficients (strengths and weaknesses), a lower concentration of data around the average was considered in preparation for the workforce (with one positive coefficient). Asymmetry obtained two negative values, indicating asymmetrical distribution to the left and one positive value with asymmetrical distribution to the right.

### **3. Results**

For analysis, the data from the pre-test and post-test were paired for each participant. Participants were then eliminated from the study if they did not complete both questionnaires. The answers were analyzed using the statistical package SPSS, v.20, with a low confidence level of 95%. We highlight that 60.9% of students responded don't know/no answer on the questions related to the EP in the pre-test, as opposed to only 1.5% in the post-test.

### *3.1. Strengths of the EP*

This category included five subcategories that the prior literature considers as useful aspects of the EP as a tool for anticipating professional practice and for improvement in working people. In Table 1 we observe the results of the analysis.

**Table 1.** Initial and final questionnaires' means and differences.


The results obtained for the Student's *t*-test show that these changes were statistically significant in all subcategories on strength of the EP as a tool for labor market entry (*p* > 0.05). We also confirmed that the effect size was very large in the categories indicated (−0.79 to −1.81) [42].

This analysis supports the previously reported data on the progression of means.

The category Beliefs underwent a statistically significant change, with a higher *r* and *d* [42]—that is, in the compatibility of EP use with studies and a more positive attitude toward its use; and the perception of the EP as a creative and innovative tool increased greatly.

The students found fewer changes related to the strength of the EP for ease of use in finding a job, their motivation, and their learning, although the changes were statistically significant. The high values of d may stem from the high number of students in the pre-test who were initially unfamiliar with the EP as a tool for labor market entry.

All categories showed a change in the students' perception (Figure 1). The category in which the mean of the items increased most was Beliefs (composed of compatibility of making an EP with studies, positive attitude toward its use, fit of the EP to professional development needs, and especially the conception that the EP is a creative and innovative tool), with an increase of 1.94 points.

**Figure 1.** Strengths of the EP: trends over time.

The students' perception of Utility of the EP for the job search also increased considerably from the pre-test to the post-test—by 1.7 points (utility for the job search, broadening network of contacts, developing professional competencies, greater visibility, contact with employers, opening new professional horizons, and especially perception that the EP could be a tool for permanent education). Third, the category Self-knowledge increased by 1.49 points in all of its items (critical thinking, self-reflection, self-evaluation, self-direction of learning, self-efficacy, reflection, and especially fit of the EP with professional development needs). Finally, the category that changed the least (0.77 points) was Community. The item that increased the most was Making the EP visible to classmates, but no change was observed in the students' perceptions of enjoying sharing information and learning by working with classmates.

Correlations among the Elements in the Category Strengths

The relationship between the subcategories of strengths was investigated using the Pearson product–moment correlation coefficient. Preliminary analyses were performed to ensure no violation of the assumptions of normality, linearity, and homoscedasticity [43]. There was a strong, positive correlation between the subcategories (Self-knowledge/Utility, Beliefs/Utility, Beliefs/Self-knowledge, Beliefs/Community) and medium (Community/ Utility, Community/Self-knowledge). As Table 2 shows, the subcategory strengths presented the largest changes in correlation, and these were significant in all cases.


**Table 2.** Strengths of the EP.

\* Values calculated using the means and standard deviations of the two groups.

Given the results obtained [42], we can conclude Beliefs in the compatibility of the EP with studies, positive attitude, and fit with professional development needs. Belief that the EP is creative and innovative also correlates more strongly with perception of Utility, increase in Self-knowledge, and Collaboration with classmates.

### *3.2. Weaknesses of the EP*

This category includes four subcategories that, based on the prior literature, have been considered as weaknesses for the EP as a tool for labor market entry (ease of use, technical advising, time in preparing and updating is worthwhile relative to the benefits, technological knowledge). Table 3 shows the results of the analysis comparing pre-test and post-test data.


**Table 3.** Weaknesses of the EP.

\* Values calculated using the means and standard deviations of the two groups.

Knowledge of the digital tools for building the EP did not increase as much as did other subcategories in weaknesses, although the change was statistically significant. Initially, the students ranked themselves at the level of user, and this category increased by 0.21 points following the preparation of their EP. In this subcategory, it is noteworthy that the increase is generally very low (with the exception of storage of EP), in which the students did believe that they improved from having almost no knowledge (M = 1.31) to having above-basiclevel knowledge (M = 2.43).

As in the case of strengths, the category weaknesses shows that the differences between the entrance and exit questionnaires were statistically significant, with a very large effect size (*d* between 0.51 and 1.43). On the one hand, the subcategory related to the ease of building the professional EP increased considerably, as did the need for technical advising. The students believed that the time invested in preparing the EP was worthwhile relative to the benefits. The perception of benefits relative to the time invested, both in creating and in updating the e-portfolio, received a higher proportion of responses on the post-test than on the pre-test. In the opposite direction, the subcategory Technological Knowledge (which includes specific computer-based knowledge to build the EP) was not as extensive, although the change was still statistically significant. As in strengths, it is possible that the high results obtained in *r* and *d* are due to the initial lack of familiarity with the EP tool for labor market entry.

We can see that the means of the categories Ease of Use and Time increased greatly (1.52 and 1.27 points, respectively) (Figure 2). Aspects such as productivity of the time used in preparing and updating the EP relative to benefits increased statistically (Table 3), as did perception of the ease of building the professional EP. We see that the initial perception of time as a problem shifts to more moderate positions, with a greater balance between time invested and benefits obtained from making the EP. In the open-response question on Technological Knowledge, the students in the pre-test and post-test indicated other tools (5.7% and 11.3%, respectively).

Correlations among the Elements in the Category Weaknesses

The preliminary scatterplot analysis suggests a very low correlation and the distribution of scores on the scatterplot do not establish the direction of the relationship [43]. The relationship between the subcategories of weaknesses was investigated using Pearson product–moment correlation coefficient. According to Table 4, there are no correlations among the different subcategories of weaknesses, since in all cases *p* > 0.5.

**Table 4.** Pearson product–moment correlation measures of subcategories of strengths.


\* *p* < 0.02 (2-tailed).

Based on the results obtained from the correlation analysis, we can conclude that the different weaknesses analyzed did not increase or decrease together. Neither did the need for advising, time employed in building, time employed in updating the EP, nor Technological Knowledge have a linear relationship to the other or to the perception of ease of use in building the EP.

### *3.3. Preparation for the Workforce*

This category includes two subcategories that, based on the prior literature, have been considered useful and valid in making the EP a tool for preparing for the workforce (planning for labor market entry, university training).

Although the changes were significant in this category (Table 5), they were not as pronounced as in most of the subcategories mentioned above. The items Students' perception of the help that the university can give them and Belief that they have the education needed for employability increased the most (0.87 and 0.80, respectively), following their experience creating the professional EP. However, the students still did not plan their job searches (an increase of 0.35 points).

**Table 5.** Workforce preparation.


\* Values calculated using the means and standard deviations of the two groups.

The results for the Student's *t*-test followed the same direction as the analysis of means (Figure 3). The item Planning for Labor Market Entry obtained a small effect size—that is, after the experience of building their professional EPs, the students did not plan the job search further. The students' conviction that they need education in employability and the help that they could receive from the university did increase, however.

**Figure 3.** Preparation for the workforce: trend over time.

We found a more sustained trend in the category Preparation for the workforce, which included aspects of planning the job search and supported the student to obtain university services for labor market entry (Figure 3).

Correlations among Elements in the Category Preparation for the Workforce

The subcategories Training and Planning correlated significantly, positively, and moderately (*r* = 0.404, *p* < 0.01) [42]. We thus believe that the greater planning and reflection on the professional future, the higher the levels of conviction of the need for training from the university.

### *3.4. Intercategory Correlations*

The relationship between the categories strengths, weaknesses, and workforce preparation was investigated using the Pearson product–moment correlation coefficient. Preliminary analyses were performed to ensure no violation of the assumptions of normality, linearity, and homoscedasticity [43]. The scatterplot preliminary suggests differences in the correlations between categories.

Based on the results obtained (Table 6), we conclude that there is no linear relationship between weaknesses and the categories analyzed. The same does not hold for strengths or Preparation for the workforce, for which we found an average positive correlation.

**Table 6.** Pearson product–moment correlation measures of subcategories of weaknesses.


### **4. Discussion and Conclusions**

This study analyzes the view of students on the strengths and weaknesses of the EP as a tool for labor market entry in higher education, the level of planning to enter the labor market, and the training that the educational institution provides to students.

In this study, building a professional EP was shown to be useful in preparing students in higher education for the workplace. As in Kabilan (2016) [5] and Ciesielkiewicz (2019) [13], our students expressed skepticism and were not familiar with the EP's potential as a tool for labor market entry. After the experience, however, the students perceived that the possible benefits of this tool were great.

As in Chang (2006) [1], the results of this study show that students perceive the EP as being useful for entering the labor market, extending one's network of professional contacts, promoting greater visibility on the job market, and helping to contact employers. These aspects reinforce results obtained by Kinash et al. (2016) [6], who show that the best strategies are related to membership in professional associations and participation in professional networks.

Although our study showed that the students value the EP as a tool for self-knowledge (critical thinking, self-evaluation, self-directed learning, self-efficacy) (Machado and Urbanetz 2019) [33], they did not generally value it as a tool in planning their job search. These results differ from those of Carl and Strydom (2017) [3]. An increase in self-knowledge was also detected in Kabilan (2016), indicating that students' experiences using the EP led to richer ideas that facilitate reconstruction and reconfiguration of the personal and significant knowledge facilitated by autonomous learning and self-development. In Schneider et al. (2016) [32], 54% of the students reported that the EP was effective for autonomous learning. Similarly, Ahmed and Ward (2016) [2] argued self-efficacy as one of the factors in acceptance of the professional EP, an aspect that the students in our study evaluated positively.

The study by Carl and Strydom (2017) [3] agrees with ours in the social aspects of collaboration and sharing of information with classmates. In both studies, this subdimension obtained lower levels than other aspects of the EP as a tool for labor market entry. Although we obtained a statistically significant difference before and after the experience, our study shows that students do not welcome collaboration, perhaps due to the scarcity of jobs in the context in which the research was performed, which increases competition among classmates. These results contrast with the analysis of Kabilan (2016) [5], in which collaboration and sharing of experiences increased with the use of the EP.

On the other hand, the students' beliefs changed significantly from the pre-test to the post-test. After making their own EPs, the students believed that the EP was a tool compatible with their studies, a belief that generated a positive attitude toward its use, since they can continue to adjust the EP to their professional development needs. Further, the students considered the EP as a creative and innovative element that, as in Carl and Strydom (2017) [3], generates innovative learning opportunities while also increasing perception of utility and encouraging self-knowledge among classmates.

The category weaknesses in our study included aspects that the prior literature indicated as such. In the light of our results, however, we conclude that the perception of students on building the EP served to change the students' perception in some respects (ease of use, time/benefits ratio), while also increasing their perception of the need for advising but not changing the levels of technical knowledge mastered.

Chang (2006) [1] has already mentioned the difficulty involved in building an EP and noted this issue as one of the factors that affected the acceptance of preparing the professional EP. Since our study demonstrates that the students' improved their perception of the ease of preparing the EP after making one, we believe that the training received was a determining factor in this change of perception.

Our study agrees with the prior literature [3,5,8,31,32,34] in concluding that the development of the EP requires technical advising. This need arises from the challenging character of the creation of the EP, which requires continued, constant help to facilitate its planning and structuring. On the other hand, as found by Andre (2010) [35], time can be a determining factor leading to the perception of the EP as irrelevant if it is not well implemented.

As in Beckers et al. (2016) [20] and Carl and Strydom (2017) [3], the students in our study had low levels of technological skills for building the EP. During this experience, they did not advance substantially in this matter. Greater prior preparation of students in basic skills (word processor, editing of videos and images, presentations, webpage editing, etc.) would have made the development of other aspects analyzed in this study easier.

In our study, we did not observe that the experience served to increase the students' planning of their entry into the labor market. These results agree with those of previous studies indicating the difficulty of development in this area. Perhaps, as Faulkner et al. (2013) [4] indicate, this is because the students are automatically oriented to their comfort zone. Or perhaps, as Slepcevic–Zach and Stock (2018) [7] indicate, interest in this issue develops in the last stretch of their university studies, since we observed that 64.5% of the students claimed that the EP is useful for orienting their professional career.

In recent years, entry into the labor market has become a challenge for higher education students. As Jorre and Oliver (2018) [19] and Reddy (2019) [25] indicate, it is necessary to seek advice from employers, professionals, and recent graduates. However, as in Faulkner et al. (2013) [4] and Marinho, et al. (2021) [44], developing EPs trains students to articulate their achievements and understand the professional structures through which entering the professional world may become more accessible. In this sense, the research work by Ciesielkiewicz (2019) [13] also highlights that students recognize the EP as a valuable tool in job search and as an effective resource in their professional development, becoming aware of its strengths and weaknesses, as this study has revealed.

The expectations for training at the university expressed by the students in the study were also statistically significant in this study, reinforcing some aspects of the results obtained by Carl and Strydom (2017) [3]—the conviction that the university both played and plays an important role in their education in this area. Moreover, this study has highlighted the importance of higher education in the development of skills and competencies that prepare graduates not only to work, but also to learn throughout their professional careers, as Bennet (2018) [17] points out.

The correlations observed point to the conclusion that strengths of the EP and Preparation for the workforce co-varied, enabling us to establish a positive linear relationship between the two. We might thus conclude that both dimensions are necessary to make the EP a useful tool for labor market entry.

The development of this research in the field of social sciences constitutes a major drawback in terms of the sample size and the context in which it was carried out. Nevertheless, further research on improving the employability of university graduates and their

incorporation into the labor market is a serious concern for higher education and other public administrations.

**Author Contributions:** Conceptualization, E.G.-S., S.G.-S. and M.Á.O.-G.; methodology, E.G.-S. and E.G.-A.; validation, E.G.-S. and E.G.-A.; formal analysis, E.G.-S.; investigation, E.G.-S., S.G.-S. and M.Á.O.-G.; data curation, E.G.-S.; writing—original draft preparation, M.Á.O.-G. and E.G.-S.; writing—review and editing, S.G.-S.; supervision, E.G.-S. and S.G.-S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** This study followed the ethical principles of the Declaration of Helsinki in terms of confidentiality, anonymity, and use of information for research purposes only.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**


### *Article* **Examining Technology Acceptance in Learning and Teaching at a Historically Disadvantaged University in South Africa through the Technology Acceptance Model**

**Clever Ndebele 1,\* and Munienge Mbodila <sup>2</sup>**


**Abstract:** The exponential growth in the use of technology for learning and teaching in the higher education sector has imposed pressure on academics to embrace technology in their teaching. The present study sought to examine factors underlying technology acceptance in learning and teaching at a historically disadvantaged university in the Eastern Cape Province of South Africa. Premised on the mixed methods approach and undergirded by the Technology Acceptance Model (TAM), both a pre-coded and an open-ended questionnaire were used to collect data. Data from the precoded questionnaire were analysed through the descriptive statistical approach. The qualitative data from the open-ended questionnaire were analysed through content analysis. The study found that most academic staff believe and see the value that ICTs bring in their teaching and learning practices. In addition, they are aware that technology use in education improves learning and teaching, and they are willing to embrace the use of technology to improve their practices. Based on the findings, we recommend intensification of lecturer training in the use of technology for teaching and learning to enable them to embrace it in their teaching practice. Furthermore, the institution needs to put in place support systems for academic staff to empower them to have continuous access to devices and internet connection for technology integration in teaching and learning. We recommend establishment of e-learning communities of practise in the university that will allow lecturers to assist each other as well as share best practices in the use of technology for teaching and learning.

**Keywords:** e-learning; technology acceptance; learning management system; behavioral intention e-learning; technology acceptance; learning management system; behavioral intention

### **1. Introduction**

The exponential growth in the use of technology for learning and teaching in the higher education sector has imposed pressure on academics to embrace this technology in their teaching. In South Africa, in 2015, with the onset of the '#FeesMustFallMovement' in universities, even more pressure has mounted to embrace technology in learning and teaching during times of disruption. Across the system, as Czerniewicz, Trotter and Haupt [1] show, university leadership engaged to varying degrees with protestors' demands, while simultaneously considering and using measures that would allow teaching to continue, or at least for the curriculum to be completed, to circumvent the effects of the disruptions with blended learning emerging as one of these measures.

The outbreak of the COVID-19 pandemic at the end of 2019 led to the closure of the schooling and university education system worldwide in 2020 and again foregrounded the need for multi-modal teaching approaches that ensure that teaching and learning takes place virtually to mitigate any challenges related to face to face tuition. Nearly every university in South Africa was forced to re-evaluate its teaching and learning approaches

**Citation:** Ndebele, C.; Mbodila, M. Examining Technology Acceptance in Learning and Teaching at a Historically Disadvantaged University in South Africa through the Technology Acceptance Model. *Educ. Sci.* **2022**, *12*, 54. https:// doi.org/10.3390/educsci12010054

Academic Editor: Maria Limniou

Received: 16 October 2021 Accepted: 8 December 2021 Published: 14 January 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

with the Department of Higher Education, Science and Technology [2] calling on public universities to produce plans that show how the 2020 academic year would be saved. This was followed by the publication of the 'Quality Assurance Guidelines for Emergency Remote Teaching & Learning and Assessment During the COVID-19 Pandemic' by the Council on Higher Education/CHE [3]. The big question remains: Do South African Institutions of Higher Learning and academics believe or see the value that ICTs bring to education to improve learning and teaching, and are they willing to embrace the use of technology that will transform HE, or is recourse to the use of technology in teaching and learning reactive because of the unforeseen circumstances alluded to above?

The present study sought to examine factors underlying technology acceptance in learning and teaching at a historically disadvantaged university in the Eastern Cape Province of South Africa. The university is a result of a merger of two polytechnic colleges and a university, which operates under a divisional governance model and has four semiautonomous campuses. The university identifies itself as an impactful and technology infused African university, foregrounding technology as a critical tool for learning and teaching [4]. Although the university introduced blended learning in 2006 as the learning and teaching strategy in the Centre for Learning and Teaching Development Founding Document [5], a very low adoption rate has been witnessed over the years, from less than 20% in 2014 to 48% in 2019 [4]. The Centre for Learning and Teaching Development as the academic development support center in the university is responsible for capacity building of lecturers in integrating information communication technology in learning and teaching in the university. It seems several academics are still far more comfortable with the traditional face to face way of teaching. Evidence shows that more than 75% of the students admitted in any particular year have never had any exposure to learning using technology [6]. This has been exacerbated by the COVID-19 pandemic that struck the nation and the world during the 2020 academic year forcing the university to introduce emergency remote teaching and learning. Given the cultural and contextual challenges identified above, it is imperative that research be conducted to examine the factors underlying acceptance of technology for teaching and learning by university lecturers. This will assist the university to design interventions that will increase such acceptance.

### *1.1. Technology Acceptance in Learning and Teaching in Higher Education* 1.1.1. Benefits of ICT Use for Teaching and Learning

The urge to use technology has generally not been embraced with the ease that would have been expected despite the widely reported benefits of integrating information communication technologies in teaching in higher education. Blended learning reduces online transactional distance, increases the interaction between teachers and their students and offers flexibility [7]. This is corroborated by [8] who argue that under ideal conditions technology has promoted flexibility in the place and time to study, accessibility of different teaching and learning resources, personalised ways of teaching and learning and readiness for future digital demands. Similarly, in a study by [9] the teachers described positive experiences regarding independence of place, time and the possibility of individualising the learning environment when using e-learning. [10] argue that as e-learning is not timebound or static, it has helped the students to access the material from anywhere and at any time. Teachers may develop, improve, and check the learning contents anytime. In South Africa, where this particular study is located, a study by [11] concluded that e-learning provided students with opportunities to manage their own task in their own time which therefore took personal learning to a whole new level. Furthermore, they argue that time and location limit students considerably while [12] avers that the use of e-learning allows lecturers access to a wide range of students anytime and anywhere. The significance of e-learning in mitigating the constraints of time and space is also corroborated by [13,14].

### 1.1.2. Teacher Beliefs and Pedagogical Use of ICT

While the literature is abounded with several benefits for integrating information communication technologies in learning and teaching, acceptance of technology should not be taken as a given as teacher beliefs on the use of technology can have an impact of technology acceptance in the higher education sector. Hew and Brush [15] noted that the challenge associated with technology acceptance comprises not only specific technology usage knowledge but also lack of technology-based pedagogical information. Rasheed, et al. [7] indicated that skepticism about the effectiveness of online instruction in improving learning is one of the reported negative perceptions and beliefs from blended learning teachers regarding using technology for teaching in the literature. In the same vein, Pan [16] reported that previous studies have highlighted that students' beliefs on the utility of technology influenced attitude toward technology use implying that both teacher and student beliefs can affect technology acceptance. Sometimes beliefs may not necessarily only be about the technology but may also be because of a group's culture, norms, and direct influences with respect to use of an educational technology. Kemp, et al. [17] suggested that how one will be perceived by others as a result of using the technology and the degree to which use of the educational technology will augment the esteem or image of the user within a social group may influence technology acceptance.

Belief in the pedagogical value of using technology in enhancing learning may also have a bearing on whether lecturers adopt technology in their teaching. A study by [9] discovered some barriers amongst many teachers in the use of technology such as the lack of direct, personal interaction, which they found unsettling and frustrating in using technology in their teaching and learning. This is in line with the assertion by [18] that failure to examine teachers pedagogical beliefs would lead to limited understanding of the factors of militating against incorporating ICT in classroom teaching. A study by [19] confirms that teachers whose pedagogical approaches are aligned to constructivist beliefs and learner-centred strategies are likely to incorporate ICT in their classroom instruction easily. In the same vein, a study in [17] confirmed that some teachers' beliefs about their inability to use ICT for teaching and learning made them feel insecure resulting in feelings that that ICT was difficult to use for teaching.

Models that attempt to theorize technology acceptance which can apply to the higher education sector are abound in the literature, among them the Theory of Reasoned Action [20] Theory of Planned Behavior [21], Technology Acceptance Model [22] and the Unified Theory of Acceptance and Use of Technology [23]. The study is premised on the Technology Acceptance Model.

### *1.2. The Technology Acceptance Model*

This study is premised on Davis [22]'s Technology Acceptance Model (TAM), as an analytical framework for determining factors which influence acceptance of technology in teaching and learning environments. TAM adapts and makes use of the Theory of Reasoned Action [20,24].

The Theory of Reasoned Action (TRA), a model wisely used in social psychology studies [25,26] postulates that an individual's attitude toward behavior is influenced by his/her beliefs [27]. Building on TRA, TAM specifically focused on analyzing "users' willingness to accept and use new technology or media in the field of information system management [27]. "The two most important individual beliefs about using information technology according to TAM are Perceived Usefulness (PU) and Perceived Ease of Use (PEOU) that are able to explain individual's Intention to Use (IU) the technology [28]. Perceived Usefulness is defined as the potential user's subjective likelihood that the use of a certain system will improve his/her action and Perceived Ease of Use refers to the degree to which the potential user expects the target system to be effortless in [15,22,28,29].

"An individual's salient beliefs about a system (perceived usefulness and perceived ease of use) determine his/her attitude towards using the given system [29]". Therefore, as Taherdoost [30] shows, recognition and realization of the needs and factors that drive users' acceptance or rejection of technologies at the introduction stage would be helpful so that they are taken into account during the development phase. Figure 1 depicts the original TAM model.

**Figure 1.** Original Technology Acceptance Model.

Through further development, the TAM model was refined to TAM II through provision of more detailed explanations for the reasons users found a given system useful at three points in time: pre-implementation, one-month post-implementation and threemonth post implementation [31]. The four major variables of Perceived Usefulness (PU), Perceived Ease of Use (PEOU), Behavioral Intention (BI) and Actual System Usage remain. According to Lee, et al. [32] through synthesis of previous efforts, and reflection on the need for the model's elaboration, Venkatesh and Davis [31] defined the external variables of PU, such as social influence and cognitive instruments which include job relevance, quality, and result demonstrability while Venkatesh [31] provided the external variables of PEOU, such as, "anchor (computer self-efficacy, perceptions of external control, computer anxiety, and computer playfulness) and adjustments (perceived enjoyment and objective usability). Computer self- efficacy, referred by some authors as technological self-efficacy [16], technological pedagogical knowledge (TPK) self-efficacy [8] and digital literacy [33] refers to all those skills, attitudes and knowledge required by teachers in a digitalized world. It also refers to the belief in one's capability to organise and execute internet-related actions required to accomplish assigned tasks [34]. If university teachers have strong ICT-related knowledge, they will be able to overcome ICT related barriers and thus successfully incorporate technology into their teaching practice [33,35]. While computer efficacy significantly predicts continuance intention in e-learning among university lecturers [36,37] it also significantly affects students' behavioural preferences to use technological tools for learning [16,38,39]. This calls for adequate capacity building to build computer self-efficacy among lecturers and students alike in universities.

A study by Lee, et al. [32] that traced TAM's history, investigated its findings, to determine its future trajectory through a review of one hundred and one articles published by leading Information Systems journals and conferences over an eighteen year period concluded that it had been, "elaborated by researchers, resolving its limitations, incorporating other theoretical models or introducing new external variables, and being applied to different environments, systems, tasks, and subjects. It is against this background that the original TAM model is used together with its subsequent refinements in this study to examine technology acceptance among academic staff.

### **2. Materials and Methods**

An explanatory sequential mixed methods paradigm approach was selected for this research study to examine technology acceptance for e-learning among academics. The explanatory-sequential approach which is a chronological approach is used when the researcher is interested in following up the quantitative results with qualitative data [40]. The sequential mixed-method (incorporating collection of both quantitative and qualitative data with quantitative data collected first followed by collection of qualitative data) was used in

order to triangulate the data, as well as to solicit rich data from respondents [41]. While the mixed methods approach was adopted for purposes of triangulation of data, the predominant approach used was the qualitative approach through the open-ended questionnaire. Quantitative data was used mainly to help construct the qualitative questions.

### *2.1. Population and Sampling*

The Learning and Teaching with Technology (LTwT) Unit in the university periodically conducts e-learning workshops and related training on the integration of information communication technologies in learning and teaching. Purposive sampling was used to select participants for the study. Purposive sampling involves identifying and selecting individuals or groups of individuals that are especially knowledgeable about or experienced with a phenomenon of interest [42]. The phenomenon of interest in this case was lecturer integration of information communication technologies in learning and teaching. Records from LTwT indicated that one hundred and three lecturers had attended these workshops in the period under review. The 103 lecturers were contacted through emails with an explanatory note on the purpose of the study requesting them to indicate if they would be willing to participate in the study. A total of 50 lecturers expressed willingness to participate and constituted the sample for the study. Fifty lecturers were considered adequate as the primary purpose of the study was not generalisation but to identify the factors underlying acceptance of technology for teaching and learning at the chosen university to assist the university to design interventions that would increase such acceptance. Notwithstanding the sample size however, the findings which are corroborated in the literature appear generalisable.

### *2.2. Data Collection*

Two sets of online questionnaires, one structured with pre-coded questions and one with open ended questions were sent to lecturers through an online link using university emails and the WhatsApp platform for them to complete. Their contact details were readily available through the email addresses and WhatsApp numbers they had provided in attendance registers during the training. Validity and reliability were ensured through content validity and inter-rater reliability [43] where experts in the Learning and Teaching with Technology Unit (LTwT) were asked to complete the questionnaires and give their opinion about whether the questionnaires captured the topic under investigation effectively and whether or not there were any confusing questions. Based on feedback from the LTwT unit, questions were modified accordingly. The two questionnaires were then converted into 'google docs' and the links emailed to all selected participants. In addition, the google docs links were sent through their WhatsApp platform using their cellphone numbers. A total of 42 questionnaires were received out of the total number of 50 questionnaires sent out. Using the sequential mixed methods approach, the pre-coded questionnaire was sent to respondents first and based on the preliminary analysis of responses, the open-ended questionnaire which had already been designed was amended where necessary to probe on issues emerging from the pre-coded questionnaire.

Building on the assertion that the belief of the person towards a system may be influenced by other factors referred to as external variables [15] the pre-coded questionnaire was designed to solicit information from participants on external factors to the system itself, but which could impact acceptance of the technology, such as ease of internet access, device ownership, availability of and ease of access to system technical support. Rather than use the traditional Likert scale, with items ranging from strongly disagree to strongly agree, actual variables were used as coded responses. The variables used are indicated as a key in the results section on the Figures for the pre-coded questions asked. Taking a cue from previous TAM research [26,29,41] the following constructs; Attitude Towards Using (AU), Perceived Usefulness (PU), and Perceived Ease of Use (PEU) were used in constructing the second open ended questionnaire which sought specifically to examine the determinants of technology acceptance at the university under study. Questions under

'Attitude towards using' sought lecturer views on the use of technology for teaching and learning and lecturer likelihood to use WiSeUp (or any other technology integration tool) for learning and teaching if given freedom to opt out. Under the TAM category of perceived usefulness questions solicited information on reasons why academics used the WiSeUp Learning Management System and lecturer views on suitability of the use of WiSeUp for interaction with students both in an out of class. Questions measuring perceived ease of use gathered data on lecturer access to devices for integration of technology in teaching and learning, ease of access to assistance with challenges associated with using the WiSeUp learning management system and computer literacy/competence skills that impeded lecturer effective use of WiSeUp.

### *2.3. Ethical Considerations*

To ensure informed consent an explanatory letter was sent to all participants explaining the purpose of the study prior to commencement. After agreeing to participate, participants then signed consent to participate forms. To ensure anonymity and confidentiality, although the link to the two questionnaires was sent through emails and the WhatsApp platform, participants responded on 'google docs' through the links and this made their identities anonymous. Emails and the WhatsApp platform were thus used only as points of contact and not as points of response to questionnaires. All data was reported as aggregated group data without any reference to individual participant identities.

### *2.4. Data Analysis*

As noted under data collection, both quantitative and qualitative data were collected using two sets of online questionnaires one structured with pre-coded questions and one semi-structured with open ended questions completed by 42 lecturers. Data from the pre-coded questionnaire were analysed through the descriptive statistical approach where the raw data was organized and summarized by use of graphical representation [44] using the Statistical Package for the Social Science (SPSS). This was followed by analysis where inferences, interpretation and conclusions were drawn from the quantitative data. The qualitative data from the semi-structured questionnaire was analysed through thematic analysis. Thematic analysis (TA), is a method for systematically identifying, analyzing, organizing, describing and reporting patterns of meaning (themes) found within a data set [45,46]. The thematic analysis involved an idiographic process that started with an iterative and detailed examination of all the individual responses several times for each question and identifying and coding emerging patterns and themes. Open-coding, axial-coding and selective-coding techniques to identify similarities and differences as well as contradictions was done [47]. Through inductive analysis [48], recurring patterns and common themes were identified. Glaser and Strauss [48], developed this approach that has been used widely in qualitative and mixed methods research studies. This approach enables participants' themes to appear from data rather than pushing the data into pre-existing categories.

### **3. Results**

These results are presented according to the three TAM categories of Attitude Towards Using (AU), Perceived Usefulness (PU), and Perceived Ease of Use (PEU). Under Attitude Towards Using, lecturer views on the use of technology for teaching and learning and results on lecturer likelihood to use WiSeUp (or any other technology integration tool) for learning and teaching if given freedom to opt out are presented. Perceived Usefulness is presented under the following subheadings: Reasons why academics use the WiSeUp Learner Management System, how the use of WiSeUp affects lecturer productivity and lecturer views on suitability of the use of WiSeUp for interaction with students both in and out of class. Under the category of Perceived Ease of Use results are presented under the subheadings; Lecturer access to devices for integration of technology in teaching and learning, computer literacy/competence skills that impede lecturer effective use of WiSeUp and ease of access to assistance with challenges associated with using the WiSeUp Learner Management System.

*3.1. Attitude towards Using (AU)*

• Lecturer Views on the use of technology for teaching and learning

To gauge participants' attitudes towards the use of technology for teaching and learning a question soliciting their views on the use of technology for teaching and learning was paused. Thirty-five out of the forty-two participants view the use of technology in positive light seeing it as necessary in the wake of the COVID-19 outbreak where students had to leave campus. Technology is therefore seen as an opportunity to ensure that students could continue to learn remotely. The context of the 4th industrial revolution was also given as one reason lecturers felt technology should be embraced in order not to be left behind. The following excerpts are sample responses in this regard:

*I think it will really assist since we were faced with COVID-19 and it will help to be aligned with other institutions, so that we won't be left behind in this new era of 4th Industrial revolution.*

*As we approach 4th industrial revolution, technology is becoming a core competency in offering fast and efficient services.*

*It a necessary tool for teaching and learning in this day and age.*

There was also a group of academics who felt apprehensive regarding the use of technology citing capacity to use the technology, fearing that students from disadvantaged backgrounds might be left behind. Some questioned the timing of accelerating the use of technology during times of crises (in this case under COVID-19) arguing that such interventions should be introduced under normal circumstances. There was anxiety around the issue of training as shown in these sample responses:

*My view is that we need lots and lots of training for us to use technology for T&L.*

*We must be cognizant of trying to introduce new ways of teaching and learning during the time of crisis like this. New ways of doing things must be introduced and be mastered while things are still normal.*

*It is convenient during this time of the Pandemic; however, it is less convenient for students who are in the most rural areas.*

*4IR requires of us to use technology. It is good but not fair to students.*

A related question sought to ascertain participant views on fear of being de-skilled with the introduction of the technology. Most of the participants (35 out of 42) had no underlying fears at all regarding the use of the technology indicating their willingness to learn where need be.

*No, I don't fear using technology, as humans we are always learning new things in life, it is not alien that change is inevitable and systems are always evolving, and one needs to always be willing to adapt and be trained on using new systems.*

*Not really, a necessary ongoing learning process for personal development as well.*

*I don't share the same sentiment especially if there is training taking place that will equip everyone to use technology.*

*No. We are living in an era that is becoming more digital by the day and thus is it necessary to adapt to the world of 4th industrial revolution.*

*No. There are academics who use WiSeUp. There is no doubt that it is challenging but it also gives academics several functions to explore for teaching.*

Five out of the forty-two participants did indeed fear that technology would deskill them as they felt they did not have the craft literacy and craft competency to embrace and use the technology. There were some, who although did not fear introduction of technology in teaching and learning, nevertheless saw training as a precondition before e-learning could be rolled out.

*Yes, many people are not trained on WiSeUp up and this makes it difficult to use it.*

*For me I see as a good system, but again thorough training must be provided.*

*Yes, I agree but with proper training not a problem.*

*For me I see it as a good system, but again thorough training must be provided.*

*I don't share the same sentiment especially if there is training taking place that will equip everyone to use technology.*

• Lecturer likelihood to use WiSeUp (or any other technology integration tool) for learning and teaching if given freedom to opt out

Further probed to indicate what they would do if they were left to decide on whether to use WiSeUp (or any other technology integration tool) and there was no compulsion from the university on the use of technology in teaching and learning, all the forty-two participants would opt to integrate technology in teaching and learning anyway. Cited reasons included the fact that students tended to be more actively engaged when learning online when compared to face to face tuition. The need to ensure that the university's students would compete equally in the technologically biased global economy was also cited as a reason for opting for technology even if this was not legislated in the university. The need to ensure learning continued actively beyond the classroom was another reason participant would opt for technology integration out of their own volition.

*Use of technology is good. I would choose technology over any other way. It forces one to learn especially if monitored and eases the work of the lectures.*

*To be quite honest, face to face teaching should be necessary only if there are specific topics that need both the lecturer and students. Most students come to class because of the "attendance register" and do not quite engage so much in class compared to when we're discussing something on an online platform e.g., WiSeUp or WhatsApp.*

*I think it would be unfair for students in our institution if we do not use technology, because the quality of students we will be graduating will not have the competencies and skills required by organizations.*

There was however caution not to abandon the traditional methods implying a blended learning approach to accommodate those students who could be late adopters. Coupled with this again the need for training was given as a pre-condition for voluntarily deciding on whether to use the technology.

*I would decide on using WiSeUp as it makes teaching and learning much accessible and easier. But I would not abandon the traditional ways of teaching because some students are late adaptors.*

*Will choose WiSeUp and other technology integration with training or assistance back up. I would use WiseUp if properly trained.*

### *3.2. Perceived Usefulness*

• Reason why Academics use the WiSeUp Learning Management System

To solicit lecturer responses on the perceived usefulness of the university's learner management system, lecturers were asked to explain why they used the system. The need to reach as many students as possible within a short space of time, technology's ability to allow lecturers to work remotely and reach their students, and its ease of access wherever students are beyond the classroom were some of the justifications given for using technology in teaching and learning. The fact that once uploaded, material remained on the system and students, including those who might have missed the lecturers could access material at their convenience.

It's easier to manage and you can see who is participating or not based on the design of the system.

*To promote effective teaching and learning outside the classroom and easy access to all irrespective of where you are.*

*For its convenience. Firstly, some information uploaded will always be accessible for the rest of their academic year.*

*Secondly if I'm not able to meet students physically I can always upload notes or work on WiseUp.*

*We are living in an era that is becoming more digital by the day and thus is it necessary to adapt to the world of 4th industrial revolution.*

From those who did not derive satisfaction from using WiSeUp, the main reason given was the issue of challenges with connectivity:

*The challenge of accessibility to technology makes me disinterested.*

*I have students in remote areas with internet access challenges.*

• Lecturer views on suitability of the use of WiSeUp for interaction with students both in and out of class

A probing question on the usefulness of the learner management system regarding the issue of student lecturer interaction in an online environment was included in the open-ended questionnaire. The fact the LMS enabled teaching and learning to continue beyond the classroom, enabled students to prepare for face-to-face lectures in advance resulting in greater engagement in class, quicker response rate from students and the opportunities for offering continuous feedback to student students at their convenience were among the reasons cited under perceived usefulness. The opportunities offered by the LMS to help mitigate teaching and learning disruption during times of crises such the national lockdown promulgated in 2020 was also cited by 29 out of 40.

*It is good because I get quick responses from my students before and after class.*

*I find it to be very relevant as teaching and learning continues outside class.*

*Students nowadays use smartphones, tablets, and laptops. The university has implemented WiFi services across the university premises. Keeping connected with students both in class & outside class makes learning easier.*

*WiSeUp is much relevant because students who have managed to interact with the content on WiSeUp are usually coming for lectures prepared*.

*Promotes continuous feedback and assessment for student performance improvements.*

Eleven of the forty-two participants appeared pessimistic on the issue of usefulness of the LMS. Some argued that it was effective out of class but in class, while one some indicted that it was useless as students were inactive on the system with very minimal participation. The issue of training was again brought up as a condition before the system could be found useful.

*Very inactive from students.*

*Effective out of class not so much in class.*

*Out of class-I can send a link for submissions and restrict the duration time and create discussions platform but I have not used the discussion forum platform as yet with my students, still need training.*

*Very minimal.*

*3.3. Perceived Ease of Use (PEOU)*

• Lecturer access to devices for integration of technology in teaching and learning

Under the pre-coded questions there was a question that sought to ascertain provision of resources by the university for ease of use of the learner management system. 90% of the participants used their own laptops and accessed internet facilities online at their own homes and not at the university. 13% of the participants did not have private internet accessibility at their homes meaning they could not work on the LMS at home. This foregrounds the need for data provision for academics beyond the university precincts.

Asked on a pre-coded question on whether they had access to reliable internet at work, it was concerning to note that over half of the participants (54.5%) as shown in Figure 2 either often or very often had challenges with internet access. The prerequisite for e-learning is reliable internet connectivity to be able to use the learner management system and unreliable internet has a negative impact on ease of use

**Figure 2.** Reliability of internet access for teaching and learning.

• Computer literacy/competence skills that impede lecturer effective use of WiSeUp

To follow up on specifics about perceived ease of use, participants were asked to enumerate computer literacy/competence skills they found to impede effective use of WiSeUp. Twenty-five of the participants did not experience any impediments while 17 of the 42 participants have grey areas they felt could be mitigated through training; The main training need given was the need for Microsoft excel training:

*Preparation of online assessments. Basic Computer Literacy. Microsoft Excel for assessments. Loading all work to monitor learner progress.*

• Ease of access to assistance with challenges associated with using the WISeUp learner Management system

Asked what mode they used to seek technical support, (Figure 2), 54.5% indicated that they relied on email communication, 18.2 percent on telephone support and technicians on site respectively and 9.1% on call centre support.

A probing pre-coded question on satisfaction with time normally taken to receive the support requested after logging a query was worrying to note that only 18.2% (Figure 3) were receiving immediate support upon request.

**Figure 3.** Method used to get technical support.

Regarding time taken to receive support, eleven of the participants felt it was challenging to receive support when they experienced challenges with using the WiSeUp Learning Management System while 31 were in the affirmative and one sat on the fence arguing that

it depended on circumstances at the time. As shown in Figure 4, it is concerning that 40.9% of the participants never received the required support and 31.8% only received support after following up several times when they logged requests with the technical support department. Only 18.2% received support timeously. When lecturers do not receive the required support their perceptions of ease of use of the technology declines leading to rejection of technology. For these who were positive reasons given included the fact that the e-learning specialists were readily available when needed:

**Figure 4.** Time taken to get support.

*It's easy. The e-teaching and learning specialist is very diligent. It is easy. CLTD are always willing to go above and beyond to assist with all the challenges that I face.*

*Not difficult. Staff is always ready to help.*

*Not difficult at all for me. Key colleagues and CLTD are on speed dial and always ready to assist even after hours.*

The fact that while some felt support was adequate others felt there was no support could probably be due to the divisional model (where individual campuses are semiautonomous) where support on some campuses could have been adequate while not adequate at other campuses.

*Difficult to get any kind of help related to WiseUp. It easy to get assistance but sometimes I run out of data. Difficult because of staff shortage and time constraints. It is difficult not enough support.*

### **4. Discussion**

The contiguous approach to integration was used in the preceding section on presentation of results, which comprised the presentation of findings with the qualitative and quantitative findings reported in separate subdivisions [40]. In this discussion of results section, the weaving approach to integration was used where both qualitative and quantitative results are discussed simultaneously on a theme-by-theme basis [40] using the three TAM concepts of Attitude Towards Use, Perceived Ease of Use and Perceived Usefulness.

*4.1. Attitude towards Using (AU)*

• Lecturer views on the use of technology for teaching and learning

As Surendran [49] shows the attitude towards use is concerned with user's evaluation of the desirability of employing a particular e-learning system and is a measure of the likelihood of the person using the system. Regarding attitudes towards using, qualitative results of the study show general positive attitudes towards the use of technology for teaching and learning. This is corroborated in the quantitative data where thirty-five out of the forty-two participants view the use of technology in positive light seeing it as necessary in the wake of the COVID-19 outbreak where students had to leave campus. Technology is therefore seen as an opportunity to ensure that students could continue to learn remotely. Thus, users' mental assessment of the match between important goals at work (successfully completing the academic year in this instance) and the consequences of performing job tasks using the system (using technology to ensure students learn remotely from home due to COVID-19 restrictions) serve as a basis for forming perceptions regarding the usefulness of the system [15,28,31]. Hoong, Thi and Lin [28] further alluded that individuals rely on the fit between their job and the performance outcomes of using the system before concluding on usefulness of the system. They argue that if the system does not produce any desirable output to enhance individual performance, the user acceptance rate is likely to drop. The concern about teacher fear in the use of ICT is confirmed by [50], who indicated that even though some teachers believe in constructivist pedagogy, they are still reluctant to use technology because of various constraints such as lack of adequate time to design lessons for online delivery, insufficient specialized help, absence of physical contact with students as well as challenges related to internet connectivity. [34] advise that factors that should be taken into consideration, besides the teaching process and instructional content, are e-competencies of students and educators, as well as the attitudes toward this mode of learning and the usability of the system. In this study, the opportunities provided by the e-learning system to save the academic year through moving to online teaching and learning to mitigate the effects of COVID-19 in face-to-face tuition seems to have cultivated positive attitudes towards use.

Five of the participants in this study however, though few, as shown in the results, felt apprehensive and therefore had negative attitudes towards the use of technology citing capacity to use the technology. To mitigate this challenge of poor technology acceptance, Ibrahim and Nat [51] call for provision of professional development programmes specifically for pedagogical and technological skills. A study [52] concludes that the level of competence regarding the use of technological tools can be improved in the need to understand that the development of virtual teaching also entails the need to develop and enhance the competence part linked to interaction and communication with students. Similarly, [33] argue that more ICT teacher training means better training conditions for students and recommend that teachers be trained in both technological and pedagogical areas in order to develop digital teaching skills.

Some lecturers in this study questioned the timing of accelerating the use of technology during times of crises (in this case under COVID-19) arguing that such interventions should be introduced under normal circumstances. This finding corroborates findings by Johnson et al. [53], which showed that teachers most of the time viewed technology as an imposition. People need to feel at ease when implementing interventions as opposed to feelings of pressure and compulsion. As Rossouw and Alexander [54] pointed out, people also need to feel they have some individual control over change as group needs, organizational needs and individual needs are not synonymous and should be addressed differently.

• Computer literacy/competence skills that impede lecturer effective use of WiSeUp resulting in lecturers being de-skilled with the introduction of the technology

Fear of the unknown can sometimes affect attitudes towards emerging innovations. While many of the participants (35 out of 42) as shown in the quantitative results had no underlying fears at all regarding the use of the technology indicating their willingness to learn where need be, five out of the forty-two participants as shown the qualitative data felt that technology would deskill them as they felt they did not have the craft literacy and craft competency to embrace and use the technology. The results assert findings by Portz et al. [26], who found in their study that perceived ease of use was impacted by participants' level of computer anxiety and computer self-efficacy. Thus, the absence of technological literacy slows blended learning applications among lecturers, and frequent interaction with technology encourages the intention to blend among instructors [51]. Ibrahim and Nat [51] further argued that capacity building in relation to training is the most critical support that a lecturer can tap from the institution. Reporting from China, [55] concluded based on the data gathered in their study that factors affecting Chinese English teachers' online

teaching provide suggestions for policymakers and teacher professional development, such as improvement in technical support, and provision of technology training.

One way of enhancing this capacity building would be through encouraging the formation of communities of practice so academics can share their practice and support each other in integrating information communication technologies in teaching. Communities of practice are groups of people who are willing to spend time together to share information, insight and advice where members ponder common issues, explore ideas and act as sounding boards for each other's ideas [45,56]. Members in a community of practice engage in joint activities and discussions, help each other and share information and build relationships that enable them to learn from each other [57–60]

• Lecturer likelihood to use WiSeUp out of own (or any other technology integration tool) for learning and teaching if given freedom to opt out

Of consensus in the results (qualitative data) is the assertion by all lectures in the study that they would continue to use WiSeUp (or any other technology integration tool) even if there was no compulsion from the university on the use of technology in teaching and learning. Cited reasons included the fact that students tended to be more actively engaged when learning online when compared to face to face tuition. The belief that technology promoted student engagement was thus a motivator that led to justification for use of the technology as it was believed this would lead to active learning in the classroom. A study by Johnson [53], shows that teachers attitudes and beliefs in the use of technology are crucial factors that determine the role and effectiveness of technology in the classroom. The need to ensure that the university's students would compete equally in the technologically biased global economy was also cited as a reason for opting for technology even if this was not legislated in the university. The need to ensure learning continued actively beyond the classroom was another reason lecturers would opt for technology integration out of their own volition. The results demonstrate that participants have assessed the potential that technology has on their work and resolved that the introduced system responds to both their current needs and those of their students. As Hoong, Thi and Lin [28], conclude, individuals will assess whether the technology constitutes a threat or an opportunity and how it can adapt into their daily tasks by changing their working behavior. In this instance the participants have resolved to use the technology out of their own free will.

### *4.2. Perceived Usefulness (PU)*

• Lecturer views on suitability of the use of WiSeUp for interaction with students both in and out of class

In line with the Perceived Usefulness (PU) tenet of the Technology Acceptance Model (TAM), results of this study have demonstrated that indeed the adoption of technology in learning teaching at the university under study depended on the extent to which it was seen as relevant and useful in the learning and teaching process. Hoong, Thi and Lin [28], argued that perceived usefulness (PU) is characterised as how much individuals trust that utilising a specific tool would improve their performance and, "is the key determinant that emphatically influences users' convictions and expectation to utilize the innovation." With regards to the perceived usefulness of the university's learning management system (WiSeUp), lecturers cited the need to reach as many students as possible within a short space of time, technology's ability to allow lecturers to work remotely and reach their students and its ease of access wherever students were beyond the classroom as factors that would enhance their performance under the COVID-19 circumstances. This is, firstly, because the easier a user feels it is to use a new technology or service, the more useful lecturers perceive it to be [61] and, secondly, because the time and effort required to use online educational services are reduced, thus making the service more convenient [55]. The fact that once uploaded, material remained on the system and students, including those who might have missed the lecturers could access material at their convenience constituted justification for usefulness of the WiSeUp. This is in line with a study by Ertmer et al. [62]

who discovered that teachers are able to enact technology integration practices that are closely aligned with their beliefs.

Further perceived usefulness, from the results can be seen in the manner lecturers saw the use of WiSeUp positively affecting their productivity and effectiveness. The flexible accessibility of the system by students and the efficacy it brought, saved valuable time. The ability to send additional links to students after online teaching, the workload reduction in assessment, specifically for Multiple Choice Questions (MCQs) were lauded. WiSeUp also ensured that even if a lecturer had to be away, for example to attend a meeting, they could upload the lessons online and learning would continue in their absence. As e-learning is not time-bound or static, it helped the students to access the material from anywhere and at any time (Patra, Sundaray and Mahapatra 2021) What emerges here is the perceived impact of technology on productivity. The attitude of an individual is not the only factor that determines his/her use of new technology, as the impact the tool or system will have on his/her performance is also significant [22,41]. Literature shows that when teachers believe that technology connects directly with their specific content areas and/or grade levels, as well as allowing them to more readily meet their classroom goals they likely have a tendency to use it frequently [63,64].

### *4.3. Perceived Ease of Use (PEOU)*

• Lecturer access to devices for integration of technology in teaching and learning

Perceived ease of use (PEOU) is defined as the degree to which the prospective user expects the intervention or system being introduced to be free of effort [49]. Results of this study show that the amount of effort or resources that lecturers must find on their own to use a system has a dent on perceived ease of use of the system. The fact that participants used their own laptops and internet facilities to be online at their own homes as shown in the quantitative data and not at the university meant that when they ran out of resources to purchase such accessibility, they could not work on WiSeUp at home. In addition, quantitative findings of the study revealed that half of the participants had no access to reliable internet at work. The prerequisite for e-learning is reliable internet connectivity to be able to use the learning management system and unreliable internet has a negative impact on ease of use. Some studies have identified issues such as lack of lecturer preparation for online learning, constraints on learning facilities that are not fully ready and complete for students and technical obstacles such as the internet network that many students complain about during online learning [17,18]. Ibrahim and Nat [51] contend that lack of access to appropriate hardware and software can slow and suppress the highest motivation. Participants in focus group discussions in a study by Chigona [65] asked for more digital resources such as reliable software and Wi-Fi and believed that making available such requisite resources could be the answer to educators' adoption and use of connected classrooms effectively. It is indisputably disappointing for the educators when they do not have adequate resources to implement their ideas or work with the system [65].

### *4.4. Ease of Access to Assistance with Challenges Associated with Using the WISeUp Learner Management System (Training and Technical)*

Tied to the issue of internet connectivity, the findings revealed that the extent to which lecturers feel they are comfortable to navigate the WiSeUp learning management system had a bearing on perceived ease of use. While it is laudable as the quantitative data revealed, that 25 of the 42 participants did not experience any impediments, the 17 participants who indicated need for training will need to be prioritized to improve their perceived ease of use. In the literature, studies show that if teachers do not have necessary competencies in using technology, they are unlikely to explore new possibilities to utilize technology compared to those who have the knowledge and skills in the use of technology frequently [33,56,64,66]. A study by Nair and Das [67] revealed that teachers would find the information technology (IT) tools more useful and will have a positive attitude towards integration of technology in teaching if through adequate training they are made more proficient in using such tools.

Further to the issue of knowledge of the system, the provision of the requisite technical support to navigate the system when need arose and the extent of satisfaction with such support was found to influence perceived ease of use. As shown in the quantitative results only 18% of the participants normally received support requested immediately after logging a query with 28% indicating that it was challenging to receive support at all when they experienced challenges with using the WiSeUp Learning Management System. Rossouw and Alexander [54], suggested that users experience the system as technology and if the system functions without any problems, then the technology is not a problem. For those who are not technology-savvy, time and effort must be invested to perform these operations in addition to ensuring that the pedagogical aspects of the course are managed effectively and lack of support creates stress and increases teachers' perceptions of complexity of the technology system [55]. In the same vein, Mbodila Ndebele and Muhandji [68] confirm that the integration of new technology for the purpose of teaching and learning depends on level of support and guidance that is provided to both teachers and students in the use of the new technology. A different study by Hu and Garimalla [69] confirms that professional development such as training to support teachers in the use of technology is one way to promote technology adoption. The differential views in satisfaction with support in this study could be attributed to the divisional model in the university (where campuses are semi-autonomous) where support is provided per campus and where support on some campuses could have been adequate while not adequate at other campuses.

### **5. Conclusions**

In conclusion, the integration of technology in teaching and learning has seen increased focus in the higher education systems around the world and continues to be a significant area of research today. Most higher education institutions around the world and in South Africa have integrated various learning management systems (LMSs) to deliver teaching and learning in a blended fashion. However, there are still challenges of slow adoption amongst academics in many institutions. The results of this study show that most academic staff still believe and see the value that ICTs bring in their teaching and learning practices. In addition, they are aware that technology use in education improves learning and teaching, and they are willing to embrace the use of technology to improve their practices. However, there is a need for the HEI to provide requisite training, support, resources and tools of trade to enable lecturers to make continuous use of technology in teaching and learning even beyond COVID-19 pandemic. Based on the above findings, the following recommendations are put forward:


**Author Contributions:** Conceptualization, C.N.; methodology, C.N.; formal analysis quantitative data, M.M.; formal analysis qualitative data, C.N. writing—original draft preparation, M.M. writingreview and editing, M.M. and C.N. All authors have read and agreed to the published version of the manuscript.

**Funding:** The work described in this paper was partially supported from the Directorate of Learning and Teaching, and Department of Information Technology Systems, at the university under study through payment of page fees

**Institutional Review Board Statement:** Ethical review and approval were waived for this study, since individuals were contacted individually in their individual capacity and gave consent in their individual capacity and not in their capacity as members of a particular university. Secondly, the name of the university is not mentioned in the study to maintain the anonymity assured to participants.

**Informed Consent Statement:** As detailed in the sub-section on ethical considerations in this article, informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The datasets used/or analysed during the current study are available from the corresponding author on reasonable request.

**Acknowledgments:** The authors acknowledge the students and staff of the university, the Directorate of Learning and Teaching, and Department of Information Technology Systems, Walter Sisulu University who participated in the study and consented to being acknowledged.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

### **References**


### *Article* **Exploring the Relationship between Saber Pro Test Outcomes and Student Teacher Characteristics in Colombia: Recommendations for Improving Bachelor's Degree Education**

**Paola Sáenz-Castro 1, Dimitrios Vlachopoulos 2,\* and Sergi Fàbregues <sup>3</sup>**


**Abstract:** This explanatory sequential mixed methods study explores the perceptions of academic and administrative managers responsible for teacher training at a public university in Colombia, as well as their views on improving such training after learning about the performance of teachers student teachers in the 2019 Saber Pro test, the differences in their test scores, and the relationships and statistical correlations between these outcomes and the students' personal, family, socioeconomic and academic characteristics. Our findings show significant differences in the student teachers' mean scores and performance when data are grouped according to personal, socioeconomic and academic conditions; a significant relationship between performance and student teacher characteristics; and correlations between critical reading scores and the other competencies assessed. Our data also highlight the lack of knowledge among academic and administrative managers about students' life circumstances and the diversity of factors that may impact their performance; the importance of correlational data; the difference between expected and true outcomes; the inequity under which students seem to pursue their education; the limitations in access to resources; the training required for teachers to be able to analyze quantitative data and use specific software; the impact of teachers' critical reading skills on student outcomes; the importance of data-driven decision-making; and the need for teachers to engage in quantitative research practices.

**Keywords:** standardized test; Saber Pro; student characteristics; mean score differences; correlations between competencies; academic performance; improvement

### **1. Introduction**

In what is commonly known as the Coleman Report, renowned sociologist James Coleman argued that sociodemographic factors could account for differences in students' scores and that school resources and teachers' education do not have a noticeable effect on student performance. Since the Coleman Report was published in 1966, researchers have debated the impact of education policies and the school system on student performance, in addition to other environmental and sociodemographic effects [1].

Improving the quality of education is an important factor for development, especially in Latin America, a part of the world with large student achievement gaps that are a reflection of substantial income inequalities [2]. Previous research has claimed that correlations exist between a district's level of development and student outcomes in Costa Rica; that a student's neighborhood has the greatest impact on educational performance; that the quality of services such as electricity and telecommunications is strongly related to student performance in Costa Rica; that information and communication technologies have

**Citation:** Sáenz-Castro, P.; Vlachopoulos, D.; Fàbregues, S. Exploring the Relationship between Saber Pro Test Outcomes and Student Teacher Characteristics in Colombia: Recommendations for Improving Bachelor's Degree Education. *Educ. Sci.* **2021**, *11*, 507. https://doi.org/ 10.3390/educsci11090507

Academic Editor: Maria Limniou

Received: 24 July 2021 Accepted: 27 August 2021 Published: 6 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

a positive effect on educational quality in Mexico; that individual and family variables are more closely linked to academic performance than school variables; and that social development has a positive effect on students' scores [2–6]. Other international studies [7] highlight the effect of professional community on student achievement, as well as the importance of student engagement [8–10] and quality management/culture [11]. Exploring the relationship between quality and equity in education and identifying factors that can reduce the impact of students' backgrounds on learning outcomes are important issues for research, policy development and teaching practices, especially when the focus is on enhancing educational quality [12].

Meanwhile, educational data mining (EDM) and learning analytics have the ability to influence current teaching and learning models. The primary goals of EDM include improving assessment techniques, reducing dropout rates, recommending resource materials based on prior learning, performing measurements, and ensuring students' self-actualization [13]. In this vein, the Colombian Institute for Educational Evaluation (http://www.icfes.gov.co accessed on 10 May 2020) has annually or biannually published the results of Saber Pro, a standardized test taken by Colombian higher education students, since 2006.

### **2. Literature Review**

In order to draw an accurate picture of quality in education, we need a set of criteria that allows us to assess the procedures used to determine it [14,15]. In this regard, the analysis of standardized test results must overcome one of its greatest obstacles: the inability to delve into the reality of the individual, the student or the institution. Studies show that children living in extreme poverty, deprived of basic needs such as food, clothing or shelter, or lacking utilities such as electricity or internet access are more likely to perform poorly in school and drop out [16,17]. Knowing each item assessed on a standardized test and what it measures is as important to the practice of teaching as defending the idea that quality in education goes far beyond quantitative results [14].

With respect to data analysis, universities' digital culture involves the adoption and use of information and communication technologies, as well as the creation of methodologies and models based on these technologies [18]. In this data-driven landscape, teachers will serve as catalysts for learning success, as they will be capable of analyzing data to better comprehend their own teaching actions, the type of students in their classrooms, and the learning outcomes their students achieve [19,20].

Today, education-focused research is supported by advanced equipment and software that are able to apply statistical techniques to optimize the collection of valuable information from massive datasets [21]. This, in turn, makes it possible to come up with improvement actions and plans that account for the link between individual and family variables and student performance in Latin America [2] and the impact that sociodemographic factors have on learning outcomes. Data analysis could be used to bridge teaching pathways, with joint plans combining curricular proposals and economic resources mapped out to boost students' progress and encourage them to stay in school. This could help meet the need to lay down common objectives in different areas in order to make progress in complex contexts [22–25].

### *2.1. Quality in Education*

In 1998, UNESCO defined quality in higher education as a multidimensional concept that should recognize and include functions or activities such as teaching and academic programs, research and scholarships, infrastructure, and the academic environment. According to the Colombian Ministry of National Education in 2009, educational quality meant ensuring that all students have opportunities to be productive and engage in lifelong learning. Then, in 2011, it was reworded to mean the creation of legitimate opportunities for progress. The National Accreditation Council (CNA) defines quality as the synthesis of

characteristics that allow a program or an institution to make judgments on the relative distance between how things are and how they should optimally be.

### *2.2. What to Do with Standardized Tests in Education*

Research has shown that teachers, schools and countries that are effective in terms of quality tend to be effective in terms of equity as well [26]. However, standardized tests in higher education seem more focused on assessing education itself, rather than assessing for educational purposes. So, rethinking assessment methods could lead to changes in curricula [27]. Multilateral organizations are promoting education reform, and standardized tests offer the statistical indicators that these organizations are looking for in order to devise strategies that eventually become government policies in different countries [14,28–30]. When this type of test yields unfavorable results, government agencies deliver recommendations without a critical understanding of the solutions they are proposing, and they prioritize reproducing quantitative test data over exploring the implications for education [31–35]. In addition to the above problem, the Colombian law allows professionals from any area to be recognized as teachers at any level of education. As a result, the education system expects professionals who are unfamiliar with data analysis and data-driven decision-making to propose reform strategies that will have an impact on millions of children and adolescents. A change in this regard requires broad political and academic resolve, as well as strategies for qualifying teachers at all levels.

There has been considerable research into Saber Pro tests in recent years, and the consensus is that results-oriented courses are scarce [35–37] and that they depend on developments in other program cycles and on teachers themselves [38–40]. Previous research has called for updated pedagogical, methodological and didactic strategies in undergraduate education and for a critical look at how learning and teaching styles affect results [40,41]. Researchers have also suggested that statistical data analysis is key to improving educational quality [36,37,42–44] and that discussions are needed to assess educational quality [41,42].

### *2.3. Political Action from Educational Data Mining and Learning Analytics*

The International Conference on Learning Analytics & Knowledge defines learning analytics as "the measurement, collection, analysis, and reporting of data about students and their contexts for the purpose of understanding and optimizing learning and the environments in which it occurs" [45]. Educational data mining (EDM), for its part, is an approach centered around developing methods for exploring unique data types to understand students and the environments in which they learn, so as to make lessons more effective. Although learning analytics are an essential part of online learning environments, teachers must also understand how to use data mining in education to make decisions based on otherwise unknown and potentially useful data patterns [46–48]. Bearing in mind that student success is a quality parameter of higher education institutions and that EDM techniques help to find relevant patterns and data, information about students' life circumstances should be included to improve learning [49]. This is also called learning analytics [13] and is a source of political actions.

The importance of interpreting student data lies in establishing progress parameters, detecting problems in order to identify social connections, integrating pedagogy into data mining [50–52] and allowing for better decision-making [18,52]. Learning analytics can help to reduce educational achievement gaps [19,53,54]. Moreover, if they were to extend their interpretive scope beyond outcomes, scores and performance to account for students' life circumstances as well, they could become a key tool for shedding light on and helping to close the invisible divide at different educational levels. Such analytics could lead to actions such as intervention, optimization, alerts and warnings, guidelines and pathways, and systemic improvements in planning or teaching [55]. Whatever the case may be, learning analytics face some important challenges, namely, assuring the quality and scope of the data gathered and the privacy and ethics of any analyses carried out thereon.

Educational institutions can restructure learning design processes based on analyses, meaning that teachers can incorporate feedback from these analyses into the future design of learning content and also customize content by including their personal understanding of a topic and previous experience [55]. In short, they can use analysis-derived information to make informed pedagogical decisions [56].

Currently, data analysis is not part of the required curriculum for undergraduate student teachers in Colombia, who therefore lack the necessary depth and rigor to use data to make education-related decisions. However, if students' socioeconomic level and teachers' subject knowledge are to be considered predictors of academic performance [57], this type of training has good reason to be enhanced.

This study is driven by our interest in acknowledging the characteristics of student teachers assessed in the 2019 Saber Pro standardized tests, and in offering insight to the academic and administrative managers responsible for teacher training in a Colombian public university into how these students' scores/performance statistically relate to their personal, family, socioeconomic and academic characteristics. This approach, which was new to some of our interviewees, opens the door to actions that promote change or improvements for the benefit of future education professionals. It is here where educational data mining and learning analytics surface as a source of political action.

### *2.4. Learning Analytics and the Link between Higher Education and Other Educational Levels*

While increased attention has been given to higher education in Colombia, it is unfortunately not under any quality education plan involving continuous education for individuals. This is evidenced by the percentage of students dropping out of higher education, as well as by the direct relationship between these dropout rates and the competencies, knowledge and skills achieved by students at previous educational levels. Although efforts to improve national productivity are important, they will not be sufficient if student development is not considered a continuous process that seeks to strengthen cognitive development. Colombia has proposed bridging the gap between secondary and higher education, albeit from a perspective of coordination with the productive sector. This is at odds with the fact that curricular connections are meant to act as a two-way link between universities and their surrounding context, as well as between theory and practice. A curriculum whose design is based on competencies, active pedagogies and flexibility allows students to choose according to their interests and lays the foundation for new interinstitutional alliances that promote collaboration across any field, subject area, time of year, semester or credit system [58–61]. If connections between secondary and higher education facilitate progress in complex contexts and curricula become bridges for activity between universities and their surrounding context, administrative support will be more likely to focus on students' education [58,61].

The challenge is to optimize learning, satisfy political interests and exploit available data to descriptively, predictively or prescriptively analyze solutions [19,54]. In this way, it is possible to envision the bridging of teaching pathways as a process in which educators coordinate plans and programs, accounting for previous learning achievements and paving the way for students to make progress. Curricular proposals, learning pathways, pedagogical models, institutional agreements and economic resources would all come together in this scenario [22].

### **3. The Saber Pro Standardized Test in Colombia**

The purpose of the Colombian Institute for Educational Evaluation (ICFES) is to assess the quality of education at all levels through standardized external examinations. It is the Ministry of National Education (MEN) that determines what should be tested in these examinations (Law 1324 of 2009). The ICFES has worked on aligning the National System of Standardized External Evaluation, making it possible to compare results at different educational levels, since different examinations evaluate the same competencies, particularly general competencies, in some areas. Saber Pro is a requirement for students

to earn their bachelor's degree. It aims to verify that students have correctly developed the target competencies of their degree, to produce value-added indicators, and to track indicators that assess the quality of higher education programs and institutions.

### *Exam Structure*

The Saber Pro test has a mandatory first sitting consisting of five modules that assess general competencies: critical reading (30 questions); quantitative reasoning (30 questions); citizenship skills (30 questions); written communication (1 question); and English (45 questions). Test takers may also opt for a second sitting made up of specific modules, which, in the case of teachers in training, are educating, teaching and evaluating. For the written communication module, the test poses an open-ended question or a topic, based on which students are asked to write an argumentative text. The other modules pose multiple choice questions with only one possible answer.

The ICFES website [62] provides an overview of the consolidated test results, including personal information (gender); geographic location (department, municipality and area of residence); academic data (cost of tuition and form of payment, semester in progress at the time of test submission); socioeconomic data (parents' level of education, parents' type of employment, socioeconomic stratum, access to the internet, television, a computer and other services, number of people with whom they share a bathroom, hours of work per week); information on the higher education institution (degree program); and results (scores and performance in the general and specific competencies assessed).

Based on data on residential properties from the DANE ("Statistics and data of Colombian government"), socioeconomic strata is a proxy measure of the economic and social development of the different areas of a municipality, as it classifies households according to their characteristics and living conditions. Socioeconomic strata range from one to six, with stratum one indicating lower living conditions and stratum six indicating higher living conditions.

### **4. Objectives and Research Questions**

Learning analytics are a necessary tool for defining improvement strategies and actions. This holds true both for higher education institutions engaged in quality assessment (measured by means of standardized test results) and the teachers who assume the new role of analysts. Through our analysis, we hope to lay the groundwork for methods and techniques that will help decision makers find meaningful insights in the data gathered in the Saber Pro test.

This explanatory sequential mixed methods study seeks to provide academic and scientific communities in higher education with input on how to exploit the results of standardized tests, focusing on the relationship between student teacher characteristics and their scores and performance in the saber Pro test. In doing so, we aim to strengthen learning processes at all levels of education in Colombia by addressing the differences in subjects and contexts, identifying contextual needs and closing the gap between educational levels. We believe that the findings of our study may serve as a springboard for improving teacher training programs and higher education institutions, as well as for setting viable objectives to promote educational quality and equity. In pursuit of these goals, our study also explores the perceptions of academic and administrative managers responsible for teacher training at a public university, as well as their initial ideas for improving such training after learning about the performance of student teachers in the 2019 Saber Pro test, the differences in their test scores, and the relationships and statistical correlations between these outcomes and the student teachers personal, family, socioeconomic and academic characteristics.

Considering the possibilities for establishing interinstitutional alliances and thus fostering curricular engagement, the following research questions are addressed:


### **5. Research Methods**

We used an explanatory sequential mixed methods design, based on an initial quantitative phase and a subsequent qualitative phase. In the quantitative phase, we aimed to find any statistically significant differences, relationships and correlations between the 2019 Saber Pro test outcomes and the student teachers' family, socioeconomic, personal and academic characteristics. Then, in the qualitative phase, we showed the results of our quantitative analyses to a sample of academic and administrative managers (Figure 1). We sought not only to gather their perceptions, but also to learn about their initial ideas for improving teacher training and fortifying the bridge between higher education and other educational levels in light of the quantitative results.

**Figure 1.** Explanatory mixed methods sequential design.

### *5.1. Quantitative Phase*

Quantitative secondary data were taken from a public university in the southwest region of Colombia. The file containing the national results of the 2019 Saber Pro test (*n* = 187,469) was downloaded, filtered by institution (*n* = 1763) and academic program. As a result, our data analysis focused on the outcomes of 298 student teachers. The independent variables were: (1) personal characteristics (gender); (2) socioeconomic characteristics (area of residence, cost of tuition, form of payment, parents' level of education, parents' type of employment, socioeconomic stratum, services available, number of people with whom the bathroom is shared, hours of work per week, pay); and (3) academic characteristics (semester in progress, name of the academic program). The dependent variables were: (1) scores in general competencies; (2) performance in general competencies; (3) scores in specific competencies; and (4) performance in specific competencies. A description of the values of the variables used in this study is provided in Table A1.

JASP statistical software was used to analyze the data. Data were entered into the software and the variables were organized. Missing values and outliers were processed. Descriptive analyses were carried out, including frequency, central tendency values and variance. For significant mean difference, a Student's *t*-test was used for a factor and another Student's *t*-test was run for the independent samples and the ANOVA test factor. To select the appropriate test for comparing differences between two independent samples, we used the Shapiro–Wilk test to assess the normality of the variables. Based on these results, we used Student's t-test when the two samples had equal variances, Welch's test when the variances were unequal, and the Mann–Whitney test when the distribution was not normal. For the ANOVA analysis, we used Levene's test to assess the homogeneity of the variances between the groups. When the homogeneity assumption was not met, we used the Kruskal–Wallis test to assess the significance of differences across the participants' scores. Finally, to test the significance of the relationship between the qualitative variables, we ran a chi-squared test, and for the quantitative variables, we used the Pearson correlation coefficient, Spearman's rho or another test that was deemed relevant given the attributes of the variables to be related.

### *5.2. Qualitative Phase*

Purposive sampling was used to bring together a group of individuals in charge of drawing up program and institutional improvement plans at the abovementioned public university in Colombia. The resulting sample was made up of a quality manager from the Central Curricular Committee, the academic vice president, the head of the Office of Accreditation and Quality, and eight coordinators of the following nine bachelor's degree student teacher programs: physical education, recreation and sport, art education, Spanish and literature, modern languages (English and French), Spanish and English, mathematics, music, and ethno-education. Since the work of these eight coordinators is evaluated by the head of each respective Department and not by the three managers attending the meeting, they could express their views openly and transparently.

Participants were recruited by the first author who coordinated an institutional project in which regular meetings were held with the bachelor's degree coordinators and the quality manager. In one of these meetings, a group interview was carried out using Google Meet, which also included the academic vice president of the university and the head of the Office of Accreditation and Quality. During the interview, the results from the first quantitative phase were presented to the participants, bearing in mind that students at this higher education institution sat the 2019 Saber Pro test. The aims of the interview were twofold: firstly, to examine participants' perceptions of the quantitative results; secondly, to explore their views on the utility of these results in driving actions to improve their degree programs and the institution as a whole, as well as in building bridges between higher education and other educational levels. The interview guide included the following two questions: (1) What results particularly caught your attention? (2) In light of the results presented, what strategic actions do you think could be included in program and

institutional improvement plans to strengthen ties between higher education and other educational levels?

The group interview was audio and video recorded and transcribed verbatim. Transcripts were uploaded into the qualitative data analysis software QDA Miner Lite v 2.0.8. We followed Braun and Clarke's [63] approach to thematic analysis, which consists of six phases: (1) familiarization with the data; (2) initial code generation; (3) search for topics; (4) review of topics; (5) designation of topics; and (6) preparation of the report. During this process, we repeatedly read the transcribed data in search of patterns, and then coded and qualified these patterns to form themes. Codes and issues were identified in the explicit or superficial meanings of the data, without looking at anything beyond what had been said when the participants mentioned something directly related to the questions posed. We have included extracts from our data below in order to substantiate the answers to our research questions. The coding and analysis of the qualitative data were made by the first author under the supervision of the second author. Any disagreements arising at this stage were discussed by the first and second author until a consensus was reached.

The interviews took place in May 2020. All participants gave their consent before being interviewed (Appendix A).

### **6. Results**

### *6.1. Statistical Description of the Analyzed Data*

Our study population had the following characteristics: 87.6% of the students were enrolled at the main campus; 93.6% studied on-site; 3% lived outside the department where the main campus is located and 21.5% lived in municipalities other than where the two campuses are located; 17.4% lived in rural areas; 85.5% paid less than USD 129 in tuition fees; 21.8% of the students' fathers did not finish elementary school and 20.7% finished their secondary school education (*bachillerato*); 25.1% of the students' mothers did not finish elementary school and 26.9% finished their secondary school education (*bachillerato*); 48.3% of students belonged to stratum 1 and 33.6% to stratum 2; 9.2% of students shared a bathroom with more than six other people; 18.1% of the fathers worked as farmers, fishermen or day laborers; 41.1% of the mothers were homemakers and neither worked nor studied; and 31% of the students worked between 11 and 20 h a week before registering for the exam.

As shown in Table 1, evaluating (Ev) and quantitative reasoning (QR) had the highest and lowest mean scores in our population, respectively. The population means exceeded the national means for all general competencies, with the exception of QR. Compared to the institution-wide means, only English (E) and written communication (WC) came out ahead.



Note: The institutional and national average of the competencies Educating, Evaluating and Teaching is not included since not all students of the institution nor all students in the country present these modules.

> Table 2 shows a breakdown of student performance in these general competencies. A large proportion of students fell into performance levels 1 and 2. Specifically, 71.4% were at level 1 or 2 for QR; 67.1% for citizenship skills (CS); 55.8% for WC; and 51.4% for critical reading (CR). With respect to the specific competencies, 75.9% of the students fell

into levels 2 or 3 for evaluating (Ev), 71.8% for teaching (T) and 67.8% for educating (Ed). However, we should also point out that almost a fourth of the students (24.8%) were in level 1 for Ed. Institutional regulations require students to be at level A2 in English (E) and national regulations require them to graduate at level B1 or higher if they have a foreign language focus. However, 62.3% of the students in our population were at level A2 or below and only 37.7% were at level B1 or above.


**Table 2.** Percentage of students by performance level.

Note: \* The student missing from the total population did not take the module and was excluded from the analysis. English is the only module to have five levels of performance, following the Common European Framework of Reference for Languages. The "0" means that the student did not take the test or that their knowledge did not reach level A1.

### *6.2. Statistically Significant Differences in Mean Scores When Students Were Grouped by Their Characteristics (RQ1)*

Tables 3 and 4 show statistically significant differences in mean scores when students were grouped by characteristics. A significant difference was found when grouping students according to gender (QR, Ev); if they pay the tuition with credit (CS, E); if the tuition is paid by parents (CR); according to some services the student has, such as internet (CR, E), a pc or laptop (E), a washing machine (E), or a tv (E); the cost of tuition (QR, E, WC, T, Ed, Ev); semester in progress (QR, E, WC, T, Ed, Ev); degree program (all skills evaluated); municipality of residence (E, T); father's level of education (CR, E); mother's level of education (T, Ev); socioeconomic stratum (E); mother's type of employment (Ed).



Note: S: Student. MW: Mann-Whitney. W: Welch. A p value that is less than or equal to 0.05 denotes a statistically significant difference. Significant differences between mean scores when students are grouped by gender, enrollment data, available services and mode of instruction.



**Table 4.**ANOVA tests.

182

program, municipality

 of residence, father's or mother's level of education,

socioeconomic

 stratum and mother's type of employment.

 \* The

Kruskal–Wallis

 test was used for

non-parametric

 data.

### *6.3. Statistically Significant Relationships between Students' Performance and Characteristics, and Correlations between the Scores Achieved*

Table 5 displays the significant relationships between categories. Among the competencies assessed, E is associated with the greatest number of student characteristics, while degree program is the characteristic that is associated with the greatest number of competencies. Table 6 shows the correlation between the scores achieved for the competencies assessed.

**Table 5.** Significant relationships between students' characteristics and their performance in the competences assessed.


Note: P is performance; QR is quantitative reasoning; CR is critical reading; CS is citizen skills; WC is written communication; E is English; Ed is educating; Ev is evaluating; and T is teaching. A *p* value that is less than or equal to 0.05 denotes a statistically significant difference.

**Table 6.** Correlations between scores.


Note: The greater the number of asterisks, the greater the strength of the correlation: \* *p* < 0.05; \*\* *p* < 0.01; and \*\*\* *p* < 0.001.

### *6.4. Perceptions of the Academic and Administrative Managers of Teacher Training Regarding Our Data Analysis of the 2019 Saber Pro Test Results, and Their Initial Ideas for Improvement (RQ2)*

For the qualitative phase of our explanatory mixed methods sequential design, we held a group interview with managers responsible for teacher training, as explained in the methods section. After being apprised of the results from the first quantitative phase, the participants expressed a general lack of knowledge about the diversity of factors affecting students' test outcomes ("as far as how well we know our students (...) this is revealing a very great lack of knowledge", P2). They noted the impact of family and socioeconomic characteristics on these outcomes ("the sociocultural and socioeconomic part showing how students' living and material conditions differ, that's where you can see how these conditions affect students' learning and academic performance", P2). When describing the potential reasons leading to these outcomes, participants pointed out a number of factors including the services available to students ("with internet and TV, you have greater opportunities for interaction", P11); the time spent on extracurricular activities such as work ("if their work is more in line with their professional aspirations, we can be more efficient and attain better results than if their work is far removed from what they have to do in their university studies", P11); the inequality with which students pursue their education ("students who belong to high strata achieved better results, while those belonging to low strata face difficulties", P4); and the limitations in access to resources ("in private universities, students receive a latest-generation mobile phone with all the apps they need, which keeps them from facing connectivity problems. We have to look at where to find the money to try and sort out these material issues that have an impact on learning", P2).

Teachers should be trained to analyze quantitative data and use specific software, especially if the higher education institutions they work for ask them for numerical data to support documents presented in program quality accreditation processes ("these reports, and the conditions under which [institutional] bodies request them, are more in line with this [data-driven] mindset", P1). This approach would provide crucial insights into the positive correlations between students' scores in CR and the other competencies assessed. Likewise, it would help explain the differences between expected and actual outcomes, as well foster data analysis-based decision-making. "It is not common to find such analyses, despite how important they are for decision-making" said one participant (P11). Indeed, they are essential for devising program improvement plans, defending decisions, identifying weaknesses, designing strategies and engaging experts ("I'm particularly struck by the use of mechanisms such as statistical analysis software to establish these relationships and links with information that we don't normally take into account", P1).

Teachers' critical reading skills may have an impact on student outcomes, which "leads us to think about how well we handle reading in our teaching practices, and not just any reading, but critical reading" (P11). Accordingly, teachers should be given the means to understand the aims and features of the Saber Pro test, as well as the requirements and conditions for taking it, "even if we teachers need to undergo training in critical reading in order to adequately guide our students" (P7).

Teachers must engage in quantitative research practices ("we're from the field of humanities, so it's very special for us to see this type of statistical analysis of the tests because it's objective, which is not something we consider very much as people with a more human-centered way of thinking", P7). Given how the institution handles standardized test results and the general lack of knowledge of the relationship between quantitative analysis and the humanities, the data given to the participants was "another vantage point from which to look at the tests, because what we had been doing was examining percentages and percentiles to review performance and think about how to improve our classes" (P6). Setting up interdisciplinary research projects "raises the likelihood that other colleagues will join in, allowing us to take advantage of all this information that we don't know exists but could prove extremely useful in decision-making" (P11). It would pave the way for universities to further analyze data from the Saber tests; to characterize student profiles ("it's a fundamental aspect upon which the university should develop research

processes so that we can get a more accurate and detailed picture of our students' living conditions", P2); to explore the impact of emotional factors and classroom activities on the outcomes achieved ("we stick to the academic and cognitive side of teaching and neglect the emotional part", P1); to examine the student–teacher relationship and its impact on student outcomes ("I still maintain that people are who they are regardless of their conditions. I believe that there are pedagogical encounters that inspire and challenge students to transcend their careers", P12); and to test the relationships between Saber 11 and Saber Pro test scores ("a comparative analysis of the results of the Saber 11 and Saber Pro tests would be vital, as it would show us how the students have improved in the various competencies after going through the program", P7).

### *6.5. Perceptions of the Academic and Administrative Managers of Teacher Training on How the Information Presented Might Contribute to Bridging Higher Education and Other Educational Levels, and Their Initial Ideas for Improvement in This Regard (RQ3)*

Participants pointed out that communication between educational levels was poor ("there is a gap between secondary education and higher education, which makes students leave high school and take a very complex leap to university; these are two completely different ways of teaching and learning", P2). This study makes it possible to work jointly with other levels ("by teaming up with elementary and secondary education we can offer up our support and knowledge of the test, because English is tested in Saber Pro and English is tested in Saber 11", P4), or to plan, execute and evaluate strategies built around the development of competencies assessed on the Saber tests at various levels ("this should be discussed with the teachers in charge of elementary and secondary education. There is a path there, an action, so that we don't do it endogenously but rather by engaging with the other educational levels", P3). It also makes it possible to collect and systematize data from the test, as well as about students' life circumstances and the use of training scenarios ("there is a high percentage of students who are working (...) What can be done there in relation to graduates or with culture and well-being? To the extent that their work is more in line with their professional aspirations (...) we can be more efficient and obtain better results than if their work is far removed from their schoolwork, from the academic work they have to do at university", P11). In order to improve the practical training that university students receive in school at other educational levels, "one possible way to bridge the gap is through internships, setting up projects along those lines" (P9). The training that university teachers can give to teachers at other levels, discussing these subjects, can lead to actions and transformation ("once we as university teachers feel that we are competent and well trained, we can support that close link that should exist between universities and elementary and secondary schools, and also provide training support; then we can establish that link", P4). Schools should consider the academic community and the characteristics of higher education ("it seems to me that the university should establish a relationship with schools in such a way that they can better prepare final-year students, giving them a better understanding of what university is, what it's like and what academic processes take place at this level of education", P2).

### **7. Discussion**

Using an explanatory sequential mixed methods design, this study explores the perceptions of academic and administrative managers responsible for teacher training at a public university, as well as their initial ideas for improving such training after learning about the performance of student teachers in the 2019 Saber Pro test, the differences in their test scores, and the relationships and statistical correlations between these outcomes and the students' personal, family, socioeconomic and academic characteristics.

As mentioned above, learning analytics can help to reduce educational achievement gaps [19,54]. Thus, following Patil and Gupta [13], who claim that information about students' life circumstances can shed light on and help improve learning and learning environments, we set out to analyze the 2019 results of the Saber Pro standardized test taken by all bachelor's degree students in Colombia in relation to the characteristics of the students who took it. Taking into account that education-focused research is now supported by software that optimizes the collection of valuable information from massive datasets [21], that universities' digital culture involves the adoption and creation of models of the information and communication technologies they possess [18], and that understanding each item assessed on a standardized test and what it measures is as important to the practice of teaching as defending the idea that quality in education goes far beyond quantitative results [14], we began by statistically describing the characteristics of the 298 bachelor's degree students from a public university in Colombia who sat the Saber Pro test in 2019.

*What are the differences and relationships between student teachers' scores and performance on the 2019 Saber Pro test and their personal, sociodemographic, socioeconomic and academic characteristics? (RQ1)*. We identified statistically significant differences in the mean scores in the assessed competencies by grouping students according to their personal, family, socioeconomic and academic characteristics (RQ1). Our findings coincide with the claim that individual and family variables are linked to student performance in Latin American contexts [2]. Specifically, we identified significant mean score differences when students were grouped by gender, the father's or mother's level of education, and the mother's type of employment, besides the mode of instruction. Our interviewees' perceptions of the results emphasized the general lack of knowledge about what students who sit the Saber Pro test are going through in their lives, about the diversity of factors that may affect their scores and performance, and about the impact that some family and socioeconomic characteristics have on their outcomes. Identifying these factors is important for research, policy making and teaching, especially when the focus is on boosting quality [7]. It is also vital when development hinges on improving the quality of education, which is especially true for Latin America, a part of the world with large student achievement gaps that are a reflection of substantial income inequalities [2]. These claims are borne out by the statistical differences we uncovered when grouping students by stratum, by cost of enrollment, and by the services available to them (e.g., internet, computer, washing machine and TV). Our findings also support the idea that students' sociocultural, socioeconomic and living conditions affect their learning and academic performance.

In addition to finding a significant difference in mean scores in English when grouping students by stratum, we also identified a statistically significant relationship between stratum and student performance in QR. This coincides with the claim that there are correlations between a district's level of development and student results [3]; these comments are relevant because, for instance, seven of the nine degree programs we analyzed do not have foreign language on their study plans. It also bears out the findings of Vivas Pacheco et al. [4], who discovered a close link between the quality of Colombian students' local environment and the educational outcomes they are able to achieve. In particular, these authors claimed that a student's neighborhood has the greatest impact on educational performance. Our study found that students belonging to high socioeconomic strata achieved better outcomes, while those belonging to low socioeconomic strata seemed to struggle more. Giménez and Castro Aristizábal [5] emphasize the importance of seeking a level of infrastructure that covers basic housing needs such as electricity and telecommunications, arguing that the quality of these services is strongly related to student performance in Costa Rica. Similarly, Jiménez et al. [6] found that information and communication technologies (ICTs) have a positive effect on economic growth, innovation and high educational quality in Mexico. Studies show that children living in extreme poverty, deprived of basic needs such as food, clothing or shelter, or lacking utilities such as electricity or internet access, are more likely to perform poorly in school and drop out [16,17]. In this vein, we found a statistically significant relationship between having certain services (e.g., internet, an oven, a TV, a video game console or a washing machine) and students' performance in QR, E, WC, Ed and T. One of our participants stated that access to the internet or television gives rise to opportunities for interaction. In line with this, Gimenez et al., [2] shed light on the interaction between academic performance and social development, a scarcely explored

relationship due to a lack of data availability. These authors conclude that individual and family variables are more closely linked to academic performance than school variables, and that social development has a positive effect on scores. Our data point to a statistically significant relationship between student performance and the student's degree program, mode of instruction and semester in progress (school variables), as well as between student performance and the student's gender and the mother's and father's level of education (personal and family variables). Altogether, we found more statistically significant relationships between school variables and students' performance in the assessed competencies, thus contradicting Coleman [1].

Finally, we identified correlations between students' scores in the assessed competencies. The positive correlation between the CR scores and the scores in the other assessed competences caught our interviewees' attention, as success on the test appears to depend largely on knowing how to read critically.

*What are the views of academic and administrative managers of teacher training at a public university in Colombia regarding the quantitative results? (RQ2).* It is vital that we explore the relationship between the Saber Pro test and the indicators used to track the quality of university degree programs. Indeed, as stated by Rosero and Montenegro [63], educational quality implies developing an organizational culture oriented towards assessment and continuous improvement and innovation, which in turn implies deploying strategies that promote teaching, research and social outreach. This study supports the idea that testing in higher education can focus on assessment for educational purposes, so rethinking assessment methods could lead to changes in curricula [27]. Proof of this lies in the proposals made by our interviewees after being apprised of our data analysis. They underlined the importance of harnessing data analysis to make and defend decisions, map out plans to improve academic programs, identify weaknesses and design improvement strategies. This is directly related to the role that teachers acquire as catalysts for learning success, as they will be capable of analyzing data to better comprehend their own teaching actions, the type of students in their classrooms, and the learning outcomes their students achieve [19,20].

Siemens [55] proposed a learning analytics model (LAM) cycle that includes action. Actions resulting from this analysis can be systemic improvements in planning or teaching. This study contributed to shaping an initial decision-making proposal based on analyzed data. Decision-making also hinges on the need to devise strategies to improve student retention. Education systems have made limited use of available data to improve teaching, learning and student success, although there is a special interest in analysis as a solution to challenges such as student retention and support [46,55]. Educational institutions can restructure learning design processes based on analyses, meaning that teachers can incorporate feedback from these analyses into the future design of learning content and also customize content by including their personal understanding of a topic and previous experience [55]. In short, they can use analysis-derived information to make informed pedagogical decisions [58] and significantly improve student learning [27].

*In what ways do these academic and administrative managers believe those results can help bridge the gap between higher education and other educational levels? (RQ3).* First of all, the fact that a significant difference was found when grouping the results of the students according to the modality of education they receive (face-to-face or distance education) feeds the dialogue about how the institutional conditions and teacher preparation may be related to student learning outcomes in addition to the relationship reported by Coleman [1] between student socioeconomic conditions and outcomes. Research has shown that teachers, schools and countries that are effective in terms of quality tend to be effective in terms of equity as well [26]. Teaching should be adapted to suit students' interests and, above all, to account for their social and cultural environment and individual differences [64]. In this regard, curricular connections are meant to act as a two-way link between universities and their surrounding context, as well as between theory and practice. A curriculum whose design is based on competencies, active pedagogies and flexibility allows students to choose according to their interests and aspirations. It also lays the foundation for new interinstitutional alliances that promote collaboration across any field, subject area, time of year, semester or credit system [58,59,61]. Our interviewees suggested the possibility of working together with other educational levels to develop the competencies assessed on the various Saber tests. This coincides with the challenge of optimizing learning, satisfying political interests and exploiting available data to descriptively, predictively or prescriptively analyze solutions. [19,54]. This proposal is in line with the claim that data analysis can be used to bridge teaching pathways, with joint plans and programs mapped out to lay down common objectives in different areas in order to make progress in complex contexts [22–25]. This bridging process has various administrative implications, including overseeing labor practices, socio-occupational guidance, labor intermediation and higher education funding [58,61]. In this regard, the interviewees suggested having teachers in training face different work scenarios in their practical activities.

The challenge is to optimize learning, satisfy political interests and exploit available data to descriptively, predictively or prescriptively analyze solutions [19,54]. In line with this, our interviewees suggested fortifying the bridge between higher education and other educational levels by teaming up to develop the competencies assessed on the various Saber tests. This allows teachers to incorporate the feedback they receive in the future design of the learning content and also personalize the content, including their personal understanding of a topic and previous experience [55]. Furthermore, they can use information derived from the analysis to make informed decisions [56], such as systematizing Saber test data and student details gathered during their first enrollment; having university students carry out their practical training in nursery, elementary and secondary schools, with an eye to strengthening students' competencies at those levels; encouraging university professors to take on a leadership role in training teachers from other educational levels; designing opportunities for reflection and action where nursery, elementary and secondary school teachers come together; or raising awareness among secondary school students about the higher education experience.

The use of tools driven by learning analytics enables educational institutions to gain an insightful understanding of their processes and governance. Such tools can also be used in tandem with data and information systems for the educational system and quality assurance [65]. In this data-driven landscape, teachers will serve as catalysts for learning success, as they will be capable of analyzing data to better comprehend their own teaching actions, the type of students they have in their classrooms, and the learning outcomes their students achieve [19,20]. According to our interviewees, quantitative data analysis is not a frequent practice among prospective and practicing teachers in the Colombian educational system, even though higher education institutions request numerical data from teachers to support the documents they present as part of their programs' quality accreditation processes. For this reason, the interviewees expressed the need to train teachers to analyze quantitative data and use specific software. This is in line with current literature [57], which confirms that if students' socioeconomic level and teachers' subject knowledge are to be considered predictors of academic performance, this type of training has good reason to be enhanced.

Although the 1966 Coleman Report argued that differences in students' scores could be explained by sociodemographic factors and that school resources and teachers' education do not have a notable effect on student performance, our interviewees raised questions about the influence of teachers' critical reading skills on student outcomes, and before this the OECD affirmed that in most PISA-participating countries and economies the relationship between teachers' participation in professional development activities and students' performance in reading is weak [8]. These statements support the need for debate about the impact that educational policies and the school system have on student performance, in addition to environmental and sociodemographic effects [1]. Although this is not an objective of our study, it is pertinent to highlight here that the OECD in its PISA 2018 results report affirms that teachers are perhaps the most important of the school resources that are

needed to boost students' learning, because improving the quality and equity of education is more likely if teachers are adequately qualified and able to support their students' needs. However, in Colombia, Georgia, Mexico and the United Arab Emirates, less than half of teachers in schools attended by 15-year-olds were fully certified [8].

The above matters were brought up by our interviewees after we showed them the results of our statistical analysis of data from students who took the 2019 Saber Pro test. In addition to these topics, they were also asked questions about the possibility of developing research projects to further analyze data from tests taken at different educational levels; to characterize student profiles; to explore the impact of emotional factors and classroom activities on the outcomes achieved; to examine the student–teacher relationship and its impact on students' outcomes; and to test the relationships between Saber 11 and Saber Pro test scores.

### **8. Study Contributions, Conclusions and Recommendations**

This study makes a significant contribution to the otherwise limited exploration of the relationship between students' academic performance and their individual, family and school variables, owning to problems of data availability [2]. It also helps to fill the research gap on the Saber Pro test and on analyzing test outcomes to drive improvement actions [36,37,41,42,44,66–68]. In this regard, we statistically analyze data to provide insights into how to improve education quality [36,37,42–44].

Our study gave participants the chance to make proposals on topics that could be addressed in future research, such as analyzing results from previous years; reviewing the impact of classroom activities on test results; investigating the level of student willingness to take the test or the reasons for academic success in those who do not have the best socioeconomic or family conditions; reviewing the impact of the student–teacher relationship and teaching practices on test results; exploring institutional recognition of student characteristics; working with other areas of knowledge and designing pedagogical, educational and didactic strategies alongside other bachelor's degree programs; comparatively analyzing the results of the Saber Pro and Saber 11 tests (added value); analyzing results to make decisions; looking into the characteristics and backgrounds of the students starting the degree programs; and reviewing the behavior of correlations in previous years.

Our study set out to perform a statistical analysis of the data from the 2019 Saber Pro standardized test in relation to the students' personal, family, socioeconomic and academic characteristics, in order to devise strategies and actions for improving the academic programs and the institution as a whole. This could have a positive effect on the quality of education. Indeed, by means of a descriptive statistical analysis and an exploration of the relationships between the data, our research signifies a first step in helping the academic and scientific community of higher education to consider the results of standardized tests—relating students' test results and performance with their characteristics—in order to improve teaching and learning processes at all levels of education in the country, taking into account the differences between students and their individual life circumstances, identifying context-based needs and closing the current gap between higher education and other educational levels, as well as paving the way for future research. We hope that our findings will lay the foundation for analyzing standardized test data for the purpose of improving teacher training programs and higher education institutions, and that they will provide insightful input for setting feasible goals with respect to the promotion of quality and equity.

In order to improve teacher training based on the data published by the ICFES, which discloses students' Saber Pro test results and their personal, family, socioeconomic and academic characteristics, we make the following recommendations:


### **9. Study Limitations**

Studies related to the Saber Pro test in recent years have underlined the scarcity of research addressing the test results [35–37]. In this study, we examined the results of the Saber Pro test taken in 2019 by 258 bachelor's degree students from a specific public university, and there is a wealth of data available on different programs and universities and from different years, so the amount of data to be analyzed is still immense. Likewise, students' varying characteristics make each group and context different. The teacher training processes are different in every program or institution. The sample from the qualitative phase of the study is rather small compared to the number of teacher education programs in Colombia. So, the data cannot be generalized to draw conclusions about the entire population of bachelor's degree student teachers or all public higher education institutions. The interview data clearly depend on the conversations between the participants and the first author; therefore, a second interview to those summoned to the presentation of results would have been enriching and would have enriched the perceptions and the actions proposed for the strengthening of the programs that train teachers. However, time was an important limitation within the new dynamics of online communication generated by the pandemic.

**Author Contributions:** Conceptualization, P.S.-C. and D.V.; methodology, P.S.-C. and D.V.; validation, P.S.-C., S.F. and D.V.; formal analysis, P.S.-C.; investigation, P.S.-C.; data curation, P.S.-C. and D.V.; writing—original draft preparation, P.S.-C.; writing—review and editing, D.V. and S.F.; supervision, D.V. and S.F. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** See Appendix A.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to identity protection/confidentiality reasons.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **Appendix A**



**Figure A1.** Participant Informed Consent Template.

### **Table A1.** Variables used in the study.



### **Table A1.** *Cont.*


**Table A1.** *Cont.*

### **References**


### *Article* **Perceptions of Digital Device Use and Accompanying Digital Interruptions in Blended Learning**

**Juliana Pattermann 1,\*, Maria Pammer 2, Stephan Schlögl <sup>3</sup> and Laura Gstrein <sup>1</sup>**


**Abstract:** Using various digital devices, and being faced with digital interruptions is a given for students not only in traditional university classes but also in blended learning courses. Hence, this study (N = 201) at an Austrian university of applied sciences investigated students' perceptions of digital device use and the digital interruptions that they face during webinars and on-campus sessions. Results show that students primarily use the same types of digital devices during webinars and on-campus sessions, i.e., computers for course-related (CR) activities, and smartphones for non-course-related (NCR) activities. Results further indicate that while the majority of students are aware of the interruptive impact that NCR activities have on their learning, the effect on others seems to be a blind spot. The reasons for NCR activities are manifold. Moreover, results suggest that students have difficulties in assessing the actual time spent on NCR activities during webinars.

**Keywords:** digital interruptions; online learning; mobile learning; blended learning

**Citation:** Pattermann, J.; Pammer, M.; Schlögl, S.; Gstrein, L. Perceptions of Digital Device Use and Accompanying Digital Interruptions in Blended Learning. *Educ. Sci.* **2022**, *12*, 215. https://doi.org/10.3390/ educsci12030215

Academic Editor: Maria Limniou

Received: 31 January 2022 Accepted: 11 March 2022 Published: 17 March 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

### **1. Introduction**

Before the COVID-19 pandemic hit and changed learning settings all around the world, laptops, tablets, and mobile phones had already become students' permanent companions in university classrooms. As technology becomes more compact and portable, highereducation institutions need to pay more attention its effects on learning processes [1–4].

While strategies on how to deal with the use of digital devices in traditional classrooms (e.g., laptop bans) are frequently discussed [2], university programs are conducted in a blended learning model, and their students are dependent on these devices. Broadbent describes blended learning in the context of her study as "the adoption of educational web-based technology (e.g., a learning management system) for online learning, which is used in combination with face-to-face located instruction from teaching practitioners" [5] (p. 25). In other words, blended learning can be considered to be the combination of synchronous online sessions (webinars), synchronous face-to-face (on-campus) sessions, and asynchronous independent learning. Thus, owning or at least having access to a digital device is one of the major requirements for students to participate and complete courses in a blended learning program.

Despite the several advantages of digital devices for learning in higher education, such as having quick access to online information, taking pictures of important content during class [6], taking notes and organizing content, or downloading necessary resources [7], the downsides of digital media usage in class cannot be ignored: prior research disclosed that students who use digital devices in class show worse overall performance compared to students who do not use the respective technologies [2,8–10]. However, although the data about the exact user behavior vary, there is much proof for the distracting effect of media use while learning [9,11]. Considering that, in a webinar, instructors do not have all their students within their full range of vision (and/or hearing) at all times, they can never know whether students are actually following the content, are lost, are distracted by their phones, or are simply watching another video.

As a consequence, we need to accept that digital devices play a relevant role in class (online and on-campus), and that students might use them not only to participate in webinars and take notes, but also to interrupt themselves or disrupt others. Aiming at a better understanding of students' awareness levels and behavior in a pre-COVID-19 period, the presented study in this paper thus investigates the following research question: how do students perceive the use of digital devices and accompanying digital interruptions faced during webinars and on-campus sessions?

Our report of this investigation is structured as follows: Section 2 introduces the main topics and concepts we used to frame this study. Section 3 refers to the research design, and outlines the sampling method, the instruments, and data collection and analysis. Section 4 presents the results and analysis, followed by the discussion of the results in relation to theory and previous studies in Section 5. Lastly, Section 6 draws a conclusion, acknowledges the limitations of this study, and introduces possibilities for future research.

### **2. Digital Devices and Digital Interruptions**

Digital devices pose many advantages for learning, yet they are also a source of interruption. While these interruptions can occur for several reasons [12], McFarlane emphasized in 1997 that computers are a more prevalent source of interruption than the person themselves, another person, or animate or inanimate objects [13]. Since then, the number of applications that run on digital devices has substantially grown (e.g., instant messaging, social media, emails, shopping, and reading the news online), increasing the potential for digital interruptions even further [12,14,15]. At the same time, however, this diversity in applications allows for individual study preferences and consequently supports different learning strategies. According to Biggs' students' approaches to learning (SAL) theory [16], this variety in learning approaches is central to individual learning progress. Others even refer to it as the reason "why students are more or less successful in their learning" [17] (p. 3). Hence, in order to promote and facilitate learning, the faculty needs to recognize SAL and offer support for individuality [18]. To what extent the lecture format (i.e., webinar vs. on-campus) affects SAL and the respective use of digital applications, however, is less researched.

### *2.1. Use of Digital Devices in University Classrooms*

The incorporation of digital devices in the university classroom has several benefits. For example, it allows forstudents to deepen their knowledge or question presented facts with the use of additional online content, and to support their learning with taking pictures of important matters [6]. Thus, digital media, defined as "always-on, socially interactive, technologically mediated communication artefacts" [19] (p. 86), offer a variety of ways to improve student learning.

Initiatives such as Anywhere Anytime Learning [20] or Bring Your Own Device [21,22] have promoted the complementary nature of digital devices to traditional teaching and learning tools. Digital devices that might be used for course-related (CR) work include laptops, tablets, desktop computers, hybrid devices, and smartphones [23,24]. Educational technologies such as learning management systems, game-based learning platforms, or polling tools can enhance the learning experience, but students need access to suitable technologies. Students increasingly use e-books instead of hard copies due to the given cost advantages [25], which requires them to use digital devices in class.

In cooperation with the Educause Center for Analysis and Research (ECAR), Brooks and Pomerantz [23] conducted an international survey with 43,559 participants from 124 institutions, highlighting that 95% of all participating students owned at least two digital devices [23], such as a laptop, tablet, desktop computer, or smartphone. In another ECAR study, Galanek, Gierdowski, and Brooks [24] looked at the usage frequency of digital devices and their perceived impact on academic success. They found that 98% of students did not only use their laptops, but almost as many students (94%) considered the device to be very or extremely important for their success [24]. Students' mindsets have also changed, so that mobile-phone or laptop usage in class is no longer perceived as signaling a lack of respect or attention [21,22]. May and Elder [26] even found that 40% of the study participants thought it was acceptable to send text messages during class. Hence, it does not come as a surprise that students stay in contact with their peers via social media and texting applications during phases of self-study and class [27].

### 2.1.1. Laptop/Tablet Use

Laptops bring many functionalities and advantages to a university classroom. Students bring them to class to "connect with the lecture" [26] (p. 7), take notes [8], and generally engage in class activities [28]. They provide note-taking applications (e.g., Microsoft OneNote, Evernote, Apple Notes), note storage, and constant access to class materials [7]. Houle, Reed, Vaughan, and Clayton [29] observed that students are aware of "the usefulness of the laptop as enhancing their own participation in the course" [29] (p. 89).

However, students use laptops not only for course-related activities but also for off-task or non-course-related (NCR) activities. With a multimethod approach, Ragan, Jennings, Massey, and Doolittle [28] examined students' laptop usage in class. In an online survey, 59% of students brought their laptops to class, mostly to take notes and engage in off-task activities such as social media or surfing the web [28]. While laptops provide many advantages for class time, they also present a major source of distraction [26]. Kay, Benzimra, and Li [30] asked their students to rate the frequency in which they engaged in distracting activities during class. About two-thirds of the students stated that they most frequently engage in emailing and web surfing, closely followed by social media activities, instant messaging (IM), and playing games [30].

This distracting nature of laptops in class not only leads to increased multitasking [2], but also to potentially lower academic performance [2,3,7,31,32]. Aguilar-Roca, Williams, and O'Dowd [8] found a performance difference between students who took notes on their laptops and those who used paper, showing that the latter scored significantly better in tests and received significantly more A grades. Sana, Weston, and Cepeda [33] used an experimental setting to analyze the consequences of laptop use for NCR tasks in class and found that students who were asked to engage in NCR activities while being in a lecture scored 11% lower than their peers did [33]. Even though Carter, Greenberg, and Walker [7] found evidence that unrestricted use of laptops and tablets had a negative impact on academic performance, Elliot-Dorans [1] countered this by showing that forbidding students to use their devices did not help them to improve their performance.

### 2.1.2. Mobile Phone Use

Mobile phones are not only used for making calls, but also for texting, sending emails, participating in video conferences, engaging in social media channels, taking pictures and sharing videos, and using other software-driven applications, as laptops are [34]. A study carried out by Lepp et al. [34] shows that, on a daily basis, students used 278.67 min on their mobile phones and sent 76.68 messages on average. While this figure gives no indication as to the number of texts sent during class, Kay, Benzimra, and Li [30] reported that 80% of students stated that they were "on-task" often or regularly while using their digital devices during a lecture. Texting is now so mundane that students "simply text irrespective of circumstances and rules" [35] (p. 26), even during class.

Even though students also use their mobile phones for activities related to the content of the class [6], the negative effects on learning prevail [9–11,36].

### *2.2. Students' Perceptions and Awareness of the Use of Digital Devices*

While university faculties understand the implications of technology use in the classroom, students' awareness of their own behavior and their perceptions about respective effects vary [26,33,37].

May and Elder [26] observed that students exhibit "poor awareness of how media multitasking affects their learning" [26] (p. 10). In an exploratory study, Clayson and Haley [35] found that 68% of their respondents were convinced of their ability to actively participate in class and text at the same time, even though this behavior was negatively correlated with their grades. Generally, students tend to underestimate their time used for NCR activities in class and its impact. For example, Kraushaar and Novak [11] showed that students' estimation regarding their time spent on instant messaging in class was too low by 40%. In a study by Kirschner and Karpinski [38], 73.8% of students did not perceive any impact of Facebook on their learning and some even saw a positive effect. These findings contradict McCoy's [37] outcomes proposing that the majority of students is well aware of the negative impact of digital devices on their attention in class. Given their opposing character, these previous findings suggest that first, students have an inaccurate impression of their own interrupting behavior, and second, that they are not capable of fully estimating the consequences of their behavior.

### *2.3. Concept of (Digital) Interruptions*

An interruption can be described as a new, additional action that interferes with an ongoing action [39]. Despite this new action, there is the intention of returning to the first activity later [40–42]. Contrary to distractions, which can only be triggered by external stimuli [43], the sources of interruptions can be both, external and internal [40,44].

External aspects causing interruptions, such as a ringing phone, are usually unintended, out of the respective person's control, and a compulsion to shift one's attention to the new stimulus [40,44,45]. However, internal interruptions describe a process of self-interruption due to physical needs, such as the urge to eat something, mental state, or thoughts, as, for example, the desire to check social media [40]. When an external interruption occurs due to an external trigger, one must reorganize and keep in mind the current goal to resume it at a later point [45]. Internal interruptions, on the other hand, lead to a suspension of the current goal because of a conscious decision to stop the primary task [45]. As they require active decision making prior to the interruption itself, they are more disruptive than external interruptions [44]. While external interruptions cannot be controlled, self-initiated interruptions may either be controllable or uncontrollable [40].

Reasons for internal interruptions are difficult to observe [40], pose some challenges regarding classification, and few studies have focused on them. Boredom, frustration, and low-workload moments [44–46] are possible reasons for self-interruption. Students often justify this self-interrupting behavior on grounds of its self-inflicting nature and argue that "they should be allowed to do whatever they wanted as long as it did not negatively affect other people" [47] (p. 107). In addition to the origin of and reasons for interruptions, the point in time when the interruption occurs is decisive [44]. To this end, a variety of studies showed that interruptions during low-workload moments are less disruptive than during high-workload moments [44,48].

Overall, several studies indicate that interruptions negatively impact the performance of the main activity in four major ways [44,49–52]. First, finishing the main task is more time-consuming [44,53]. Second, the longer an interruption is, the more challenging it is to come back to the original task [44]. Third, even if the main task is continued, the interrupted action is more prone to error [51,53]. Fourth, the additional time required for the main task can cause stress and anxiety [53,54].

Focusing on digital interruptions, laptops and other digital devices are used in either productive or distractive ways [2,11]. Kay and Lauricella [4] concluded that social networking and IM are the foremost interrupting activities during class. Besides social media, IM, and online surfing, Kay, Benzimra, and Li [30] observed another significant type of

interruption: 41% of students regularly used mobile devices during class for emailing. In 2015, email checking seemed to be one of the most common activities [55]. Other types of interruptions include shopping, checking sport scores [56], reading the news, watching videos, and chatting [15].

### **3. Context and Methods**

We conducted this study with two cohorts of first-semester business administration bachelor's students at a university of applied sciences in Austria (pre-COVID-19). The respective blended learning program was based on synchronous online sessions (webinars in the evenings), synchronous on-campus sessions, and asynchronous independent learning. Each course consisted of six webinars (1 webinar = 120 min), a full day on campus (360 min), and a significant amount of guided self-study.

Studying the use patterns of digital devices and digital interruptions in a blended learning setting, we adopted a similar categorization of types of interruptions as that of Ravizza et al. [15]. That is, we looked for the previously highlighted interrupting activities and investigated their frequency and duration during class. We focused on time spent on laptops, since students of this blended learning program are explicitly encouraged to use private computers instead of mobile phones to participate in class.

In order to answer the question of how students perceive the use of digital devices and accompanying digital interruptions faced during webinars and on-campus sessions, we developed the following set of assumptions, all of which were deduced from the previous work discussed above:

**Assumption 1a.** *Students primarily use the same type of device (e.g., their laptop or tablet) for CR activities during webinars and on-campus sessions (cf. Section 2.1.1)*.

**Assumption 1b.** *Students primarily use the same type of device (e.g., their smartphone) for NCR activities during webinars and on-campus sessions (cf. Section 2.1.2)*.

**Assumption 2a.** *Students perceive the use of computers for NCR activities during webinars as interruptive to their own learning (cf. Section 2.3)*.

**Assumption 2b.** *Students perceive the use of computers for NCR activities during on-campus sessions as interruptive to their own learning (cf. Section 2.3)*.

**Assumption 3a.** *Students perceive the use of computers for NCR activities by other students during webinars as interruptive (cf. Section 2.3)*.

**Assumption 3b.** *Students perceive the use of computers for NCR activities by other students during on-campus sessions as interruptive (cf. Section 2.3)*.

**Assumption 4.** *Students use their computers for NCR activities during webinars more than during on-campus sessions (cf. Section 2.2)*.

To evaluate these assumptions, students first completed a self-assessment questionnaire after the third webinar of a course (Survey 1) and a second questionnaire at the end of the on-campus day (Survey 2), which marked the end of the course. Questionnaires included mainly quantitative data elements, but were enriched by some open-ended questions, for which we used thematic content analysis to expand on the quantitative results [57].

### *3.1. Sample*

In total, we asked 211 first-semester students to participate in this study, of whom 176 completed Survey 1, and 144 Survey 2 (a total of 201 distinct students completed either Survey 1 or Survey 2). Table 1 provides an overview of the gender distribution of the sample.


**Table 1.** Gender distribution of sample (n = 201 distinct students).

Descriptive analysis of the data showed that participants were between 19 and 55 years old (M = 27.6 years, SD = 6.775), and the majority (i.e., 83.6%) had never studied at the tertiary level before. Participants also reported an average of 8 years of work experience (SD = 6.91 years) and that they currently work an average of 33.88 h per week next to their studies.

### *3.2. Instruments*

In Survey 1, we asked students about the types of digital devices that they use during webinars. Next, we focused on their self-estimated use of digital devices for course-related (CR) and non-course-related (NCR) activities during webinars based on a survey instrument by Ravizza, Uitvlugt and Fenn [15]. To this end, we inquired on their estimated usage of digital devices for the following interrupting activities: checking social media, sending SMS, messaging on WhatsApp, shopping online, reading the news or checking sport scores, watching videos, playing games, and other activities. Next, several questions were asked to establish an understanding of the students' learning environment (e.g., location and in the company of someone or alone). Types of and reasons for interruptions were investigated by a mixture of closed and open-ended questions. Furthermore, we inquired on the students' perception of how their digital device usage for NCR activities affected their own and their peers' learning during webinars and on-campus sessions [15]. Survey 2 asked the same questions but slightly reworded, so as to fit to a physical classroom setting. A pilot test with five participants from various backgrounds (i.e., academic faculty, business professionals, and students) was conducted for each questionnaire to identify ambiguous formulations, guarantee sufficient clarity and full understanding of all used terms, and test whether the surveys were suitable to investigate previously outlined assumptions. An excerpt of the final questionnaires is available in the Supplementary Materials (File S1).

### *3.3. Data Collection and Analysis*

We informed the students before the beginning of courses Accounting and Controlling I (Cohort, 2018) and Fundamentals of Law (Cohort, 2019) about the survey. These two courses were chosen due to their early position in the curriculum during the first semester, their similar structure, and comparable assessment mode, namely, a final written exam. Survey 1 was conducted after the third of six webinars, and Survey 2 took place at the end of the on-campus day. We used frequency analysis to explore students' engagement in NCR activities.

After reading through the answers to the open-ended questions, we started an inductive coding process. Our team focused on keywords, nominal phrases, and sentence parts, and only allocated one code per selected text unit. We developed a separate coding scheme with main and subcategories for webinars and on-campus sessions (Supplementary File S2). Exemplary answers were selected to represent each category. Memos were written to summarize the meaning of each code to avoid a lack of consistency in coding and to ensure interand intrarater reliability [58]. Next, two of the researchers coded the answers independently and compared their allocations. Lastly, we adapted the coding scheme as a consequence of our varying agreements on the basis of oral discussions and in accordance with the literature (Supplementary File S3).

The previously described characteristics of the two student year groups concerning age and gender represent the total student population's characteristics. The overall population's diversity in work and educational experience, and current working hours is reflected in the sample. Therefore, nonsampling bias could be excluded. A nonresponse bias was also excluded, as 83% (n = 176) of the two student cohorts responded to Survey 1 and 68% (n = 144) to Survey 2.

### **4. Analysis and Results**

Our goal was to investigate the participants' use of digital devices and to explore the digital interruptions they face during webinars and on-campus sessions. A better understanding thereof can help lecturers in blended learning settings to support their students in overcoming distracting behavior and reducing the time of non-course-related activities. We first established an understanding of the participants' learning environment and focused on the types of digital devices they use during webinars and on-campus sessions. Next, we examined the perceived effects that non-course-related activities that are carried out with digital devices have on learning. Third, we explored the use of digital devices, relevant digital interruptions, and the extent to which students engage in NCR activities during webinars and during on-campus days.

### *4.1. Learning Environment*

As webinars took place after typical working hours, 85.8% of the students indicated that they were always or very often alone in a room when they participated in the webinars, whereas 14.2% of the participants answered that they were never, rarely, or only sometimes alone. Of the participants, 94.9% indicated that they typically joined the webinars from home, and only 2.8% from the office.

During webinars, 59.43% of the students took notes on paper; 22.64% on their laptops; 13.21% using either a desktop computer, a tablet, a smartphone, or a hybrid device; and 4.72% did not take notes at all. During the on-campus day, 59.32% of them took notes on paper; 27.68% used their laptops; 7.34% either a tablet, a smartphone, or a hybrid device; and 5.65% refrained from taking notes.

### *4.2. Types and Use of Digital Devices*

In this particular blended learning program, webinars are typically hosted using a web conferencing platform. To this end, 85.8% of the participants indicated that they used their laptops to log in, followed by 6.8% using their desktop computers, 5.7% using a hybrid device, and 1.7% using a tablet or smartphone.

To compare the difference between the use of digital devices for CR and NCR activities during webinars and on-campus sessions, we asked students to select one of the following devices that they primarily use: desktop PC (only in Survey 1), laptop, tablet, smartphone, hybrid device, or other.

For course-related activities during webinars, 80.1% of the participants indicated that they used their laptops, followed by 7.4% using a desktop PC, 6.3% a smartphone, 5.7% a hybrid device, and 0.6% a tablet. During the on-campus day, students primarily used their laptops (83.3%) to engage in CR activities, followed by 6.9% using a hybrid device, 5.6% a smartphone, and 4.2% a tablet (see Table 2). These numbers show that the use of digital devices for CR activities during webinars is similar to its use during on-campus lectures, so that Assumption 1a is clearly supported (cf. Section 3).

Regarding non-course-related activities, 54.5% of the students stated that they used their smartphones during webinars and 34.1% that they used their laptops, while the remaining 11.4% opted for one of the other previously listed devices. During the oncampus day, 67.4% of the students resorted to their smartphones for NCR activities, 27.8% to their laptops, and 4.8% used one of the other devices (cf. Table 2). Although there seems to be a slight difference between the use of digital devices for NCR activities during webinars compared to their use during on-campus lectures, Assumption 1b (cf. Section 3) still seems to be supported.


**Table 2.** Primary use of digital devices for CR and NCR activities during webinars and on-campus sessions.

<sup>1</sup> n = 176; <sup>2</sup> n = 144.

### *4.3. Perceived Effects of NCR Activities on Learning*

When investigating the perceived impact of NCR activities on students' learning, 75.66% of them stressed that it somewhat or strongly disrupted their learning of course material during webinars, whereas only 60.42% indicated that it disrupted them during the on-campus day (see Table 3). These numbers support both Assumption 2a (students perceive the use of computers for NCR activities during webinars as interruptive to their own learning) and Assumption 2b (students perceive the use of computers for NCR activities during on-campus sessions as interruptive to their own learning).

**Table 3.** Student perception of impact of own computer use for NCR activities on own learning.


<sup>1</sup> n = 176; <sup>2</sup> n = 144.

We also investigated the perceived effect of other students' computer use on learning. To this end, nearly three quarters of students perceived that other students' use of computers for NCR activities during webinars as well as on-campus sessions made no difference to their learning (cf. Table 4). Consequently, neither Assumption 3a nor Assumption 3b (Section 3) are supported by the analytical results. Nonetheless, nearly one-quarter of student perceived a somewhat or strongly disrupting effect during webinars (23.87%) and during on-campus sessions (23.61%).


**Table 4.** Student perception of impact of computer use by other students for NCR activities on own learning.

<sup>1</sup> n = 176; <sup>2</sup> n = 144.

### *4.4. Relevant Digital Interruptions*

To assess the average time that students spend on NCR activities with their computer, we asked them to estimate the number of minutes dedicated to checking social media, reading or writing emails, texting, etc. (Table 5).

**Table 5.** Average time spent on computer on non-course-related activities.


It follows that, during an average webinar, students reported that most of their off-task time was spent on checking social media, reading or writing emails, texting, and other unspecified activities. During an on-campus day, they estimated that most of their time off-task during class was dedicated to texting, reading the news, reading or writing emails, checking social media, and other activities. However, the high standard deviations indicate that the behavior of students varies greatly.

In total, students reported to spend on average 15.06% (18.08 min) of an entire webinar on NCR activities, compared to 8.12% of an entire on-campus day (29.23 min). By running a one-tailed t-test, results indicated that students spent significantly more time on digital interruptions during webinars than they did during on-campus sessions (t(271) = 4.43, *p* = 0.000).

Surprisingly, when asked to estimate the total percentage of time they spend on NCR activities, they reported having dedicated on average 8.48% of the webinar time and 7.39% of the on-campus day to these interruptions. Here, no significant difference was found between webinars and on-campus sessions (t(255) = 0.88, *p* = 0.190). Students' estimates of NCR activities during on-campus lectures were rather stable, whereas their estimates of NCR activities during webinars greatly varied depending on whether they were asked to provide percentages or absolute minutes.

Thus, depending on which approach to self-assessment is considered (estimating the number of minutes or estimating a percentage of time), Assumption 4 (Section 3) may be supported or not.

### *4.5. Reasons for Interruptions*

The open questions of the surveys showed that students have various reasons for spending time on non-course-related activities. During on-campus sessions, students feel internally interrupted by thoughts and worries about their families and about organizational matters, such as plans for the evening or the weekend. Internal interruptions also stem from curiosity about possible incoming messages. Students mention concentration difficulties as a further reason for non-course related activities due to tiredness or the need for a break, for example, "On-campus sessions are very long and sometimes you're losing concentration and the first thing you do is to check your phone for messages". Physical needs, such as the urge to eat something, were additionally mentioned as sources of interruption. One participant indicated that "I want to get to know my peers better as this makes working together easier". This statement is exemplary for participants' strong need to interact with their peers due to the limited time on campus inherent to the blended learning approach, which represents another reason for interruptions.

External interruptions during on-campus sessions can mainly be summarized as family and work issues. The students' statements show the challenges of reconciling work, education, and family. For example, they mention stress at work while being on campus, urgent job-related tasks, and family emergencies interrupting their on-campus sessions. Nonetheless, in addition to individual impact factors, general factors within the classroom play a role as well. For example, noise in the classroom, distractions on other students' screens, the content of which students believe to have prior knowledge, and redundant questions of fellow students seem to lead to non-course-related activities. Furthermore, the teaching style in class is seen as a potential source of interruption: "If the course is boring, then I spend my time on other things".

During webinars, internal interruptions can also cause students to engage with noncourse-related matters. Students also indicated that reasons such as participation within the group chat on WhatsApp, concentration difficulties, preknowledge on the presented topic, the feeling of not being able to follow the lecturer, and technical difficulties led them to interrupt themselves during webinars. For example, one participant wrote, "The students often write together on WhatsApp during the webinars, but this is both for course-related and non-course-related stuff. This gives it more of a classroom vibe, both explaining the topics and small jokes on the topics". One statement shows that interrupting behavior is sometimes kept up despite better knowledge: "I'm not happy to say that, but I guess I'm just addicted to social media in some way".

Interruptions by family members play a significant role: "My Children want something or my wife wants something". This reasoning was to be expected, as the majority of the students participate in webinars from home (Section 4.1). In the webinar setting, pressure to interrupt course-related activities due to responsibilities at work is present: "One reason is that if, for example, an e-mail pops up, I want it done immediately. What is done is done." Teaching style seems to play an even greater role in webinars than that during on-campus sessions. Hence, the students list factors such as the lack of focus and structure, missing interaction, too much information on slides, and the lecturer's monotonous voice as reasons for non-course-related activities. Lastly yet importantly, when asked about any distracting behavior of their fellow students in the classroom, participants provided a diverse set of answers. One participant, for example, highlighted that "a lot of unnecessary questions from fellow peers ould have been sent via mail to the lecturer to not take up our already limited webinar time", suggesting that a great number of questions by peers during webinars and the consequent endeavor of lecturers to answer these questions are perceived negatively by other students.

Summarizing the answers to the open-ended questions, students must deal with internal and external interruptions in both webinars and on-campus sessions. What is unexpected, however, is that the reasons for digital interruptions are alike in both contexts.

### **5. Discussion**

Together, results presented above provide important insights into students' perceived use of digital devices and the accompanying digital interruptions they face during webinars and on-campus sessions in the pre-COVID-19 period. They primarily use the same types of digital devices during webinars and on-campus sessions. While they mainly use laptops for course-related activities, they take out their smartphones to engage in non-course-related activities, such as checking social media, texting, and reading or writing emails. Overall, one might ponder why students use one device for each activity, namely, laptops for courserelated activities, smartphones for non-course-related activities, and paper for taking notes; this might give them a sense of successful multitasking by simultaneously engaging in several activities [59].

When taking notes, students employ a similar approach regardless of whether they are at home in front of their laptops or sitting in a classroom on campus. Contrary to Kay and Lauricella [60], who found that students saw the function of note taking as the largest advantage of using laptops in class, almost 60% of our study participants reported taking notes on paper both during webinars and during the on-campus session. We can only guess reasons for this behavior, but an explanation might be that working professionals may appreciate the haptic aspect of writing on paper for a change.

Similar to results reported in Ravizza et al. [15], the participants of our study stated to have spent most of their time on checking social media, texting, reading or writing emails, and reading the news both in webinars and during on-campus sessions. Even though messaging services such as WhatsApp are available on laptops, Clayson and Haley [35] argued that texting on phones is now so normal that students might simply do as they please, not considering the context. Almost one-quarter of students indicated that they felt disrupted by others using their laptops in class (23.7%). A possible explanation for this might be that students stick to their phones as an act of social nicety to avoid distracting others.

We assumed that students would use their computers more during webinars than during the on-campus session for non-course-related activities. When asked to estimate the minutes spent on off-task activities, students' responses indicated that digital interruptions during webinars lasted significantly longer than during on-campus sessions. While the estimation of the total percentage of NCR activities and the percentage calculated from the accumulated minutes for the on-campus day are similar, the participants provided different estimations when asked about webinars. This mismatch was unexpected; one explanation might be that students lack awareness of their self-interrupting behavior when at home and online.

While the majority of the participants are aware of the interruptive impact that NCR activities conducted on their computers have on their learning, but 25.69% of them are convinced that NCR activities have no impact on their own learning when on campus, whereas only 15.34% see no impact during webinars. The effect of their computer use for NCR activities on others seems to be a blind spot, with 70.8% of students not perceiving its impact during the on-campus sessions on others at all.

Although prior studies show that their peers' activities on digital devices catch the students' attention [2,6,33], they do not seem to be fully aware of the related consequences.

The reasons that students mentioned for non-class-related activities are manifold. Some indicated that they felt like getting in contact with other students when online, and others mentioned boredom as a reason for their laptop use. On the other hand, being interested in the topic or having lively discussions keeps them from being interrupted [4]. Taken together, the reasons for non-course-related activities are various and call for further investigation.

### **6. Conclusions**

Developing competencies to critically reflect one's behavior and to apply corrective measures if necessary is an essential part of higher education. This applies to physical and digital contexts alike. Therefore, supporting our students in recognizing potential interruptions is the first important step towards a potential evolvement of self-regulating measures. To inspire these necessary changes in our students, which are increasingly important in a world of national and regional lockdowns, and a departure from traditional teaching methods, lecturers first need to be well aware of students' behavior and potential sources of interruptions.

Thus, this study set out to explore students' perceptions and use of digital devices and accompanying digital interruptions in webinars and on-campus sessions. Our results show that digital interruptions are an issue in both webinars and on-campus sessions. On the one hand, students claim to be somewhat aware of the interrupting potential of digital devices during class time. On the other hand, this awareness has very little impact on students' behavior. The use of digital devices in higher education offers several benefits that students are already well on their way to integrating into their lives. However, considering that social media, emails, and instant messaging easily steal the students' attention from what is happening during class, we as lecturers need to ask ourselves how we can avoid a future in which our teaching becomes the main interrupting element during class time.

These results are based on a study that had been conducted pre-COVID-19. We do not know whether students' behavior has since changed. Another limitation may be seen in the rather small sample of students completing both surveys. All our results are based on students' self-assessment and self-estimations. Pairing reported perceptions with real data from tracking software or an objective performance indicator would allow for a better and more indepth understanding of students' behavior. Furthermore, one issue that was not addressed by this study regards the students' preference for using their smartphones for non-course-related activities. This reasoning and further aspects related to behavioral causes should be explored in additional focus groups and in-depth interview sessions.

Despite these limitations, our study contributes to the groundwork for future research on students' behavior and their dealing with digital interruptions, for which we would push for the development of self-regulation skills as an essential next step in dealing with digital interruptions. Especially when taking the context of COVID-19 into consideration, further studies could focus on the development and validation of specific solutions to support students at different levels (primary, secondary, and tertiary education) in becoming higherlevel self-regulated learners.

**Supplementary Materials:** The following supporting information can be downloaded at: https: //www.mdpi.com/article/10.3390/educsci12030215/s1, File S1: excerpt from Survey 1 and Survey 2; File S2: first draft of coding scheme; File S3: final coding scheme.

**Author Contributions:** Conceptualization, J.P., M.P. and S.S.; data curation, J.P. and L.G.; formal analysis, J.P., M.P. and S.S.; investigation, J.P., M.P., S.S. and L.G.; methodology, J.P., M.P. and S.S.; project administration, M.P.; resources, J.P., M.P., S.S. and L.G.; software, J.P., M.P. and S.S.; supervision, M.P.; validation, J.P., M.P. and S.S.; visualization, J.P.; writing—original draft, J.P., M.P., S.S. and L.G.; writing—review and editing, J.P., M.P. and S.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted in accordance with the school's guidelines outlining ethical considerations regarding research with human participation. The fulfillment of the respective protocol was approved by the respective Research Ethics group.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Data are contained within the article or the Supplementary Materials.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Education Sciences* Editorial Office E-mail: education@mdpi.com www.mdpi.com/journal/education

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel: +41 61 683 77 34

www.mdpi.com

ISBN 978-3-0365-6114-1