*Article* **Assessment of Cognitive Student Engagement Using Heart Rate Data in Distance Learning during COVID-19**

**Andrea Catalina Ladino Nocua 1, Joan Paola Cruz Gonzalez 1,\*, Ivonne Angelica Castiblanco Jimenez 2,\*, Juan Sebastian Gomez Acevedo 3, Federica Marcolin <sup>2</sup> and Enrico Vezzetti <sup>2</sup>**


**Abstract:** Student engagement allows educational institutions to make better decisions regarding teaching methodologies, methods for evaluating the quality of education, and ways to provide timely feedback. Due to the COVID-19 pandemic, identifying cognitive student engagement in distance learning has been a challenge in higher education institutions. In this study, we implemented a non-self-report method assessing students' heart rate data to identify the cognitive engagement during active learning activities. Additionally, as a supplementary tool, we applied a previously validated self-report method. This study was performed in distance learning lessons on a group of university students in Bogota, Colombia. After data analysis, we validated five hypotheses and compared the results from both methods. The results confirmed that the heart rate assessment had a statistically significant difference with respect to the baseline during active learning activities, and this variance could be positive or negative. In addition, the results show that if students are previously advised that they will have to develop an a new task after a passive learning activity (such as a video projection), their heart rate will tend to increase and consequently, their cognitive engagement will also increase. We expect this study to provide input for future research assessing student cognitive engagement using physiological parameters as a tool.

**Keywords:** student engagement; heart rate; active learning; non-self-report; self-report; distance learning; higher education; COVID-19

#### **1. Introduction**

Education allows society and its territories to be transformed. As society changes, the education style must adapt to its demands [1]. Educational institutions focus their resources on achieving quality teaching and professional development for their students, who expect education to be suitable, reasonable, interactive, agreeable, and practical [2]. Achieving these expectations directly impacts student engagement and become a matter of concern for teachers and institutions; therefore, measuring student engagement becomes a valuable and necessary tool for institutions [2]. Student engagement has been defined as the physical or mental energy and effort students apply and invest in their academic environment [3]. Also, student engagement envelops the disposition and energy that students maintain in a learning process, persistence in the face of obstacles, and the value given to learning [4]. A higher level of student engagement generates deep learning, active participation, and a positive response to challenges [4].

The interest in raising student engagement has led to the importance of its measurement [5,6]. So far, in Colombia, student engagement has been measured using self-report methods, such as questionnaires, in which students directly report their perceptions; some-

**Citation:** Ladino Nocua, A.C.; Cruz Gonzalez, J.P.; Castiblanco Jimenez I.A.; Gomez Acevedo J.S.; Marcolin F.; Vezzetti E. Assessment of Cognitive Student Engagement Using Heart Rate Data in Distance Learning during COVID-19. *Educ. Sci.* **2021**, *11*, 540. https://doi.org/10.3390/ educsci11090540

Academic Editor: Kelum A. A. Gamage

Received: 2 August 2021 Accepted: 7 September 2021 Published: 14 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

times, these methods can be subjective and divert attention [7]. Furthermore, according to the literature review, a different approach has not yet been implemented in this country.

Higher education has been affected by the pandemic of COVID-19, generating impacts on education systems. Institutions have had to adapt their methodologies to a virtual environment and implement new teaching and online assessment strategies [8]; this transition has received different terms like remote teaching, online learning, or distance education [9]. Some obstacles students face with this situation are an unstable internet connection, restricted access to technological tools, and inadequate space to take classes [9]. Lassoued et al. have classified these difficulties into four categories: personal (self-imposed), pedagogical, technical, and financial obstacles [10].

This paper presents an investigation conducted in Colombia to measure cognitive student engagement by measuring students' heart rate during the COVID-19. In the United States, Darnell D. K. and Krieg P. A. [7] implemented a similar method in a group of medical students; the researchers measured cognitive student engagement using wristwatch-style monitors, which detected and recorded the heart rate of the study subjects during lecture classes that applied active learning activities [7]. In India, Senthil and Lin performed a study using a wireless wearable sensor to collect the heart rate of university students for measuring engagement, comparing the heart rate in a state of rest against the heart rate during active learning activities in lecture classes [11].

The structure of this paper is as follows: first, we introduce a background of previous works focused on the diverse methods for measuring cognitive student engagement. Next, we present the methodology, divided into three phases: (1) research preparation, which describes the instruments and tools necessaries for began the data collection, (2) execution of the research. This phase includes the collection of three types of data, quantitative data of heart rate, qualitative data of annotations taken by the researchers during the development of each virtual lesson, and the qualitative data of the questionnaire, (3) and data treatment, in this phase we proceed to the cleaning and processing of data for validating five hypotheses, as well as the comparative analysis of the non-self-report method vs. the self-report method using a quadrant model. Then, we proceeded to the analysis and results of the three phases of the methodology. Finally, we reported our conclusions and future directions of research.

#### **2. Background**

Student engagement can be perceptible in the behavioral, cognitive, or affective dimensions [3]. Behavioral engagement occurs when students get involved in their learning, for example, selecting times or places to study [12]. Cognitive engagement happens when students make a mental effort on the topic they are learning; when students use deep or superficial study methods, they can identify the type of mental effort [12]. Finally, affective engagement refers to the students' emotions, such as interest, enjoyment, or frustration [12].

Cognitive engagement is defined as the level of psychological investment that students make in the learning environment [13]. It considers mental efforts focused on reflecting, implementing strategies, and willingness to carry out the requirements for understanding complex ideas and train themselves in skills with a high degree of difficulty to master the material and achieve new knowledge [14]. Literature also describes cognitive engagement as the preference from the student of performing challenging tasks and being self-regulated, a condition that leads to planning and monitoring their own learning [15]. Cognitive engagement contributes the flexibility in problem-solving and a positive attitude toward facing failures [14].

Cognitive engagement has two levels depending on the student's effort: superficial and active engagement [16]. Superficial engagement is a minimal effort in the learning like simple memorization, help-seeking, and effort-avoidant strategies; on the other hand, active engagement implied deep understanding and expertise in activities such as attention, planning, connecting, and monitoring the learning [14].

In the literature, the studies found have implemented different methods to measure student engagement evaluated from the perspective of the study subject, the student. These methods can be classified into two categories: self-report methods and non-self-report methods [17]. One of the characteristics of the self-report methods is that the study subjects report the data directly. The most used tools are questionnaires and interviews. On the other hand, the non-self-report methods use tools that allow collecting the data without consulting directly with the study subject, such as capturing a video during class, which can later be used to analyze the emotions that the person experiences [17]. It is important to highlight that various studies have been developed for measuring student engagement from a third-party agent perspective, like a professor, a psychologist, or a researcher; these studies include teacher ratings, observation, and many others. Our study implemented a method that determines the cognitive student engagement using the data measured directly from students. Consequently, the background is focused on giving examples of good practices implemented to determine student engagement examining the student point of view and behavior directly. In this sense, it isolates studies that assessed engagement from a third-party perspective only.

Different countries have used self-report methods; in most cases, the researchers used questionnaires or surveys created or adapted from previous investigations. Table 1 shows some studies that have used self-report methods.


#### **Table 1.** Self-report measurement methods.

In the USA, the researchers have used surveys and questionnaires to measure student engagement; they carried out activities different from the traditional class. In one case, Lei et al. applied the National Survey of Student Engagement in a lesson composed of six people; three of them were physically present, and the other three in telepresence [18]. In another study, the students developed a mobile game using a tool (GameSalad) created by the researchers, and then they measured student engagement using a questionnaire [19].

In Malaysia, Awang-Hashim R. et al. used a survey to evaluate the scale of participation in student learning at the University of Malaysia (MUSLIS) consisting of 24 items; later, they performed the data analysis using specific software [20]. In Turkey, Alion and Delialio ˘glu implemented m-learning material in a computer networks course to measure student engagement and motivation using the NSSE- National Survey of Student Engagement and the MSLQ- Motivated Strategies for Learning Questionnaire, m-learning is the method that allows students to acquire certain types of knowledge anywhere and anytime utilizing wireless technologies, here is important to state that e-learning materials are designed to be watched on computers, while the visualization of m-learning materials is better on tablets or smartphones [21].

In China, Zhoc et al. applied the HESES-Higher Education Student Engagement Scale survey, which allowed to evaluated five facets of student engagement: academic engagement, cognitive engagement, social engagement with classmates, social engagement with teachers, and affective engagement [22]. Another study applied a survey to measure student engagement, Ma J. et al. used a guide for the formulation of projects of innovation and technological development in a mixed environment, that is, the combination of faceto-face and virtual class; the survey evaluated three modules: behavioral, cognitive and emotional behavior [23].

Although questionnaires and surveys are the most common tools, some researchers have used other tools; for instance, in the United Kingdom, Balaam et al. developed a handheld orb named Subtle Stone for implementing an interactive method with the students, the study subjects, had to press the Subtle Stone to indicate the emotion that they were experimenting in a class [24].

Table 2 presents some studies that have used non-self-report methods in different countries. These methods usually collect and analyze data on the physiological characteristics of the study subjects, such as facial expressions, eye movement, heart rate, among others.

Some studies conducted in China, the USA, Canada, and Ireland analyze facial expressions to determine factors such as emotions or student engagement. In Hong Kong, Zeng H. et al. developed an analytical system that recognizes emotions through facial expressions, named EmotionCues; they use a camera and software that detects which emotion a student is expressing [17]. In another study conducted in China, Zhang H. et al. identified affective states such as boredom, confusion and engagement through a facial recognition system [6]. In the USA, Alkabbany et al. recorded a video during a reading of 10 to 15 min; this video focused on the angle of the posture of the face and the gaze, after, the researchers analyzed these characteristics to identify student engagement [25]; in another study also carried out in the USA, Whitehill J. et al. analyzed facial expressions using Machine Learning to determine the level of engagement [5]. In Canada, Sakulchit et al. identified the emotions that children experienced before, during and after taking a blood sample through facial expressions [26] and in the case of Ireland, using an application, Farrell et al. analyzed the subject's head posture to determine if the student was looking at the screen. Thus, they detected the person's emotional state, which finally allowed them to determine the student engagement [27].


**Table 2.** Non-self-report measurement methods.

Other physiological characteristics analyzed in the studies presented include the study of brain signals and eye movement. In Brazil, Herpich F. et al. measured the signals emitted by the brains of students when interacting with educational technology [28]. In Japan, Hayashi et al. used lenses with integrated electrodes to analyze the user's eye movement and thus determine student engagement [29].

Finally, the heart rate is another physiological characteristic that researchers have used to measure student engagement [7,11]. In a study performed in the United States, Darnell D.K. and Krieg P.A. measured student engagement, specifically cognitive engagement, analyzing the students' heart rate. Researchers analyzed the heart rate behavior using wristwatch-style monitors during active learning activities [7].

Heart rate has been a tool widely used for identifying and monitoring emotions, attention, autonomic process and mental conditions [30–34]; heart rate is associated with the autonomic nervous system, which responds to a stimulus or resting states [33,35]. Emotions have three components: cognitive, physiological and behavior [34]. These components allow researchers to use the heart rate to measure more advanced factors like cognitive engagement [7,35]. Cognitive engagement can be determined through the heart

rate since the exposure to stimulus or tasks activates neural mechanisms and, consequently, triggers an acceleration or deceleration in the heart rate, which has been an indicator of alertness and drowsiness [36–38].The advantages of using heart rate include it is noninvasive, easy, and cheap to get [33,34], experiments with heart rate are simple to set up and can be used in conjunction with other biometric measures like facial expressions and respiration [38,39]. The disadvantages associated with the heart rate are the conditions of the environment under which happen the data collection since they are challenging to eliminate, and also the response time to a stimulus is long; these two points generate more uncertainty [30,40,41].

Active learning activities allow student engagement to be improved and induced [42,43]. The concept of active learning refers to activities that induce students in a thinking process about the new information and connect it with experience or knowledge [42,44]. There is a great variety of activities that can be considered as active learning, such as discussion in small groups, peer activities, individual activities that required a mental effort, interactions, study cases, problem resolution, laboratories, quizzes, and games, all of them have to guarantee that students think effectively [42–46]. The advantages include that students feel more secure sharing their ideas in small groups, developing critical thinking, retaining new knowledge, developing communication and leadership skills, and feeling more motivated and interested [42,44–47].

In Colombia and Latin America, as some researchers stated, the investigations regarding education and student engagement have focused on the following topics: study habits and motivation for distance learning [48], academic dropout and its relationship with the student's conditions and the organizational context of the academic institution [49] the relationship between student engagement and academic performance determined through the academic average [50], and the relationship that emotional intelligence and happiness orientation have with student engagement [51].

Other research examines the opinion of students in schools regarding how they experience student engagement after conducting, recording, and analyzing surveys [52]. Bertel-Narváez M. P. et al. performed a literary review of education in Latin America, highlighting that the motivation of learning is a fundamental aspect of developing research in Latin America [1]. Studies and research developed in Colombia have used traditional methods such as interviews, surveys, questionnaires, and the corresponding analysis. However, in this country, the studies developed have not yet used more advanced methods, which measure student engagement with physiological characteristics.

In Colombia, the studies performed have not implemented a method that uses physiological characteristics to measure cognitive engagement. This work developed exploratory research, similarly to the analysis proposed in the United States by Darnell D.K. and Krieg P.A. [7] and adapted for this study during the pandemic. We used the heart rate for measuring cognitive student engagement. In some studies, the heart rate is also used for determining the emotional aspect [38], which means that emotional and cognitive engagement generate a variation in the heart rate. For that reason, a good way to differentiate these dimensions is through the activity or stimuli to which the student is exposed. In this study we selected active learning activities, which are focused on activating the cognitive engagement. For collecting data, the students used a heart rate band linked to a mobile application while participating in virtual lessons that contained active learning activities; every student collected her or his heart rate during four lessons; the total data collection took three months. This research took place at Escuela Colombiana de Ingeniería Julio Garavito in Bogotá, Colombia, in a sample of 20 students, from which we discarded four students after data cleaning.

#### **3. Materials and Methods**

This study adopted the methodology developed by Carrasco and cited by Viñan J.A. et al. [53]. It consists of three phases (Table 3); the first phase entails the research preparation, the second phase the execution of the research, and the third phase regards the data treatment. The objective of this study was to determine the cognitive student engagement using the heart rate as the primary measurement tool and using a questionnaire as a complementary tool.

**Table 3.** Research methodology.


The development and detail of each phase are described below.

#### *3.1. Phase 1. Research Preparation*

This phase consists of four activities to prepare instruments and tools necessary for began the data collection. Figure 1 details each activity.

**Figure 1.** Research preparation.

Below, we explain the considerations made for developing each activity.

#### 3.1.1. Active Learning Activities

All the activities planned with the professor include active learning activities performed in virtual lessons during the situation of COVID-19. In each lesson they developed the method Think Pair Share proposed by Kaddoura [54] and adapted it to this study, which we detail below:


3. Share: then, the professor asked each pair of classmates to share their answers with the whole lesson. However, in some cases, the students did not share their answers but gave the solution to the professor directly.

#### 3.1.2. Participants

The following characteristics were important for selecting the participants:


In this study, we measured the heart rate of 20 students, from which we discarded four students after cleaning the data. We defined the number of participants after reviewing the sample used in similar studies carrying out non-self-report methods. (Table 4).

Ethical considerations were taken into account following the guideline given by Escuela Colombiana de Ingeniería Julio Garavito, the area of *Política de Tratamiento de Datos* (Data Processing Policy Department) reviewed and approved the document of data treatment used in this study, that posteriorly was signed by each student. The following ethical guidelines were implemented in the research:



**Table 4.** Number of participants in studies of non-self-report methods.

#### 3.1.3. Heart Rate Band Preparation

This investigation used heart rate bands, which can measure the heart rate with a frequency of one minute; these have a Photoplethysmography (PPG) heart rate sensor, which applies low-intensity infrared (IR) light on the skin. An optical sensor measures light reflection, which changes depending on the blood flow through the illuminated spot. Since blood flow changes during a heartbeat, it is possible to measure the heart rate.

The considerations for preparing the heart rate band were the following:


#### *3.2. Phase 2. Execution of the Research*

We divided the execution of the investigation into three categories, which composed the data collected: quantitative data, qualitative data of annotations, and qualitative data of the questionnaire. The quantitative data (obtained through a non-self-report method) refers to data of heart rate collected through the heart rate app during the development of each virtual lesson. The annotations of qualitative data refer to the data collected by the researchers while attending each lesson and taking note of the activities carried out and their respective time. Finally, the qualitative data refers to the questionnaire (self-report method) that each student filled out at the end of each lesson, which evaluated the cognitive student engagement. Figure 2 presents each data category collected in this phase. Below, we explain the conditions required in each data category.

**Figure 2.** Execution of the investigation.

3.2.1. Quantitative Data (Non-Self-Report Method)

The conditions verified in the quantitative data collection were the following:


#### 3.2.2. Qualitative Data of Annotations

To collect annotations qualitative data, the researchers considered the following aspects:


A questionnaire is a self-report tool to measure student cognitive engagement. However, using this method will allow us to compare the report made by the students and the results obtained with the measurement of heart rate and helping to identify factors that may affect the student's heart rate behavior during lessons.

As described above, the participants filled out a questionnaire after completing each virtual lesson; the main objective is to identify the students' cognitive engagement reported, allowing us to compare the participants' answers and the heart rate behavior.

To identify student cognitive engagement in different contexts, the queries in the questionnaire were selected following three studies, which applied questions focused on measuring student engagement. For the design of the questionnaire for this study, we selected questions focused on measuring student cognitive engagement.

• Study one: Exploring Factors and Indicators for Measuring Students' Sustainable Engagement in e-Learning [55]

In this research through a questionnaire, they analyzed six factors in student engagement in the e-learning environment: factor 1. psychological motivation, factor 2. peer collaboration, factor 3. cognitive problem solving, factor 4. interactions with instructors, factor 5. community support, and factor 6. learning management. For our study, we selected and adapted factor 3 (Cognitive problem solving).

• Study two:Examining engagement in context using experience-sampling method with mobile technology [12]

The following measures were applied in the above survey: study time, study location, reasons for study, behavioral engagement, cognitive engagement, self-efficacy, academic motivation, and prior academic achievement. For our questionnaire, we adapted the measures used in cognitive engagement.

• Study three:Initial Development Process of a Student Engagement Scale in Blended Learning Environment [23]

In this study, Ma J. et al. designed and constructed a student engagement scale framework in higher education during blended learning environment. They evaluated the three dimensions of student engagement (cognitive, behavioral, and emotional). For our questionnaire, we selected the items used in cognitive student engagement.

Table 5 presents the format used in the questionnaire, the self-report method used in this study, which the students filled after finishing each virtual lesson. We used a Likert scale from 1 to 5 (1: totally disagree and 5: totally agree) to rate each item. The interpretation given to this scale related to cognitive student engagement was the following: An answer of 1 indicated a low cognitive engagement, an answer of 5 indicated a high cognitive engagement, and an answer of 3 was taken as a neutral position regarding the question.

#### **Table 5.** Self-report method the questionnaire format.

#### **Self-Report Method: The Questionnaire**


12 I make connections between the things that I have learned in today's lesson.

	- *3.3. Phase 3. Data Treatment*

This phase included three activities that allowed us to perform the data treatment (Figure 3). The first consisted of cleaning data, the second entailed data processing, and the last compared the non-self-report method with the self-report method.

**Figure 3.** Data treatment.

Below, we present the requirements considered in each activity.

#### 3.3.1. Data Cleaning

First, we organized the data, classifying it into three categories: participants, lessons, and activities. The criteria used for cleaning the data were the following:


#### 3.3.2. Data Processing

Intending to have a general view of data, we applied a clustering of all data. After, we processed and organized the data to verify the following hypothesis:

**Hypothesis 1.** *The heart rate may increase during an active learning activity or rest and then return to the mean level.*

**Hypothesis 2.** *The heart rate decreases from the beginning to the end of the lesson.*

**Hypothesis 3.** *The drop in heart rate is biphasic, further decreasing during the early stages of the lesson.*

**Hypothesis 4.** *The heart rate decreases at the beginning and increases at the end of the lesson.*

**Hypothesis 5.** *The heart rate decreases in passive learning activities, such as watching a video.*

For each hypothesis, we developed the following data processing:

For Hypothesis 1 we selected and labeled active learning activities in each lesson for each participant; we also labeled activities before and after each one, considering only those corresponding to theoretical explanation, beginning of a lesson, or end of a lesson. Subsequently, we plotted the mean heart rate before, after, and during each active learning activity using a data science platform.

For Hypothesis 2 for each participant in each lesson, we plotted the heart rate against time. We applied linear regression with the aim of identifying if the heart rate trend decreased or increased.

For Hypothesis 3 to analyze the biphasic behavior, we divided the data into two sections. We showed the first 20 min of each lesson against the next 70 min. We plotted the heart rate against time in each section; finally, we applied linear regression.

For Hypothesis 4 we selected the first and last minutes of each lesson, ranging between 2 to 10 min, to analyze the behavior at the beginning and end of each lesson. The time range is variable since each lesson started or finished the topic and activities at a different time. Then, we graphed each time range and applied linear regression to identify the behavior at the beginning and end of each lesson.

For Hypothesis 5 we performed the same processing of the first hypothesis, with the difference that at this phase, we selected and labeled the passive learning activities.

#### 3.3.3. Non-Self-Report Method vs. Self-Report Method

In the research developed by Nonis et al. the results suggested to combine two types of methods for measuring engagement, they proposed to combine the User Engagement Scale questionnaire with an additional physiological measurement (a facial expression recognition system) [56]. In this section, for analyzing the behavior of quantitative data of the heart rate (non-self-report method) against qualitative data of the questionnaire (self-report method), we performed a quadrant model to show the variation of these two data types (Figure 4).

**Figure 4.** Quadrant model of heart rate variation vs. mean response of questionnaires.

For quantitative data, we calculated the mean heart rate for each participant and subtracted the heart rate baseline to identify whether there was a positive or negative variation. For the qualitative data of the questionnaire, we calculated the mean response to the questions of questionnaires filled out in each lesson; each question indicates a high or low level of student cognitive engagement, thus if the student's mean response was below 3, it indicated a low engagement, and if it was above 3 it indicated a high cognitive engagement. Quadrant I represented a positive variation of the heart rate and a high level of cognitive engagement reported in the questionnaire by students, quadrant II indicated a positive variation of the heart rate and a low level of cognitive engagement reported in the questionnaire by students, quadrant III represented a negative variation of the heart rate and a low level of cognitive engagement reported in the questionnaire by students. Finally, quadrant IV indicated a negative variation of the heart rate and a high level of cognitive engagement reported in the questionnaire by students.

#### **4. Analysis and Results**

#### *4.1. Phase 1. Research Preparation*

Four groups of five students of the subject Human Talent Management participated in the research, led by two professors. The activities planned and developed in each lesson followed the method: *Think Pair Share* [54]. Also, other activities considered were online quizzes, group activities, activities in pairs and individual activities. The lessons were in remote teaching using a virtual platform for communication and online programs for the development of the activities.

The 20 participants belong to the industrial engineering program; Figure 5 presents the characteristics associated with them:

**Figure 5.** Information participants.

The participants followed the instruction and conditions for the preparation of the heart rate band. Due to compatibility issues and connection issues, we discarded the data from participant 0 and 19.

#### *4.2. Phase 2. Execution of the Research*

Using the heart rate band, we measured the heart rate of each participant during the virtual lessons. We took note of different conditions that could alter the student's heart rate and the moment in which the professor developed active learning activities. Also, we collected the baseline heart rate of each participant (Table 6). In this study, we used the baseline for identifying the heart rate response to activities developed in each lesson; the difference between the baseline and the heart rate could be negative or positive depending on the engagement of each participant. The behavior of the heart rate compared with the baseline is the response to a stimulus, according to the study of Siennicka et al. [33]. We do not show the baseline heart rate of four participants (participant 0, participant 9, participant 18, and participant 19) in Table 6, due to the reasons explained above. In the cleaning data section, we will explain the reason why participants 9 and 18 were also discarded.

On the other hand, we applied the questionnaire described in the methodology, we used a Likert scale to rate each item, as follows: (1) totally disagree, (2) in disagreement, (3) partially agree, (4) I agree, and (5) totally agree. In Table 6, also we show the mean according to the response given by participants in the questionnaire.


**Table 6.** Mean response in the questionnaire and baseline heart rate for each participant.

*4.3. Phase 3. Data Treatment*

4.3.1. Data Cleaning

Applying the criteria described in the methodology, we transformed the data into the same time format requested by the data science platform; we used this tool to perform some steps of data cleaning and data processing. We discarded two participants because they had data from fewer than three lessons (participant 9 and participant 18). Then, we applied the normalization criterion to the participants and lessons shown in Table 7. Finally, we analyzed the behavior of each participant in each activity and eliminated the outliers.

**Table 7.** Data normalized and discarded under the criteria of 3SD.


#### 4.3.2. Data Processing

Before analyzing each hypothesis, we clustered all the data dividing each group to identify the general and predominant behavior of the heart rate. First, we applied the elbow method. We obtained that, in all groups, the best value of K was seven, and then we proceed to the data clustering.

Table 8 shows the cluster for each group, the number of items classified in each cluster, and the centroid based on the heart rate. We presented the table organized in descending order according to the centroid, and we highlight the two clusters with the most items.

The clusters highlighted indicate the predominant behavior of the heart rate in each group, varying from between 66 and 77, except for Group 1, which is between 70 and 82. This behavior shows that the predominant data was not in the extremes, which contained the minority of items.

Figure 6 presents the distribution of items in each cluster in the four groups graphically. In each graph, the behavior of each cluster is similar in each lesson. Although each participant had a different baseline, it did not alter the classification in the clustering. For this reason, we were able to perform a general analysis of data. We conducted an ANOVA test afterwards; for better data treatment, we conducted additional analysis by categorizing the heart rate according to participants, lessons, and activities.


**Figure 6.** Distribution of items in each cluster for the four groups.

For each hypothesis, after applying the data processing we obtained the following results:

• Hypothesis 1.

After selecting and labeling the mean heart rate (MHR) of each active learning activity with its activities before and after, we obtained four behaviors, which described the mean heart rate before, during, and after an active learning activity. We show an example of each behavior in Figure 7.


**Figure 7.** The behavior of the mean heart rate before, after, and during an active learning activity.

For this analysis, we examined 58 active learning activities; Table 9 presents the results obtained in each behavior. It is important to note that the activities that we considered before an active learning activity were theoretical explanation, the beginning of a lesson, or instruction given by the professor. If the activity is theoretical explanation, the student could be in a rest state; thus, the heart rate may be lower than an active learning activity. On the other hand, if the activity is the beginning of a lesson, the heart rate may vary due to external factors depending on the activity that each student was doing before starting the lesson. Finally, if the previous activity is the instruction given by a professor, the student can be attentive without doing any activity that involves deep learning. These three activities have in common that they are not active or passive learning activities; for this reason, they can be grouped for the analysis of H1.

It is important to note that the activities that we considered before an active learning activity was either theoretical explanation, the beginning of a lesson, or an instruction given by the professor. If the activity is *theoretical explanation*, the student could be in a rest state; thus, the heart rate may be lower than an active learning activity. On the other hand, if the activity is *the beginning of a lesson*, the heart rate may vary due to external factors depending on the activity that each student was doing before starting the lesson. Finally, if the previous activity is *an instruction given by a professor*, the student can be attentive without doing any activity that involves deep learning. These three activities have in

common that they are not active or passive learning activities; for this reason, they can be grouped for the analysis of H1.


**Table 9.** Number of active learning activities according to each behavior.

Behavior one (low MHR - high MHR- low MHR) coincides with H1. As shown in Table 9, 41.4% of the activities presented this behavior, which has the highest percentage compared to the other behaviors. On the other hand, behavior two (High MHR-low MHRhigh MHR) represents the contrary situation to H1. The main reason why this situation could happen was the activity: interaction between students and professor, which could generate an immersive state in the students, causing a decrease in the heart rate.

Behavior three (low MHR - high MHR - high MHR) is not common and it is consistent with the results of the study of Darnell D.K. and Krieg P.A., who stated that after an active learning activity the heart rate did not continue to increase but returned to the behavior it had before this activity [7]. Finally, behavior four (high MHR - low MHR - low MHR) represents a resting, due to the heart rate decreased during an active learning activity. This result indicates that the participant did not need to make a significant cognitive effort because of previous knowledge and clarity regarding the discussed topic.

Since MHR behavior during an active learning activity is the most relevant factor in this study, we applied an ANOVA (Table 10) with *p* < 0.05 between two variables: the baseline heart rate and the mean heart rate during active learning activities. If the value F is greater than the critical value for F, it implies that the means of the two variables are significantly different.


**Table 10.** ANOVA-Active learning activities.

Except for the quiz, all active learning activities had statistically significant differences in heart rate when participants developed these activities compared to each participant's baseline. The case of the quiz had a particularity, a set time was given to finish it, but some participants could finish earlier, so they could carry out other activities in the remaining time, such as taking a break; this condition could be the main reason why in the ANOVA this activity was not statistically significant.

Overall, we partially accept H1 adapting the initial approach to the following: the mean heart rate has a significant variation with an active learning activity, indicating cognitive engagement in the student experiment. This approach is also supported by the statement made by Mayson and Oleksy, "heart rate is useful in the detection of cognitive attention because it changes when cognitive attention is directed to a particular situation" [57].

• Hypothesis 2.

For this hypothesis, we graphically represented the heart rate values of each virtual lesson and applied a linear regression. In each graph it was possible to identify if the heart rate trend (HRT) increased or decreased during each lesson (Figure 8). We were able to identify two behaviors:


**Figure 8.** Heart rate trend decreased or increased from the beginning to the end of the lesson.

In total, we evaluated 15 lessons; Table 11 shows the summary of each behavior.

The expected trend of the heart rate was not evident in all lessons; although in the study of Darnell D. K. and Krieg P. A., the authors found a decreasing trend among all the lecture lessons analyzed [7], the results we obtained in virtual lessons suggest that the trend in heart rate from the beginning to the end of the lesson may depend on external or internal factors. The external factors are related to connectivity problems or the environment in which the participant takes the lesson; the internal factors are related to the moment in which the students develop active learning activities and their duration, either at the beginning, in the middle or the end of the class. For these results, we partially accept H2 with the following adaptation: the trend of the heart rate from the beginning to the end of the lesson depends on external and internal factors, the internal factors are linked with the active learning activities, mainly the duration and the moment in the lesson that the professor develops them.

**Table 11.** Summary of heart rate trend.


• Hypothesis 3.

For this hypothesis, we selected the virtual lessons that presented a decrease in the heart rate trend according to H2; after that, we divided the data into two sections—the first 20 min of each lesson against the next 70 min. Figure 9 shows an example of a lesson divided into two sections with its linear regression.

**Figure 9.** Behavior biphasic during a lesson.

A negative slope means a decrease in the heart rate trend, in the Table 12 we show the slope of each section of the lessons analyzed and its correspondence with H3; for the correspondence, we evaluated if the first and last section had a negative heart rate trend and if the slope in the first section was minor to the slope in the last section, which indicated a further decrease during the early stages of the lesson.



We show the summary of correspondence with H3 of the 15 lessons in Table 13, which displays the nine lessons analyzed in Table 12 and the six lessons that had an increase in H2 and which do not correspond with H3 (Table 11). H3 has a strong dependency on H2 since it requires a drop in heart rate throughout the lesson, a condition that was not met in all lessons; for this reason, we included the six lessons for the analysis in the correspondence with H3 (Table 11). Since 73% of the lesson did not show the behavior expected, we reject H3.

**Table 13.** Correspondence with hypothesis 3.


• Hypothesis 4.

We present the behavior that occurred at the beginning and end of each virtual lesson in Table 14. We also show the correspondence with H4.


**Table 14.** Behavior of the heart rate at the beginning and end of each lesson.

The heart rate decreased at the beginning and increased at the end in 20% of lessons; for that reason, we reject H4.

We analyzed each heart rate behavior presented at the beginning and the end of the lessons (Table 15). The percentage for each behavior is similar between them, so it is possible to conclude that there is no standard pattern. In another study, the researchers verified a standard heart rate behavior during various lessons. However, they evaluated it during lecture classes and the students were physically present [7]. In our study, the lessons were virtual, with different factors that could influence the heart rate of the participants, like connection problems, and the influence of external factors according to the place where they take the lessons that could affect their level of attention, etc.


**Table 15.** Summary of each behavior at the beginning and the end of lessons.

#### • Hypothesis 5.

In this hypothesis, we applied a process similar to H1, considering the activities before, during, and after passive learning activities, in this case, a video projection. We obtained four behaviors, which described the mean heart rate (MHR) before, during, and after a passive learning activity.


However, this hypothesis focuses on the behavior before and during a passive learning activity since that the first two behaviors represent a decrease and the last two an increase in heart rate during a passive learning activity. Table 16 presents a summary of the combination of these behaviors and their percentage.



The increased in the heart rate could occur because, before each video, professors instructed their students to pay attention and to make an activity after the video projection, which could prevent students from entering a resting state during the video projection. These results allow us to rebuild H5 with this approach: if students are previously advised that they will have to develop an activity after a passive learning activity (such as a video

projection), their heart rate could increase and, consequently, so could their cognitive engagement. Overall, we partially accept H5 with the exposed approach. We present a summary of the results of each hypothesis in Table 17.


**Table 17.** Hypotheses results.

4.3.3. Non-Self-Report Method vs. Self-Report Method

In this section, we constructed a quadrant model for relating the non-self-report method with the self-reported method. We analyzed two scenarios, the first related the mean responses of the questionnaire with the general heart rate variation (HRV). In this last variable, we took the data for the heart rate during whole lessons, that is, all activities. In the second scenario, we analyzed the same relationship of variables but in the second variable, we only considered the HRV of active learning activities (ALA).

• First scenario: General heart rate variation vs. mean response of the questionnaire

In this analysis we evaluated the 16 participants classifying them in the four quadrants (Figure 10). Ten were in the quadrant I, one in the quadrant II, one in the quadrant III, and four in the quadrant IV.

The classification of the 10 participants in the quadrant I indicates that they had a positive HRV, that is, an increase in the general mean heart rate (MHR) concerning the baseline. These students also reported a high cognitive engagement in questionnaires. This quadrant is the best scenery in this study because it proved that a positive HRV is related to a high engagement.

One participant was in the quadrant II; she/he had a positive HRV but reported a low level of engagement in the questionnaire. This result indicates that the student did not feel a high cognitive engagement during the whole lesson, but he could have moments or activities that caught their attention, increasing their heart rate. If a participant was in quadrant III, she/he had a negative HRV, and in the same way, the student reported a low level of engagement in the questionnaire.

**Figure 10.** First scenario—Quadrant of general heart rate variation vs. mean response of questionnaire.

Finally, four participants were in the quadrant IV; they reported a high engagement in the questionnaire. However, the general MHR decreased with respect to the baseline, a circumstance that may indicate that students were in an immersive state and not necessarily a low level of attention or a low level of cognitive engagement; this affirmation coincides with the results of the study made by Ronney et al., who analyzed the behavior of the participant's heart rate while watching a film; the results demonstrated that a drop in the heart rate might suggest an immersive environment followed by an increase in the attention paid to the film [58].

• Second scenario: Heart rate variation of active learning activities vs. mean response of the questionnaire

In the same way as in the first scenario, we evaluated the 16 participants with the difference that here we related the HRV during active learning activities (ALA) and the mean responses in the questionnaire (Figure 11).

Nine participants were in quadrant I, seven of them presented a positive HRV below three points when an ALA was developed. This behavior is similar to the first scenario, which indicates that ALA had a great influence on the behavior of a whole lesson. The quadrant I represents that participants were making a mental effort for acquiring a high level of attention, a circumstance that resulted in a positive variation of the heart rate. The other two participants had a positive HRV greater than three points. This quadrant represents the best scenario; with these results, we can affirm that a positive HRV is linked to a high cognitive engagement.

Two participants were in the quadrant II, unlike the first scenario where we found one participant in quadrant II and the other participant in quadrant III. This behavior indicates that, although the participant in quadrant III reported a low cognitive engagement in the questionnaire, the ALA developed by the professor caught her/his attention, which meant an increase in the HRV and caused her/him to be in the quadrant II in the second scenario.

Five participants were in the quadrant IV. As we explained, this quadrant represents an immersive state of the participants. The participant who was in the quadrant I in the first scenario is now in the quadrant IV in the second scenario. This change means that she/he was in an immersive state when the professor carried out ALA. On the other hand, one of the participants had a significant negative HRV, below eight points, which indicates that her/his concentration was higher during ALA.

**Figure 11.** Second scenario—Quadrant of heart rate variation of active learning activities vs. mean response of questionnaires.

Overall, with the analysis performed and illustrated in the first and second scenario, the study shows that although there were heart rate variations (causing location changes for some participants inside the quadrant), the number of subjects for each quadrant is similar. This condition allows us to suggest that ALA developed by professors determinea the behavior of the MHR throughout the whole lessons.

Using the quadrant analysis it was possible to ascertain that combining a non-selfreport method (heart rate) with a self-report method (questionnaire) is advantageous for measuring cognitive student engagement, since the comparison of the heart rate against the report given by each participant provides more information about the influence of ALA on the heart rate.

#### **5. Discussion**

In this study, we conducted exploratory research implementing a non-self-report method to measure the cognitive student engagement, with a physiological characteristic, the heart rate. We have measured and analyzed this variable, using a heart rate band, in 16 participants from the Industrial Engineering program, in a Colombian University located in Bogota. With the collected data, we evaluated five hypotheses. Below, we will present and discuss the results of each hypothesis.

• Hypothesis 1.

To analyze H1, we evaluated the behavior of the heart rate before, during, and after an active learning activity (ALA); we found that 41.4% of data correspond with H1. Since ALA was an important matter in this study, we applied an ANOVA test to evaluate the heart rate during an ALA against each participant's baseline, an analysis that demonstrated a statistically significant difference between these variables. We partially accepted H1 adapting the approach to the following: the mean heart rate has a significant variation with an active learning activity, indicating students experiment with cognitive engagement.

In comparison with similar activities developed in a face-to-face environment, the heart rate had a variation if the student was exposed to active learning activities; however, in the study performed by Darnell et al. [7], the heart rate had a significant uptick, while in our study, in virtual lessons, the heart rate had a significant variation that could be positive or negative. With this perspective, we could deduce that one of the main factors that affect the student engagement is the environment and not the activity that is developed. For that reason, we conclude that it is valuable to design activities that consider the learning environment to capture the student's attention and improve her/his knowledge.

• Hypothesis 2.

We evaluated H2 with the heart rate trend; if it was negative indicated a decrease and if it was positive indicated an increase in the heart rate from the beginning to the end of the lesson. From a total of 15 lessons, 60% presented the behavior expected by H2. For these results we partially accept H2 with the following adaptation: the trend of the heart rate from the beginning to the end of the lesson depends on external and internal factors, the internal factors are linked with the active learning activities, especially the duration and the moment in the lesson that the professor develops them.

In the study of Darnell et al. [7], the heart rate had a decreasing trend in the morning lessons; they found that this behavior was caused mainly by external factors as mental fatigue, and the accumulation of multiple classes, among other reasons. We found that these aspects happened in virtual lessons, but internal factors also affected the trend. These factors are linked with the active learning activities that professors can plan and monitor during each lesson.

• Hypothesis 3.

H3 had a strong dependence on H2, because H3 requires the trend during a lesson to decrease. After evaluating the 15 lessons, we found that only 27% of them had the behavior expected in H3, and for this result we rejected H3.

The biphasic shift with a faster decline in the first minutes and a slower decline in the remainder of the class was not confirmed in the study developed in the United States [7]; we did not find the biphasic behavior described in the development of virtual lessons.

• Hypothesis 4.

After analyzing the behavior of the heart rate at the beginning and the end of each lesson, we found that only 20% of the lessons achieved H4. After the analysis we concluded that here is not a standard pattern related to this hypothesis, and for this reason we rejected H4.

The behavior at the beginning and the end of the lesson in the study of Darnell et al. [7] showed a drop at the first three minutes and an uptick in the last three minutes of the lesson; for this reason, they excluded these three minutes for the analysis of the others hypothesis. They determined that this behavior could happen for external factors related to a face-to-face environment, such as arriving in a classroom and speaking with other students or thinking about the next class or next activities after finishing each lesson. In our study, we did not find a notable change at the beginning and the end of each virtual lesson; we take into account the data of these moments for other analysis.

• Hypothesis 5.

We analyzed 11 passive learning activities and found that 45% of them presented the behavior described in H5. All activities were a video projection; however, before students watched the video, the professor indicated to them that there would be an activity after the video, a factor that could influence the results. We partially accepted H5 with the following adaptation: if students are previously advised that they will have to develop an activity after a passive learning activity (such as a video projection), their heart rate could increase and, consequently, so could their cognitive engagement.

The study performed in student of medicine in the United States [7] found that during a short video projection, the heart rate appears to decrease greatly since it is categorized as a passive activity, the difference with our study consisted in the instruction that the professor gave to students, which was developed a task after a video projection, a condition that modified the behavior of the heart rate.

In addition to the analysis of the hypotheses, we made the relationship between the quantitative data vs. qualitative data from the questionnaire.

#### *5.1. Non-Self-Report Method vs. Self-Report Method*

We related the heart rate variation (HRV) of whole lessons and the HRV during the active learning activities (ALA) against the response that participants gave in the questionnaire; the VHR refers to the difference between mean heart rate and the baseline, we explored the behavior of the 16 participants.

We used a quadrant model (Figures 10 and 11); two quadrants stood out over the others; Quadrant I reflected that the HRV was positive and the responses in the questionnaire indicated a high level of cognitive engagement, the majority of data was located in this quadrant; the first stage, HRV of whole lessons, had 10 participants, and the second stage, HRV during ALA, had nine participants. Quadrant IV was the second predominant one. It represented an HRV negative, but the questionnaire's response indicated a high level of cognitive engagement; this situation reflected that participants could be in an immersive state during the ALA, and not necessarily a low level of engagement. Finally, the results suggest the behavior of the heart rate during a whole lesson might be determined for the ALA that professors developed in each lesson.

#### *5.2. Limitations, Recommendations, and Future Research*

We used the heart rate as a tool for measuring student engagement in virtual lessons during the COVID 19 pandemic, and we had limitations regarding the difficulty of eliminating environmental and external influences generated by the inconveniences of having access to resources and tools necessary for the development of the lessons, such as internet connection problems, unstable electricity, lack of an appropriate space without interruptions, a computer without the required capacity to run several programs and software at the same time (lesson connection software and programs to develop activities during class), and inconvenience with the phone that collected heart rate data (the reason for removing some data from the study). These limitations affected the level of attention of the participants and sometimes caused them to carry out other activities in parallel to the lessons. The difficulty of eliminating environmental and external influences is a known limitation in the use of heart rate as a tool for identifying engagement; for this reason, for future research, we recommend using the heart rate as a non-self-report method in combination with a self-report method such as a questionnaire that allows the identification of external factors that could affect the participant's attention in addition to the perception that she/he had of the lesson and the activities developed.

Another alternative we recommend in order to reduce the influence of external factors is to use supplementary biometric measures that allow us to identify additional characteristics which affect the heart rate, like emotions, motivation or level of concentration, factors that have a strong influence on student engagement. The supplementary biometric measures we suggest are: detection of facial expression, breath rate, skin temperature, and conductance or brain signals.

In this study, we made an exploratory investigation limited to the number of participants, the quantity of heart rate bands, and to a specific subject (human talent management). The sample number of participants was between the ranges of other studies in the sector; however, we recommend that for future studies the researchers increase the sample and evaluate the student engagement in other engineering subjects. We hope that our study can serve as the basis and input for future research related to the analysis of cognitive student engagement using heart rate as a non-self-report method.

Using the heart rate for analyzing the student engagement has the advantage that data is collected in real-time. However, the data processing and the analysis require additional time. This limitation means that teachers do not receive feedback in real time to make the necessary decisions to modify or adapt the methodology of the course. For this reason, a future challenge could be the development of a model that processes and evaluates the data of the heart rate in a shorter time or even in real-time, which can be used by all students in a lesson and provide feedback to the professor.

The heart rate is a physiological parameter used mainly by enterprises to evaluate customer behavior. Carrying out this study, we aim to demonstrate that the heart rate is a prominent tool that can determine cognitive student engagement. We expect that this study could be a base for future research looking to deepen the evaluation of the variables that influence heart rate in an academic environment.

#### **6. Conclusions**

Student engagement allows educational institutions to improve their decisions regarding methodologies, evaluation of quality, and feedback. The methods to determine student engagement are divided into self-report methods, such as questionnaires, surveys and analysis of these, and non-self-report methods, which use physiological characteristics such as heart rate, brain signals, analysis of facial expressions, etc. In Colombia, the investigations only used self-report methods.

In this study, we develop an exploratory investigation to determine the level of student cognitive engagement through heart rate, during the development of active learning activities. We used heart rate bands and a mobile application to collect the data, in a sample of 16 students who were taking the Human Talent Management subject of the industrial engineering program of a higher education institution in Bogotá, Colombia.

The results confirm that heart rate can be used as a tool for measuring cognitive student engagement in distance learning, especially if the professor develops an active learning activity, since statistically, the heart rate has a significant variation with respect to the baseline heart rate during the development of these activities. At this point, it is important to clarify that this difference could be positive or negative. A positive variation implies an increase in the heart rate because the study subject is making a mental effort, and a negative variation means an immersive state, i.e., the active learning activity captured the student's attention and immersed her/him in this environment, causing her/him to isolate herself/himself from any other activity. In the development of virtual lessons, one of the main concerns of the professors is to identify activities that encourage the participation of students, capture their attention, and allow the transmission of knowledge. In this study, we found that active learning activities achieve these objectives, because spaces are generated within the lessons that allow students to concentrate on the activities developed, resolve doubts and connect their past experiences with new knowledge.

Using a quadrant model, we confirmed that combining a non-self-report method and a self-report method allows us to analyze the engagement in a broader perspective. This association allows us to relate the engagement reported by the students with the results obtained from the physiological characteristic. In our study, one of the results suggested that, during an active learning activity, most participants reported a high level of cognitive engagement, and the heart rate variation had a significant difference with respect to the baseline, a result that was evident in the ANOVA. However, in this analysis it was possible to identify that this difference could be positive or negative.

In the development of this research we found that students feel more comfortable, feel less pressure, and are more willing to participate in a study if factors such as student engagement are measured using a non-self-report method, because they sent us the heart rate data immediately after the end of the lesson without having to remind them to send the data. However, the opposite happened with the self-report method, because we had to ask them to fill out each questionnaire at least twice and sometimes up to four times. The above shows that students are interested in participating in non-self-report methods and shows the importance of rethinking the way self-report methods are implemented.

We confirm that the heart trend from the beginning to the end of the lesson depends on external and internal factors. The external factors include instability in the internet connection and activities that students may be developing in parallel to lessons, and the internal factors, which professors can control, are linked with the active learning activities. The internal and external factors that are presented in a face-to-face lesson are very different from those that are presented in a virtual lesson. In a face-to-face lesson, the external factors that affect the concentration of students are reduced due to the learning environment that a classroom provides. However, in a remote environment, the external factors are greater and variable. For this reason, the active learning activities that professors develop become more relevant, especially in terms of the duration and time of the lesson in which they are performed. These activities provide a balance at the beginning, in the middle and at the end of the lesson.

Concerning passive learning activities, if students are previously advised that they will have to develop a task after a passive learning activity (such as a video projection), their heart rate could increase and, consequently, so could their cognitive engagement. The instructions of the activities to be carried out after the video projection were focused on the topic that the students were learning and generated a discussion between the work groups or between the students and the professor. This particularity should be considered if professors want to implement a passive learning activity to promote cognitive engagement.

We find that heart rate did not present a pattern in the biphasic analysis or a pattern behavior at the beginning and the end of the lesson. The four heart rate behaviors found at the beginning and at the end of the lesson allow us to deduce that the variation depends on external factors that may occur before and after the class, and that the professor cannot control; for example, other academic activities of other subjects or personal activities that can cause a variation in heart rate.

We expect that this study can provide input for future research assessing student cognitive engagement in higher education and motivate researchers to use physiological characteristics as a measurement tool.

**Author Contributions:** Conceptualization, A.C.L.N., J.P.C.G. and I.A.C.J.; methodology, A.C.L.N., J.P.C.G., I.A.C.J. and J.S.G.A.; validation, A.C.L.N., J.P.C.G., I.A.C.J., J.S.G.A., F.M. and E.V.; formal analysis, A.C.L.N., J.P.C.G., I.A.C.J., J.S.G.A., F.M. and E.V.; investigation, A.C.L.N., J.P.C.G., I.A.C.J., J.S.G.A., F.M. and E.V.; resources, A.C.L.N.; data curation, A.C.L.N., J.P.C.G., I.A.C.J., J.S.G.A., F.M. and E.V.; writing—original draft preparation, A.C.L.N. and J.S.G.A.; writing—review and editing, J.P.C.G., I.A.C.J., F.M. and E.V.; visualization, A.C.L.N., J.P.C.G., I.A.C.J., J.S.G.A., F.M. and E.V.; supervision, J.P.C.G., I.A.C.J., J.S.G.A., F.M. and E.V.; project administration, A.C.L.N., J.P.C.G., I.A.C.J., J.S.G.A., F.M. and E.V. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Escuela Colombiana de Ingeniería Julio Garavito of Autorización del tratamiento de datos participación investigación de Maestría.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Acknowledgments:** Thanks to Escuela Colombiana de Ingeniería Julio Garavito, professors and participants for their effort and support.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Abbreviations**

The following abbreviations are used in this manuscript:

MUSLIS Malaysian University Student Learning Involvement Scale NSSE National Survey of Student Engagement MSLQ Motivated Strategies for Learning Questionnaire HESES Higher Education Student Engagement Scale ALA Active learning activities


#### **References**

