Comparative Study on Students’ Engagement and Academic Outcomes in Live Online Learning at University

Abstract

Many studies have assessed students’ engagement with learning and academic outcomes and studied the influence of the engagement level on academic outcomes. Nevertheless, there are few studies that demonstrate the results of live online learning and reveal whether gender or/and education level mediates the relationships between students’ engagement and academic outcomes. A total of 301 under- and postgraduate students from the Humanity Institute of Peter the Great Polytechnic University voluntarily participated in this study. We conducted online surveys to reveal the level of students’ engagement and academic outcomes in the process of live online learning and analyzed the associations between these variables. Results displayed higher levels of engagement and academic outcomes for females, and the difference between the results was significant, particularly for behavioral and emotional engagement (p < 0.001). Significant differences of the measurement factors including behavioral engagement (p < 0.01), cognitive engagement (p < 0.01), and academic outcomes (p < 0.05) were revealed between undergraduate (UG) and postgraduate (PG) students. The findings obtained by using the linear regression analysis technique indicated that engagement scores significantly predict the students’ academic outcomes for both females and males, while UG and PG students demonstrated different results. It was detected that cognitive engagement has a greater effect on academic outcomes for PG students, while behavioral engagement has a greater effect on academic outcomes for UG students.

1. Introduction

Digital online education was formerly regarded as an important and generally progressive form of learning. However, it only complements and revitalizes analog formats. The COVID-19 pandemic has highly influenced the entire education system. The transition to distance learning was sudden and forced for all levels of education and for all participants in the educational process, regardless of their degree of technical readiness, level of digital skills, or desire [1].

The transition to distance learning in March 2020 was immediate and unexpected for most higher education institutions in Russia. Between 14 and 20 March 2020, the Ministry of Education and Science issued several orders and letters legitimizing the transition of universities to a distance learning format [2]. The next academic year (September 2020–June 2021), universities were transferred mostly to a blended learning format that included face-to-face classes, live online learning on the Microsoft Teams platform, and online learning on the university’s Moodle platform. These rapid transformations obviously had a great influence on students and teachers, because the phase of adaptation was skipped. The goal of the current study is to analyze the differences in students’ engagement and academic outcomes in live online learning by gender and by education level.

This study addresses a valuable contribution to the Russian and international literature as it comprehensively evaluates students’ engagement and academic outcomes in live online learning, and investigates whether gender and education level mediates the relationships between students’ engagement and academic outcomes. The following research questions are presented to address the need to generate actions to set an effective educational environment:

• Are students’ engagement and academic outcomes in live online learning affected by gender?
• Are students’ engagement and academic outcomes in live online learning affected by the education levels of students’ degrees, including undergraduate (UG), and postgraduate (PG) degrees?
• To what extent does students’ engagement predict students’ academic outcomes for males and females?
• To what extent does students’ engagement predict students’ academic outcomes for different education levels?

The remainder of this paper is organized as follows. The second section describes the theoretical background of the research; the research model and research hypothesis are presented. In Section 3, the methodology is described, including the demographic profile of respondents illustrated. In Section 4, the validity testing is presented and the comparative analysis of students’ engagement and academic outcomes in live online environment is made and discussed. Lastly, the conclusion and future research directions are indicated in Section 5.

Theoretical Background

E-learning and blended learning. E-learning, virtual learning, or online learning can be defined as learning that can take place functionally and effectively in the absence of a traditional classroom environment [3]. Some scholars distinguish virtual learning as learning with more interaction between students and instructors in comparison with e-learning and online learning [4,5,6]. We will use the term e-learning as an embracing descriptor of technology-based learning, which reflects the power and presence of new information and communication technologies. Previous studies have indicated the anti-role benefits and drawbacks of e-learning. While undergraduate students require face-to-face contact with instructors in order to adequately assimilate the required knowledge [4,5], many students responded positively to elements of e-learning such as a more comfortable learning environment, or efficient use of time through the repetition of video content [1,6]. Unlike e-learning, blended learning is an integration of the virtual learning described earlier with elements of face-to-face learning [7,8,9]. Benefits of blended learning approaches may include higher education flexibility [1], increased student engagement, and improved self-monitoring and regulation of learning [10,11]. Previous authors have also noted greater potential for practical application of this approach [12].

Live online learning. Live online learning refers to teaching and learning conducted through live online streaming in real time [13,14]. Teachers should post learning materials on the learning platform in advance and conduct lessons, including lectures and tutorials, in real time, answer students’ questions, and allow discussion in the classroom. While motivation and engagement are among the factors that are key to the success of online learning, the situation has changed with the outbreak of the coronavirus pandemic. Teachers must adapt regardless of their teaching style, participatory role, or technology barriers. For the introduction of online learning, universities provided dedicated online learning tools, infrastructure, and technical support from the Information Technology (IT) department to support real-time learning. In this live stream, classroom interaction is facilitated with the support of teachers, resulting in classrooms where students can interact and collaborate on an equal footing with on-site students [15]. According to research of Yilmaz, students studying in live online classes have higher academic achievements than students studying in a completely remote format [16]. Generally, there are insufficient studies that analyze live online learning, most of them including results on e-learning or blended learning.

Students’ engagement. A significant assessment for teaching students in a digital environment is engagement. Fredricks and McColsky noted that while some researchers explain it in terms of views and principles about the significance of learning, others define it as an effort beyond the bare minimum [17].

According to the analysis of digital learning publications, half of the studies mention the term “engagement” [18]. Their outcomes also demonstrated that, despite the common usage of the term, in exceptional cases, research is directly linked to involvement in digital learning.

Several scientists have proposed a multidimensional interaction model [19,20,21]. According to this theoretical model, engagement is multidimensional, which can embrace behavioral, emotional, and cognitive aspects [22,23]. According to Fredicks and McColskey, behavioral engagement underlines involvement, stubbornness, and participation in academic activities [17]. Emotional engagement centers on positive and negative reactions to peers, professors, and institutes, in addition to evaluating learning results. Cognitive collaboration comprises the student’s engagement and the understanding of the topic. “Cognitive engagement draws on the idea of investment; it incorporates thoughtfulness and willingness to exert the effort necessary to comprehend complex ideas and master difficult skills” [17] (p. 73).

Academic engagement expects numerous long-standing positive effects such as pursuit of higher education, steadiness in learning modes, enhanced job prospects, positive self-perception and welfare, and less depressive indications [24,25,26,27,28]. Thus, participation in academic activities can have positive and wide-ranging results even outside the educational environment. In addition, academic participation was discovered to be strongly associated with academic inspiration and performance; students rate their studies, score higher, and report lower levels of academic abstinence and effort avoidance [29].

Log records from electronic learning management systems can keep independent and related data, such as how many times a student has logged in. Although scholars have concentrated on log files for various points, they infrequently used these log data for analysis interactions. It is only lately that Gobert, Baker, and Wixon have established procedures to notice engagement in the online study environment for examining scientific questions [30]. Hence, other methodologies, such as interviews, have their benefits. Scholars can use interview techniques to obtain comprehensive material about why students partake or do not partake in certain events, why students vary in communication performance, and contextual aspects that can lead to student engagement or disengagement [31]. Considering the point that each method to dimension has its pros and cons, several academics recommend using numerous techniques to measure students’ engagement [32,33,34,35].

It should be noted that in the previously mentioned studies, much attention is paid to the availability of an electronic educational platform, namely, the availability of the necessary resources for active and effective educational activities of students. An electronic educational platform with a convenient interface and content can motivate students and increase their engagement.

Engagement and academic outcomes. In recent years, there has been interest in exploring the role of engagement as a key factor in academic success. It has been suggested that the influence of positive emotions may be indirect, through motivational processes such as engagement [36,37]. Engagement is associated with motivational processes. It is assumed that engagement plays an important role in achieving work goals. For example, interested students tend to devote more effort to academic tasks. In such situations, it is more likely that the task will be completed successfully and the academic performance will be improved. Engagement is a construct that has mostly been studied in work environments and is considered to be a work-related state of mind characterized by energy, dedication, and absorption [38]. Energy is characterized by a high level of mental mobility during work. Dedication refers to a sense of worth, enthusiasm, inspiration, pride, and challenge. Absorption is characterized by full concentration and satisfaction with one’s work, so that time passes quickly. This concept has also been applied to a scientific context and defined with a focus on student tasks and activities [38]. Engaged students feel energized, passionate about their studies, and involved in their academic life [38]. Empirical evidence has proven that engaged university students performed better [39,40,41,42,43,44]. This pattern of outcomes was also obtained using experimental designs, where a positive relationship was found between engagement and academic performance [45].

2. Materials and Methods

2.1. Participants and Procedures

A total of 301 under- and postgraduate students studying at Humanity Institute of Peter the Great Polytechnic University voluntarily participated in this study.

Table 1. Demographic profile of respondents (n = 301).

Demographic Variables		Number	Percentage
Gender	Male	131	43.52
	Female	170	56.48
Education level	Undergraduate	245	81.40
	Postgraduate	56	18.60
Field of study	Linguistics	96	31.90
	Psychology	64	21.26
	Law	141	46.84

displays the demographic profile of the students. The gender distribution of the students was quite balanced, with 56.48% females and 43.52% males. The age of the respondents is between the ages 21 to 27. The humanitarian fields of study included legal, psychological, and linguistic sciences. Students from the legal department study Russian and international law, existing legal systems, sources of law, etc. Students from the department of psychology study psychology of behavior, conflict resolution techniques, etc. Students from the linguistic department study several foreign languages, phonetics, and semantics of languages. The results were collected at the end of the spring semester 2021 after a year of live online learning due to COVID-19. Analyses were carried out on the depersonalized data.

2.2. Measures

The present study measured students’ academic engagement scale using the three most common dimensions identified by researchers (behavioral, emotional, and cognitive engagement). We measured behavioral engagement (BE) through students’ attendance records of online lectures in MS Teams and records of tasks completed in MS Teams (results are presented on a 10-point scale). To identify emotional engagement (EE), we used the motivation questionnaire. We created specific statements to clarify the students’ perceptions about teaching and learning offered, defining five indicators: (a) desire for learning after university, (b) anxiety, (c) positive attitude to learning, (d) self-esteem, and (e) self-demand. The answers were measured by the five-point Likert scale. To investigate students’ cognitive engagement (CE), we conducted a survey consisting of three items defining cognitive criteria, which index the extent to which students are attending to and expending mental effort in the learning tasks encountered (“I put a lot of effort into preparing for classes on MS Teams”; “I was engaged with the topics at hand in lectures”; “I spend much time accomplishing all assignments). The answers were also measured by the five-point Likert scale. We used the students’ semester grades in professional disciplines (four courses) as measures of academic outcomes (AO). To analyze the data we used IBM SPSS Statistics, version 21 for ANOVA, correlation, and regression analyses.

3. Results

3.1. Validity Test

First, we performed the reliability analysis using Cronbach’s α and research unit reliability (CR) to measure the internal consistency of the variables used in the research. According to

Table 2. Validity test results (n = 351).

Indicator	Items	Factor Loadings	$\mathit{\alpha }$	C.R.	AVE
BE	2	0.787-0.821	0.854	0.875	0.748
EE	5	0.881-0.921	0.933	0.944	0.879
CE	3	0.873-0.905	0.903	0.922	0.868
AO	4	0.821-0.845	0.897	0.907	0.790

, Cronbach’s α values of each variable ranged from 0.854 to 0.933, and CR values ranged from 0.875 to 0.944. Thus, it was detected that the reliability between the measurement items was more than standard value (0.70). Next, to measure the validity of the concept, the validity of convergence and the validity of discrimination were determined and verified.

To evaluate the instrument reliability and validity, we applied the confirmatory factor analysis (CFA) approach for our seven-factor research model. To load the items on their prespecified factors, each item was restricted, while the factors themselves were permitted to correlate freely. Table 2 and

Table 3. Discriminant validity (n = 301).

Indicator	BE	EE	CE	AO
BE	0.903
EE	0.619 ***	0.823
CE	0.415 **	0.345 *	0.901
AO	0.475 **	0.720 ***	0.319 *	0.885

Note: * p < 0.05; ** p < 0.01; *** p < 0.001.

present results of the analysis.

Convergence validity is indicated as a high correlation between the same concepts. The factor loadings were observed to be between 0.787 and 0.921, and the AVE value was 0.748 or more, which confirms the convergence validity of research units in Table 2.

3.2. Gender Difference on Academic and Behavioral Outcomes

In this study, 131 (43.52%) males and 170 (56.48%) females participated in live online learning. The distribution of males and females within fields of study and degrees is displayed in

Table 4. Gender breakdown for fields of study and degrees (N = 301).

		Gender
		Males	Females
Education level	Undergraduates	108 (44.08%)	137 (55.92%)
	Postgraduates	23 (41.07%)	33 (58.93%)
Fields of study	Linguistics	15 (15.63%)	81 (84.37%)
	Psychology	23 (35.94%)	41 (64.06%)
	Law	93 (65.96%)	48 (34.04%)

.

To investigate the differences between male and female students’ engagement and academic outcomes in live online learning, an independent sample t-test was used. The results are summarized in

Table 5. Summary of the gender differences (n = 301).

Factors	Gender		t-Value	p-Value
	Male (SD)	Female (SD)
BE	7.12	8.34	−9.17 **	0.00071
EE	3.66	4.13	−8.21 **	0.00087
CE	3.69	3.94	−5.05 *	0.0043
AO	3.87	4.06	−4.77 *	0.0079

Note: * p < 0.01; ** p < 0.001.

.

It was revealed that the mean scores for females were generally higher than males for all considered indicators in live online learning. Moreover, the differences (p-value range from 0.00071 to 0.0079, p < 0.01 or p < 0.001) between the two tested groups were highly significant.

3.3. Education Level Difference on Academic and Behavioral Outcomes

To investigate the differences between undergraduate and graduate students’ engagement and academic outcomes in live online learning, an independent sample t-test was used. The results are summarized in

Table 6. Summary of the education level differences (n = 301).

Factors	Education Level		t-Value	p-Value
	Undergraduate (SD)	Postgraduate (SD)
BE	7.93	6.67	6.89 **	0.0018
EE	3.97	4.02	−1.12	0.258
CE	3.89	3.71	4.66 **	0.0091
AO	3.94	4.05	3.91 *	0.017

Note: * p < 0.05; ** p < 0.01.

.

The positive value of the mean difference demonstrated that the mean scores of the UG students were higher than the PG students. The results revealed that there were significant differences between UG and PG students in most of the measurement factors, including behavioral engagement (p < 0.01), cognitive engagement (p < 0.01), and academic outcomes (p < 0.05). The higher result on emotional engagement of postgraduate students can be explained by their positive attitude toward such format of learning.

3.4. Regression Analysis

The results of the linear regression analysis conducted to analyze whether the engagement scales of the university students predicted the academic outcomes are presented in

Table 7. Engagement scales as a predictor of academic outcomes for male and female students.

	Males (N = 131)
	B	SEB	$\mathit{\beta }$	t	F	R2	Adjusted R2
Constant	4.77	0.11	0.41	11.02 ***	51.29 ***	0.57	0.50
BE	0.01	0.00		9.89 ***
	B	SEB	$\beta$	t	F	R2	Adjusted R2
Constant	3.27	0.09	0.34	7.14 **	34.73 **	0.32	0.30
EE	0.01	0.00		6.64 **
	B	SEB	$\beta$	t	F	R2	Adjusted R2
Constant	2.02	0.10	0.23	3.34 *	19.29 *	0.19	0.16
CE	0.01	0.00		3.18 *
	Females (N = 170)
	B	SEB	$\beta$	t	F	R2	Adjusted R2
Constant	4.86	0.11	0.45	12.11 ***	60.29 **	0.60	0.55
BE	0.01	0.00		10.87 ***
	B	SEB	$\beta$	t	F	R2	Adjusted R2
Constant	3.71	0.09	0.35	7.78 **	37.73 *	0.36	0.31
EE	0.01	0.00		6.14 **
	B	SEB	$\beta$	t	F	R2	Adjusted R2
Constant	2.09	0.10	0.24	4.89 *	20.29 *	0.20	0.16
CE	0.01	0.00		3.36 *

Dependent variable: Academic outcomes. Note: * p < 0.05; ** p < 0.01; *** p < 0.001.

.

According to Table 7, the engagement scores predict the academic achievement levels of the students in a significantly positive way (β ranged from 0.24 to 0.45, t ranged from 3.34 to 12.11). The results between males and females are similar and academic outcomes were mostly predicted by behavioral engagement (R² = 0.57, adjusted R² = 0.50 for males, R² = 0.60, adjusted R² = 0.55 for females).

The results of the linear regression analyses conducted to analyze whether the engagement scales of the university undergraduate and graduate students predicted the academic outcomes are presented in

Table 8. Engagement scales as a predictor of academic outcomes for undergraduate and graduate students.

	Undergraduates (N = 245)
	B	SEB	$\beta$	t	F	R2	Adjusted R2
Constant	4.98	0.11	0.44	11.14 ***	58.29 ***	0.59	0.54
BE	0.01	0.00		9.97 ***
	B	SEB	$\beta$	t	F	R2	Adjusted R2
Constant	3.72	0.09	0.35	7.78 **	38.90 **	0.35	0.31
EE	0.01	0.00		6.81 *
	B	SEB	$\beta$	t	F	R2	Adjusted R2
Constant	2.02	0.10	0.22	3.25 *	18.14 *	0.18	0.16
CE	0.01	0.00		2.87 *
	Postgraduates (N = 56)
	B	SEB	$\beta$	t	F	R2	Adjusted R2
Constant	4.24	0.11	0.37	8.11 **	51.29 **	0.38	0.35
BE	0.01	0.00		6.87 **
	B	SEB	$\beta$	t	F	R2	Adjusted R2
Constant	3.33	0.09	0.30	6.12 **	34.73 *	0.32	0.30
EE	0.01	0.00		5.81 **
	B	SEB	$\beta$	t	F	R2	Adjusted R2
Constant	3.94	0.10	0.34	7.73 **	41.29 **	0.38	0.34
CE	0.01	0.00		6.18 **

Dependent variable: Academic performance. Note: * p < 0.05; ** p < 0.01; *** p < 0.001.

.

Table 8 summarizes the engagement scores predicting the academic outcomes of the undergraduate and postgraduate students in a positive way (β ranged from 0.22 to 0.44, t ranged from 3.25 to 11.14). A difference was revealed between undergraduate students’ results and postgraduate students’ results. 38% of the total variance of the academic outcomes scores for postgraduate students can be explained by the CE scores (R² = 0.38, adjusted R² = 0.34), while only 18% of the total variance of the academic outcomes scores for undergraduate students can be explained by the CE scores (R² = 0.18, adjusted R² = 0.16). Thus, cognitive engagement has a greater effect on academic outcomes for postgraduate students. 59% of the total variance of the academic outcomes scores for undergraduate students can be explained by the BE scores (R² = 0.59, adjusted R² = 0.54), while only 38% of the total variance of the academic outcomes scores for postgraduate students can be explained by the BE scores (R² = 0.38, adjusted R² = 0.35). Thus, behavioral engagement has a greater effect on academic outcomes for undergraduate students than for postgraduate students.

4. Discussion

Research [32,33,34] on students’ engagement assessment allowed us to choose the best methods for analyzing this factor. For the analysis, we used students’ attendance records of online lectures in MS Teams and records of tasks completed in MS Teams, the motivation questionnaire, and cognitive criteria survey. Also, we analyzed students’ semester grades in professional disciplines as measures of academic outcomes.

There are several studies that analyze whether students’ engagement in online learning predicts academic outcomes [3,9,12,46]. According to Dunn and Kennedy, engagement in online learning when defined as comprising emotional, cognitive, and behavioral components confers a direct benefit to educational attainment, while there was no association found between behavioral engagement alone and academic achievement [46].

The current analysis revealed that the mean scores of the engagement level and academic outcomes for females were generally higher than for males and the difference was significant. The results also indicated that there were significant differences between UG and PG students in most of the measurement factors, including behavioral engagement (p < 0.01), cognitive engagement (p < 0.01), and academic outcomes (p < 0.05). The findings obtained by using the linear regression analysis technique indicated that engagement scores predict the academic outcomes of the students significantly in a similar and positive way for both females and males, while undergraduates and postgraduates demonstrated different results. It was detected that cognitive engagement has a greater effect on academic outcomes for postgraduate students, while behavioral engagement has a greater effect on academic outcomes for undergraduate students.

In contrast to earlier studies [3,9,12], we both assessed the level of students’ engagement and determined the relationship between engagement and academic outcomes, and made a comparative analysis of these factors and their relationships in different student groups (males and females; undergraduates and postgraduates). We confirm that the level of student engagement affects the academic outcomes of students during the educational process, namely, the professional disciplines’ outcomes in a live online learning environment. The results of the current study are consistent with Mogus, Djurdjevic, and Suvak’s [47] research, which observed that a strong relationship exists between students’ online activity logs and their final marks recorded in the online learning database.

Avcı and Ergün conducted analysis of undergraduate students and revealed that students with high participation had both high engagement and high performance [48]. Other scholars [49] defined several factors that have influences on students’ engagement in an online environment, among which was work/life commitments, so it can explain lower engagement results of postgraduate students in the current study.

5. Conclusions

The findings of the current study have several important implications for educators when implementing live online learning in the future. Based on the comparison of each factor influencing the fundamental moderating factors, it gives educators the basis for developing the learning context, learning strategy, lecture organization, and assessment method, which enhance the effectiveness of students’ live online learning. As behavioral engagement has a greater effect on academic outcomes for students at lower levels of education, additional efforts should be made to increase their motivation for live online learning by, for example, adding interactive technologies and developing more learning activities such as virtual games.

It should be noted that our study has limitations. Only students from Russia were analyzed, therefore, these students had a similar mentality, which can differ from representatives of other countries and affect the results of the study. Also, the study was conducted on a short time frame basis since live online learning was introduced in September 2020, so the sample size was quite limited. In addition, the study included only humanities students. Education in technical areas has its own characteristics, and the influence of engagement on academic outcomes in live online learning for technical specialties may differ.

The results of the study will be useful for future scientific work. This direction is promising, since live online education continues at the present time. It is important to study separately the adaptability of students to the digital environment over time and the selection of educational technologies specifically for the digital environment, in order to improve the academic outcomes of students. In addition, it appears important to compare students’ engagement and academic outcomes in live online learning, blended learning, and total e-learning formats.