Evaluating of Education Effects of Online Learning for Local University Students in China: A Case Study

Abstract

The spread and persistence of the global COVID-19 pandemic has caused online education to gradually become the “new normal” in higher education, and a comprehensive and systematic study of the online course learning of college students in the context of the normalization of online teaching is urgently needed. Higher education is the main stage of cultivating innovative talents, and the evaluation of collaborative education through science and education in the context of online education is also a key content. Therefore, this study established a hierarchical evaluation system based on the Analytic Hierarchy Process (AHP), including three second-level evaluation factors and fifteen third-level evaluation factors. Then, a judgment matrix for each level of evaluation factor was constructed and passed the consistency test. Finally, the weights of each factor were determined, and the feasibility and effectiveness of this method in online learning quality evaluation were verified through a case study. The evaluation results indicate that the cultivation of online learning ability for local college students is more important than online course resources and the online learning environment. In order to promote the realization of educational equity, it is not only necessary to focus on improving students’ academic performance, but also to find ways to enhance students’ learning abilities. In addition, the cultivation of communication and collaboration abilities between teachers and students is an important way to improve the quality of online learning for college students in the process of online learning for college students.

1. Introduction

In the past three years, numerous nations and territories have temporarily shuttered their schools in reaction to the COVID-19 pandemic [1,2,3]. Due to the global wave of pandemic lockdowns, there was an unprecedented rise in online learning. As of 1 April 2022, nearly 1.5 billion children in 173 countries were affected by school closures, as shown in Figure 1. In Denmark, children up to the age of 11 are returning to nurseries and schools after initially closing on 12 March 2022, but in South Korea students are responding to roll calls from their teachers online. In China, the largest “online movement” in the history of education occurred in mid-February 2022 after the government instructed a quarter of a billion full-time students to resume their studied online [4,5,6]. As a result, education has changed dramatically, with the distinctive rise of online learning, whereby teaching is undertaken remotely and on digital platforms. Fernando et al. [7] analyzed the opportunities and challenges of emergency remote teaching based on experiences of the COVID-19 emergency, and a qualitative research method was undertaken in two steps. Based on the lessons learned from this worldwide emergency, challenges and proposals for action to face these same challenges, which should be and sometimes have been implemented, were provided. Octavian and Nicoleta [8] carried out a comparative scientific research regarding the students’ perception of e-learning platforms and tools based on 144 respondents from Romania and the Republic of Moldova. Mohammed et al. [9] contributed to the body of knowledge on educational technologies adoption during COVID-19 by providing a comprehensive model that captures the most significant determinants of Madrasati platform adoption among Saudi public schools.

The difference between online education and traditional education lies not only in the physical campus structure and teaching time and space, but also in the teaching modes, organizational forms, and challenges faced [10]. This determines that the online education quality evaluation system is not a complete replication of the traditional education quality evaluation system. Therefore, as a social activity aimed at cultivating people in a purposeful, systematic and organized manner, education is faced with unprecedented sustainable development challenges [11,12,13]. In order to establish a scientific and reasonable evaluation system for online learning quality, many institutions and scholars internationally have conducted research and discussions. For example, the European Online Education Quality Assurance Assessment Framework was launched in 2019 to improve the quality of online education in European universities [14]. This assessment framework not only retains the core factors of ESG (2015) and reiterates the student-centered philosophy and the quality culture of continuous improvement, but also highlights the concept and technology of learning analytics in line with the characteristics of online education. This assessment framework is of great significance for the guarantee and improvement of the quality of online education in European universities. Based on the performance oriented theory, Sun constructed and applied the online learning performance evaluation model to study the evaluation and improvement of online learning quality of students majoring in economics and management in a university in Beijing [15]. The Professional Committee for Instructional Design and Application in the United States has proposed the “Online Learning Certification Standards” to evaluate online learning from three dimensions: usability, technicality, and teaching effectiveness [16]. In 2000, the American Institute of Higher Education Policy published “Quality Online the Line Benchmarks for Success in Internet based Distance Education”. This standard proposes seven factors to evaluate and supervise online education activities to ensure the quality of online education. These seven factors include: school support, curriculum development, teaching and learning process, curriculum structure, student support, teacher support, and evaluation and assessment. Professor Badrul from the University of Washington proposed “A framework for Online Learning”, which evaluates online learning from three dimensions: resources, management, and teaching [17]. Overall, learning resources, teaching management, course design, and support services are important components for evaluating the quality of online learning.

Online education in Chinese universities has developed rapidly in recent years, but the construction of online teaching quality assurance system needs to be strengthened, especially in local universities. The number of local colleges and universities in China is large and the scale is relatively large, but the comprehensive strength is weak, the debt is high, and the anti-crisis ability is poor. In the post COVID-19 era, the sustainable development pressure of local colleges and universities is huge. The core tenet of sustainable development is high-quality education [18]. Differences in course resources, learning equipment, and learning environment, on the other hand, will certainly impair the learning quality and long-term development of educational equality of local colleges and universities students when they learn online. At present, there are many different methods for evaluating students’ online learning quality in China, but most of them have certain limitations. Some evaluation standards are too high, evaluation models and methods are too simple, and the influence of multiple factors is not comprehensively considered [19]. Some are single evaluation results that cannot comprehensively reflect the actual level of students. Therefore, we wanted to analyze and investigate the quality online education of local colleges and universities from both the theoretical and conceptual points of view, as well as from the quantitative and statistical viewpoint [20].

At present, most evaluation methods in China rely on comments given by a certain evaluation factor to determine whether a teacher’s teaching quality or a student’s learning quality meets the standards, making it difficult to achieve the goal of comprehensively evaluating of the educational quality. Therefore, how to scientifically and reasonably evaluate the teaching or learning quality of teachers or students is an urgent problem that needs to be solved in the current education industry, especially in the current era of widespread online education in China [21]. By establishing a scientific and effective evaluation system for student’s online learning quality, one can promptly identify problems in school or teacher work and propose effective solutions and suggestions. Online education quality evaluation is a complex issue with numerous and interconnected influencing factors [22]. The AHP combines qualitative and quantitative factors, determines factors and evaluation levels with a goal oriented approach, and assigns weights. When using this method to evaluate the online learning quality of students, it can avoid using objective scoring standards simply and mechanically. When determining the weight of each factor, there is no need to use subjective or objective scoring methods, and the weight of the factor is consistent with the subjective evaluation weight of the decision maker for the factor. Therefore, this evaluation method has certain feasibility and effectiveness in evaluating the online learning quality of local university students in China [23].

This article is based on the AHP method and constructs a student online learning quality evaluation system. First, the AHP method was introduced in the analysis and evaluation process to quantify and analyze various factors of online learning quality. Then, the weights and levels of each factor were determined and validated based on expert survey data and questionnaires. In order to ensure a certain degree of objectivity in the evaluation of online learning quality, the selection of influencing factors should take into account the relatively consistent opinions of all parties, and also consider the use of expert scoring methods when opinions from all parties are not very consistent or there are relatively large contradictions. However, there are no strict requirements for scoring standards. We assume that the decision criteria or variables are independent of each other and no interaction scenarios occur. This article has certain reference value for promoting the realization of educational equity in local universities in China and clarifying the future development direction of online education in China.

2. Methodology

2.1. Multi-Level Factor Valuation System

AHP, also known as the Expert Evaluation Method, is a scientific multi-objective decision-making method developed in the 1970s. AHP is a comprehensive mathematical method that combines qualitative and quantitative analysis, is goal-oriented, and takes into account the allocation of factor weights. In practical applications, this method considers not only qualitative factors but also quantitative factors. Therefore, AHP is an effective mathematical method that converts qualitative and quantitative problems into mathematical models for solving [24,25,26]. It transforms multiple interrelated qualitative problems into quantitative problems for solving. It decomposes the problem into several levels and comparison steps, and then judges and sorts them one by one based on the dependency relationship between each level and the overall goal [27,28,29]. First, determine the problems to be solved at each level and the goals for each level. Then, establish a multi-level factor valuation system. Finally, determine the weight of each factor.

Here, we first briefly introduce the establishment of a multi-level factor system. AHP is a qualitative analysis method, whose basic principle is to divide a problem into several interconnected levels based on its actual situation, and determine the weight values of each factor or the total ranking of each level at each level. We know that an online learning quality evaluation standard with high value is multi-level and comprehensively evaluates the education and teaching process and results [30]. In order to achieve this goal, it is very important to establish a scientifically and reasonably evaluated factor system. In this article, we only consider factors related to the education and teaching process, and do not consider factors unrelated to or less related to the education process [31]. Therefore, we incorporate the sustainable evaluation of online learning quality and fairness among local university students into the primary factors. For the factors in the first-level factors, multi-level and comprehensive evaluation criteria should be established based on the corresponding relationship between the evaluation criteria. The first-level factors can be divided and analyzed into second-level factors, third-level factors, fourth-level projects, sixth-level projects, and seventh-level projects in this layer, in order to determine the corresponding first- or second-level standards for each level. The sustainable evaluation that affects the quality and fairness of online learning for local university students mainly includes three aspects: online course resources, online learning ability, and online learning environment. Determine the second- and third-level factor systems based on the requirements of the first-level factors [32,33,34].

When establishing a multi-level structural model at this level, attention should be paid to the following issues: AHP mainly focuses on qualitative analysis, and when establishing a multi-level structural model, multiple considerations should be considered, different methods should be used for modeling models with different hierarchical structures, and each level of factor should have clear regulations. The hierarchical evaluation model established for the evaluation of online learning quality among local university students is shown in

Table 1. Multi-level factor valuation system for online learning quality for local university students in China.

Evaluating of online learning quality for local university students in China A	Online course resources B1	Online learning platform operation capability C1
		Ability to autonomously select online learning resources C2
		Enrichment of online courses C3
		Ability to arrange online learning plans C4
		Online learning exam scores C5
	Online learning ability B2	Online learning and interactive communication skills C6
		Collaborative communication skills in online learning C7
		Online learning reflection and evaluation ability C8
		Innovative thinking and abilities in online learning C9
		Online learning application ability C10
	Online learning environment B3	Online learning network fluency C11
		Stability of learning equipment C12
		The environment surrounding online learning C13
		Online learning classroom atmosphere C14
		Autonomous control in online learning C15

, which includes three two-level evaluation factors (B1~B3) and fifteen three-level evaluation factors (C1~C15).

2.2. Determine the Weight of Each Factor

The main calculation process of AHP is shown in Figure 2. On the basis of the established hierarchical evaluation system, a judgment matrix is constructed. Based on the nine-level scale scoring method proposed by professor Thomas L. Saaty (

Table 2. Nine-scale scoring method of the AHP evaluation system.

Scale	Meaning
1	Indicates that two factors have the same importance
3	Indicates that one factor is less important than the other
5	Indicates that one factor is obviously more important than the other
7	Indicates that one factor is more important than the other
9	Indicates that one factor is absolutely more important than the other
2, 4, 6, 8	Represents the intermediate value of the adjacent judgment
Reciprocal matrix	If element i/element j = rij, then if element j/element i = 1/rij

), the factors are compared in sequence, and the corresponding judgment matrix can be established based on the comparison results [35,36,37,38]. The established judgment matrix is shown in Equation (1), the elements on the diagonal are all 1, and the elements on both sides of the diagonal are reciprocal to each other, i.e., r_ji = 1/r_ij.

$$R={\left[r_{ij}\right]}_{n\times n}=\left[\begin{array}{ccc}{r}_{11}& \cdots & r_{1n}\\ ⋮& \ddots & ⋮\\ r_{n1}& \cdots & r_{nn}\end{array}\right],\left(r_{ij}>0;r_{ij}=\frac{1}{r_{ji}};r_{ii}=1;i=1,2,\cdots ,n;j=1,2,\cdots ,n\right)$$

(1)

In Equation (1), R is the constructed judgment matrix, and the normalized feature vector of R can be used as the weight vector. According to the nine-scale scoring method, the r_ij value in R can be obtained by comparing each evaluation factor in pairs, namely r_i and r_j. Secondly, calculate the maximum eigenvalue λ_max of the constructed judgment matrix R through Equation (2) and its corresponding normalized feature vector W.

$$R_W={\lambda }_{\max }W,(W={(w_1,w_2,\cdots ,w_n)}^T)$$

(2)

In Equation (2), λ_max is the maximum eigenvalue of the judgment matrix R, and W is the corresponding normalized eigenvector. Then, the weight values of each factor in the AHP evaluation model can be obtained through Equations (3)–(5).

$$u_j=\frac{1}{n}{\displaystyle \sum _{j=1}^n{c}_{ij}^{\prime }},(i=1,2,\cdots ,n)$$

(3)

$$c_{ij}^{\prime }=\frac{c_{ij}}{{\displaystyle \sum _{i=1}^n{c}_{ij}}},(i=1,2,\cdots ,n)$$

(4)

$$W={\left[w_1,w_2,\cdots ,w_n\right]}^T={\left[\frac{1}{n}{\displaystyle \sum _{k=1}^n{c}_{1n}^{\prime },\frac{1}{n}{\displaystyle \sum _{k=2}^n{c}_{2n}^{\prime },\cdots ,\frac{1}{n}{\displaystyle \sum _{k=n}^n{c}_{kn}^{\prime }}}}\right]}^T$$

(5)

Finally, it is necessary to perform consistency checks on the judgment matrix R to ensure that it reaches the threshold level. The consistency test is conducted according to Equation (6). If the consistency test is passed, the decision can be made according to the calculation results obtained from the combination weight vector. Otherwise, the hierarchical model needs to be rebuilt or the paired comparison matrix with a large consistency ratio needs to be reconstructed.

$$CI=\frac{{\lambda }_{\max }-n}{n-1}$$

(6)

In Equation (6), CI is the consistency factor, and the smaller the CI, the greater the consistency. If CI = 0, there is complete consistency; if CI is close to 0, there is satisfactory consistency; the larger the CI, the more severe the inconsistency.

Using the eigenvector corresponding to the maximum eigenvalue as the weight vector of the influence of the compared factor on a certain factor in the upper layer, the greater the degree of inconsistency, the greater the judgment error caused.

To measure the size of CI, the random consistency factor RI is introduced, which can be obtained from

Table 3. Standard value of average random consistency factor RI.

Matrix order	1	2	3	4	5	6	7	8	9	10	11
RI	0	0	0.58	0.90	1.12	1.24	1.32	1.41	1.45	1.49	1.51

. In order to avoid consistency testing errors caused by random reasons, it is necessary to compare CI with the random consistency factor RI when testing whether the judgment matrix R has satisfactory consistency, and obtain the test coefficient CR, as shown in Equation (7):

$$CR=\frac{CI}{RI}$$

(7)

In Equation (7), the random consistency factor RI is related to the order of the judgment matrix R. In general, the higher the order of the matrix, the greater the likelihood of random deviation from consistency. The corresponding relationship is shown in Table 3.

3. Case Study

During the COVID-19 pandemic, Xuchang University organized two online thematic learning sessions for the whole school, namely “Thematic Seminar on Teaching Management Mode Innovation and Teachers’ Teaching Ability Improvement under the Normalization of Epidemic Situation” and “Integrated Teaching Seminar on Response to the Epidemic Situation”. At the same time, the academic affairs office organized and invited platform technical engineers and experienced teachers on campus to form a technical team, establish an online classroom communication group, provide technical support for class opening and live teaching, and take multiple measures to help teachers improve their information application capabilities and online course design and operation capabilities. A scientific research group from Xuchang University used AHP to analyze and comprehensively score the selected evaluation factors when evaluating the online learning quality of students, and determined the weights of each factor based on expert opinions. We conducted a survey of 25 students in a class and obtained the data results. We screened these original survey questionnaires and then summarized them to obtain the results. The judgment matrix of the evaluation factors at all levels constructed is shown in

Table 4. Judgment matrix A-B_i(i = 1~3).

A	B1	B2	B3
B1	1	1/3	1/2
B2	3	1	3
B3	2	1/3	1

,

Table 5. Judgment matrix B1-C_i(i = 1~5).

B1	C1	C2	C3	C4	C5
C1	1	1/3	1/5	1/5	3
C2	3	1	1/3	1/3	5
C3	5	3	1	1/3	5
C4	5	3	3	1	5
C5	1/3	1/5	1/5	1/5	1

,

Table 6. Judgment matrix B2-C_i(i = 6~10).

B2	C6	C7	C8	C9	C10
C6	1	1/2	4	3	3
C7	2	1	7	5	5
C8	1/4	1/7	1	1/2	1/3
C9	1/3	1/5	2	1	1
C10	1/3	1/5	3	1	1

and

Table 7. Judgment matrix B3-C_i(i = 11~15).

B3	C11	C12	C13	C14	C15
C11	1	1	1	4	1
C12	1	1	2	4	1
C13	1	1/2	1	5	3
C14	1/4	1/4	1/5	1	1/3
C15	1	1	1/3	3	1

.

According to the constructed judgment matrix of evaluation factors at all levels, consistency testing of the judgment matrix is carried out, and the test results are shown in

Table 8. Consistency inspection results of A-B_i(i = 1~3).

λmax	CI	RI	CR	Consistency Inspection Results
3.054	0.027	0.520	0.052	Pass

,

Table 9. Consistency inspection results of B1-C_i(i = 1~5).

λmax	CI	RI	CR	Consistency Inspection Results
5.397	0.099	1.120	0.089	Pass

,

Table 10. Consistency inspection results of B2-C_i(i = 6~10).

λmax	CI	RI	CR	Consistency Inspection Results
5.073	0.018	1.120	0.016	Pass

and

Table 11. Consistency inspection results of B3-C_i(i = 11~15).

λmax	CI	RI	CR	Consistency Inspection Results
5.266	0.067	1.120	0.059	Pass

. Generally, the smaller the CR value, the better the consistency of the judgment matrix. Generally, if the CR value is smaller than 0.1, the judgment matrix meets the consistency test. If the CR value is greater than 0.1, it indicates that there is no consistency, and the judgment matrix should be adjusted appropriately before further analysis. The constructed judgment matrices have all passed consistency testing, indicating that the constructed judgment matrices are scientific, reasonable, and effective, and can be used for sustainable evaluation of the quality and fairness of online learning for local university students.

Based on the constructed judgment matrix, the weight calculation results of the second-level factors are shown in Figure 3. The weight of online course resource B1 is 0.1593, the weight of online learning ability B2 is 0.5899, and the weight of online learning environment B3 is 0.2519. The weight calculation results of the tertiary factors are shown in Figure 4, Figure 5 and Figure 6. In the evaluation factors under the second-level factor of online course resources B1, the order of weight ranking is online learning plan arrangement ability C4 > online course richness C3 > online learning resource independent selection ability C2 > online learning platform operation ability C1 > online learning exam score C5. In the second-level factor of online learning ability B2, the ranking of weights is online learning collaborative communication ability C7 > online learning interactive communication ability C6 > online learning application ability C10 > online learning innovative thinking and ability C9 > online learning reflection and evaluation ability C8. In the second-level factor of online learning environment B3, the order of weight ranking is in the online learning environment C13 > in the learning device stability C12 > in the online learning network fluency C11 > in the online learning autonomous control C15 > in the online learning classroom atmosphere C14. The final weight calculation results are shown in

Table 12. Weight calculation results.

First-Level Factor A	Second-Level Factors (B1~B3) and Weights		Third-Level Factors (C1~C15) and Weights
Evaluating of online learning quality for local university students in China A	Online course resources B1	0.1593	Online learning platform operation capability C1	0.0822
			Ability to autonomously select online learning resources C2	0.1674
			Enrichment of online courses C3	0.2766
			Ability to arrange online learning plans C4	0.4256
			Online learning exam scores C5	0.0483
	Online learning ability B2	0.5899	Online learning and interactive communication skills C6	0.2623
			Collaborative communication skills in online learning C7	0.4744
			Online learning reflection and evaluation ability C8	0.0545
			Innovative thinking and abilities in online learning C9	0.0985
			Online learning application ability C10	0.1103
	Online learning environment B3	0.2519	Online learning network fluency C11	0.2231
			Stability of learning equipment C12	0.2673
			The environment surrounding online learning C13	0.2714
			Online learning classroom atmosphere C14	0.0562
			Autonomous control in online learning C15	0.1820

.

4. Results and Discussion

From the weight calculation results of the second-level factors, it can be seen that the weight of online learning for local college students is the highest. It can be seen that the cultivation of online learning ability for local college students is more important than online course resources and online learning environment. Only by fully mastering the ability of online learning can college students significantly improve the quality of online teaching for teachers. The cultivation of college students’ online learning ability should be approached from multiple aspects, mainly including online learning interactive communication ability, online learning collaborative communication ability, online learning reflection and evaluation ability, online learning innovative thinking and ability, and online learning application ability. From the calculation results of the third-level factors, the highest weight is online learning collaborative communication ability C7, followed by online learning interactive communication ability C6. From this, it can be seen that in the process of online learning for college students, the cultivation of communication and collaboration abilities between teachers and students is an important way to improve the quality of online learning for college students.

In the calculation of the weight results of the three-level factors, the weight of the online learning exam score is only 0.0077. It can be seen that the exam scores of online learning are not very important. In recent years, many universities in the United States have cancelled the grading of freshmen’s grades. The purpose of abolishing the grading system is to achieve a smooth transition to higher education, especially for new students who are studying alone for the first time or who are not fully prepared for higher education in high school and urgently need more time to become familiar with it. Jodie Green, who serves as a special advisor to the Education Equity and Academic Achievement Department of UCSC, said that "Grades are not a representation of student learning, as hard as it is for us to break the mindset that if the student got an A it means they learned". If a student has mastered and learned the course content before class, and obtained an A, then they have actually learned nothing. And if this student works hard in class and gains a C, they may actually have gained a lot. It can be seen that in the trend of online learning and exams, the cultivation of college students’ learning ability is far more important than exam scores. From this, it can be seen that in order to promote the realization of educational equity, we cannot solely focus on improving students’ academic performance, but should find ways to enhance students’ learning abilities. Only the improvement of learning ability can greatly enhance the development of students’ comprehensive qualities [39].

According to the calculation results of the three-level factors in online learning environment B3, the fluency of online learning network C11, the stability of learning devices C12, and the surrounding environment C13 have roughly equal weights. It can be seen that in the process of online learning, differences in learning equipment can to some extent lead to the emergence of educational unfairness. In China, it is difficult for students from remote areas to access smooth wireless networks and stable learning devices, such as iPads, laptops, and mobile phones. Therefore, schools need to focus on improving students’ learning equipment and learning environment during online learning, in order to promote the realization of educational equity and enhance students’ learning ability and quality [40].

5. Limitation and Future Work

This article mainly focuses on a local university in Henan Province, China. The research results have certain reference value for the quality of online learning for students in this local university. In the future, we will conduct a broader survey and research on the quality of online learning for more local university students. On the other hand, when there are too many evaluation factors, the data statistics are large and the weights are difficult to accurately determine. In the future, we will actively explore more practical and efficient evaluation methods.

6. Conclusions

This article studies the online learning quality of local university Students in China, taking Xuchang University as a case study. We established a hierarchical evaluation factor system and determined the weights of each factor. Based on the AHP method, a hierarchical evaluation model was constructed, and a judgment matrix for evaluation factors at all levels was established. After consistency testing, the comprehensive weights of each factor were calculated. The main conclusions drawn are as follows:

(1) The weight calculation results of the second-level factors indicate that the weight of online course resource B1 is 0.1593, the weight of online learning ability B2 is 0.5899, and the weight of online learning environment B3 is 0.2519. It can be seen that the cultivation of online learning ability for local college students is more important than online course resources and online learning environment. Only by fully mastering the ability of online learning can college students significantly improve the quality of online teaching for teachers.

(2) In the evaluation factors under the second-level factor of online course resources B1, the order of weight ranking is C4 > C3 > C2 > C1 > C5. It can be seen that the exam scores of online learning are not very important. In order to promote the realization of educational equity, it is not only necessary to focus on improving students’ academic performance, but also to find ways to enhance students’ learning abilities. Only the improvement of learning ability can greatly enhance the development of students’ comprehensive qualities.

(3) In the second-level factor of online learning ability B2, the ranking of weights is C7 > C6 > C10 > C9 > C8. From this, it can be seen that in the process of online learning for college students, the cultivation of communication, and collaboration abilities between teachers and students is an important way to improve the quality of online learning for college students.

(4) In the second-level factor of online learning environment B3, the ranking of weights is C13 > C12 > C11 > C15 > C14. Therefore, schools need to focus on improving students’ learning equipment and learning environment during online learning, in order to promote the realization of educational equity and enhance students’ learning ability and quality.