A Comprehensive Evaluation Framework to Assess the Sustainable Development of Schools within a University: Application to a Chinese University

Abstract

Higher education institutions have long played a critical role in society. The sustainable development of universities—wherein they consistently maintain a high level of performance in teaching, research, innovation, and stewardship of talent—is increasingly viewed as critical to driving social change and building a sustainable future. Every year, many organizations publish rankings to assess and compare the performance of universities across a nation or the world. However, few of these rankings focus on the differences in performance between individual schools within a particular university, which is crucial to improving the quality of the institution as a whole. This study attempts to fill this knowledge gap by proposing a comprehensive evaluation framework to allow for systematic and standardized analysis of performance at the school/college level according to any combination of relevant indicators. The framework builds upon existing work related to the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and Grey Relational Analysis (GRA) methods and proposes an improved model that mitigates defects in previous models while enhancing interpretability and stability. An applied example in which this framework is used to evaluate schools of humanities and social sciences disciplines at Wuhan University in China is provided. The results of the applied example show that the framework enables an in-depth analysis of performance levels through multiple perspectives, thereby providing valuable insights for formulating targeted strategies to improve school performance and enhance the sustainable development of higher education institutions.

1. Introduction

With the growth in demand for a skilled workforce and the accompanying rise in individual need to receive higher education, the 20th century witnessed a rapid expansion and qualitative change in higher education. During that period, higher education developed at an unprecedented speed and transformed through three stages, from serving elites to popularization among the masses and, finally, to universalization [1]. In the present era, higher education is increasingly viewed as a key player in driving economic progress and social change [2]. As one of society’s main pillars for knowledge production and dissemination, higher education institutions are tasked with educating the workforce and training future decision-makers, thereby contributing to economic mobility and wellbeing for individuals and improved equity and productivity for nations [3,4]. At the same time, higher education institutions are often the purveyors of innovation through their research activities and can help formulate solutions, provide guidance in policy-making, and increase the overall capacity of communities to build a sustainable future [5,6].

Taking into context the important role of higher education in achieving a sustainable society, the sustainable development of higher education institutions in and of itself also emerges as a critical topic. Namely, higher education institutions must consistently maintain a high level of performance in teaching, research, innovation, and stewardship of talent in order to effectively serve their aforementioned role expected by modern society.

Since the 1980s, the practice of ranking higher education institutions has become widespread among the popular press [7]. At present, different organizations have various strategies for assessing and comparing the performance of higher education institutions across a country or the world [8,9]. Some rankings mainly use a single index, such as the Nature Index (Ni), which only considers an institution’s research productivity and impact via the number of publications in a select group of prominent natural-science journals [10]. More often, a ranking system depends on a group of indicators that includes not only achievements in research but also factors like teaching quality, the institution’s reputation and internationalization, environmental sustainability, and so on [11].

It should be noted, however, that few of these rankings focus on the differences in performance between individual schools within a particular institution. Higher education institutions—especially major universities—are usually comprised of multiple schools, the success of which is tied to the success of the institution as a whole. Therefore, if university managers want to improve the quality of the institution, a significant problem lies in how to evaluate the performance of different schools in the same university. With evaluation results in hand, university managers can be better equipped to put forward targeted improvement measures and achieve the overall sustainable development of the university.

As mentioned previously, an institution’s sustainable development level is affected by myriad factors, many of which can influence and restrict each other. Therefore, it is difficult to base a conclusion on intuition or a simple analysis of a handful of individual indicators. This paper proposes the use of a comprehensive evaluation model, the basic idea of which is to convert multiple indicators into a single comprehensive indicator in order to provide a systematic and standardized assessment system.

The comprehensive evaluation method has been widely used to solve problems in different fields. Zhao et al. [12] established a comprehensive evaluation system of Xiangtan water-saving measures using an analytic hierarchy process and grey correlation analysis. Jin et al. [13] established a gray comprehensive model to evaluate the sustainable development level of building industrialization in the Beijing-Tianjin-Hebei Region. Cheng et al. [14] proposed a fuzzy multi-criteria decision-making approach for sustainable ferry operator selection based on the fuzzy analytic hierarchy process, entropy weight method, and fuzzy technique for order preference by similarity to the ideal solution method. Wang et al. [15] used an analytic network process, entropy weight method, and fuzzy comprehensive evaluation method to construct an elastic evaluation model to assess the project safety management of the high-speed railway subgrade construction system in goaf sites. Xu et al. [16] assessed the reliability of highway bridges by applying hierarchical analysis, entropy weight, the Spearman consistency coefficient, and the Technique for Order of Preference by Similarity to Ideal Solution.

In this study, we first established a comprehensive evaluation framework for assessing the performance of individual schools within a university according to any combination of indicators. Our framework builds upon existing work, such as those mentioned previously, related to the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and Grey Relational Analysis (GRA) methods and proposes an improved model that mitigates defects in previous models while enhancing interpretability and stability. This established comprehensive evaluation framework is then applied to evaluate schools of humanities and social sciences disciplines at Wuhan University in China as a demonstrative example. This paper concludes by discussing the results and the implications of the study.

2. Comprehensive Evaluation Framework

In this section, we introduce a comprehensive model to evaluate the sustainable development level of m schools based on n selected indicators.

2.1. Model Preparation

2.1.1. Data Collection and Processing

Data collection plays a key role in building models. The accuracy and authenticity of data can make the model more reliable. The independence between indicators and the reliability of the collected data are considered in choosing the indicators and the objects.

Assuming that we have selected the data of n indicators for m schools, we can construct them into matrix $\mathit{X}=\left[x_{ij}\right]\left(i=1,2,\cdots ,m,j=1,2,\cdots ,n\right)$. The original matrix is forwarded, which means all indicator types are uniformly converted into extremely large indicators. We then standardized the forwarded matrix to matrix $Z=\left[z_{ij}\right]\left(i=1,2,\cdots ,m,j=1,2,\cdots ,n\right)$ to eliminate the influence of data dimension.

2.1.2. Weight Decision

We used the Analytical Hierarchy Process (AHP) to determine the weight of each indicator. AHP was first introduced by Professor Thomas L. Saaty in the 1970s and gradually became a multi-attribute decision-making method [17,18]. For several decades, the AHP has been applied to many fields, such as urban planning [19], education for sustainability [20], measuring punishment [21], steel casting manufacturing [22], telecommunications [23], and so on. Considering its simplicity, ease of use, and great flexibility [24], we used AHP to determine the weights of different indicators in this study.

We constructed a judgment matrix $A=\left[a_{ij}\right] \left(i,j=1,2,\cdots ,n\right)$, where $a_{ij}$ represents the importance of the indicator $u_i$ to that of $u_j$. We then calculated the maximum eigenvalue of the matrix and its corresponding eigenvector. By normalizing the calculated eigenvector, we can get the weights.

To check the consistency of the judgement matrix, we further calculated the consistency index ($CI$), which is defined as:

$$CI=\frac{{\lambda }_{max}-n}{n-1},$$

(1)

where ${\lambda }_{max}$ is the maximum eigenvalue. We can claim that the consistency is acceptable if $CI<0.1RI$, where $RI$ is the given index related to $n$. If the consistency is unacceptable, we adjust the judgment matrix.

2.2. Model Building

The Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) [25,26] and Grey Relational Analysis (GRA) [27,28] are commonly used comprehensive evaluation methods. They can reflect the distance and curve shape similarity between a vector and the positive (negative) ideal vector, respectively. TOPSIS is notable as an effective method for solving multi-objective problems—the principle is to use the relative proximity of the ideal solution as the standard of comprehensive evaluation [25,26]. GRA is useful for focusing on the similarity between sequences [27,28]. However, each of the two methods has certain defects. In the process of evaluation information aggregation, GRA can easily cause information loss, thus affecting the accuracy of results. TOPSIS is more accurate and convenient for information aggregation, but the results can be influenced by different aggregation operators. Therefore, we established an evaluation model that integrates the two methods. The steps are as follows:

We first defined the vector of the maximum and minimum values of each indicator:

$$\{\begin{array}{c}{Z}^{+}=\left(Z_1^{+},Z_1^{+},\cdots ,Z_n^{+}\right)=\left(max\left\{z_{11},z_{21},\cdots ,z_{m1}\right\},max\left\{z_{12},z_{22},\cdots ,z_{m2}\right\},\cdots ,max\left\{z_{1n},z_{2n},\cdots ,z_{mn}\right\}\right)\\ Z^{-}=\left(Z_1^{-},Z_1^{-},\cdots ,Z_n^{-}\right)=\left(min\left\{z_{11},z_{21},\cdots ,z_{m1}\right\},min\left\{z_{12},z_{22},\cdots ,z_{m2}\right\},\cdots ,min\left\{z_{1n},z_{2n},\cdots ,z_{mn}\right\}\right)\end{array}$$

(2)

The gaps from the status of the ith school to the maximum and the minimum were respectively defined as $D_i^{+}$ and $D_i^{-}$ (also called Euclidean Distance):

$$\{\begin{array}{c}{D}_i^{+}=\sqrt{\sum _{j=1}^n{\omega }_j{\left(Z_j^{+}-z_{ij}\right)}^2}\\ D_i^{-}=\sqrt{\sum _{j=1}^n{\omega }_j{\left(Z_j^{-}-z_{ij}\right)}^2}\end{array},$$

(3)

where ${\omega }_j$ is the weight of the jth indicator. A larger $D_i^{-}$ represents a smaller distance from $Z^{+}$.

We also calculated the absolute difference $\left|Z_j^{+}-z_{ij}\right|$ and $\left|Z_j^{-}-z_{ij}\right|$, which represents the difference between the evaluated object indicator sequence and the reference sequence. After defining the minimum difference $\underset{i}{\mathit{min}}\underset{j}{\mathit{min}}\left|Z_j^{+}-z_{ij}\right|$, $\underset{i}{\mathit{min}}\underset{j}{\mathit{min}}\left|Z_j^{-}-z_{ij}\right|$ and the maximum difference $\underset{i}{\mathit{max}}\underset{j}{\mathit{max}}\left|Z_j^{+}-z_{ij}\right|$,$\underset{i}{\mathit{max}}\underset{j}{\mathit{max}}\left|Z_j^{-}-z_{ij}\right|$, we calculated the correlation coefficient of each comparison sequence and the reference sequence:

$$\{\begin{array}{c}{\zeta }_{ij}^{+}=\frac{\underset{i}{\mathit{min}}\underset{j}{\mathit{min}}\left|Z_j^{+}-z_{ij}\right|+\rho ·\underset{i}{\mathit{max}}\underset{j}{\mathit{max}}\left|Z_j^{+}-z_{ij}\right|}{\left|Z_j^{+}-z_{ij}\right|+\rho ·\underset{i}{\mathit{max}}\underset{j}{\mathit{max}}\left|Z_j^{+}-z_{ij}\right|}\\ {\zeta }_{ij}^{-}=\frac{\underset{i}{\mathit{min}}\underset{j}{\mathit{min}}\left|Z_j^{-}-z_{ij}\right|+\rho ·\underset{i}{\mathit{max}}\underset{j}{\mathit{max}}\left|Z_j^{-}-z_{ij}\right|}{\left|Z_j^{-}-z_{ij}\right|+\rho ·\underset{i}{\mathit{max}}\underset{j}{\mathit{max}}\left|Z_j^{-}-z_{ij}\right|}\end{array},$$

(4)

where the resolution coefficient $\rho$ is usually set to 0.5.

The Grey Correlation Degree can then be calculated according to the above correlation coefficient:

$$\{\begin{array}{c}{r}_i^{+}=\sum _{j=1}^n{\omega }_j·{\zeta }_{ij}^{+}\\ r_i^{-}=\sum _{j=1}^n{\omega }_j·{\zeta }_{ij}^{-}\end{array},$$

(5)

where ${\omega }_j$ is the weight of the jth indicator. A larger $r_i^{+}$ represents a smaller shape in relation to $Z^{+}$.

By combining the Euclidean Distance and Grey Correlation Degree, we can avoid the defect of using a single evaluation method, as noted previously. The combined indicator was defined as:

$$\{\begin{array}{c}{\gamma }_i^{+}=\alpha \langle D_i^{-}\rangle +\left(1-\alpha \right)\langle r_i^{+}\rangle \\ {\gamma }_i^{-}=\alpha \langle D_i^{+}\rangle +\left(1-\alpha \right)\langle r_i^{-}\rangle \end{array},$$

(6)

where $x$ denotes the standardized value of $x$, $\alpha \in \left[0,1\right]$ reflects the preference for distance, and $1-\alpha$ reflects the preference for curve shape. In this study, we set $\alpha =0.5$.

Finally, we defined the comprehensive performance score of the ith school as:

$$Scor{e}_i=\varnothing \left(\frac{\langle {\gamma }_i^{+}\rangle }{\langle {\gamma }_i^{+}\rangle +\langle {\gamma }_i^{-}\rangle }\right)\times 40+60,$$

(7)

where $\varnothing \left(x\right)=\frac{x-min\left(x\right)}{max\left(x\right)-min\left(x\right)}$. In this model, the score of each school is distributed between 60 and 100. A higher score indicates a better performance and sustainable development level overall based on the selected indicators and the assigned weights.

2.3. Expanding Model Interpretability

Using the model described in Section 2.2, we can calculate the comprehensive performance score for every school with respect to its overall performance and sustainable development level. To further expand the model’s interpretability, we can evaluate the variability in performance among schools for each indicator. We defined an indicator ${\sigma }_j=\frac{{\sum }_{i=1}^m{\zeta }_{ij}^{+}}{m}\left(j=1,2,\cdots ,n\right)$, where a larger ${\sigma }_j$ suggests that the average level of all schools for an indicator is close to the highest level, and vice versa. This can help us understand in which aspects the performance of these schools is similar and in which elements the gap among them is significant.

We can also employ the hierarchical cluster method [29] to find schools with similar characteristics but varying scores. Specifically, we can use the selected indicators as inputs, using the sum of squares of deviations (Ward method [30,31]) to cluster. If the classification is appropriate, the sum of squares of deviations between samples or variables in the class should be relatively small. By applying this cluster analysis, we can identify schools that have low scores despite sharing similar characteristics with higher-scoring schools and identify which indicators low-scoring schools can focus on to improve their performance.

2.4. Stability Analysis

Considering that the selection of indicators is subjective, we tested the stability of the model. This was done by reducing the number of indicators in the model and examining how the results would change with the reduction. Specifically, we deleted the indicators one by one and calculated the mean absolute error of the score before and after subtracting the indicator. We then averaged the m obtained values. We finally sorted the vectors containing n averages from small to large and verified the robustness of the result.

3. An Application Example

In this section, we provide an applied example in which our comprehensive evaluation model is used to assess and compare the performance of schools of humanities and social sciences disciplines at Wuhan University in China. Wuhan University, established in 1893, is a major university directly under the administration of the Education Ministry of the People’s Republic of China. It was a Project 985 and Project 211 institute. Currently, it is a Class A Double First Class University. In 2022, the university ranked 157 in the Times Higher Education World University Rankings, 101–150 in the Academic Ranking of World Universities, and 194 in the QS World University ranking.

There are a total of ten schools of humanities and social sciences disciplines with available data: School of Philosophy, School of Liberal Arts, School of Foreign Languages and Literature, School of Journalism and Communication, School of History, School of Economics and Management, School of Marxism, School of Sociology, School of Political Science and Public Administration, and School of Information Management. In the following analysis, we randomly replaced the names of these schools with the letters A–J.

3.1. Indicator Selection and Refinement

When evaluating higher education, the evaluation criteria are generally self-explanatory if the indicators are given. The main problem is that different researchers usually choose different indicators. For example, Mikryukov and Mazurov [32] adopted the same strategy as the QS World University ranking. They selected six indicators, including academic reporting, reporting with the employee, the ratio of the number of students to the number of scientific and biomedical workers, cities per teacher, and international teachers and international students. Ranjan et al. [33], however, selected the faculty/student ratio, number of papers/books published, number of successful graduate and postgraduate students, number of doctoral theses completed, yearly costs related to teaching and non-teaching staff, and departmental operating costs/student in a year for evaluating. Wu et al. [34] proposed nine evaluation dimensions, including Teaching Resources, Internationalization, Extension Education Service, Discipline and Guidance, General Education, Administrative Support, Faculty, Teaching, and Research. Each dimension is controlled by several indicators.

Wuhan University publishes an archival record every year to summarize the achievements in a particular year. In this study, we used all the available indicators (28 in total) provided by the archival record for the year 2020, as shown in

Table 1. Preliminary indicators to assess the performance of schools.

Primary Index	Indicators	Abbreviation
Faculty Strength	Number of full-time teachers	FTT
	Number of professors	NPR
	Number of associate professors	NAP
	Number of nationally and internationally recognized experts	NE
Education Outcome	Number of PhD degree graduates	NP
	Number of Master’s degree graduates	NM
	Number of Bachelor’s (Undergraduate) degree graduates	NU
	Employment rate of Bachelor’s (Undergraduate) degree graduates	EU
	Employment rate of Master’s degree graduates	EM
	Employment rate of PhD degree graduates	EP
	Number of approved “college student innovation and entrepreneurship training” projects	CTP
	Number of undergraduate students receiving honors above the provincial level	UHP
	Number of graduate students receiving academic innovation awards	GAA
Discipline Development	Number of “double first-class” disciplines and nationally designated key disciplines	DD
	Number of provincially designated key disciplines	DP
	Number of “national excellence” courses and “first-class” undergraduate courses	EFU
	Number of teaching and research equipment and instruments	TRI
Talent Attraction	Number of “young talent” from the country and abroad	YTE
	Number of postdoctoral fellows	POS
Achievement in Research	Number of papers included in the top three citation index systems	PTS
	Number of published works	PW
	Number of outstanding achievement awards received from the Ministry of Education, the province, or the city	OAA
	Number of projects approved by the National Social Science Foundation or the Humanities and Social Sciences Research Division of the Ministry of Education	PNS
	Number of nationally or provincially commissioned research centers and think tanks	RC
	Sum of research funds	SRF
	Number of academic journals sponsored	AJS
Academic Exchange	Number of academic conferences and lectures	ACL
	Number of long-term foreign experts employed	FEE

. The indicators can be roughly categorized into six main aspects contributing to the sustainable development of a school: Faculty Strength, Education Outcome, Discipline Development, Talent Attraction, Achievement in Research, and Academic Exchange. These indicators can undoubtedly reflect the development of a school from multiple perspectives. Since we only give an example to show how the framework proposed in Section 2 works, the discussion of the optimal solution for indicator selection, which is an essential issue during practical application, is beyond the scope of this study.

Based on the data of the ten schools, we first calculated the correlation coefficients of the 28 indicators (Figure 1). We found a strong correlation between certain indicators. For example, if a school has a larger number of teachers, the number of students and the number of student awards are expected to be larger. To avoid the potential issue that the weight of a single indicator is too significant in the comprehensive evaluation model, we filtered the indicators according to the correlation coefficients. Specifically, an indicator was kept only if the correlation coefficients between it and other indicators were all less than 0.70. After filtering, we retained 14 indicators whose weights, as determined by AHP, are shown in

Table 2. Filtered indicators and weights.

Indicator	Weight
FTT	0.1118
NE	0.0373
EU	0.0373
EM	0.0373
EP	0.0373
GAA	0.0745
DD	0.1118
EFU	0.1491
TRI	0.1118
POS	0.0123
PW	0.0745
RC	0.0373
SRF	0.1491
FEE	0.0186

.

3.2. Comprehensive Evaluation Results & Discussion

In accordance with Equation (7), as explained in Section 2.2, we calculated the comprehensive performance scores of all ten schools included in the analysis (Figure 2). We found that School F far outperformed the other schools. Among the remaining schools, Schools B and H scored the lowest.

We also analyzed the extent of variability in the performance of schools for each indicator. We found that the employment rate of Master’s degree graduates (EM) and PhD degree graduates (EP) were relatively similar across all schools (Figure 3). In contrast, greater variability in performance was more apparent for the following four indicators: number of full-time teachers (FTT), number of graduate students receiving academic innovation awards (GAA), number of teaching and research equipment and instruments (TRI), and number of postdoctoral fellows (POS). In other words, the differences in the schools’ comprehensive performance scores largely resulted from the differences in performance in these four indicators among the schools.

Shifting our attention to low-scoring schools, we can gain further insight through a hierarchical cluster analysis of these ten schools. Taking School H as an example, we found that this school is categorized in the same cluster as Schools D and I (Figure 4). This indicates that the three schools’ development and performance may share similar patterns.

We further compared the scores of these three schools (Schools D, H, and I) in each of the 14 filtered indicators against each other and the average score of all ten schools (Figure 5). We found that School H is below average in all respects, suggesting that it has great room for improvement. Unlike School H, Schools D and I have one or two aspects where they stand out, although they are also below-average for many indicators. Namely, School D has the highest employment rate of PhD degree graduates (EP) out of the ten schools; School I ranks first both in the employment rate of Master’s degree graduates (EM) and the number of nationally and provincially commissioned research centers and think tanks (RC).

The above findings provide several hints for policies that can help improve performance and enhance the sustainable development of School H. For instance, learning directly from Schools D and I, School H can expect to raise its comprehensive score by increasing the employment rates of its graduates and establishing more nationally or provincially commissioned research centers and think tanks. University managers may take this as guidance to establish strategies aimed at achieving these objectives, such as by making career counseling resources more robust, creating new courses focused on the practical application of knowledge and skill training in line with expectations of the job market, and strengthening outreach with government agencies and policy-makers.

The final step of the analysis involved testing the stability of the model in accordance with the method detailed in Section 2.4. We found that some of the 14 filtered indicators had a greater impact on the results. However, if a certain indicator was replaced, the possible score difference was less than two, which would not significantly affect the rankings (Figure 6). Hence, the indicators we selected have wide coverage, and the evaluation results obtained through the established model have high reliability.

4. Concluding Remarks

The sustainable development of higher education institutions is critical to enhancing students’ personal development and fostering economic change and social progress. Ranking and comparing the performance of entire institutions (such as world university rankings) have long remained a hot topic, but there is little research focusing on the differences in performance among schools within a particular university. Aiming at filling this research gap, this paper proposed a comprehensive evaluation framework to assess the performance of individual schools within a university. Based on selected indicators, our framework uses the analytic hierarchy process to determine the weight of each indicator, constructs an improved TOPSIS−GRA comprehensive evaluation model, and establishes a comprehensive performance score for each school. We further provided interpretation strategies via variability and hierarchical cluster analyses and verified the reliability of the framework via a stability analysis. As an application example, the framework was used to study the performance of schools of humanities and social sciences disciplines at Wuhan University in China. The results showed that the established comprehensive evaluation framework could provide robust, interpretable, and reliable data to compare performance between different schools from multiple perspectives.

The comprehensive evaluation framework established in this paper enables an in-depth study of the performance levels of individual schools within a university. Ultimately, the results of a study based on our framework can serve as a valuable reference for university managers to make targeted suggestions to enhance the sustainable development of low-performing schools. Of course, there is still a long way to go before our framework can be genuinely applied to reality for managers to evaluate and make decisions. First, an important but neglected problem is the selection of indicators. Here, we directly used all the indicators we could access and did not consider other derived indicators. For example, the ratio between faculty and students may be more reasonable than the number of teachers. Therefore, the focus of future research is to comprehensively evaluate the impact of various indicators on the results, similar to our work in Figure 3 and Figure 6 but more comprehensive and detailed. Second, in this study, we applied the method to schools of social science and humanities. Therefore, it is vital to test the effectiveness of our model in schools of natural science or other majors. Third, the original intention of our model is to compare the development of different colleges. However, it can also be used to compare the development of the same school in different years. The data we used are from the archival record of Wuhan University for the year 2020, which is a unique year due to COVID-19. Therefore, in the future, another application prospect of this model is to evaluate the epidemic’s impact on the development of schools.