An Academic Performance Indicator Using Flexible Multi-Criteria Methods

Abstract

Composite indicators are a very useful tool for conveying summary information on the overall performance of institutions and facilitating decision-making. Increasingly, there is a demand for indicators that allow performance to be assessed after the implementation of a strategy. This has several difficulties, and in this paper, we address three of them: how to evaluate at different points in time, how to estimate the weighting of the criteria and how to normalize the data. Our proposal is based on multicriteria techniques, using a recent method, uwTOPSIS, and is applied to data collected from 2975 students enrolled in the first year of science and engineering at the Industrial University of Santander (Colombia) from the first semester of 2016 to the first semester of 2019. In the paper, we show that our proposal makes it possible to measure and evaluate the academic performance of students at two points in time, and this allows the University to know whether its student support policy has been successful and to what degree it has been effective. Due to the large amount of data handled, data management has been done using R programming language, and model implementation has been done with Python.

1. Introduction

Academic institutions, both public and private, are interested in knowing the overall student performance, as it is essential to improve management results. Some institutions apply complementary programs to achieve better performance and an efficient use of physical and human resources. Considering that performance is a multidimensional concept and that the relative importance of each dimension is not always the same, techniques must be looked for that allow us to measure the overall performance. Determining how much each variable or dimension has contributed to performance improvement (partial performance) may also be interesting.

This work aims to construct a composite indicator that measures and evaluates total performance and partial change, focusing on the design and construction of an indicator that measures academic performance and allows the evaluation of student performance at two points in time. Thus, the institution will be able to know if the policies/actions to support students have been satisfactory and detect weaknesses that can be transformed into improvements, increasing the allocation of resources and, therefore, the management of the institution. Since decisions affect future periods, an important aspect to consider is the study of the evolution of one or more dimensions at two points in time. This paper proposes a composite indicator to study the evolution of students’ academic performance when they enter university and once they have completed their first academic year.

The construction of composite indicators to synthesize information from different dimensions is very useful and allows institutional managers to make decisions. However, the construction of indicators has its pros and cons, and it is sometimes problematic to define an indicator that adequately reflects or conveys all the dimensions considered. However, if all the necessary steps to construct an indicator are fulfilled (definition of a theoretical framework, adequate selection of variables, standardization of variables if they are of different nature and are measured in different units, weighting and aggregation of criteria in an appropriate manner), indicators can be useful tools that allow summarizing information in a relatively simple way [1,2]. This is the case of the proposed performance, adequacy and academic excellence indicators, as they allow educational institutions to make decisions based on the results of these indicators.

The determination of academic performance, by its very nature, is a multidimensional concept that includes several variables or simple indicators. Multicriteria analysis techniques (MCMD) are suitable for solving problems when various criteria from a set of alternatives are obtained. These techniques are widely known and used to solve highly complex problems. In particular, it is recognized by resolving the conflict of interests posed by problems with several criteria [3], where each of these may have very different characteristics with respect to the others.

As a consequence, multicriteria decision-making methods (MCDM) have become an essential tool for agents involved in the selection of alternatives [4]. One of the methods used for its robustness is the TOPSIS method proposed by [5]. This method has been used in the area of education for sustainable development practices [6] and in previous works, in which students of the Industrial University of Santander (UIS) were characterized and in which indicators of adequacy and excellence were proposed [7,8,9,10,11], and it is commonly applied in decision making in different areas of knowledge [12].

However, this method requires knowledge of the weights of each criterion, which are not always easy to assign, and the assignment of each of its values based on the experience of the agents can be highly controversial [13,14]. It is clear that the values of each weight determine the relative importance of each criterion, but there is not always a consensus when it comes to establishing them [15,16]. Numerous researchers investigated the effect caused by the subjective assignment of these values [17,18,19,20,21] and the difficulty of determining a set of weights according to the type of problem [22,23,24,25]. However, some studies objectively analyze the selection of weights to avoid this type of problem [26]. Now, if the institutions have difficulties and cannot determine what these weights are, the uwTOPSIS method proposed by Liern and Pérez-Gladish [27] can be applied, which evaluates the different criteria of each alternative considering the weights as variables [27,28,29].

In addition, another problem to be addressed is data collection. For this reason, some approaches study the uncertainty of these problems with fuzzy techniques [30,31,32,33], while others consider the imprecision subject to data collection by defining indicators and evaluators of the results [7,34]. It should be noted that the assessment of intangible magnitudes, such as personal perspectives, is analyzed by many authors [35,36]. However, weights, normalization, metrics and ranking functions can also be decisive and significantly affect the final result.

This article presents a methodology, which can be applied to the construction of indicators in any area. However, it will be used to evaluate the academic condition of students at the Industrial University of Santander (Colombia). The Student Academic Excellence Support System (its acronym in Spanish is SEA) was created at this university to coordinate and manage programs and strategies and aims to improve the academic performance of students to reduce university dropout and failure. The institution needs to recognize those students with adverse situations that may directly affect their outcomes. The data that will be used in this work are those collected by the SEA of students entering the university in the academic, cognitive, economic, health and social dimensions before entering the university. At the end of the semester, the same data will be considered for all dimensions, except for the academic dimension, where each student’s final grades in calculus and lineal algebra will be taken into account. It is assumed that the rest of the dimensions do not vary since it is unlikely that in one semester, the economic, social, health and cognitive circumstances vary significantly. To conduct this work, data are considered from 2975 students enrolled in the first year of science and engineering from the first semester of 2016 to the first semester of 2019.

The rest of the paper is structured as follows. In the second section, how to construct the academic performance indicator will be described. Before defining it, a brief description of the steps necessary to guarantee the development of a good indicator will be given. At the end of the section, the properties of the academic performance indicator will be studied. In Section 3, a real case applied to the Industrial University of Santander (Colombia) will be presented. In Section 4, the discussion will be carried out. The paper will be completed with the main conclusions of the work.

2. Construction of the Academic Performance Indicator

In this section, a composite indicator will be constructed that allows us to relate the evaluation of the alternatives and assess performance over two time periods when only one of the criteria varies [7]. The steps to be followed to construct a composite indicator are as follows:

Step 1.

Development of the conceptual framework or theoretical model, which will allow us to identify the phenomenon to be measured, the groups involved and identify the variables.

Step 2.

Selection of the decision criteria. It is essential to identify the variables or simple indicators. Determining how to obtain the data, studying whether there are missing data and how to resolve it and analyzing the data structure is fundamental to constructing a quality.

Step 3.

Comparison of alternatives. When a composite indicator is constructed, in most cases, the units in which the variables are measured and the nature of these are very diverse [37]. To compare and aggregate variables and simple indicators, it is necessary to express them on a similar scale, i.e., normalize [38,39,40]. However, the normalization method must be carefully assessed because the results may vary depending on the method used. In this work, the method proposed by Liern et al. [9] will be used to normalize it, which will be described in more detail in Section 2.2.

Step 4.

Assignment of weights and aggregation. One of the problems widely discussed in the literature is the weighting of variables according to their relative importance [26,41,42,43]. Sometimes, the assignment is done based on expert judgment, but at other times, as we will develop in this paper, we will not assign the weights a priori but apply the uwTOPSIS method, in which it is not necessary to have the weights showing a priori the relative importance of the criteria [27].

Step 5.

Fifth and final step: Comparison of the alternatives and sensitivity analysis of the composite indicator. In this case, the total and partial performance of each student will be evaluated using the indicator proposed in this work. When an indicator is proposed, it is necessary to test its usefulness and study whether it meets the properties of the indicators [44,45], and finally, the robustness of the indicator must be determined.

In the next section, we provide a detailed description of how the academic performance indicator is obtained.

2.1. Academic Performance Indicator

Given the values of the m dimensions considered at time t, a multicriteria method is applied, and the relative proximity interval is determined for each alternative, obtaining the performance range $R_i^I$ = $[R_i^L,R_i^U]$ per each $i\in \{1,\cdots ,n\}$. If at time $t + 1$ of all dimensions considered in the previous period, only one has varied the variables that compose it, the score of the modified dimension is considered and the values obtained are normalized for each alternative using min-max normalization [40].

Once the score at time $t + 1$ has been obtained for each alternative, it is compared with the score at time t. Thus, the results for each alternative are obtained at two points of time, which allows a priori–posteriori contrast to be carried out.

Suppose an individual has an initial score $[R^L,R^U]$ and a final score x previously normalized. When comparing x with $[R^L,R^U]$, a scalar $\lambda > 0$ would be expected to exist, so $x = R^L + \lambda R^U$ since its initial interval indicates its reference framework. However, it can be that x is outside this range, which would indicate that the individual has obtained a score lower or higher than the performance range. Therefore, the next indicator is defined.

Definition 1

(Academic performance indicator). Given a set $\mathcal{D} = \{(x,R^L,R^U)\in {[0,1]}^3:R^L \le R^U\}$, the academic performance indicator is defined as $P_A:\mathcal{D}\to [0,1]$ such that

$$P_A\left(x,R^L,R^U\right) = \left\{\begin{array}{ccc}0\hfill & if& x \le R^L\hfill \\ {\displaystyle \frac{x-R^L}{R^U-R^L}}\hfill & if& R^L < x <R^U\hfill \\ 1\hfill & if& x \ge R^U\hfill \end{array}\right.$$

(1)

In the case $R^L = R^U$, the academic performance indicator is given by the characteristic function of the interval $[R^U,1]$.

Proposition 1.

Given $0 \le R^L < R^U \le 1$, for each $y\in [0,1]$ exists $x\in [0,1]$; thus $P_A(x,R^L,R^U) = y$.

Proof. 

The proof is trivial by taking $x = R^L + (R^U-R^L)y$ where it exists and $0 \le R^L + (R^U-R^L)y \le R^L + (R^U-R^L) = R^U \le 1$. □

From Proposition 1, it is concluded that the indicator $P_A$ is surjective in $\mathcal{D}\backslash \Pi$, where $\Pi = \{(x,R^L,R^U)\in \mathcal{D}:R^L = R^U\}$.

2.2. Determination of Performance Range Values

To determine the performance range $R_i^I$ = $[R_i^L,R_i^U]$, the multicriteria analysis technique uwTOPSIS will be used because unlike the TOPSIS algorithm, in which it determines the ranking of the alternatives, minimizing the distance to the positive ideal and, at the same time, maximizing the distance to the negative ideal, for weights assigned a priori, uwTOPSIS considers the weights as variables. The score obtained for each alternative is converted into a function that depends on the weights, where the minimum and maximum values are obtained by solving an optimization problem for each case. Therefore, the score for each alternative is given as an interval that describes the minimum and maximum possible score that could be obtained by varying the set of weights if they were calculated by applying the TOPSIS method. The steps of the uwTOPSIS method are described below:

Step 1.

Decision matrix is determined $\left[x_{ij}\right]$ so that $i\in \{1,\cdots ,n\}$ and $j\in \{1,\cdots ,m\}$, where n and m are the number of alternatives and criteria, respectively.

Step 2.

Decision matrix is normalized $\left[r_{ij}\right]$, where each $r_{ij}\in [0,1]$ for each $i\in \{1,\cdots ,n\}$ and $j\in \{1,\cdots ,m\}$.

Step 3.

Positive ideal and negative ideal are determined as $A^{+} = (v_1^{+},\cdots ,v_m^{+})$ and $A^{-} = (v_1^{-},\cdots ,v_m^{-})$, where

$$v_j^{+} = \left\{\begin{array}{ccc}{\displaystyle \underset{1\le i\le n}{max}{r}_{ij}}\hfill & \mathrm{si}& j\in J_{max}\hfill \\ {\displaystyle \underset{1\le i\le n}{min}{r}_{ij}}\hfill & \mathrm{si}& j\in J_{min}\hfill \end{array}1 \le j \le m\right.$$

$$v_j^{-} = \left\{\begin{array}{ccc}{\displaystyle \underset{1\le i\le n}{min}{r}_{ij}}\hfill & \mathrm{si}& j\in J_{max}\hfill \\ {\displaystyle \underset{1\le i\le n}{max}{r}_{ij}}\hfill & \mathrm{si}& j\in J_{min}\hfill \end{array}1 \le j \le m\right.$$

where $J_{max}$ is the set of criteria to be maximized and $J_{min}$ is the set of criteria to be minimized.

Step 4.

Given $\mathsf{\Omega } = \{w=(w_1,\cdots ,w_m)\in {[0,1]}^m:{\sum }_{j=1}^m{w}_j=1\}$ and d a distance defined in ${[0,1]}^m\times {[0,1]}^m$, we consider the separating functions $D_i^{+},D_i^{-}:\mathsf{\Omega }\to [0,1]$ for each of $i\in \{1,\cdots ,n\}$ as

$$D_i^{+}\left(w\right) = d\left((w_1{r}_{i1},\cdots ,w_m{r}_{im}),(w_1{v}_1^{+},\cdots ,w_m{v}_m^{+})\right).$$

(2)

$$D_i^{-}\left(w\right) = d\left((w_1{r}_{i1},\cdots ,w_m{r}_{im}),(w_1{v}_1^{-},\cdots ,w_m{v}_m^{-})\right).$$

(3)

Step 5.

The relative proximity function to the ideal solutions is defined as $R_i:\mathsf{\Omega }\to [0,1]$ for each $i\in \{1,\cdots ,n\}$

$$R_i\left(w\right) = \frac{D_i^{-}\left(w\right)}{D_i^{+}\left(w\right)+D_i^{-}\left(w\right)},1 \le i \le n.$$

(4)

Step 6.

For each $i\in \{1,\cdots ,n\}$, the values $R_i^L$ and $R_i^U$ are calculated when solving mathematical programming problems over $R_i$ considering the set of weights as the problem variables

$${\displaystyle R_i^L = \underset{1\le i\le n}{min}\left\{R_i\left(w\right):\sum _{j=1}^m{w}_j=1,l_j\le w_j\le u_j\right\},1 \le i \le n.}$$

(5)

$${\displaystyle R_i^U = \underset{1\le i\le n}{max}\left\{R_i\left(w\right):\sum _{j=1}^m{w}_j=1,l_j\le w_j\le u_j\right\},1\le i\le n.}$$

(6)

where l_j is the lower bound and u_j the upper bound within w_j, $\forall j\in \{1,\cdots ,m\}$.

Then, the interval $R_i^I$ = $[R_i^L,R_i^U]$ is obtained, which represents the worst and best possible scores for each alternative $i\in \{1,\cdots ,n\}$. However, each of these $R_i^I$ are subject to the normalization type (ϕ), separation function (D), proximity function (R) and the set of lower and upper bounds of the weights ($\mathrm{\Lambda } = {\{l_j,u_j\}}_{j=1}^m$). Thus, each $R_i^I$ can be written as $R_i^I(X_i^0,\mathrm{\Lambda },\varphi ,D,R)$ thanks to the implicit function theorem, where $X_i^0$ is the vector with the dimensions of the i-th student a priori.

In this article, in Step 2, the normalizations to be used are those proposed by [9], where the normalizations η and ξ are given by the following expressions:

$${\eta }_{A,a,b,B;k_1,k_2}\left(x\right) = \left\{\begin{array}{ccc}{\displaystyle \frac{1-e^{k_1\frac{x-A}{a-A}}}{1-e^{k_1}}}\hfill & \mathrm{if}& A \le x < a\hfill \\ 1\hfill & \mathrm{if}& a < x < b\hfill \\ {\displaystyle \frac{1-e^{k_2\frac{B-x}{B-b}}}{1-e^{k_2}}}\hfill & \mathrm{if}& b < x \le B\hfill \\ 0\hfill & & \mathrm{otherwise}\hfill \end{array}\right.$$

(7)

$${\xi }_{A,a,b,B}\left(x\right) = \left\{\begin{array}{ccc}{\displaystyle \frac{x-A}{a-A}}\hfill & \mathrm{if}& A \le x < a\hfill \\ 1\hfill & if& a < x < b\hfill \\ {\displaystyle \frac{B-x}{B-b}}\hfill & if& b < x \le B\hfill \\ 0\hfill & & \mathrm{otherwise}\hfill \end{array}\right.$$

(8)

In both normalizations, the (A,a,b,B) array represents the trapezoidal fuzzy shape, and the (k₁,k₂) coefficients are the left and right exponents that determine whether each spread is convex or concave. For a better understanding of their functionality, Figure 1 depicts how η and ξ transform the data.

In Step 3, the positive and negative ideals have been considered to be $A^{+} = (1,\cdots ,1)$ and $A^{-} = (0,\cdots ,0)$, thus having a data-independent comparative model.

The distance used in Step 4 is the Euclidean distance, and the weights of Step 6 have been considered based on expert judgment.

Remark 1.

For the particular case where the previously described uwTOPSIS method presents a set of weights Λ of the form $l_j = u_j$, $\forall j\in \{1,\cdots ,m\}$, one has an application of the classical TOPSIS method in the same way as the authors in [11]. This is because Step 6 of the unweighted method would return us the same $R_i^L = R_i^U$ for each $i\in \{1,\cdots ,n\}$.

2.3. Properties of Academic Performance Indicator

When constructing an indicator, it is necessary to make sure that it verifies properties that guarantee its usefulness [7,46,47]. In the following, the properties that verify P_A will be checked.

• Existence and determination. For every tern $(x,R^L,R^U)\in \mathcal{D}$, the function P_A is well-defined, and the value of $P_A(x,R^L,R^U)$ exists due to Proposition 1.
• Uniqueness. By the indicator’s construction, each element of its domain returns a unique value.
• Monotonicity. Given $(x,R^L,R^U)\in \mathcal{D}$ for the cases where $x \le R^L$ and $x \ge R^U$, the indicator is monotonically a constant. For the case that $R^L < x < R^U$, the gradient of the indicator can be defined as

  $${\displaystyle \nabla P_A(x,R^L,R^U) = \left(\frac{1}{R^U-R^L},-\frac{x-R^L}{{(R^U-R^L)}^2},\frac{x-R^U}{{(R^U-R^L)}^2}\right).}$$
  Therefore, the function is monotonically increasing with respect to x and monotonically decreasing with respect to R^L and R^U.
  Figure 2 shows a particular case of the behavior of the excellence indicator. Three graphs are represented where one of the variables varies and the rest remain fixed.
• Continuity. The indicator is continuous except for the case where $R^L = R^U$. This is because, for each $(x,R^L,R^U)\in \mathcal{D}$, where $R^L < R^U$ is satisfied

  $$\underset{x\to {(R^L)}^{+}}{lim}{P}_A(x,R^L,R^U) = 0\mathrm{and}\underset{x\to {(R^U)}^{-}}{lim}{P}_A(x,R^L,R^U) = 1.$$
• Decomposability. Given ${\mathcal{D}}_1,{\mathcal{D}}_2\subset \mathcal{D}$ such that ${\mathcal{D}}_1\cup {\mathcal{D}}_2 = \mathcal{D}$ and ${\mathcal{D}}_1\cap {\mathcal{D}}_2 = \varnothing$, it is easy to see $P_A = P_A{{|}_{{\mathcal{D}}_1} + P_A|}_{{\mathcal{D}}_2}$. For the case that $R^L = R^U$, the proof is trivial. Now if in each ${\mathcal{D}}_1$ and ${\mathcal{D}}_2$ is satisfied $R^L < x < R^U$, we have the same decomposition since the function has no discontinuities in $\mathcal{D}$ as is well demonstrated by the property 4.
• Normality. By definition, the indicator P_A takes values between 0 and 1, where, in addition, the same scale is guaranteed for each of the variables in $\mathcal{D}$.
• Scale invariance. Given an element $(x,R^L,R^U)\in \mathcal{D}$ and a scalar $\lambda \in \mathbb{R}$ such that $(\lambda x,\lambda R^L,\lambda R^U)\in \mathcal{D}$, so $P_A(\lambda x,\lambda R^L,\lambda R^U) = \frac{\lambda x-\lambda R^L}{\lambda R^U-\lambda R^L} = \frac{x-R^L}{R^U-R^L} = P_A(x,R^L,R^U)$.
• Translation invariance. Given an element $(x,R^L,R^U)\in \mathcal{D}$ and a scalar $\lambda \in \mathbb{R}$ such that $(x+\lambda ,R^L+\lambda ,R^U+\lambda )\in \mathcal{D}$, so $P_A(x+\lambda ,R^L+\lambda ,R^U+\lambda ) = \frac{(x+\lambda )-(R^L+\lambda )}{(R^U+\lambda )-(R^L+\lambda )} = \frac{x-R^L}{R^U-R^L} = P_A(x,R^L,R^U)$.

Having verified that the academic performance indicator obeys the stated properties and in order to contrast the a priori and a posteriori results, the academic performance indicator for the set of ternaries ${\left\{(X_i^1,R_i^L\left(X_i^0\right),R_i^U\left(X_i^0\right))\right\}}_{1\le i\le n}$ will be applied to each alternative, where $X_i^0$ and $X_i^1$ are the variables of the i-th alternative at time t and $t + 1$, respectively.

3. Application of the Academic Performance Indicator to UIS Students

In this section, the academic performance indicator will be applied to data from the Industrial University of Santander, Bucaramanga campus, Colombia. For this purpose, we have obtained information from science and engineering students enrolled from the first semester of the academic year 2016 to the first semester of 2019.

First, a global study of the results obtained by applying the academic performance indicator have been carried out, thus determining the performance of students in their first semester at university. Subsequently, the results obtained are shown when grouped by gender and economic status. Finally, to analyze student performance using the academic performance indicator, a threshold of $\alpha = 0.6$ is set. It is considered that if a student has a P_A score equal to or higher than 0.6, he/she presents an adequate academic level and meets the minimum required by the university.

3.1. Dataset

For this work, from the data provided by the SEA, 2975 students enrolled in the first year of science and engineering are selected, and the data of the academic, cognitive, economic, health and social dimensions of students who have just joined the university have been obtained [8]. The variables that compose each dimension are:

(a)

Academic dimension. This dimension is composed of three items: a diagnostic test of UIS Math (DTM); EFAI-4 numerical ability (NUA); and 11-Math Knowledge Test (PSO).

(b)

Cognitive dimension. Five items are assessed: verbal reasoning (VR), numerical reasoning (NR), abstract reasoning (ABR), memory (MEM) and spatial attitude (SA).

(c)

Economic dimension. The variables that analyze this dimension are: Income from economic dependence, ED = wage/SMMLV (where SMMLV = current legal minimum monthly wages), the number of siblings (NS), the position between siblings (PS) and the payment of rent during the course (PRC).

(d)

Health dimension. The eight variables that compose this dimension are: Anxiety (ANX), Depression (DEP), Emotional Adjustment (EMA), Alcohol Dependence (ALD), Psychoactive Substance Abuse (PSA), Chronic Illnesses (CI), Disability (DI) and “Question 23” (P23), which refers to the tendency toward suicide.

(e)

Social dimension. This is determined by family dysfunction (FAD) through the “Family APGAR” [48].

A more detailed description of the dimensions considered in the study and the weights applied in each case can be found in [7,9,10]. As the dimensions are measured on different scales (

Table 1. Dimensions, ranges, ideals and normalizations.

Dimension	Original	Transf.	Ideal	Normalization
Academic	{VL, L, LM, M, MH, H, VH}	[1, 7]	[6, 7]	${\eta }_{1,6,7,7;1,0}\left(x\right)$
Cognitive	{VL, L, LM, M, MH, H, VH}	[1, 7]	[6, 7]	${\eta }_{1,6,7,7;-1,0}\left(x\right)$
Economic	[0, 1]	[0.8, 1]	[0.8, 1]	${\xi }_{0,0.8,1,1}\left(x\right)$
Health	[0, 0.65]	[0, 0.65]	0.65	${\xi }_{0,0.65,0.65,0.65}\left(x\right)$
Social	{0.1, 0.5, 0.7, 1}	[0.1, 1]	[0.7, 1]	${\xi }_{0.1,0.7,1,1}\left(x\right)$

), the transformation proposed by the authors [9] has been used, which allows us to work with data grouped by intervals and combine different types of variables.

At the end of the semester, to study the variation in academic performance, the grades of each student in the subjects of calculus and algebra are considered and not the university entrance grades, which were considered in the academic dimension before the beginning of the semester. Since the grades are measured differently from the other dimensions, they will be normalized using the min-max normalization and the mean will be calculated.

Performance on admission to college will be determined by the interval $[R^L,R^U]$, which represents the worst and best possible score for each student upon entering university and is the result of aggregating the five dimensions. The uwTOPSIS method will be applied to obtain the interval. At the end of the semester, the value of the academic dimension will be calculated, which will be given by the X¹ value, and normalized with the minimum-maximum normalization. With the data obtained before and after the end of the first semester, the students will be classified depending on whether or not the academic dimension score is contained within the interval. The student can be considered to have had an expected performance if the final score is contained in the interval $[R^L,R^U]$, excellent if he/she has obtained a final score higher than $R^U$ and, on the contrary, an insufficient score if he/she has had a final score lower than $R^L$. From now on, the academic performance of students are distinguished as shown in the

Table 2. Type of academic performance from P_A.

	Indicator PA(X1, RL, RU)
Excellence	PA = 1
Expected	0 < PA < 1
Insufficient	PA = 0

.

In this work, the dimensions as criteria have been considered to be maximized. This will cause students with worse conditions to obtain a small value in $R^L$ and, on the contrary, those with stable conditions will have a value close to 1 in $R^U$. We have considered the positive and negative ideals to be $A^{+} = (1,\cdots ,1)$ and $A^{-} = (0,\cdots ,0)$, respectively. Finally, to prevent the method from eliminating certain criteria when optimizing, we have considered the minimum and maximum weights to be, respectively, 0.1 and 0.5.

Due to the large amount of data handled, data management has been done using the R programming language [49], and model implementation has been done with Python [50]. In particular, a repository on GitHub (https://github.com/Aaron-AALG/uwTOPSIS, accessed on 19 September 2021) to be able to apply the uwTOPSIS method has been created, as indicated in Section 2.2 [51]. For implementation purposes, Appendix A contains a script sample of the method with both η and ξ normalizations.

Remark 2.

The mathematical optimization associated to the problem has been computed with the optimize module of the SciPy (https://www.scipy.org, accessed on 19 September 2021) library. We would like to highlight that this is not a unique option, although for our particular case, it was enough. In cases when we face more complex problems, it would be convenient to develop a metaheuristic algorithm that fits with the behavior of the objective function.

3.2. Classification of Students at UIS Colombia According to Gender and Economic Status

Taking the interval values obtained with the uwTOPSIS method into account, the performance of students in each semester of the academic years 2016 to 2019 was studied. The results obtained by applying the proposed academic performance indicator (Equation (1)) on the dataset offered by the SEA show us that, on average, the academic performance is 0.6498 or higher in all the periods analyzed, except in those corresponding to the two semesters in 2018 that take values of 0.2177 and 0.2795 (

Table 3. Average of the a priori, posteriori and academic performance indicator results by semester.

Course	RL	RU	X1	PA
2016_1	0.3718	0.8540	0.8346	0.8602
2016_2	0.4081	0.8533	0.7913	0.7734
2017_1	0.4225	0.8671	0.8172	0.7814
2017_2	0.5192	0.8879	0.7733	0.6498
2018_1	0.5753	0.8988	0.5868	0.2177
2018_2	0.5080	0.8835	0.5723	0.2795
2019_1	0.5334	0.8868	0.8299	0.7560

).

Thus, for example, while in the semester of 2016_1, on average, they had an academic performance of 0.8602, or in 2017_1, the value of the indicator is 0.7814; in 2017_2, the value is reduced to 0.6498 due to the number of students who had low scores. In addition, in the two periods of 2018, there was atypical behavior due to the fact that the students’ scores when considering all dimensions were not good and also the width of the interquartile range in these periods was larger (Figure 3). One possible explanation is the national university strike in Colombia in 2018. That strike forced the institution to delay classes. Some students had to cancel the academic semester and a majority failed many subjects, among them calculus and linear algebra.

If the academic performance according to gender for all periods is analyzed, information on 2040 male and 935 female science and engineering students is considered. The average academic performance for males is 0.6211 and for females 0.6632; therefore, it is seen that the values are similar when disaggregated by gender (

Table 4. Academic performance indicator values by gender.

Group	PA	Var	N
M	0.6211	0.1555	2040
F	0.6632	0.1380	935

).

When breaking down the mean values by gender, for each semester, it can be seen that the highest scores occur in semester 2016_1, where the mean for men is 0.8571 and is 0.8667 for women. The largest differences occur in semester 2017_2, where the mean for men is 0.6149, while for women in this same period, it is 0.7195. In this case, it can be seen that the variance in the men’s group for this period is the highest with 0.1454. More detail can be seen in

Table 5. Academic performance indicator values by semester and gender.

Course	Group	PA	Var
2016_1	M	0.8571	0.0681
	F	0.8667	0.0667
2016_2	M	0.7623	0.1175
	F	0.7949	0.0919
2017_1	M	0.7708	0.1012
	F	0.8046	0.0854
2017_2	M	0.6149	0.1454
	F	0.7195	0.0974
2018_1	M	0.2125	0.0839
	F	0.2297	0.0927
2018_2	M	0.2661	0.0856
	F	0.3075	0.0796
2019_1	M	0.7452	0.1095
	F	0.7848	0.0786

. In Figure 4, it can be seen that there are no significant differences in the academic performance score by gender for the different periods studied. Therefore, it can be said that academic performance is not affected by gender for these semesters.

Next, it is intended to study how the economic condition of students who access the UIS affects academic performance. The institution classifies students according to whether their economic risk is low, medium or high. However, for the periods analyzed, students are either low risk ($E{R}_l$) or high risk ($E{R}_h$). Thus, a total of 458 students are at low economic risk, while a total of 2517 are at high economic risk, which is 84.6% (

Table 6. Students by risk group and gender.

Gender	Number of Students		Percentage of Students
	ERl	ERh	ERl	ERh
M	303	1737	14.85	85.15
F	155	780	16.58	83.42

).

If we group by economic status and gender, it can be seen that 85.15% of men and 83.42% of women belong to the high economic risk group (Table 6). Therefore, most of the students accessing UIS have a hard economic situation. It is worth mentioning that UIS is one of the most important official universities in the region, and due to its public institution, it serves students from the lowest socioeconomic strata, which is reflected in the data.

If

Table 7. Academic performance indicator values by semester per economic condition.

Course	Group	PA	Var
2016_1	ERl	0.8411	0.1000
	ERh	0.8629	0.0631
2016_2	ERl	0.8276	0.0855
	ERh	0.7663	0.1116
2017_1	ERl	0.8130	0.0748
	ERh	0.7792	0.0979
2017_2	ERl	0.7525	0.0767
	ERh	0.6270	0.1412
2018_1	ERl	0.4747	0.0739
	ERh	0.1568	0.0702
2018_2	ERl	0.4968	0.0399
	ERh	0.2200	0.0795
2019_1	ERl	0.9110	0.0360
	ERh	0.7174	0.1101

is observed, where the average performance values per semester are shown, it can be seen that in all periods, except 2016_1, the scores are higher for students with low economic risk. In the 2016_2 and 2017_1 periods, the scores between the two groups do not differ by more than 0.06 points. From this period onwards, the differences are significant, reaching the maximum difference in 2018_1 as reflected in Figure 5.

The intention is to compare the results of academic performance with P_A taking into account whether they have received actions or not. If

Table 8. Academic performance indicator by semester and participation.

Course	No Participation		Participation
	PA	Var	PA	Var
2016_1	0.8977	0.0488	0.7982	0.0927
2016_2	0.9080	0.0386	0.6924	0.1336
2017_1	0.8004	0.0950	0.7641	0.0972
2017_2	0.6509	0.1352	0.6492	0.1297
2018_1	0.2030	0.1262	0.2182	0.0855
2018_2	0.6229	0.1744	0.2654	0.0756
2019_1	0.7020	0.1494	0.7577	0.0999

is observed, it can be seen that in all semesters, except in 2019_1, the mean value of the P_A indicator is higher for students who have not received complementary actions, i.e., those who had a high score in the academic dimension when they accessed the university. The differences that exist between the two groups of students in semester 2016_2 should be highlighted, where the average academic performance for those who did not participate in any action was 0.9080 and those who participated in some action was 0.6924. In 2018_2, the mean academic performance for those who did not participate in any action was 0.6229 and those who participated in any action was 0.2654. This may be due to the fact that in some semesters, the political situation in the country led to class stoppages during part of the semester and the suspension of complementary actions. In this way, students with greater needs ceased to have some support actions.

The percentages of students appearing in each group (Excellence, Fail or Expect) show disparate results depending on the semester analyzed, but in all of them, students who do not participate in actions present a higher percentage of Excellence (

Table 9. Percentage of students according to whether or not they have participated in the support program.

	Percentage of Students
	Course	Excellence	Fail	Expected	α ≥ 0.6
No Participation	2016_1	62.62	2.49	34.89	90.97
	2016_2	64.62	1.54	33.85	90.77
	2017_1	49.62	6.77	43.61	79.32
	2017_2	28.43	14.21	57.36	62.94
	2018_1	8.33	41.67	50.00	16.67
	2018_2	38.46	23.08	38.46	61.54
	2019_1	46.67	13.33	40.00	66.67
Participation	2016_1	40.21	7.22	52.58	81.44
	2016_2	39.81	12.04	48.15	67.59
	2017_1	39.38	6.16	54.45	76.37
	2017_2	27.51	13.59	58.90	64.08
	2018_1	3.00	49.50	47.50	13.00
	2018_2	1.26	40.69	58.04	13.56
	2019_1	38.84	8.37	52.79	77.25

). For example, in 2016_2, the percentage of Excellence among students who did not participate in any action was 64.62%, while that of those who participated in at least one was 39.81%. Surprising again, of the few students who have an Excellence score in the results obtained for the 2018_1 and 2018_2 semesters, 8.33% and 38.46% did not participate in complementary actions and 3% and 1.26% did participate. Above all, the percentage of students in the Fail situation, 41.67% and 23.08% in the group of those who do not participate and 49.50% and 40.69% of those who have participated in some complementary action, is surprising in these periods.

Next, the data of the students who have obtained a value of $P_A \ge 0.6$ will be analyzed. According to the UIS criteria, we consider that from this value, they present an adequate academic performance. In all periods, the percentage was higher than 61%, except in 2018_1, where the percentage of those who did not receive complementary actions was 16.67% and of those who received was 13%, and in 2018_2, where the percentage of students who received complementary actions was 13.56%. If the results are compared for each semester, it is seen that in all periods, the percentages are higher for students who do not receive complementary actions, except in 2019_1, where students who do not participate account for 66.67% and those who participate account for 77.25%. In any case, what can be observed is that the complementary actions allow a high percentage of students to be above the proposed threshold. Obviously, our intention goes beyond comparing groups. The most important thing is to prevent students in vulnerable situations from being unable to continue university studies. With these actions, a high percentage of students achieve the expected results and reduce the dropout rate, which is a success for the institution.

If the groups by gender are analyzed (

Table 10. Gender of the students that did not participate in the support program (NP) and the students that did participate, at least once, in the support program (P).

	Group	Percentage of Students
		Excellence	Fail	Expected	α ≥ 0.6
NP	M	47.07	8.24	44.69	76.55
	F	56.98	5.04	37.98	86.43
P	M	24.20	23.70	52.09	51.88
	F	26.14	17.58	56.28	55.54

), it can be seen that the results are not very different between men and women. Both in the groups that have not participated in complementary actions and in those that have participated in at least one, women obtain better results. The percentage of women with an Excellence score among those who do not participate in any action is 56.98%, while that of men is 47.07%. The same is true for those who participate in any complementary actions. The percentage of women is 26.14%, while that of men is 24.20%. If we focus on the percentage of students by gender in the case of $\alpha \ge 0.6$, we see that the pattern is repeated. In the case of females who do not participate in any action, the percentage is 86.43% compared to 76.55% for males. The same is true for students participating in complementary actions (55.54% of females have a value of P_A higher than 0.6, while the percentage of males is 51.88%).

Regarding the study of the economic conditions of the students, a higher percentage of excellent students is observed in the groups that have not participated in the SEA support program (

Table 11. Economic risk of the students that have not participated in the support program (NP) and the students that have participated, at least once, in the support program (P).

		Percentage of Students
		Excellence	Fail	Expected	α ≥ 0.6
NP	ERl	55.29	1.18	43.53	84.71
	ERh	49.38	7.96	42.66	78.86
P	ERl	32.98	3.49	63.54	61.93
	ERh	23.06	25.69	51.26	51.14

). Thus, 55.29% of students who have not participated in complementary actions and who have a low economic risk are excellent versus 32.98% of those who have participated in some action. Among those with high economic risk, $E{R}_h$, and of those who have not participated in complementary actions, 49.38% are excellent versus 23.06% who have participated in at least one action. In any case, of those students who have participated in complementary actions, 61.93% of those with a low economic risk, $E{R}_l$, and 51.14% of those with a high economic risk, $E{R}_h$, have obtained an academic performance indicator score of P_A above 0.6. Given that more than half of the students who participated in the complementary actions organized by the SEA have had a score higher than the threshold set as success of the strategy, it is justified to recommend that the UIS continues with the actions carried out and implement some more so that the success rate is close to that of the students who do not have support needs.

4. Discussion

We do not want to finish this work without including a discussion about the construction and use of indicators and the application of a MCDM method, the uwTOPSIS, where it is not necessary to assign the weights a priori.

An indicator of academic performance has been constructed that lets us see the evolution of students in two moments of time, comparing the final value with the initial score obtained using the uwTOPSIS multicriteria method. As alternatives, a lower bound and an upper bound are considered, which allows constructing an interval $R_i = [R_i^L,R_i^U]$ that represents the worst and the best possible score. In addition, comparing the information obtained by applying the uwTOPSIS at a time t with the data obtained at time $t + 1$ for the same dimension but considering different variables, we can construct an indicator of academic performance that determines whether the performance of a student has worsened, maintained or improved.

The main use of the uwTOPSIS method is that it considers the weights as variables and allows the evaluation of the alternatives without prior assignment of the weights. It shows an appreciable improvement since in previous works using data from the Universidad Industrial de Santander, the allocation of weights was made based on expert judgment. This sometimes implied difficulties in reaching a consensus, and consequently, studies were not fully implemented in the strategy. Undoubtedly, uwTOPSIS allows progressing and improving the procedures we have applied since it has several advantages:

• An interval associated with the worst and best possible results of each student is obtained, which allows knowing their aspirations.
• Results dependent on the data handled have been avoided (rank reversal [40]), using $(1,1,\cdots ,1)$ and $(0,0,\cdots ,0)$ as ideal and anti-ideal, respectively.
• A standardization has been applied that allows taking into account the criteria of the institution, which makes the future applicability of the results easier.

As happens in all studies that are used in real problems with a large amount of data and involving several decision-makers, our proposal does not resolve some issues:

• In this study, stability/regularity in the data has not been taken into account. As it was said in the different sections, there are periods with clear differences with respect to the others, and this could require an in-depth study of the homogeneity between periods. However, once the results are known, it is clear that these differences should be taken into account in a subsequent study.
• The intervals assigned to the weights, although they make the results more flexible and objective, determine (in some way) the groupings. This could be avoided by allowing weights that could span the entire interval $[0,1]$. However, this would not make complete sense in a multicriteria environment. According to [52], a weight greater than 50% in any of the criteria in a decision context with more than two criteria is not sustainable with the multicriteria character.

The choice of the TOPSIS method is based on the conclusions of Vicente et al. [9] since it was proved that this technique appropriately fit with the SEA-UIS dataset. Moreover, the approach of unweigthed MCDA techniques is a field that we must research in depth.

We also consider the use of machine learning algorithms for future works, in which we could implement supervised learning techniques to estimate future academic performance or unsupervised learning techniques for cluster analysis. Nonetheless, in this paper, we have not applied such techniques because we have followed the UIS guidelines in terms of the data preparation and both evaluation and classification of their students.

Finally, there are a few difficulties involved in applying some methods in practice due to the associated computational cost of solving uwTOPSIS; when calculating uwTOPSIS for 2975 alternatives and 5 criteria, a decision matrix with 14,875 entries is made. When calculating the optimal solutions of the relative proximity function (R), a total of 5950 mathematical optimization problems are performed to obtain each $[R^L,R^U]$. With all this, our code has been compiled [51] on a computer with AMD Ryzen 7 3700U with Radeon Vega Mobile Gfx processor and 16 GB of RAM, obtaining a computational cost of 4651.98 s (1 h, 17′ and 31.98′′) to run the code in full, which is 1.27 iterations per second. However, when adding the Python Joblib module (https://github.com/joblib/joblib, accessed on 19 September 2021) to execute parallel computation, the computational cost was 1629.83 s (27′ and 9.83′′) when using 8 CPUs simultaneously, which is 3.65 iterations per second.

5. Conclusions

By constructing composite indicators that incorporate all the variables and meet a series of indicator properties, it is possible to synthesize the information and draw simple and rapid conclusions that help institutional decision-makers. However, the proper construction of indicators requires overcoming or resolving several obstacles. Once the standardization problems have been solved by applying the proposal made by Liern et al. (2020), one of the pending issues is the assignment of weights. The objective of this work is to propose a composite indicator in which the weights are considered variable. Taking these aspects into account, the proposed indicator, called the academic performance indicator, allows academic institutions to obtain the value of the indicator for each student and to know whether or not there has been an improvement in the results.

The use of the proposed indicator allows the institution to know whether the complementary actions implemented have been successful or not since at the beginning of the semester, based on the available data of the five dimensions analyzed, it is determined which students can participate in the actions promoted by the institution. At the same time, global data on the students are provided, and groups can be established according to the result of the academic performance calculated from the indicator.

The proposed method was applied to the data obtained from the students of the Industrial University of Santander (Colombia) to draw conclusions that can be applied to decision making. The UIS committed, investing economic and human resources, to offering complementary actions and wants to determine whether this decision is adequate or needs to be modified. One way to determine this is by calculating the academic performance indicator, so if it improves or is maintained, the investment is adequate; if not, the university will have to reinforce the proposed activities, determine other alternatives or even change the policy carried out so far.

By applying the method to the UIS data, it is verified that there are no significant differences in academic performance when calculating the indicator according to gender; however, these differences are greater if the economic condition of the students is taken into account. However, by applying complementary actions, the institution guarantees an improvement in the results of students who presented deficiencies at the beginning of the semester, improving academic results and reducing the dropout rate.

This work is expected to continue deepening the construction of indicators that allow the UIS to have a global vision of the results of the policies implemented in 2014, the year in which the SEA was founded and actions to help students with academic and economic problems began to be implemented. It is also intended that the use of indicators will allow comparisons with other universities and facilitate other universities or local governments into the design of student curricula and the implementation of educational policies.