Construction of Quality Indicators Based on Pre-Established Goals: Application to a Colombian Public University

Abstract

This study creates indicators of adequacy and excellence based on multiple-criteria decision-making (MCDM) methods and fuzzy logic. The calculation of indicators presents two main difficulties: The nature of the data (numerical, interval, and linguistic values are mixed) and the objective of each criterion (which does not have to reach either the maximum or the minimum). A method is proposed, based on similarity measures with predetermined ideals, that is capable of overcoming these difficulties to provide easy-to-interpret information about the quality of the alternatives. To illustrate the usefulness of this proposed method, it has been applied to data collected from students across nine semesters at the Bucaramanga campus of the Industrial University of Santander in Colombia. This case study demonstrates that the proposed method can facilitate strategic decisions at an institution and open the way for the establishment of action policies regarding gender inequality and economic disparity, among other things.

1. Introduction

Complexity in decision-making often forces institutions to find a solution that considers multiple criteria with objectives that may conflict with each other [1]. A significant part of an institution′s competitive advantage lies in its ability to find a consensual solution [2,3]. Among the many areas of application of the multiple-criteria decision-making (MCDM) methods [1], in our work, the evaluation and identification of a company’s ability to compare their performance with that of their competitors [4], the process of personnel selection [5], and the analysis of students′ preferences in the choice of the university where they want to carry out their studies [6,7] are particularly useful.

It is essential for institutions to be aware of the adequacy of their staff and their ability to undertake a task. This requires strategic actions involving many people and usually depends on the coordinated actions of several employees from different departments, along with the adequate allocation of resources. Knowing the degree of compliance with this strategic plan, based on indicators capable of handling various criteria, the achievement of objectives can substantially increase. In short, this is achieved using decision support systems (DSS) based on MCDM methods [8].

The majority of DSS assume that the best outcome for each criterion (separately) is to try to reach the maximum or minimum value, depending on the specific case [9]. However, in many real cases, this is not the best outcome [10]; for example, in health-related models, intermediate values are often desirable, such as for weight and blood pressure. In addition to this, managers encounter other difficulties when calculating their estimates, as follows:

• The values come from several sections that use their own criteria and are expressed in different ways [11,12].
• The scales are diverse, and their sizes are extremely heterogeneous, which does not allow aggregated operators to be applied directly [12,13,14].
• The pre-established objectives for each criterion are future estimates, which are not usually precise [15].

Given this situation, it is necessary to determine appropriate indicators for each section so that the complete assessment is as faithful as possible to the actual performance of the activity, and the results are expressed clearly. In this work, we use a method that seeks to standardize and normalize the data by comparing them with the ideal objectives set by the institution. It is not only about transferring all the data to the interval [0, 1], but the intention is also to express the similarity of each value with the ideal.

Following this process, it is possible to both define a synthetic indicator that provides a global value for each alternative and to estimate its quality (Figure 1). To show the usefulness of the method, a real example is studied using data from students enrolled at the Industrial University of Santander in Colombia (its acronym in Spanish is UIS).

For almost five years, since the UIS created the Student Academic Excellence Support System (its acronym in Spanish is SEA) to coordinate and manage programs and strategies, five types of data have been collected from students enrolled at the university: economic, social, health, academic, and cognitive [16]. This is a clear example where the measurement methods, the participants in the evaluations, and even the managers of the initiative have changed; therefore, exclusively using numerical indicators does not guarantee success. The instruments used need to be sufficiently flexible to provide plausible results [17].

Once the students have been assessed in a convincing manner, they can then be subdivided into three groups, according to whether they show significant, an average amount, or no difficulty in carrying out their studies. The proposed method accounts for students in different groups and aims to determine how this can affect the management of the institution in order to achieve its objective—Establishing adequate and excellent indices that allow the entity to make strategic decisions.

This study used data from UIS students enrolled from 2014 to the first semester of 2018. It confirmed that, in the analysis period, between 65% and 73% of the students were considered to have an adequate ability to carry out their studies successfully, while the excellence index did not exceed 20%. This means that the UIS should provide additional support to approximately one-third of the enrolled students and plan activities that promote equal opportunities and high performance for one or two out of every 10 students.

2. Homogenization and Normalization of the Data Based on the Similarity Measures

Depending on the type of criteria analyzed, an evaluation using real numbers, intervals, linguistic evaluation, or finite collections of linguistic evaluations may give a more realistic result [12]. In this section, we show how to transform a set of data extracted from a finite and totally ordered discrete term (or numerical) set into real numbers or real intervals. Once transformed, we normalize the data in the interval [0, 1], taking into account the similarity of each of them with a pattern fixed a priori [12,13].

2.1. Data Homogenization

The use of intervals in decision-making processes is very common and does not usually add many operational difficulties to the use of real numbers [13,18]. However, the use of linguistic data usually requires the use of previous processes for their conversion into numbers. For that reason, it is necessary to specify the treatment that will be carried out on them. Considering the work of Yager [19], Herrera and Martínez [20,21], and Xu [12,22], it can be assumed that a linguistic evaluation scale is a set

$$l_1=\left\{s_{\alpha }:\alpha \in \left\{1,2,\dots ,H\right\}\right\},$$

(1)

which must satisfy the following conditions:

(1)

The set is ordered: $s_{\alpha }>s_{\beta }$ if $\alpha$ > $\beta$;

(2)

There is a negation operator: neg($s_{\alpha })=s_{\beta }$ such that $\beta =H+1-\alpha$;

(3)

There are max. and min. operators: max($s_{\alpha },s_{\beta }$) = $s_{\alpha }$ if $\alpha \ge \beta$, and min($s_{\alpha },s_{\beta }$) = $s_{\alpha }$ if $\alpha \le \beta .$

Given $\left\{s_1,s_2,\dots ,s_p\right\},$ a collection of elements of $l_1$, where $s_1\le s_2\le \dots \le s_p,$ according to [12], we can express the collection as an interval $[s_1,s_p]$. Formally, a linguistic interval evaluation scale can be expressed as [12]

$$l_2=\left\{\tilde{s}=[s_{\alpha },s_{\beta }]:\alpha \le \beta ,\alpha ,\beta \in \left\{1,2,\dots ,H\right\}\right\}.$$

(2)

In $l_2,$ we can also establish an order. Let ${\tilde{s}}_1=[s_{{\alpha }_1},s_{{\beta }_1}]$ and ${\tilde{s}}_2=[s_{{\alpha }_2},s_{{\beta }_2}]$ be two elements of $l_2,$ and let $len({\tilde{s}}_1)={\beta }_1-{\alpha }_1,len({\tilde{s}}_2)={\beta }_2-{\alpha }_2.$ According to Xu [12,22], the degree of possibility of ${\tilde{s}}_1\ge {\tilde{s}}_2$ is obtained as

$$p({\tilde{s}}_1\ge {\tilde{s}}_2)=\frac{max\left(0,len({\tilde{s}}_1)+len({\tilde{s}}_2)-\max \left({\beta }_2-{\alpha }_1,0\right)\right)}{len({\tilde{s}}_1)+len({\tilde{s}}_2)}.$$

(3)

To put the sets in $l_1$ and $l_2$ into arithmetic form and to convert the elements of $l_2$ into intervals of real numbers and to provide an arithmetic form in the sets $l_1$ and $l_2$, these sets are extended to continuous scales as follows [12,20,22]:

$$L_1=\left\{s_{\alpha }:\alpha \in \left[1,H\right]\right\},$$

(4)

$$L_2=\left\{\tilde{s}=[s_{\alpha },s_{\beta }]:\alpha ,\beta \in \left[1,H\right],\hspace{1em}\alpha \le \beta \right\}.$$

(5)

If the data are to be contained in a linguistic evaluation scale, $l_1=\left\{s_{\alpha }:\alpha \in \left\{1,2,\dots ,H\right\}\right\},$ as in (2), in many situations, it is more appropriate to transform linguistic variables into intervals instead of real numbers [20,23]. Taking into account the information presented in [23], given the linguistic scale $l_1$ with H values, we make a partition of the interval $\left[1,H\right]$ into H disjoint sub-intervals, ${\overline{r}}_{\alpha },\alpha =1,2,\dots ,H,$ such that $\sup {\overline{r}}_{\alpha }\le \inf {\overline{r}}_{\alpha +1},\alpha =1,2,\dots ,H-1.$ If we make

$$T\left(s_{\alpha }\right)={\overline{r}}_{\alpha },\hspace{1em}\alpha =1,2,\dots ,H,$$

(6)

we can transform each term of $l_1$ into an interval contained in [0, H].

Taking into account all the cases we have mentioned in this section, the data we are going to handle can be converted into real numbers (Type I) or into real intervals (Type II).

Table 1. Types of data.

Data	Original Data	Transformation
Type I (real numbers)	$(\mathrm{a})v\in ℝ$	$v\in ℝ$ (or $\left[v,v\right]\subset ℝ$)
	$(\mathrm{b})v\in S$	$v\to \mathrm{r}\in ℝ$ (or $\left[r,r\right]\subset ℝ$)
Type II (intervals in $ℝ$)	$(a)v=\left[v^L,v^R\right]\subset ℝ$	$v\subset ℝ$
	$(b)v=\left\{v^1,v^2,\dots ,v^p\right\}\subseteq S$	$v\to \left[1,p\right]\subset ℝ$
	$(\mathrm{c})v\in S$	$v\to {\overline{r}}_{\alpha }\subset ℝ$

shows the types of data that we will use in a schematic way.

2.2. Normalization Based on an Ideal

In Cables et al. [10] and Acuña-Soto et al. [11], the normalization of each x data contained in a range $\left[A,B\right]\subset ℝ$ is proposed by measuring the similarity with an ideal fixed by the interval $\left[a,b\right]\subset \left[A,B\right].$ The resulting normalization function is as follows:

$${\mu }_{\left(A,a,b,B\right)}(x)=\{\begin{array}{ll}\frac{x-A}{a-A},\hfill & A\le x<a\hfill \\ 1,\hfill & a\le x\le b\hfill \\ \frac{B-x}{B-b},\hfill & b<x\le B\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}$$

(7)

The normalization can be extended to data intervals [11,24]. Given $\left[p,q\right]\subset \left[A,B\right],$ the normalization is performed by measuring the similarity with $\left[a,b\right]$ by means of the Manhattan distance,

$$d\left(\left[p,q\right],\left[a,b\right]\right)=\frac{1}{2\left(B-A\right)}\left(\left|p-a\right|+\left|q-b\right|\right);$$

that is,

$$N\left(\left[p,q\right]\right)=\{\begin{array}{ll}{\mu }_{\left(A,a,b,B\right)}\left(p\right),& \mathrm{if}p=q\\ 1-\frac{1}{2\left(B-A\right)}\left(\left|p-a\right|+\left|q-b\right|\right),& \mathrm{if}p\ne q.\end{array}$$

(8)

The normalization method proposed in (10) is very useful, especially for applying multicriteria optimization methods [25], but the goal is to extend these methods in order to broaden their field of application.

To this end, two inconveniences of (8) need to be avoided:

(a)

Linearity cannot always be assumed for the manner of approaching or moving away from the ideal. The effect of having values below the ideal may be very different from that of having values greater than the ideal [26].

(b)

The normalization proposed in (10) is not very sensitive to the position of the interval $\left[p,q\right]$ with respect to the ideal $\left[a,b\right].$

In order to reduce these drawbacks, we build functions that take into account the range mentioned $\left[A,B\right]$ (it is the support of the function), the ideal [a, b], and the way to approach this ideal. To do this, given six real numbers A, a, b, B, k₁, and k₂, such that $A\le a\le b\le B$, we define the function ${\eta }_{\left(A,a,b,B;k_1,k_2\right)}\left(x\right):ℝ\to \left[0,1\right]$, given by

$${\eta }_{(A,a,b,B;k_1,k_2)}(x)=\{\begin{array}{ll}\frac{1-e^{k_1\frac{x-A}{a-A}}}{1-e^{k_1}},& A\le x<a\\ 1,& a<x<b\\ \frac{1-e^{k_2\frac{B-x}{B-b}}}{1-e^{k_2}},& b<x\le B\\ 0,& \mathrm{otherwise}.\end{array}$$

(9)

The values k₁ and k₂ determine the way to approach or leave from the ideal—for positive values, it is convex and for negative values, it is concave (see Figure 2). In addition, the higher the absolute values of k₁ and k₂, ${\eta }_{\left(A,a,b,B;k_1,k_2\right)}\left(x\right)$, the further we move away from the linear functions $f\left(x\right)=\frac{x-A}{a-A}$ and $g\left(x\right)=\frac{B-x}{B-b}$, which appear in (7).

Remark 1. 

When A = a (resp. b = B) we say that ${\eta }_{\left(A,a,b,B;k_1,k_2\right)}\left(x\right)$is degenerated. Using the same notation as in (9), we write k₁ = 0 (resp. k₂ = 0). With this, the functions${\eta }_{\left(A,A,b,B;0,k_2\right)}\left(x\right)$,${\eta }_{\left(A,a,B,B;k_1,0\right)}\left(x\right)$are as follows:

$${\eta }_{\left(A,A,b,B;0,k_2\right)}\left(x\right)=\{\begin{array}{cc}1,& A\le x<b\\ \frac{1-e^{k_2\frac{B-x}{B-b}}}{1-e^{k_2}},& b<x\le B\\ 0,& \mathrm{otherwise},\end{array}\hspace{1em}{\eta }_{\left(A,a,B,B;k_1,0\right)}\left(x\right)=\{\begin{array}{cc}\frac{1-e^{k_1\frac{x-A}{a-A}}}{1-e^{k_1}},& A\le x<a\\ 1,& a\le x\le B\\ 0,& \mathrm{otherwise}.\end{array}$$

(10)

As in ${\mu }_{\left[A,a,b,B\right]}\left(x\right),$ when the similarity to the ideal is piecewise linear, with the function ${\eta }_{\left[A,a,b,B;k_1,k_2\right]},$ we can normalize the data if they are real numbers [11]. However, to be able to normalize interval data, we need some previous results.

Proposition 1. 

Given a continuous function $f:ℝ\to \left[0,1\right]$and a non-empty interval$\left[p,q\right]\subset ℝ$, if $p<q$, then

$$n\left(\left[p,q\right]\right)=\frac{1}{q-p}{\int }_p^{q}f\left(x\right)dx.$$

(11)

In addition, $n\left(\left[p,q\right]\right)\in \left[0,1\right]$ and $c\in \left[p,q\right]$ exist such that $f\left(c\right)=n\left(\left[p,q\right]\right).$

Proof. 

The f function is continuous and restricted to $ℝ$; therefore, for any non-empty interval $\left[p,q\right]\subset ℝ$, f is Riemann integrable in $\left[p,q\right]$. In addition, since $p\ne q,$ $n\left(\left[p,q\right]\right)$ exists. As the function f satisfies $0\le f\left(x\right)\le 1,x\in ℝ$ through the mean value theorem for integrals [27] in $\left[p,q\right]$,

$$0\left(q-p\right)\le {\int }_p^{q}f\left(x\right)dx\le 1\left(q-p\right),$$

and $n\left(\left[p,q\right]\right)\in \left[0,1\right].$ In addition, the same theorem guarantees the existence of $c\in \left[p,q\right]$, satisfying $f\left(c\right)=n\left(\left[p,q\right]\right).$ □

Definition 1. 

Let$f:ℝ\to \left[0,1\right]$be a continuous function. The value$n\left(\left[p,q\right]\right)$given in (11) is called the normalization of the interval $\left[p,q\right]$ with respect to the function$f.$

In fact, $n\left(\left[p,q\right]\right)$ can be extended to the case $p=q,$ defining

$$n\left(\left[p,q\right]\right)=\{\begin{array}{ll}f\left(p\right),& \mathrm{if}p=q\\ \frac{1}{q-p}{\int }_p^{q}f\left(x\right)dx,& \mathrm{if}p\ne q.\end{array}$$

(12)

Remark 2. 

If, for operational reasons, it is preferable to avoid the calculation of the integral, the normalization could be approximated, for example, using the Simpson method [27]:

$$n\left(\left[p,q\right]\right)=\frac{1}{q-p}{\int }_p^q\mu \left(x\right)dx\approx \frac{1}{6}\left[\mu \left(p\right)+4\mu \left(\frac{p+q}{2}\right)+\mu \left(q\right)\right].$$

If $\mu$ is a piecewise function, defined for example in $\left[p,q_1\right],\left[q_1,p\right]$, so that $\left[p,q\right]=\left[p,q_1\right]\bigcup \left[q_1,p\right]$, the normalization can be approximated by

$$n\left(\left[p,q\right]\right)\approx \frac{q_{1-}p}{6\left(q-p\right)}\left[\mu \left(p\right)+4\mu \left(\frac{p+q_1}{2}\right)+\mu \left(q_1\right)\right]+\frac{q-q_1}{6\left(q-p\right)}\left[\mu \left(q_1\right)+4\mu \left(\frac{q_1+q}{2}\right)+\mu \left(q\right)\right].$$

Also, if the function is linear in pieces, the approximation is exact.

Taking into account Definition 1, we can normalize data with respect to the function ${\eta }_{\left(A,a,b,B;k_1,k_2\right)}\left(x\right)$. In the following example, we illustrate how this normalization works and compare it with the normalization N given in (8).

Example 1. 

The range of a data set is considered to be$\left[A,B\right]=\left[0,10\right],$and the ideal is $\left[a,b\right]=\left[6,7\right]$. We assume that, according to our information, the similarity with $\left[a,b\right]$can be estimated by means of ${\eta }_{\left(0,6,7,10;-2,1.5\right)}$, i.e.,

$$\eta (x):={\eta }_{\left(0,6,7,10;-2,1.5\right)}(x)=\{\begin{array}{ll}\frac{1-e^{\frac{-2\left(x-1\right)}{5}}}{1-e^{-2}},& 0\le x<6\\ 1,& 6\le x\le 7\\ \frac{1-e^{\frac{1.5\left(10-x\right)}{3}}}{1-e^{1.5}},& 7<x\le 10\\ 0,& \mathrm{otherwise}.\end{array}$$

We compare the normalizations provided by N and n (given in (8) and (12), respectively) for the data v₁ = 4, v₂ = [4, 9], and v₃ = [8, 9].

As can be observed in Figure 2, there must be differences between the two normalizations. For $v_1$, we have $n\left(v_1\right)={\eta }_{\left(0,6,7,10;-2,1.5\right)}\left(4\right)=0.808$ and $N\left(v_1\right)={\mu }_{\left(0,6,7,10\right)}\left(4\right)=0.6$.

If we calculate the normalization of v₂ = [4, 9] and v₃ = [8, 9] using N, we have

$$N\left(v_2\right)=1-\frac{1}{20}\left(\left|4-6\right|+\left|9-7\right|\right)=\frac{4}{5},\hspace{1em}N\left(v_3\right)=1-\frac{1}{20}\left(\left|8-6\right|+\left|9-7\right|\right)=\frac{4}{5}.$$

In other words, v₂ and v₃ are transformed into the same value. However, the relationships of $\left[4,9\right]$ and $\left[8,9\right]$ with the ideal are very different: $\left[a,b\right]\subset \left[4,9\right]$, whilst $\left[a,b\right]{\displaystyle \cap }\left[8,9\right]=\varnothing .$ For this reason, if the actual situation requires it, it may be important to distinguish between both situations.

If we normalize with n given in (12), we obtain

$$n\left(v_2\right)=\frac{1}{9-4}\left[{\int }_4^6\frac{1-e^{\frac{-2\left(x-1\right)}{5}}}{1-e^{-2}}dx+{\int }_6^{7}1dx+{\int }_8^9\frac{1-e^{\frac{1.5\left(10-x\right)}{3}}}{1-e^{1.5}}dx\right]=0.632133,$$

$$n\left(v_3\right)=\frac{1}{9-8}{\int }_8^9\frac{1-e^{\frac{1.5\left(10-x\right)}{3}}}{1-e^{1.5}}dx=0.327175.$$

Clearly, $n\left(\left[8,9\right]\right)<n\left(\left[4,9\right]\right)$, i.e., n reflects the reality in a more accurate way than the normalization N.

Remark 3. 

Note that, if in Example 1, linearity is assumed instead of the function ${\eta }_{\left(0,6,7,10;-2,1.5\right)}\left(x\right)$, we can normalize it by making $f\left(x\right)={\mu }_{\left(0,6,7,10\right)}\left(x\right)$in the expression (12). In this case, the normalization of v₂ = [4, 9] and v₃ = [8, 9] results in

$$n_{*}\left(\left[8,9\right]\right)=\frac{1}{9-8}{\int }_8^9\frac{10-x}{3}dx=0.5.$$

$$n_{*}\left(\left[4,9\right]\right)=\frac{1}{9-4}\left[{\int }_4^6\frac{x-1}{5}dx+{\int }_6^{7}1dx+{\int }_8^9\frac{10-x}{3}dx\right]=0.62.$$

It is clear that even in the linear case, the normalization of intervals by means of Definition 1 is better adjusted to the similarity with the ideal since it distinguishes clearly between v₂ = [4, 9] and v₃ = [8, 9].

Taking into account Example 1 and Remark 3, in this work, we use the normalization proposed in (12).

3. Ideal Similarity and Multiple-Criteria Decision-Making Methods

An institution is interested in assigning values to n alternatives A_i, i = 1,…, n to perform an expensive activity on time and with the appropriate resources and efforts of personnel. To manage it, a value is assigned to each alternative in m criteria, C_j, j = 1,…, m. Depending on the nature of the criteria, an evaluation is carried out using numerical values, intervals, linguistic values, or sets of linguistic values [17]. According to Section 2.1, all of these values could be expressed by means of intervals $\left[v_{ij}^L,v_{ij}^R\right]\subset ℝ,1\le i\le n,1\le j\le m.$ In addition, each criterion, C_j, j = 1,…, m, has a range [A_j, B_j] where the values may vary:

$$A_j\le \underset{1\le i\le n}{\min }{v}_{ij},B_j\ge \underset{1\le i\le n}{\max }{v}_{ij},j=1,\dots ,m.$$

(13)

On the other hand, the institution establishes [a_j, b_j] as an ideal (or pre-established goal) for each criterion, so that

$$\left[a_j,b_j\right]\subseteq \left[A_j,B_j\right]\hspace{1em}j=1,\dots ,m.$$

(14)

Once the ranges and ideals of each C_j criterion are set [25], a similarity function with the ideal $f_j$ could be established, like the proposal in (7). From this function, the normalization for each criterion can be performed with the following function:

$$n_j([v_{ij}^L,v_{ij}^R])\{\begin{array}{ll}{f}_j\left(v_{ij}^L\right),& \mathrm{if}{v}_{ij}^L=v_{ij}^R\\ \frac{1}{v_{ij}^R-v_{ij}^L}{\int }_{v_{ij}^L}^{v_{ij}^R}{f}_j\left(x\right)dx,& \mathrm{if}{v}_{ij}^L\ne v_{ij}^R\end{array}\hspace{1em}j=1,2,\dots ,m.$$

(15)

Consequently, by applying (15) to all of the criteria, a normalization of the original data is obtained, which expresses, by means of a real number r_ij of the interval [0, 1], similarity with the ideal set for the criterion.

In order to obtain a multi-criteria global assessment of each alternative, the Technique for Order of Preference by Similarity (TOPSIS) method is used [9]; this is one of the most popular sorting methods and fits perfectly with the needs of this work [28]. Specifically, data normalization based on the similarity with the ideal solution has already been used successfully with the TOPSIS method in other contexts [10,11].

Next, the method′s steps are expressed algorithmically.

Step 1. Determine the decision matrix $\left[v_{ij}\right],1\le i\le n,1\le j\le m,$ where each data point is an interval $v_{ij}=\left[v_{ij}^L,v_{ij}^R\right]$, as shown in Table 1, where n is the number of alternatives, and the number of criteria is m.

Step 2. According to (15), we construct the normalized decision matrix by means of

$$\left[r_{ij}\right],\hspace{1em}{r}_{ij}\in \left[0,1\right],\hspace{1em}1\le i\le n,\hspace{1em}1\le j\le m.$$

Step 3. Determine the weighted normalized decision matrix. Given $\left(w_1,w_2,\dots w_m\right),$ $w_j\in \left[0,1\right]$, with ${\sum }_{j=1}^m{w}_j=1$, we calculate

$$s_{ij}=w_j{r}_{ij},\hspace{1em}1\le i\le n,\hspace{1em}1\le j\le m.$$

Step 4. Calculate two separation measures for each alternative:

$$S_i^{+}=\sqrt{{\displaystyle \sum }_{i=1}^m{\left(s_{ij}-w_j\right)}^2},\hspace{1em}\hspace{1em}{S}_i^{-}=\sqrt{{\displaystyle \sum }_{i=1}^m{s}_{ij}{}^2.}$$

(16)

Step 5. Calculate the relative proximity to the ideal solution using the relative index:

$$R_i=\frac{S_i^{-}}{S_i^{+}+S_i^{-}},\hspace{1em}1\le i\le n.$$

(17)

Once the TOPSIS method has been applied, a sorting of the alternatives is obtained.

Definition 2. 

Given the alternatives ${\left\{A_i\right\}}_{i=1}^n$and the relative proximity ${\left\{R_i\right\}}_{i=1}^n$obtained in (17), it can be said that $A_i$is preferred to $A_j\left(A_i\ge A_j\right)$only if $R_i\le R_j.$

In addition to the sorting, a cluster in the set of alternatives could be established. If two values, $S_0$ and $S_1$, are set, so that

$$\underset{1\le i\le n}{\min }{R}_i\le S_0\le S_1\le \underset{1\le i\le n}{\max }{R}_i,$$

(18)

a partition of the alternatives can be established:

$${\mathcal{A}}^1=\left\{A_i:R_i\ge S_1\right\},\hspace{1em}{\mathcal{A}}^2=\left\{A_i:R_i\le S_0\right\},\hspace{1em}{\mathcal{A}}^3=\left\{A_i:S_0<R_i<S_1\right\}.$$

(19)

In fact, if we consider

$$n_1=\mathrm{cardinal}({\mathcal{A}}^1),\hspace{1em}{n}_2=\mathrm{cardinal}({\mathcal{A}}^2),\hspace{1em}{n}_3=\mathrm{cardinal}({\mathcal{A}}^3),$$

(20)

the set of n alternatives can be decomposed as ${\mathcal{A}}^1\bigcup {\mathcal{A}}^2\bigcup {\mathcal{A}}^3$, i.e., $n=n_1+n_2+n_3$.

4. Construction of Quality Indices

For the managers of any entity, having an overall estimate of the percentage of people in their institution that are considered adequate for carrying out an activity could be very useful. This allows them to determine whether they should consider complementary actions and modifications in their strategies for the activity to be successful.

From now on, the alternatives ${\left\{A_i\right\}}_{i=1}^n$ are considered to be people who must perform a task, and S₀ represents a value below which individuals are not suitable for the task, while S₁ represents the threshold above which adaptation is excellent. Considering (20), the institution knows that n₁ individuals will perform the task successfully, n₂ people will be inadequate, unless they undertake additional training activities ($\beta$ activities per person, $\beta \ge 1$), and n₃ individuals can perform the task, although some of them might have some difficulty with it ($\alpha n_3$ people are adequate, $\alpha \in \left[0,1\right]$).

It is clear that the percentage of individuals that are part of ${\mathcal{A}}^1$ provides a measure of absolute quality.

Definition 3: 

With the previous notation, the index of absolute excellence is defined as

$$E_A\left(n\right)=\frac{n_1}{n}\times 100.$$

(21)

Using similar reasoning, from $n_1,n_2$, and $n_3$, an adequacy indicator that increases as n₁ increases and decreases as n₂ increases was constructed in Parada et al. [17]. With A(n), the institution can estimate who is suitable for the activity.

Definition 4: 

With the previous notation, the adequacy index is defined as

$$A\left(n\right)=\frac{n_1+\alpha n_3}{n_1+\beta n_2+n_3}\times 100,\hspace{1em}\mathsf{\alpha }\in \left[0,1\right],\hspace{1em}\mathsf{\beta }\ge 1.$$

(22)

By construction, $A\left(n\right)\in \left[0,100\right],$ and its variation is as follows [17].

Proposition 2: 

The indicator$A\left(n\right)$verifies:

(a)

It is increasing with respect to n₁ with n₂ and n₃ being constants.

(b)

It is decreasing with respect to n₂ with n₁ and n₃ being constants.

(c)

When n₃ varies, and n₁ and n₂ remain constant,

$A\left(n\right)$ is increasing if $n_2\in \left]\frac{\left(1-\mathsf{\alpha }\right)}{\mathsf{\alpha }\mathsf{\beta }}{n}_1,n\right[$ and decreasing if $n_2\in \left[0,\frac{\left(1-\mathsf{\alpha }\right)}{\mathsf{\alpha }\mathsf{\beta }}{n}_1\right[$.

On the other hand, $A\left(n,\mathsf{\alpha },\mathsf{\beta }\right)$, which is seen as a function of α and β, can also be represented as

$$A\left(n,\mathsf{\alpha },\mathsf{\beta }\right)\in \left[0,\frac{n_1+n_3}{n}\right]\times 100\subseteq \left[0,100\right].$$

(23)

However, this is not enough to determine the quality of an institution; it also depends on the estimate of its percentage of excellence. In Parada et al. [15], a measure of excellence called ILSAE (Integral Level of Student′s Academic Excellence) is defined, which, in the present work, is called weak excellence.

Definition 5: 

With the previous notation, the weak excellence index is defined as

$$E_W\left(n\right)=\frac{100}{n}\sqrt{\left(n_1+n_3\right)n_1}.$$

(24)

By means of a direct calculation, it can be proved that

$$E_W\left(n\right)\in \left[E_A\left(n\right),100\right].$$

(25)

In accordance with (21), $E_W\left(n\right),$ is the geometric mean between the absolute excellence, $E_A\left(n\right)$, and the $\underset{\mathsf{\alpha },\mathsf{\beta }}{\max ,}A\left(n,\mathsf{\alpha },\mathsf{\beta }\right)$,

$$E_W\left(n\right)=\sqrt{\frac{n_1}{n}\frac{\left(n_1+n_3\right)}{n}}\times 100=\sqrt{E_A\left(n\right)\frac{100\left(n_1+n_3\right)}{n}}.$$

(26)

Therefore, it constitutes an optimistic estimate of excellence, hence the nickname ‘weak’. For this reason, a more representative excellence index of the real situation is introduced.

Definition 6: 

With the previous notation, the excellence index is defined as the geometric mean between adequacy $A\left(n\right)$and absolute excellence $E_A\left(n\right)$,

$$E\left(n\right)=\sqrt{E_A\left(n\right)A\left(n\right)}.$$

(27)

The three indices of excellence that have been introduced are related.

Proposition 3. 

With the previous notation, if $\beta \le 1+\frac{\alpha n_{3}n}{n_1{n}_2},\hspace{1em}{n}_1{n}_2\ne 0,$it can be proven that

$$0\le E_A\left(n\right)\le E\left(n\right)\le E_W\left(n\right)\le 100.$$

(28)

Proof. 

By construction, $E_A\left(n\right),E\left(n\right),E_W\left(n\right),A\left(n\right)\in \left[0,100\right].$ If $n_1$ or $n_2$ are nil, the result is obvious. If $n_1{n}_2\ne 0,$ by a direct calculation, when $\beta \le 1+\alpha n_{3}n/(n_1{n}_2)$, then $E_A\left(n\right)\le A\left(n\right)$ is obtained. Therefore, if ${\left[E_A\left(n\right)\right]}^2\le E_A\left(n\right)A\left(n\right),$ then, $E_A\left(n\right)\le E\left(n\right).$ For (23), if $A\left(n\right)\le \frac{n_1+n_3}{n_1+n_2+n_3}$, then,

$$E_A\left(n\right)A\left(n\right)\le E_A\left(n\right)\frac{n_1+n_3}{n_1+n_2+n_3},$$

and therefore, $E\left(n\right)\le E_W\left(n\right).$ □

Remark 4. 

The condition$\beta \le 1+\frac{\alpha n_{3}n}{n_1{n}_2}$usually does not involve any restrictions. For example, the data in Section 5 only imply that no more than 452 actions should be performed for each individual at risk,$n_2.$

5. Case Study

The proposed methods were applied to the Industrial University of Santander (UIS) at its main campus in Bucaramanga, Colombia. The Academic Excellence Support System (SEA) analyses data of newly enrolled students in five areas: economic, social, health, academic, and cognitive [16]. The aim of this work was to help the UIS design strategic plans that improve the results of its students based on data from nine semesters (from 2014 to 2018).

5.1. Data Used

The SEA assigns values to the areas using five synthetic indicators, which are the ones used here.

(a) Economic dimension (P_E): This is obtained from three variables: income from economic dependence, $\mathrm{ED}=[\mathrm{wage}/\mathrm{SMMLV}]$ (where $⎡x⎤$ is the greatest integer that is less than or equal to x and SMMLV is the current legal minimum monthly wage), the number of siblings (NS), and the position between siblings (PS). The economic dimension indicator is

$$P_E=0.5\mathrm{ED}+0.3\mathrm{NS}+0.2\mathrm{PS},$$

(29)

where each variable can take the values that appear in

Table 2. Assigning values to all dimensions.

(a) Economic dimension (PE): PE = 0.5 ED + 0.3 NS + 0.2 PS
ED	Value	PS	Value	NS	Value
1	0.1	>5	0.1	>3	0.1
2	0.2	Fifth	0.2	3	0.3
3	0.4	Fourth	0.4	2	0.5
4	0.6	Third	0.6	1	1
5	0.8	Second	0.8
>5	1	First	1
(b) Social dimension (Ps): PS = FAD
FAD			Value
$\left[0,9\right]$	(Severe situation)		0.1
$\left[10,13\right]$	(Moderate situation)		0.5
$\left[14,17\right]$	(Slight situation)		0.7
$\left[18,20\right]$	(Good situation)		1
(c) Academic dimension (PA): PA = 0.5 DTM + 0.25 NUA + 0.25 PSO
DTM, NUA		Value	PSO		Value
Very Low		1	Low Level [0, 30]		1
Low		1.3	Medium Level (30, 70]		2
Low to Medium		1.6	High Level (70, 100]		3
Medium		2
Medium to High		2.3
High		2.6
Very High		3
(d) Cognitive dimension (PC): PC = 0.2 VR + 0.2 NR + 0.2 ABR + 0.2 MEM + 0.2 SA
VR, NR, ABR, MEM and SA			Value
$\left[40,80\right]$ (High Risk)			0.1
$\left[81,90\right]$ (Medium Risk)			0.5
$\left[91,160\right]$ (Low Risk)			1
(e) Health dimension (PH): PH = 0.25 ANX + 0.2 DEP + 0.2 EMA + 0.2 ALD + 0.15 PSA
ANX, DEP, EMA			Value	ALD, PSA	Value
$\left[56,80\right]$ (Serious problems)			0.1	Yes	0.1
$\left[48,55\right]$ (problems)			0.5	No	1
$\left[0,47\right]$			1

.

(b) Social Dimension (Ps): This is determined by family dysfunction (FAD), measured using the Family APGAR (Adaptability, Partnership, Growth, Affection, and Resolve) [29],

P_S = FAD,

(30)

which can take four values {0.1, 0.5, 0.7, 1}, as shown in Table 2.

(c) Academic dimension (P_A): This is obtained through three valuations [17]: a diagnostic test of UIS Math (DTM), the EFAI-4 numerical ability (NUA), and the 11-Math Knowledge Test (PSO), such that

$$P_A=0.5\mathrm{DTM}+0.25\mathrm{NUA}+0.25\mathrm{PSO},$$

(31)

where each variable takes a value between 1 and 3. Once the assessment of the academic indicator is known, P_A, the level of mathematical performance of each student is determined using a linguistic scale, as expressed in Table 2.

(d) Cognitive dimension (P_C). This is estimated through five tests: verbal reasoning (VR), numerical reasoning (NR), abstract reasoning (ABR), memory (MEM), and spatial awareness (SA). Therefore,

$$P_C=0.2\mathrm{VR}+0.2\mathrm{NR}+0.2\mathrm{ABR}+0.2\mathrm{MEM}+0.2\mathrm{SA}.$$

(32)

As shown in

Table 3. Range, ideal, and weighting of each dimension.

Dimension	Ranges		Ideals		Weights
	Original *	Transf.	Original	Transf.
Economic	[0, 1]	[0, 1]	[0.8, 1]	[0.8, 1]	0.25
Social	{0.1, 0.5, 0.7, 1}	[0.1, 1]	[0.7, 1]	[0.7, 1]	0.125
Academic	$\left\{\mathrm{VL},\mathrm{L},\mathrm{LM},\mathrm{M},\mathrm{MH},\mathrm{H},\mathrm{VH}\right\}$	[1, 7]	$\left\{\mathrm{H},\mathrm{VH}\right\}$	[6, 7]	0.3
Cognitive	$\left\{\mathrm{VL},\mathrm{L},\mathrm{LM},\mathrm{M},\mathrm{MH},\mathrm{H},\mathrm{VH}\right\}$	[1, 7]	$\left\{\mathrm{H},\mathrm{VH}\right\}$	[6, 7]	0.2
Health	[0, 0.65]	[0, 0.65]	0.65	0.65	0.125

* VL = Very Low, L = Low, LM = Low to Medium, M = Medium, MH = Medium to High, H = High, VH = Very High.

, the values of the variables involved in the cognitive dimension are discrete quantities from the set {0.1, 0.5, 1}, so that the closer a student is to 0, the greater their risk and the greater the need for action.

(e) Health dimension (P_H). This is made up of five variables: anxiety (ANX), depression (DEP), emotional adjustment (EMA), alcohol dependence (ALD), and psychoactive substance abuse (PSA), which are measured using qualitative variables (Table 2):

$$P_H=0.25\mathrm{ANX}+0.2\mathrm{DEP}+0.2\mathrm{EMA}+0.2\mathrm{ALD}+0.15\mathrm{PSA}.$$

(33)

An in-depth study should consider circumstances that may influence a student′s academic performance. For example, students with parents who are highly educated but lack financial resources may perform better than students who are very well-off but whose parents are less educated. However, we think that the Industrial University of Santander has these issues included in some way because of the way it values the economic and social dimensions. Thus, while in the economic dimension, they try to determine the economic resources from the variables described previously, in the social dimension, by analyzing the Family APGAR [29], they want to determine the affection, adaptation, family life, and help that students receive from their family members. On the other hand, adding these circumstances imply a new institutional consensus that would involve a new scenario and a complete reformulation of the institution′s inclusion policies.

5.2. Calculation of Indicators

First, the ranges, ideals, and weighting of each criterion were agreed on with the UIS specialists [16]. In Table 3, these values are shown in their original form and also transformed into intervals, while in column 6, it can be confirmed that the cognitive and academic part accounts for 50% of the value, and the remaining dimensions account for the other 50%.

The cognitive and academic dimensions represent 50% of student assessment (see column 6 of Table 3). The SEA experts proposed a non-linear normalization of the assessments of these dimensions. In the words of the experts, “reinforcement of a student with very low scores usually results in much less improvement than a student with high scores”; hence, the convex form is preferred for the function (see Figure 3). In the cognitive case, the opposite is true; a concave function is preferred. This is due to the inclusive policies of the Industrial University of Santander for students with problems and to attempt a trade-off between the two dimensions (see Figure 3).

Taking into account the ranges and ideals for these dimensions (see Table 3), according to (10), the similarity to the ideal was estimated by means of the following functions:

$${\eta }_A\left(x\right)=\{\begin{array}{cc}\frac{1-e^{\frac{x-1}{5}}}{1-e},& 1\le x<6\\ 1,& 6\le x\le 7\\ 0,& \mathrm{otherwise},\end{array}\hspace{1em}\hspace{1em}\hspace{1em}{\eta }_C\left(x\right)=\{\begin{array}{cc}\frac{1-e^{\frac{1-x}{5}}}{1-e^{-1}},& 1\le x<6\\ 1,& 6\le x\le 7\\ 0,& \mathrm{otherwise}.\end{array}$$

Using the notation of (10), the above functions were ${\eta }_A\left(x\right):={\eta }_{\left(1,6,7,7;1,0\right)}\left(x\right),$ ${\eta }_C\left(x\right):={\eta }_{\left(1,6,7,7;-1,0\right)}\left(x\right)$

For the other dimensions, the similarity with the ideal was estimated with piecewise linear functions (see (7)). Column 5 of

Table 4. Range, ideal, and similarity functions of each dimension.

Dimension	Range = [A, B]	Ideal = [a, b]	Type of Data	Similarity Function	Weights
Economic (E)	[0, 1]	[0.8, 1]	I	${\mu }_{\left(0,0.8,1,1\right)}\left(x\right)$	wE = 0.25
Social (S)	[0.1, 1]	[0.7, 1]	I	${\mu }_{\left(0.1,0.7,1,1\right)}\left(x\right)$	wS = 0.125
Academic (A)	[1, 7]	[6, 7]	II	${\eta }_{\left(1,6,7,7;1,0\right)}\left(x\right)$	wA = 0.3
Cognitive (C)	[1, 7]	[6, 7]	I	${\eta }_{\left(1,6,7,7;-1,0\right)}\left(x\right)$	wC = 0.2
Health (H)	[0, 0.65]	0.65	I	${\mu }_{\left(0,6.5,0.65,0.65\right)}\left(x\right)$	wH = 0.125

shows the similarity functions for each dimension.

Economic, social, cognitive, and health dimension data are expressed as real numbers (column 4 of Table 4). However, academic dimension data are presented as intervals. The following process used to value this dimension varied greatly in the period analyzed (from 2014 to 2018). The steps followed were as follows: (a) qualitative assessment, (b) numerical transformation, (c) calculation of $P_A$, and (d) qualitative expression of “academic risk” using the scale {Very Low, Low, Low to Medium, Medium, Medium to High, High, Very High}. After going through this process, the SEA experts stated that it was less realistic to directly transform words into real numbers than to estimate them by means of intervals, as follows (6):

VL = [1, 2], L = [2, 2.75], LM = [2.75, 3.5], M = [3.5, 4.75], MH = [4.75, 5.5], H = [5.5, 6.25], VH = [6.25, 7].

(34)

Considering the similarity functions in Table 4 and (34), data could be normalized with (12).

Table A1. Steps of the Technique for Order of Preference by Similarity (TOPSIS) method.

2018_1	Original Numerical Data (Step 1)					Normalized Data (Step 2)					Weighted Normalized Data (Step 3)					Distances (Step 4)		Score (Step 5)
Students	E	S	H	A	C	E	S	H	A	C	w1E	w1S	w1H	w1A	w1C	D+	D-	Ri
#1	1	0.5	0.55	[3.5, 4.75]	6.00	1.000	0.667	0.846	0.508	1.000	0.250	0.083	0.106	0.152	0.200	0.379	0.144	0.725
#2	0.77	1	0.65	[3.5, 4.75]	6.00	0.963	1.000	1.000	0.508	1.000	0.241	0.125	0.125	0.152	0.200	0.390	0.137	0.741
#3	1	0.7	0.65	[3.5, 4.75]	1.00	1.000	1.000	1.000	0.508	0.000	0.250	0.125	0.125	0.152	0.000	0.342	0.242	0.586
#4	1	1	0.65	[5.5, 6.25]	4.00	1.000	1.000	1.000	0.962	0.714	0.250	0.125	0.125	0.289	0.143	0.444	0.057	0.886
#5	1	0.5	0.55	[5.5, 6.25]	5.00	1.000	0.667	0.846	0.962	0.871	0.250	0.083	0.106	0.289	0.174	0.441	0.053	0.893
#6	0.03	0.5	0.52	[3.5, 4.75]	2.00	0.038	0.667	0.800	0.508	0.287	0.009	0.083	0.100	0.152	0.057	0.209	0.315	0.399
#7	1	0.7	0.52	[2.0, 2.75]	3.00	1.000	1.000	0.800	0.185	0.522	0.250	0.125	0.100	0.055	0.104	0.320	0.253	0.558
#8	0.03	1	0.65	[3.5, 4.75]	2.00	0.038	1.000	1.000	0.508	0.287	0.009	0.125	0.125	0.152	0.057	0.241	0.311	0.436
#9	1	0.7	0.65	[5.5, 6.25]	3.00	1.000	1.000	1.000	0.962	0.522	0.250	0.125	0.125	0.289	0.104	0.434	0.096	0.819
#10	1	0.7	0.65	[2.0, 2.75]	6.00	1.000	1.000	1.000	0.185	1.000	0.250	0.125	0.125	0.055	0.200	0.370	0.233	0.613

shows the TOPSIS results for 10 students in the period 2018_1 (see Appendix A).

According to the criteria of the Academic Excellence Support System (SEA), the students ${\left\{{\mathrm{x}}_{\mathrm{i}}\right\}}_{\mathrm{i}=1}^{\mathrm{n}}$, assessed above 85% are not at risk, and those who are under 45% are at high risk. By using this, we divided the students into risk-free ${\mathcal{A}}^1=\left\{x_i:R_i\ge 0.85\right\}$, those with obvious risk ${\mathcal{A}}^2=\left\{x_i:R_i\le 0.45\right\}$, and those in an intermediate state:

$$n_1=\mathrm{cardinal}({\mathcal{A}}^1),n_2=\mathrm{cardinal}({\mathcal{A}}^2),n_3=n-n_1-n_2.$$

(35)

The values obtained are shown in columns 2–5 of

Table 5. Comparison between adequacy and excellence indices.

Period	MCDM Results				Quality Indicators Results
	n	n1	n2	n3	${\mathit{E}}_{\mathit{A}}\left(\mathit{n}\right)$	$\mathit{E}\left(\mathit{n}\right)$	${\mathit{E}}_{\mathit{W}}\left(\mathit{n}\right)$	$\mathit{A}\left(\mathit{n}\right)$
2014_1	454	10	11	433	2.20	12.59	14.66	71.99
2014_2	726	4	12	710	0.55	6.33	7.36	72.70
2015_1	700	8	47	645	1.14	8.815	10.33	65.83
2015_2	964	25	60	879	2.59	13.37	15.59	66.82
2016_1	1248	13	41	1194	3.85	16.23	19.09	70.48
2016_2	343	8	18	317	2.33	12.60	14.87	68.07
2017_1	1262	22	26	1214	1.74	11.23	13.07	72.40
2017_2	936	29	28	879	3.10	14.87	17.34	71.40
2018_1	1047	34	26	987	3.25	15.31	17.80	72.16

.

Following the expert indications, 20% of $n_3$ were predicted to end up needing help, and high-risk students would usually require two or more interventions; therefore, the values $\alpha =0.8$ were estimated to calculate the adequacy index (16). Finally, with (21), (22), (24), and (27), the excellence indicators that appear in columns 6–9 of Table 5 were obtained.

It could be observed that the adequacy indices took values between 65.83% and 72.40%, with oscillations that were not very significant (Figure 4). This means that between 27.60% and 34.17% of students could be considered to have global deficiencies when it comes to properly carrying out their studies. Although the adequacy percentage was found to be slightly lower in women than in men, between 0.5% and 4.5% (see Figure 5b), this trend was reversed for excellence. As of 2015, excellence in women was higher than in men, sometimes by up to 5% (Figure 5a).

In Figure 4b, we represent the three quality indices that were defined: $E_A\left(n\right),E\left(n\right),E_W\left(n\right)$. As stated in Proposition 3, we verified that $E_A\left(n\right)\le E\left(n\right)\le E_W\left(n\right)$. Note that absolute excellence never exceeded 3.25%, while excellence ranged from 6.33% to 15.31%, and weak excellence sat between 7.36% and 17.86%.

When the discriminant analysis was performed for economic risk, the results were much more alarming (Figure 6). The adequacy of students without economic risk showed a 54% gap to the detriment of the students at risk (Figure 6a). Regarding excellence, the differences were evident. For example, 17.69% of the students without economic risk were excellent in 2018_1, whilst none of the at-risk students were excellent in the same period (Figure 6b).

6. Discussion

Our proposal contains three processes: a normalization method based on similarity with pre-established goals, an MCDM method (in which the weights of each criterion are given by the institution) to assess each alternative, and the calculation of indicators using parameters set by the managers (see Figure 1). In general, the values of the weights and the parameters are the result of a consensus that may have been difficult to achieve and, therefore, should not be modified without the approval of the institution. This is the case for the Industrial University of Santander, in which the weights shown in Table 2 for the parameters α, β of (22), and the thresholds of inadequacy and excellence were provided by SEA experts [16].

The main utility of the TOPSIS method is to order alternatives but, in this work, we not only took into account this possibility but also used the value of the relative proximity to the ideal solution, R_i, as the global assessment for each alternative. This is justified because using (16), regardless of the data set, the ideal and anti-ideal values of the classic TOPSIS method [9] correspond to (1,1, ..., 1) and (0,0, ..., 0), respectively. This fact prevents R_i values from changing when new data are incorporated, that is, the phenomenon known as the rank reversal is avoided [10].

From a mathematical point of view, for the real case study, we generated different scenarios where we were able to verify that the indicators are much more sensitive to changes in the value of α than to changes in β (22). Furthermore, the sensitivity to weights of each dimension was also not the same. From the highest to the lowest sensitivity, the order was academic, cognitive, economic, social, and health. However, as noted, the sensitivity study could not be used because the values were set by experts.

In our opinion, it is useful to compare our proposal with others that could be similar in some parts of the process, specifically, normalization and the chosen MCDM method.

6.1. Comparison with Other Methods

Before opting for our proposal of normalization (12) and the TOPSIS method, other existing alternatives were analyzed. The only conditions that could not be modified were the values established by the experts of the Industrial University of Santander by consensus. These values were the weights and thresholds of inadequacy and excellence. Furthermore, the values obtained with the indicators had to be consistent with the institution′s academic historical results. Thus, we assume that more than 50% of students are adequate to carry out their studies, and we suppose that less than 20% is excellent. With these conditions, we tested other normalizations and another MCDM method.

Firstly, we compared the results obtained with TOPSIS with the proposed normalization given in (12) and the direct calibration used in the fsqca method [30]. In the calibration, both the numerical variables and the non-verbal categories are assigned a degree of belonging, fixed around three theoretical anchors established between the value of total membership, 1, the cross-over point, 0.5, and the value of total exclusion, 0. In this work, the thresholds and the border point were set based on the ranges and transformed ideals of each dimension (see Table 3). The calibrated values were obtained using the function calibrate in the R package QCA [31].

We compared the TOPSIS results with the results of VIKOR [32], another MCDM method that also allows n alternatives valued with m criteria to be ordered. The VIKOR and TOPSIS methods are based on an aggregation function that represents the proximity to some reference points [33], which, in our case, are (1,1, ..., 1) and (0,0, ..., 0). For the VIKOR method, we used three different normalizations: linear (proposed in the classic VIKOR method [32]), the normalization given in (12), and the calibration obtained with R package QCA. The calculations were performed using the VIKOR function in the R package MCDM v 1.2. [34].

In

Table 6. Values of n₁, n₂, and n₃ in the cases analyzed.

		TOPSIS						VIKOR
		Our Norm.			Calibration			Linear Norm.			Our Normaliz			Calibration
PERIOD	n	n1	n2	n3	n1	n2	n3	n1	n2	n3	n1	n2	n3	n1	n2	n3
2014_1	454	10	11	433	20	36	398	9	8	437	9	24	421	14	31	409
2014_2	726	4	12	710	8	60	658	21	114	591	4	81	641	4	98	624
2015_1	700	8	47	645	9	151	540	4	79	617	12	89	599	4	147	549
2015_2	964	25	60	879	25	64	875	5	264	695	35	93	836	25	263	676
2016_1	1248	13	41	1194	25	70	1153	38	79	1131	45	116	1087	44	119	1085
2016_2	343	8	18	317	9	16	318	12	35	296	12	19	312	12	24	307
2017_1	1262	22	26	1214	21	37	1204	31	14	1217	31	27	1204	31	27	1204
2017_2	936	29	28	879	38	22	876	2	29	905	2	14	920	2	16	918
2018_1	1047	34	26	987	32	22	993	41	29	977	84	28	935	176	27	844

, we show the values of n₁, n₂, and n₃ obtained with the different methods and in different periods.

Using the values in Table 6, the proposed indicator values are shown in

Table 7. Values of indicators obtained with TOPSIS method.

TOPSIS	Our Normalization				Calibration
Period	$E_A\left(n\right)$	$E\left(n\right)$	$E_W\left(n\right)$	$A\left(n\right)$	$E_A\left(n\right)$	$E\left(n\right)$	$E_W\left(n\right)$	$A\left(n\right)$
2014_1	2.20	12.59	14.66	71.99	4.41	16.92	20.14	65.00
2014_2	0.55	6.33	7.36	72.70	1.10	8.38	10.05	63.80
2015_1	1.14	8.815	10.33	65.83	1.29	7.91	10.04	48.65
2015_2	2.59	13.37	15.59	66.82	2.59	13.11	15.56	66.27
2016_1	3.85	16.23	19.09	70.48	2.00	11.63	13.75	67.51
2016_2	2.33	12.60	14.87	68.07	2.62	13.45	15.82	68.94
2017_1	1.74	11.23	13.07	72.40	1.66	10.88	12.71	71.13
2017_2	3.10	14.87	17.34	71.40	4.06	17.16	19.91	72.55
2018_1	3.25	15.31	17.80	72.16	3.06	14.90	17.30	72.66

and

Table 8. Values of indicators obtained with VIKOR method.

VIKOR	Linear				Our Normalization				Calibration
Period	$E_A\left(n\right)$	$E\left(n\right)$	$E_W\left(n\right)$	$A\left(n\right)$	$E_A\left(n\right)$	$E\left(n\right)$	$E_W\left(n\right)$	$A\left(n\right)$	$E_A\left(n\right)$	$E\left(n\right)$	$E_W\left(n\right)$	$A\left(n\right)$
2014_1	1.98	12.02	13.96	72.89	1.98	11.61	13.70	67.94	3.08	14.28	16.95	66.13
2014_2	2.89	12.64	15.62	55.27	0.55	5.75	7.00	60.07	0.55	5.62	6.90	57.28
2015_1	0.57	5.85	7.12	59.92	1.71	10.01	12.23	58.46	0.57	5.30	6.72	49.09
2015_2	0.52	4.71	6.14	42.85	3.63	15.08	18.11	62.63	2.59	10.60	13.73	43.36
2016_1	3.04	14.26	16.89	66.79	3.61	15.08	18.08	63.07	3.53	14.87	17.86	62.75
2016_2	3.50	14.72	17.72	61.90	3.50	15.42	18.18	67.96	3.50	15.20	18.04	66.01
2017_1	2.46	13.48	15.59	73.96	2.46	13.34	15.50	72.46	2.46	13.34	15.50	72.46
2017_2	0.21	3.88	4.55	70.54	0.21	3.95	4.59	72.84	0.21	3.94	4.58	72.53
2018_1	5.82	20.53	23.80	71.91	8.02	24.21	27.94	73.05	16.81	35.58	40.47	75.33

. Values of adequacy indices below 50% and values of excellence indices above 20% are highlighted with boxes.

According to the results shown in Table 6, Table 7 and Table 8, there are apparently no significant differences between the analyzed scenarios. However, after considering the historical results of the UIS, the experts of the institution decided to set a lower bound for the percentage of adequacy (50%) and an upper bound for the percentage of excellence (20%). Of all the possibilities, the only one that met such required bounds was the TOPSIS method with the normalization proposed in (12). Note that this option is the only one having no values out of the bounds (i.e., without any highlighting boxes in Table 7 and Table 8). This fact, together with the simplicity of the TOPSIS calculation and the opinions of the experts of the SEA [16], led to the construction of the indicators used in this work.

6.2. Strengths and Weaknesses of Our Proposal

When quality indicators are proposed to analyze an institution, the intention is that actions that allow the optimization of resources of all kinds can be planned. In general, the indicators obtained through the presented method meet this intention. In fact, the proposed indicators and the way to obtain them have been very useful for the managers of the Industrial University of Santander. However, our proposal is not without limitations; therefore, below, we summarize the strengths and weaknesses of the method.

The strengths of the method are as follows:

(1)

The opinions of the experts, who have been working on the question for years, are taken into account in all parts of the process: the choice of the ideal situation for each dimension, the nature of the data (numbers, intervals or linguistic expressions), the weight given to each dimension, the thresholds S₀ and S₁ that determine the inadequacy and excellence, and the parameters that appear in the indicators.

(2)

All steps of the method can be calculated with basic tools that any participant in the process can see on their own computer. Specifically, all calculations were performed with Microsoft Excel (see Appendix B).

(3)

The indicators are easily interpretable and allow comparison with other achievement indicators of the institution. For example, in the case of the Industrial University of Santander, the values of the proposed indicators are very consistent with the results obtained by its students.

As for the weaknesses, we have the following:

(1)

As can be seen in Section 6.1, the proposed method clearly depends on the normalization used and, therefore, on the choice of ideals for each dimension. This makes the consensus among experts decisive in the process.

(2)

The weights that have been set for each dimension to determine the solution to the TOPSIS method and; therefore, the values of the indicators. That these weights are accepted by the institution is a key element in the process.

(3)

To determine the evolution of the institution, the data handled in each period have not always followed the same criteria, scales, etc. In addition, some data collected over the years cannot be used, for example, because data protection laws prohibit it.

To avoid, to a large extent, the limitations of the method, we are working on making both the weights and the values of the different parameters flexible through intervals. This will greatly facilitate consensus among experts, but we must be able to avoid an excessive increase in computational complexity so that all participants can follow the process.

7. Conclusions

Measuring the quality of institutions, or some of their services, constitutes a challenge for managers, since it requires them to design strategic actions and increase their competitive advantage. To this end, the use of the indicators built using MCDM methods has become a very useful tool in the decision-making process. On the one hand, they allow the selection of the best alternatives, and on the other, they allow the synthetization of information in order to draw conclusions quickly. These were the objectives of this approach.

The first obstacle is the homogenization and normalization of data of very different natures, such as real numbers, intervals, or linguistic variables, which assign a value to very different characteristics and whose aggregation cannot be performed directly. This study proposed a method of standardization based on similarity to the ideal objectives set by the institution for each criterion.

The manner of treating quality has been gradual. It started with the construction of a synthetic indicator to measure the adequacy of each alternative and, subsequently, three indicators of excellence were defined. They report the quality in different contexts and with varying degrees of demand so that the institution can design strategic actions. Meanwhile, the methodology employed provides a classification of the alternatives into groups, depending on the adequacy for a specific task.

This proposal was applied to a real decision-making case at the Industrial University of Santander. One of the objectives of this university is to provide support to students who show deficiencies in the global value of the five dimensions assessed by the institution.

Given that the resources, both economic and human, of universities, in general, do not allow students to be offered all the necessary support, actions should be established in order to:

• Estimate the percentages of students who will not have problems carrying out their studies successfully and those who will need some support.
• Understand each student’s personal situation in order to try to guarantee support for those who need it most.

Using the UIS data, we demonstrated that gender differences (in terms of excellence) are not very significant, although, from 2016, women have slightly exceeded men. However, the differences when taking account of the economic level are very important. The intention of applying this methodology is that these actions can be planned and generalized in order to achieve optimization of resources of all kinds so that inequalities can be reduced and excellence objectives can be established in both the medium and long term.