**3. A Numerical Case Study**

This section describes the process of the selection of an e-learning course based on the opinions of twenty-four respondents by using the PIPRECIA method and the IVTFN-ARAS method. Four e-learning courses in the field of programming are evaluated, designated as: *A*1—Cubes (www.cubes.edu.rs); *A*2—ITAcademy (www.it-akademija.com); *A*3—Link-eLearning (www.link-elearning.com), and *A*4—Ok School (www.ok-school.com). The criteria used for the evaluation of the e-learning courses are obtained based on the analysis of the relevant literature [53–55], and their weights are determined by using the PIPRECIA method. Evaluation criteria and their corresponding weights that represent the attitudes of the group are shown in Table 1.


**Table 1.** The group evaluation criteria and their weights.

At the beginning of the evaluation, all twenty-four respondents evaluated alternatives using the five-point Likert scale. The Likert scale was chosen because it is easy to use and its usage is understandable for respondents. Ratings obtained from three randomly selected respondents are shown in Tables 2–4 and Figures 1–3.

**Table 2.** The ratings obtained from the fist of twenty-four respondents.


shown in Tables 2−4 and Figures 1−3.

shown in Tables 2−4 and Figures 1−3.

At the beginning of the evaluation, all twenty-four respondents evaluated alternatives using the five-point Likert scale. The Likert scale was chosen because it is easy to use and its usage is understandable for respondents. Ratings obtained from three randomly selected respondents are

At the beginning of the evaluation, all twenty-four respondents evaluated alternatives using the five-point Likert scale. The Likert scale was chosen because it is easy to use and its usage is understandable for respondents. Ratings obtained from three randomly selected respondents are

Symmetry 2020, 12, x FOR PEER REVIEW 6 of 14

Table 2. The ratings obtained from the fist of twenty-four respondents.

Table 2. The ratings obtained from the fist of twenty-four respondents.

Criteria C1 C<sup>2</sup> C3 C4 C5 C6 C<sup>7</sup> A1 4 3 3 4 2 4 5 A2 3 5 2 4 4 4 4 A<sup>3</sup> 5 5 4 5 3 3 2 A4 4 5 5 4 4 4 4

Criteria C1 C<sup>2</sup> C3 C4 C5 C6 C<sup>7</sup> A1 4 3 3 4 2 4 5 A2 3 5 2 4 4 4 4 A<sup>3</sup> 5 5 4 5 3 3 2 A4 4 5 5 4 4 4 4

Figure 1. The ratings obtained from the fist of twenty-four respondents. **Figure 1.** The ratings obtained from the fist of twenty-four respondents. **Table 3.** The ratings obtained from the second of twenty-four respondents. Figure 1. The ratings obtained from the fist of twenty-four respondents. Table 3. The ratings obtained from the second of twenty-four respondents.

Figure 2. The ratings obtained from the second of twenty-four respondents. **Figure 2.** The ratings obtained from the second of twenty-four respondents.


Figure 2. The ratings obtained from the second of twenty-four respondents. **Table 4.** The ratings obtained from the third of twenty-four respondents.

Table 4. The ratings obtained from the third of twenty-four respondents.

Criteria C1 C<sup>2</sup> C3 C4 C5 C6 C<sup>7</sup> A1 3 3 3 4 5 4 4 A2 2 4 4 3 2 5 5 A<sup>3</sup> 5 4 4 3 4 5 3 A4 5 5 5 3 3 4 4

Figure 3. The ratings obtained from the third of twenty-four respondents. **Figure 3.** The ratings obtained from the third of twenty-four respondents.

Subsequently, in the next step, a group decision matrix was formed. The elements of this matrix, shown in Table 5, are IVTFNs formed by the transformation of crisp ratings into IVTFNs, as is explained in Stanujkic [52]. Subsequently, in the next step, a group decision matrix was formed. The elements of this matrix, shown in Table 5, are IVTFNs formed by the transformation of crisp ratings into IVTFNs, as is explained in Stanujkic [52].

Based on the data from Table 5, the optimal performance ratings are determined by using Equation (5). The obtained optimal performance ratings are shown in Table 6. Based on the data from Table 5, the optimal performance ratings are determined by using Equation (5). The obtained optimal performance ratings are shown in Table 6.

In the next two steps, normalized and weighted normalized decision-making matrices are calculated using Equations (11) and (12). The normalized and weighted normalized decision-making matrices are shown in Tables 7 and 8. In the next two steps, normalized and weighted normalized decision-making matrices are calculated using Equations (11) and (12). The normalized and weighted normalized decision-making matrices are shown in Tables 7 and 8.



*A*<sup>3</sup>

*A*<sup>4</sup>

[(0.2, 0.56), 0.78, (0.94, 1)] [(0.2, 0.53), 0.74, (0.92, 1)] [(0.2, 0.46), 0.63, (0.84, 1)] [(0.2, 0.46), 0.76, (0.97, 1)] [(0.2, 0.57), 0.71, (0.85, 1)] [(0.2, 0.48), 0.73, (0.9, 1)] [(0.4, 0.51), 0.63, (0.91, 1)]

[(0.4, 0.52), 0.69, (0.87, 1)] [(0.2, 0.6), 0.8, (0.93, 1)] [(0.4, 0.52), 0.73, (0.93, 1)] [(0.2, 0.46), 0.68, (0.83, 1)] [(0.2, 0.5), 0.76, (0.89, 1)] [(0.2, 0.43), 0.63, (0.85, 1)] [(0.4, 0.5), 0.68, (0.85, 1)]

#### *Symmetry* **2020**, *12*, 928


**Table 8.** The weighted IVTF performance ratings.

(0.13, 0.14)]

(0.13, 0.14)]

(0.13, 0.13)]

(0.12, 0.14)]

(0.13, 0.15)]

(0.12, 0.14)]

(0.13, 0.15)]

Finally, the overall interval-valued triangular fuzzy performance ratings, obtained by using Equation (13), are shown in Table 9.


**Table 9.** The overall IVTF performance ratings.

In order to determine the quality of the e-courses, these values must be defuzzified using some of the well-known procedures [52].

Results obtained using the simplest of all of the considered defuzzification procedures are shown in Table 10. The relative quality, i.e., the degree of utility, of analyzed e-courses as well as their ranking orders, are also shown in Table 10.


**Table 10.** The degree of utility and ranking order of analyzed e-courses.

As can be seen from Table 10, the best alternative, i.e., e-course, is the alternative designated as *A*2. By varying the coefficient λ, greater importance can be given to *l* and *u* in relation to *l'* and *u'*, and vice versa. The results obtained using Equation (16) for some characteristic values of the coefficient λ are shown in Table 11.


**Table 11.** The degree of utility and ranking order of analyzed e-course for some characteristic values of λ*.*

In this case, the alternative denoted as *A*<sup>2</sup> remains the best alternative i.e., an e-learning course in all cases. This indicates the stability of the chosen e-learning course.

However, in many cases variation of the coefficient lambda may have an impact on the ranking order of the considered alternatives, and this approach may be useful to analyze different scenarios such as pessimistic, realistic, and optimistic.
