**5. Results and Discussion**

VAS matrix is a set of the VAS scales placed in a single question. Since multiple data like the importance value and the ranking information can be gathered from a single survey question, the VAS matrix might be successfully exploited for the survey-based criteria weighting tasks. However, biases of the respondents and the psychometric features of the VAS scales should be carefully treated to avoid uncertainties in the preference elicitation results.

Scatterplots of the two criteria with the highest mean value and with the lowest mean value were generated to illustrate the tendencies in the data collected with the VAS matrix (Figure 3). Analysis of the data distribution shows that the majority of estimates are in the range of 60–100. This is in line with the research showing that direct weighting performed with the Likert-scales or the VAS-scales shows a tendency towards the positive skew of the collected data. On the cognitive side, this may also suppose that parents have a tendency to say that most of the analyzed aspects are important for assessing the quality of kindergartens. *Symmetry* **2020**, *12*, 1641 14 of 20 of the data distribution shows that the majority of estimates are in the range of 60–100. This is in line with the research showing that direct weighting performed with the Likert-scales or the VAS-scales shows a tendency towards the positive skew of the collected data. On the cognitive side, this may also suppose that parents have a tendency to say that most of the analyzed aspects are important for assessing the quality of kindergartens.

**Figure 3.** Scatterplots of the VAS values for the criteria that were determined as the most important (**a**,**b**), and the least important (**c**,**d**). Lines determines intervals for the five importance groups: (0–10) not important at all, (10–50)—unimportant, (50–75)—important, (75–95)—very important, (95–100)– extremely important. **Figure 3.** Scatterplots of the VAS values for the criteria that were determined as the most important (**a**,**b**), and the least important (**c**,**d**). Lines determines intervals for the five importance groups: (0–10)—not important at all, (10–50)—unimportant, (50–75)—important, (75–95)—very important, (95–100)–extremely important.

criteria is neither important nor unimportant (value = 50). To prevent the uncertainties caused by the

In the numerical example presented in this paper, missing data is noticed in 72.84% of the answers. The accuracy of the survey results is usually sought to be improved by ensuring an appropriate sample of the responses. However, recently the significant decrease in the response rate of the online polls can be noticed [59]. In these circumstances, the opinion of each respondent becomes increasingly important. Pairwise comparison approaches like AHP or SWARA are not able to deal with the missing data, but it is not an issue for the VASMA weighting. On the contrary, VASMA weighting exploits the non-response values to achieve the greater accuracy of the preference

erroneous interpretation, we consider this situation as the missing data.

elicitation results.

It is also noteworthy to observe that assessments ranging from 40 to 60 were hardly ever

It is also noteworthy to observe that assessments ranging from 40 to 60 were hardly ever provided by the respondents. It might be related to the design of the VAS scales, where the default position of the marker is placed in the middle between the two linguistic anchors. A non-moved marker can be understood either as the non-response situation, or as the cognitive answer that the criteria is neither important nor unimportant (value = 50). To prevent the uncertainties caused by the erroneous interpretation, we consider this situation as the missing data.

In the numerical example presented in this paper, missing data is noticed in 72.84% of the answers. The accuracy of the survey results is usually sought to be improved by ensuring an appropriate sample of the responses. However, recently the significant decrease in the response rate of the online polls can be noticed [59]. In these circumstances, the opinion of each respondent becomes increasingly important. Pairwise comparison approaches like AHP or SWARA are not able to deal with the missing data, but it is not an issue for the VASMA weighting. On the contrary, VASMA weighting exploits the non-response values to achieve the greater accuracy of the preference elicitation results.

#### *5.1. Comparison of the Direct Rating and VASMA Weights Symmetry* **2020**, *12*, 1641 15 of 20

DR technique.

*5.2. Sensitivity Analysis* 

differences in the data collection procedure [9].

Direct weighting approaches like point allocation, direct rating, SMART, and SMARTER might be considered as the simplest criteria elicitation methods [48]. Direct rating (DR) is probably the easiest of them since criteria weights are assessed by purely asking the respondents to assign absolute values of the criteria. Since DR does not require any prior learning on the preference elicitation process, it might also be easily applied for the survey-based criteria weighting [9]. However, two important disadvantages are recurrently associated with the direct rating methodology: the high potential for biased information [46] and the tendency towards the low variance of the criteria weights [9,44]. A comparison of the DR and VASMA approaches was performed to reveal how the data processing technique integrated into the VASMA weighting methodology affects both the variability and the accuracy of the criteria weights. Both the direct rating and the VASMA weighting techniques employ VAS Matrix as the data collection technique, but the distinctive data processing procedures. While DR simply calculates the averages of the criteria weights proposed by the respondents, VASMA calculates both the subjective and objective weights for the preference elicitation. The criteria weights calculated with the direct rating and VASMA weighting approaches are compared in Figure 4. *5.1. Comparison of the Direct Rating and VASMA Weights*  Direct weighting approaches like point allocation, direct rating, SMART, and SMARTER might be considered as the simplest criteria elicitation methods [48]. Direct rating (DR) is probably the easiest of them since criteria weights are assessed by purely asking the respondents to assign absolute values of the criteria. Since DR does not require any prior learning on the preference elicitation process, it might also be easily applied for the survey-based criteria weighting [9]. However, two important disadvantages are recurrently associated with the direct rating methodology: the high potential for biased information [46] and the tendency towards the low variance of the criteria weights [9,44]. A comparison of the DR and VASMA approaches was performed to reveal how the data processing technique integrated into the VASMA weighting methodology affects both the variability and the accuracy of the criteria weights. Both the direct rating and the VASMA weighting techniques employ VAS Matrix as the data collection technique, but the distinctive data processing procedures. While DR simply calculates the averages of the criteria weights proposed by the respondents, VASMA calculates both the subjective and objective weights for the preference elicitation. The criteria weights calculated with the direct rating and VASMA weighting approaches are compared in Figure 4.

**Figure 4.** VASMA weighting and the direct rating comparison. **Figure 4.** VASMA weighting and the direct rating comparison.

Results presented in Figure 4 support the idea that DR is typically associated with the low variation of the criteria weights. DR weights calculated for the criteria C1–C10 slightly vary in the interval (0.0770, 0.0906), while the range of the VASMA weights is much wider (0.0637, 0.1104). Respect for the psychometric properties of the VAS scales and the awareness on the uncertainty of the collected data showed that VASMA weighting demonstrates the positive Results presented in Figure 4 support the idea that DR is typically associated with the low variation of the criteria weights. DR weights calculated for the criteria C1–C10 slightly vary in the interval (0.0770, 0.0906), while the range of the VASMA weights is much wider (0.0637, 0.1104). Respect for the psychometric properties of the VAS scales and the awareness on the uncertainty of the collected data

The sensitivity analysis was performed to study the consistency of the obtained ranking. Ranks of the two direct weighting techniques (point allocation and direct rating) and VASMA weighting were determined and compared (Figure 5). Two popular direct weighting techniques SMART and SWING were not included in the comparison because of the methodological showed that VASMA weighting demonstrates the positive effect for both the equal weighting and the high bias issues that are the vast disadvantages of the DR technique.
