*1.2. Matrix Questions and the Response Scales*

Matrix questions are usually constructed when several questions about a similar idea should be assessed using the chosen measurement scale. Likert-type scales are typically expressed as the set of radio buttons, representing five or more discrete categories dedicated to revealing respondents' current state, feelings, or traits [18]. Since Likert-type scales are easily understandable, they are frequently met in the online surveys. However, the ambiguous number of response categories is the important disadvantage of these scales. Moreover, intervals between values cannot be presumed to be equal, and the biases induced by the ordinal data points might cause adversative effects on the calculations of the statistical measures like mean, covariance, correlations, or the reliability coefficients [19].

Issues inherent from the Likert-type measurement increased scientists' interests in the alternative scales [17]. Research on the historical origin of the semantic differentials revealed that they were initially made from the continuous scales, also known as the visual analogue scales (VAS). A VAS is typically presented as a horizontal line, anchored with two verbal descriptors at the extremes. A respondent indicates his opinion by placing a marker at the most appropriate point. Since VAS uses a line continuum to measure latent traits and to obtain data measurements, they are able to present weighting results without the constraints raised by the limited number of the response categories [20,21]. Fine-grained responses aid in reducing measurement error for both the value-based and the rank-based valuations. Since VAS scales produce interval-level measurement data, they are also better suited for statistical and mathematical algorithms [22].

VAS matrix is a set of the VAS scales placed in a single question. Since twofold data like the importance value and the ranking information can be gathered from a single VAS matrix, it might be successfully exploited for the preference elicitation tasks [23]. Besides, the high degree of details in the VAS scales is exceptionally beneficial when small differences can be detected between the evaluated subjects [24]. For instance, if 13 criteria ought to be assessed on the 7-point Likert scales, criteria of the different importance might fall into the same category making them indistinguishable from one another (Figure 1).


*Symmetry* **2020**, *12*, 1641 3 of 20

**Figure 1.** Matrix questions where the same set of criteria is assessed with the visual analogue scales (VAS) matrix (**a**) and with 7-point Likert-scales (**b**). **Figure 1.** Matrix questions where the same set of criteria is assessed with the visual analogue scales (VAS) matrix (**a**) and with 7-point Likert-scales (**b**).

As can be seen, VAS scales are highly sensitive to the respondent's opinion. Due to this sensitivity, VAS scales are widely applied in medical studies and other areas where small differences might be significant. As can be seen, VAS scales are highly sensitive to the respondent's opinion. Due to this sensitivity, VAS scales are widely applied in medical studies and other areas where small differences might be significant.

#### *1.3. Uncertainty of the Collected Data 1.3. Uncertainty of the Collected Data*

VAS scales are easy to understand, administer, and score when implemented in online surveys [25]. Survey-based weighting processes are typically accompanied by the biases of the evaluators and the uncertainty of the experimental conditions. End-aversion bias and the positive skew are also the companions of the VAS scales [26]. End-aversion bias refers to the respondents' reluctance to use extreme categories such as "extremely important" or "absolutely unimportant". It does not affect the mean values of the respondent group, but it reduces the variance of the recorded scores [27]. Positive skew refers to the data distribution situation when the responses are not evenly distributed over the range of the scale but show a positive skew towards the favorable end [28]. VAS scales are easy to understand, administer, and score when implemented in online surveys [25]. Survey-based weighting processes are typically accompanied by the biases of the evaluators and the uncertainty of the experimental conditions. End-aversion bias and the positive skew are also the companions of the VAS scales [26]. End-aversion bias refers to the respondents' reluctance to use extreme categories such as "extremely important" or "absolutely unimportant". It does not affect the mean values of the respondent group, but it reduces the variance of the recorded scores [27]. Positive skew refers to the data distribution situation when the responses are not evenly distributed over the range of the scale but show a positive skew towards the favorable end [28].

Both the end-aversion bias and the positive skew suppose that data points belonging to the different ranges of the VAS scales should be treated unequally. Cautious attitude toward the psychometric features of the response scales and the uncertainty of the collected data is required to ensure the accuracy of the criteria weighting results. A new preference elicitation technique that uses the VAS Matrix for the survey-based data collection and employs the appropriate data processing approach to reduce the uncertainties of the collected data is going to be presented in this paper. Both the end-aversion bias and the positive skew suppose that data points belonging to the different ranges of the VAS scales should be treated unequally. Cautious attitude toward the psychometric features of the response scales and the uncertainty of the collected data is required to ensure the accuracy of the criteria weighting results. A new preference elicitation technique that uses the VAS Matrix for the survey-based data collection and employs the appropriate data processing approach to reduce the uncertainties of the collected data is going to be presented in this paper.

## **2. Criteria Weighting Approaches 2. Criteria Weighting Approaches**

Determination of the criteria weights is an important step of the decision-making processes related to the current state of the economic, social, or environmental aspects [5,29]. Since there is no unique classification of the criteria weighting methods, preference elicitation can be divided into statistical and algebraic, direct and indirect, subjective and objective, compensatory and noncompensatory techniques [30]. Determination of the criteria weights is an important step of the decision-making processes related to the current state of the economic, social, or environmental aspects [5,29]. Since there is no unique classification of the criteria weighting methods, preference elicitation can be divided into statistical and algebraic, direct and indirect, subjective and objective, compensatory and non-compensatory techniques [30].

## *2.1. Subjective and Objective Techniques 2.1. Subjective and Objective Techniques*

Subjective, objective, and integrated approaches are widely used for preference elicitation. Subjective weights are determined solely according to the preference of the decision-makers. This type of preference elicitation is mostly based on pairwise comparison methods like AHP (Analytic Hierarchy Process) [31], DEMATEL (Decision-making Trial and Evaluation Laboratory) [32], SWARA (Step-Wise Weight Assessment Ratio Analysis) [33], or PIPRECIA (Pivot Pairwise Relative Subjective, objective, and integrated approaches are widely used for preference elicitation. Subjective weights are determined solely according to the preference of the decision-makers. This type of preference elicitation is mostly based on pairwise comparison methods like AHP (Analytic Hierarchy Process) [31], DEMATEL (Decision-making Trial and Evaluation Laboratory) [32], SWARA

Criteria Importance Assessment) [34]. Objective weights are typically applied then the influence of

(Step-Wise Weight Assessment Ratio Analysis) [33], or PIPRECIA (Pivot Pairwise Relative Criteria Importance Assessment) [34]. Objective weights are typically applied then the influence of the individual decision-makers should be reduced. The most well-known objective weighting approaches are the entropy method [35], CRITIC (Criteria Importance Through Intercriteria Correlation) [36], FANMA methods [37].

Since the subjective judgments are noticeably affected by the knowledge and experience of the decision-makers, most of the time, weights determined by subjective approaches neglect the objective information [38]. The integrated preference elicitation approaches can be used to achieve the more accurate values of the criteria weights [39]. These approaches focus on the principle of integrating the subjective weights based on the expert's opinion and the information gathered from the criteria data in a mathematical form. For instance, Wang and Lee [40] proposed to integrate objective weights calculated by Shannon's entropy [35] and the subjective weights determined directly by the decision-makers. Saad et al. [41] proposed to weight the criteria combining the Fuzzy Shannon entropy and the subjective weights calculated as the averages of the direct valuations gathered from three decision-makers. The integrated approach that combines objective and subjective weights calculated from the same survey data will be presented in this paper.
