On Ordinal Information-Based Weighting Methods and Comparison Analyses

Abstract

In this paper, we focus on weighting methods within multi-attribute utility/value theory (MAUT/MAVT). In these methods, the decision maker (DM) provides ordinal information about the relative importance of criteria, but also additional information concerning the strength of the differences between the ranked criteria, which can be expressed in different forms, including precise/imprecise cardinal information, ratio-based methods, a ranking of differences, a semantic scale, or preference statements. Although many comparison analyses of weighting methods based on ordinal information have been carried out in the literature, these analyses do not cover all of the available methods, and it is not possible to identify the best one depending on the information provided by the DM. We review the analyses comparing the performance of these weighting methods based on empirical and simulated data using different quality measures. The aim is to identify weighting methods that could be recommended for use in each situation (depending on the available information) or the missing comparison analyses that should be carried out to arrive at a recommendation. We conclude that in the case of additional information in the form of a semantic scale, the cardinal sum reciprocal method can definitively be recommended. However, when only ordinal information is provided by the DM and in cases where additional information is provided in the form of precise/imprecise cardinal information or a ranking of differences, although there are some outstanding methods, further comparison analysis should be carried out to recommend a weighting method.

1. Introduction

The WEB-MAUT-DSS is a web decision support system (DSS) based on the multi-attribute utility theory (MAUT), which consists of an adaptation and improvement of the generic multi-attribute analysis (GMAA) system [1,2]. It is based on the decision analysis (DA) methodology [3], which has been widely used to address complex real-world decision-making problems on the basis of an additive multi-attribute utility model.

DA consists of the following steps: structuring the problem (building an objective hierarchy and establishing attributes to indicate the extent to which the lowest-level objectives are achieved); identifying the feasible alternatives and their performances in terms of the attributes and uncertainty (if necessary); quantifying preferences, which involves assessing single attribute utility/value functions, as well as the relative importance of criteria; evaluating alternatives by means of a multi-attribute utility/value function; and conducting sensitivity analyses to check the robustness of the results.

Regarding alternative evaluation, the additive multi-attribute utility/value function is considered a valid approach in most real decision-making problems for the reasons described in [4,5].

However, most complex decision-making problems involve imprecise information, mainly because it is impossible to exactly predict the performances of the alternatives under consideration, which may be derived from statistical methods. Additionally, it is often not easy to elicit precise weights, since decision makers (DMs) may find it difficult to compare criteria (some weighting methods are too cognitively demanding) or may not want to reveal their preferences in public. Alternatively, the imprecision of weights may be the result of a negotiation process. This situation is usually referred to as decision-making with imprecise information, with incomplete information or with partial information [6], which has been widely addressed in the literature focusing mainly on the weight elicitation process [7,8,9,10,11].

WEB-MAUT-DSS accounts for uncertainty about the alternative performance and admits imprecise information about the DMs’ preferences in both the assessment of component utilities and weight elicitation, which leads to classes of utility functions and weight intervals, respectively.

Regarding weight elicitation, the GMAA originally provided two methods to hierarchically elicit the relative importance of criteria. Local weights were first elicited at the different levels and branches of the objective hierarchy by means of a direct assignment or a method based on trade-offs, accounting for imprecision concerning DM responses by means of ranges of responses to the probability question that the DM is asked. Then, the attribute weights, $w_j$, were computed by multiplying the local weights in the paths from the corresponding attribute to the overall objective.

One of the improvements provided by WEB-MAUT-DSS over the GMAA system is the incorporation of new weighting methods apart from direct assignment or methods based on trade-offs. Specifically, weight elicitation methods based on ordinal information can be considered, where the DM has to provide ordinal information about the relative importance of attributes/criteria, i.e., a ranking of importance of the criteria. Moreover, the DM is allowed to provide additional information about the strength of the differences between the ranked criteria if they are able or consider it appropriate to do so. This additional information could be provided in different forms, including a ranking of differences, a semantic scale, precise/imprecise cardinal information, ratio-based methods, and preference statements. The DM can decide which information they are able/willing to provide (or with which they feel more comfortable), and then the respective weighting method can be used.

However, different weighting methods are available in the literature to deal with only ordinal information within MAVT/MAUT, as well as with additional information about the strength of the differences between the ranked criteria. In addition, different comparative analyses have been carried out to analyse their performance based on empirical and simulated data using different quality measures. The question is, then, is it possible to recommend weighting methods for incorporation into the WEB-MAUT-DSS on the basis of the comparison analyses in the literature? Or should further comparison analyses be carried out to arrive at recommendations for different situations depending on the available information?

In this paper, we try to answer these questions. To achieve this, we first review the different weighting methods reported in the literature dealing with ordinal information on weights or which also employ additional available information. We then look at papers including comparison analyses of weighting method performance, analyzing the conclusions drawn by the different authors in order to arrive at a recommendation or at least identify missing comparison analyses.

The paper is structured as follows: in Section 2, weighting methods accounting for ordinal information on weights are reviewed, including surrogate weighting methods and methods based on the notions of pairwise and absolute dominance, as well as other techniques like stochastic multi-criteria acceptability analysis (SMAA2) or simulation. Section 3 deals with weighting methods in which additional information is also available. A review of comparison analyses in the literature involving the weighting methods under consideration is reported in Section 4. The results of the different comparison analyses are aggregated and discussed in Section 5. Finally, some conclusions and future research lines are provided in Section 6.

2. Weighting Methods Accounting for Ordinal Information

We consider an MAUT/MAVT context in which an additive model is used to evaluate the alternatives under consideration:

$$u\left(A^j\right)=\sum _{i=1}^n{w}_j{u}_i\left(x_i^j\right),$$

(1)

where n is the number of attributes/criteria, $x_i^j$ is the impact/performance of the i-th attribute for alternative $A^j$, $u_i$ represents the single utility/value assigned to the respective attribute $X_i$, and $w_i$ is the weight of the i-th attribute.

Most authors agree with the classification of weighting methods into subjective and objective approaches [12,13]. Subjective approaches reflect the DM’s subjective opinion and intuition, which have a direct influence on the outcome of the decision-making process. Objective approaches determine the criteria weights on the basis of the information contained in a decision-making matrix by means of mathematical models, neglecting the DM’s opinion. Although both approaches are recognized as efficient in dealing with real-life multicriteria decision-making problems, most of the weighting methods in the literature are based on the DM’s cognitive preferences. The hybridization of both concepts has been discussed at length in [14,15], and other classifications are available in [12].

In this paper, we consider subjective MAUT/MAVT methods with partial/imprecise/ incomplete information where ordinal information is available on weights, i.e., the DM is able to provide a ranking of importance of criteria/attributes, arranged in descending order from the most to the least important ones:

$$\mathbf{w}\in W=\left\{\mathbf{w}=(w_1,...,w_n)|w_1\ge w_2\ge ...\ge w_n\ge 0,\sum _{i=1}^n{w}_i=1\right\}.$$

(2)

Different approaches can be found in the literature to deal with such a situation, particularly surrogate weighting methods and methods based on the notions of pairwise and absolute dominance, as well as other techniques like SMAA2 or simulation (see Figure 1).

2.1. Surrogate Weighting Methods

In surrogate weighting methods (SWMs), a weight vector is selected from a set of admissible weights to represent the set, which is then used to evaluate the alternatives on the basis of MAVT/MAUT.

The original SWMs were proposed in [16] (equal weights (EW)), [17] (rank sum (RS), rank exponent (RE) and rank reciprocal (RR) weights), and [18] (rank-order centroid (ROC) weights).

Table 1. Surrogate weighting methods (SWMs).

Equal weights (EW)	${\mathit{w}}_{\mathit{i}}=\frac{1}{\mathit{n}},$  $\mathit{i}=1,...,\mathit{n}$
Rank sum (RS)	$w_i={\displaystyle \frac{n+1-i}{{\displaystyle \sum _{j=1}^n}(n+1-j)}}={\displaystyle \frac{2(n+1-i)}{n(n+1)}},$  $i=1,\dots ,n$
Rank exponent (RE)	$w_i={\displaystyle \frac{{(n+1-i)}^p}{{\displaystyle \sum _{j=1}^n}{(n+1-j)}^p}},$  $i=1,\dots ,n$
Rank reciprocal (RR)	$w_i={\displaystyle \frac{1/i}{{\displaystyle \sum _{j=1}^n}1/j}},$  $i=1,\dots ,n$
Rank-order centroid (ROC)	$w_i={\displaystyle \frac{{\displaystyle \sum _{j=1}^n}1/j}{n}},$  $i=1,\dots ,n$
Equal ratio fixed (ERF)	$w_i\left(r\right)={\displaystyle \frac{r^i}{{\displaystyle \sum _{j=1}^n}{r}^j}},$  $i=1,\dots ,n$
Geometric weights (GW)	$w_i={\displaystyle \frac{100}{{\left(\sqrt{2}\right)}^{i-1}}},$  $i=1,\dots ,n$
Variable-slope linear (VSL)	$w_i={\displaystyle \frac{1-(3.19514+\frac{37.75756}{n})\left(\frac{i-1}{100}\right)}{{\displaystyle \sum _{j=1}^n}(1-(3.19514+\frac{37.75756}{n})\left(\frac{j-1}{100}\right))}},$  $i=1,\dots ,n$
Maximum entropy ordered (MEO)	$w_i=\sqrt[n-1]{w_1^{n-i}\ast w_n^{i-1}},$  $i=2,\dots ,n-1$
Weighted averaging (MEOWA)	w1, wn predetermined
Least-squared ordered weighted	$w_i\left(\theta \right)={\displaystyle \frac{1}{n}}+{\displaystyle \frac{6(2i-n-1)}{n(n+1)}}+({\displaystyle \frac{1}{2}}-\theta )$  $i=1,\dots ,n$
Averaging (LSOWA)
Sum reciprocal (SR)	$w_i={\displaystyle \frac{1/i+\frac{n+1-i}{n}}{{\displaystyle \sum _{j=1}^n}(1/j+\frac{n+1-j}{n})}},$  $i=1,\dots ,n$
Rank order total (ROT)	$w_i={\displaystyle \frac{3(n+1-i)(n+2-i)}{n(n+1)(n+2)}},$  $i=1,\dots ,n$
Geometric sum (GS)	$w_i={\displaystyle \frac{s^{i-1}}{{\displaystyle \sum _{j=1}^n}{s}^{j-1}}},$  $i=1,\dots ,n$
Generalized rank sum (GRS)	$w_i\left(\delta \right)={\displaystyle \frac{(\delta -1){\delta }^{n-i}}{{\delta }^n-1}},$  $i=1,\dots ,n$
Improved ROC (IROC)	$w_i({\phi }_{in},\dots ,{\phi }_{nn})={\displaystyle \sum _{j=1}^n}({\phi }_{jn}/j),$  $i=1,\dots ,n$
Generalized ROC (GROC)	$w_i\left(\phi \right)={\displaystyle \sum _{j=1}^n}{\displaystyle \frac{{\phi }^{n-j}}{n\sum _{k=1}^{n-j-i}{\phi }^{k-1}}},$  $i=1,\dots ,n$
Rank order logarithm (ROL)	$w_i={\displaystyle \frac{log\left(\frac{i}{n+1}\right)}{{\displaystyle \sum _{j=1}^n}log\left(\frac{j}{n+1}\right)}}$, $i=1,\dots ,n$

This Table summarizes the most common SMWs in the literature and their mathematical formulas.

shows how weights are computed for SWMs. The ROC and RR methods attach more importance to the best rankings, whereas all the attributes are equally important in the EW method, and the RS method places equal emphasis on all the weight rankings (i.e., $w_1-w_2=w_2-w_3=\cdots =w_{n-1}-w_n$). The RE method is a generalization of the RS method, which is reduced to the EW and the RS methods if p is 0 or 1, respectively. Note that the SMART exploiting ranks (SMARTER) method [19] incorporates a component to elicit ordinal information on weights, from which ROC weights are derived.

Different SWMs have been proposed since then. They are summarized in the order in which they were published, as follows.

In the equal ratio fixed (ERF) method [20], each weight is given a ratio to the next largest weight, and $w_1$ is determined as $\sum w_i=1$. Alternatively, it is suggested that $w_1/w_n$ be elicited, thus making $r={(w_1/w_n)}^{\left(1\right((n-1))}$ in [21]. The geometric weights (GW) method wass also proposed in [20], where weights decrease exponentially by a factor of $\sqrt{2}$.

In the rank order distribution (ROD) method [22], the relative $w_2$ through $w_n$ are uniformly distributed at the interval $(0,w_1)$, and then the surrogate weights are obtained by computing their means and through normalization. The authors provided formulas for $n=2$ to 5, whereas the corresponding surrogate weights up to $n=10$ were provided in [23].

Figure 1. Weighting methods based on ordinal information and accounting for additional information. [7,11,19,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48].

Figure 1. Weighting methods based on ordinal information and accounting for additional information. [7,11,19,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48].

In [49], the variable-slope linear (VSL) method was proposed, where the criteria weights are assumed to be linear with a variable slope, and least-squares regression is applied to identify the best of three different (quadratic, inverse, and exponential) models to estimate the slope.

Maximum entropy ordered weighted averaging (MEOWA) weights were proposed in [50,51]. These are a special class of the ordered weighted averaging (OWA) method [52] that has the maximal entropy for a specified value of attitudinal character $\theta$, which, like Hurwicz’s $\alpha$, denotes the DM’s degree of optimism. The underlying notion of the least-squared ordered weighted averaging (LSOWA) method [25] is to compute the weights that minimize the sum of deviations from EW, since entropy is maximized when all weights are equal. It also extends the OWA method, deriving the OWA weights that are, as far as possible, evenly spread out around EW. The LSOWA method ensures that weights are the weight space if $\theta \in ({\displaystyle \frac{n-2}{3n-3}},{\displaystyle \frac{2n-1}{3n-3}})$ and $\theta \ge 0.5$ (see Table 1).

Combinations of SWMs have also been proposed by different authors, such as the sum reciprocal (SR) method [26], an additive combination of RS and RR weight functions, and the rank order total (ROT) method [53], a combination of the RS method and the ROC method. In the geometric sum (GS) method [26], the resulting weights multiplicatively reflect the rank order, including the parameter s (see Table 1).

The aim of the generalized rank sum (GRS) method [54] is to maximize the distance from the closest inequality constraint of the weight space, which is tantamount to finding the middlemost solution to the constraints. The result is actually a simple extraction from the finite geometric series, and $\delta$ (see Table 1) has to be more than 1 to ensure the order relations of the weight space. Note that the GRS method reduces to the EW method when $\delta$ is close to 1.

Finally, the most recent proposals regarding SWMs consist of revised versions of the ROC method. The improved ROC (IROC) method [55] claims, contrary to the ROC method, that DM preferences at the corners of the weight space may be similar rather than equal and establishes the corresponding different coefficients ${\phi }_{jn}$ by means of simulation techniques. On the other hand, the generalized ROC (GROC) method [56] incorporates an upper bound $\phi$ for the ratio $w_i/w_{i+1}$, which implies that a given weight needs to be multiplied by $\phi$ to be equal to the next highest weight, leading to the following weight space:

$$\left\{(w_1,\dots ,w_n)|w_1\ge \phi w_2,w_2\ge \phi w_3,\dots ,w_{n-1}\ge \phi w_n,\sum _{i=1}^n{w}_i=1,w_i\ge 0\right\}.$$

(3)

Note that $\phi =1$ in the ROC method. Meanwhile, another revised version of the ROC method, the rank order logarithm (ROL) method, was proposed in [57] (see Table 1).

Note that one or more parameters have to be set in some of the above SWMs, which may imply the application of simulation techniques (IROC). Alternatively, the DM may be required to provide additional information, such as $w_1/w_n$ in the ERF method or an upper bound for the ratio $w_i/w_{i+1}$ in the GROC method. The $\theta$, $\delta$, and s parameters also have to be set in the LSOWA, GRS, and GS methods, respectively.

2.2. Methods Based on Pairwise and Absolute Dominance Notions

Pairwise and absolute dominance notions can be used to directly recommend the best alternative and fully rank alternatives rather than selecting a weight vector from a set of admissible weights. Absolute dominance considers the following linear optimization problems:

$$U_k=max\{{\mathbf{wu}}_k|\mathbf{w}\in \mathbf{W}\}\mathrm{and}{L}_k=min\{{\mathbf{wu}}_k|\mathbf{w}\in \mathbf{W}\},$$

(4)

and alternative $A_k$ absolutely dominates $A_j$ if $L_k>U_j$, i.e., the lower bound of $A_k$ exceeds the upper bound of $A_j$.

The use of absolute dominance values is exemplified by the modification of four classical decision rules to encompass an imprecise decision context concerning weights and component values/utilities [27,28]:

• The maximax or optimist, based on their maximum guaranteed value, i.e., $max\{U_j,j=1,\dots ,m\}$.
• The maximin or pessimist, based on their minimum guaranteed value, i.e., $max\{L_j,j=1,\dots ,m\}$.
• The minimax regret rule, based on the maximum loss of value with respect to a better alternative, i.e., $min\{M{R}_k,k=1,\dots ,m\}$, where $M{R}_k$ represents the maximum regret incurred when choosing alternative j, i.e., $M{R}_k=max\{max\{u\left(A_j\right)-u\left(A_k\right)|\mathbf{w}\in \mathbf{W}\}\forall j\ne k\}$.
• The central value rule, based on the midpoint of the range of possible performances, i.e., $max\{{\displaystyle \frac{U_j+L_j}{2}},j=1,\dots ,m\}$.

Two additional rules were proposed in [11]: the quasi-optimality rule and the quasi-dominance rule.

Regarding pairwise dominance, linear programming displays dominance and potential optimality for alternatives when information about weights, utilities, and alternative performances is available in the literature [29,30,31,58,59], whereas the concepts of weak potential optimality and strong potential optimality were proposed in [60].

In the rank inclusion in criteria hierarchies (RICH) method [61], the DM is allowed to provide a ranking of weights or to specify subsets containing the most important attributes, leading to decision recommendations on the basis of the computation of dominance relations and decision rules. It was implemented in a user-friendly decision support system, RICH Decisions (http://decisionarium.aalto.fi (accessed on 26 August 2024), [62]).

Another approach accounting for pairwise dominance consists of considering information about each alternative’s intensity of dominance, known as dominance measuring methods (DMMs) [7,32,63,64]. DMMs are based on the computation of a dominance matrix including pairwise dominance values, which are exploited in different ways to derive measures of dominance to rank the alternatives under consideration.

The first DMMs, OUT I, and OUT II, were proposed in [63], where dominating (OUT I) and dominated measures are computed for each alternative, and then a net dominance accounting for the difference between them is used to rank the alternatives (OUT II).

The methods in [63] were extended in [32], where dominating and dominated measures are combined into a dominance intensity reducing the duplicate information involved in the computations (dominance measuring extension 1 (DME1) method). The dominance measuring extension 2 (DME2) method was also proposed in [32], which derives a global dominance intensity index to rank alternatives on the basis that

$$D_{kl}\le {\mathbf{w}}^T({\mathbf{v}}_k-{\mathbf{v}}_l)\le -D_{lk},\forall \mathbf{w}\in W,{\mathbf{v}}_k,{\mathbf{v}}_l\in V_{kl},$$

(5)

where $D_{kl}$ is the pairwise dominance value between alternatives $A_k$ and $A_l$, and W and $V_{kl}$ define the feasible region for weights and values for each attribute, respectively.

A new dominance intensity method based on triangular fuzzy numbers and a distance notion was proposed in [7]. This method, known as dominance intensity method (DIM), incorporates the DM’s attitude toward risk into the analysis. Instead of pairwise dominance ($D_{kl}$), the DIM method uses the following values:

$$v_{kl}=\sum _{j=1}^n{w}_j^c{v}_{kj}^c-\sum _{j=1}^n{w}_j^c{v}_{lj}^c,$$

(6)

where $(w_1^c,...w_n^c)$ is the centroid of the polytope representing the weight space and $(v_{k1}^c,v_{l1}^c),$$...,(v_{kn}^c,v_{ln}^c)$ are the centroids of the polytopes in the n attributes delimited by the constraints accounting for alternatives $A_k$ and $A_j$.

2.3. Other Weighting Methods Based on Ordinal Information

Probability distributions and partial preference information are used to represent inaccurate uncertain criteria values and weights in the stochastic multi-criteria acceptability analysis (SMAA) method [65] in order to explore the weight space and identify the scores that each alternative would need to achieve if it were to be the preferred option or ranked in a specific position. The SMAA-2 method [33] extends SMAA by incorporating various types of preference information, including a priority order for the criteria.

In addition, Monte Carlo simulation techniques could also be used to deal with ordinal information on weights. For instance, the GMAA system [1,2], an MAUT decision support system (DSS) accounting for imprecision concerning DM preferences and uncertainty about alternative performances, includes a sensitivity analysis tool that randomly generates weights while preserving a total or partial attribute rank order. The GMAA system computes several statistics about the rankings of each alternative and provides multiple boxplots for the alternatives that can be useful for discarding and/or identifying the most preferred available alternatives.

3. Weighting Methods Accounting for Additional Information

In many real decision-making problems, the DM is unable/unwilling to provide additional information on the ranking of the importance of attributes, as pointed out in Section 1, in which the above methods are applicable. However, the DM might provide information about the differences between the weights of consecutive criteria in different ways (see Figure 1).

One possibility is the provision of precise/imprecise cardinal information, or ratios. Alternatively, the DM could provide a ranking of the differences between the weights of consecutive criteria, although a semantic scale could also be used. In addition, additional information on the basis of preference statements could be available. These possibilities are described in detail below.

3.1. Precise/Imprecise Cardinal Information

Different methods using scores to derive the strength of the differences between the weights of criteria can be found in the literature, including the direct rating, SWING, and simple multi-attribute rating technique using SWING (SMARTS) methods.

In direct rating [34], the DM first ranks all the criteria according to their importance and then rates each criterion on a scale of 0–100. Indeed, a high score means that the factor is important. Then, some form of normalization can be applied to this scoring operation.

The SWING method [35] explicitly incorporates the attribute ranges in the elicitation questions. The DMs are asked to consider the worst consequence for each criterion and to identify which criterion they would most prefer to change from its worst to its best outcome. This criterion is assigned the highest number of points—for example, 100. The procedure is then repeated with the remaining criteria. The criterion with the next most important swing is assigned a number relative to the most important criterion (thus the points scored by criteria denote their relative importance), and so on. Finally, the scored points are normalized to a sum of one.

The simple multi-attribute rating technique (SMART) and the SWING method were later combined into the SMARTS method [19], where SWING was used to set up the order of importance of the criteria.

Simos’ method proposes that the DM should rank criteria by placing cards containing the names of the criteria under consideration in order of the least to the most important (where cards containing equally important criteria are grouped together) [36,37]. Then, the DM is asked about the importance of successive criteria in the ranking by introducing white cards between two successive cards (or subset of cards). Finally, the collected information is processed to derive the normalized weights, $w_i$. Simos’ method was extended in [38] to overcome some objections to how the normalized weights are derived.

The cardinal and rank ordering of criteria (CROC) method was proposed in [39,40]. CROC is a two-stage distance-based method where the DM provides the magnitude of the differences between the ranked criteria.

In the first stage, the DM is asked to provide information about their preferences. This stage consists of three steps. First, ordinal information is collected in a criteria ranking with a SWING weighting style, producing a Hasse diagram. Then, the difference between the weights for the most and the least important criteria is assessed. Note that it is easier for DMs to provide information on the most and least important criteria than those in between. Finally, the criteria are equally distributed along a slider (using a graphical user interface), i.e., all magnitudes of the differences are initially equal, and each criterion is associated with a clouded region with a length equal to the default distance. Then, the DM is asked to adjust the distances between criteria to represent information of cardinal importance between them, leading in some cases to overlaps between the respective clouded regions.

In the second stage, the information provided by the DM is translated into a constraint set for weight variables, enabling the use of decision analysis methods to model imprecision by means of linear constraints. This process is explained in detail in [41].

This method is capable of handling imprecision in two ways: by means of intervals (where the DM statement $x_i$ in the slider is interpreted as an interval such that $w_i\in [x_i/\sum x_i-a_i,x_i/\sum x_i+b_i]$, with $a_i,b_i\in (0,1]$ representing the degree of confidence in the weights) and/or by means of weight comparison.

Although the step-wise weight assessment ratio analysis (SWARA) method [42,66] was originally proposed as a dispute resolution method in a group decision-making context, it is a score-based method which has also been adapted to deal with a single DM.

In SWARA, after scoring ($s{c}_i$, $i=1,...,n$), the differences in the consecutive criteria are determined for incorporation into the weight elicitation as follows:

$$Y_i={\displaystyle \frac{Y_{i-1}}{1+(s{c}_i-s{c}_{i-1})}},i=2,...,n,$$

(7)

with $Y_1=1$ and where $s{c}_i$ is the score provided by the DM for the i-th criterion.

Once the values of $Y_i,i=2,...,n$ are computed, a normalization process is performed to derive the weights: $w_i={\displaystyle \frac{Y_i}{{\sum }_i^n{Y}_i}},i=1,...,n$.

3.2. Ratio-Based Methods

In the simple multi-attribute rating technique (SMART) [67], the DM is asked to rank criteria according to their importance from worst to best. Then, 10 points are assigned to the least important criterion, and an increasing number of points are assigned to the other criteria to address their importance relative to the least important criterion. The derived weights are normalized.

The full consistency method (FUCOM) based on pairwise comparisons of criteria was proposed in [68]. FUCOM consists of three steps. In the first step, the criteria are ranked according to their significance from the highest to lowest weight. In the second step, the ranked criteria are compared, leading to comparative priorities

$$({\phi }_{1/2},{\phi }_{2/3}...,{\phi }_{n-1/n}),$$

(8)

where ${\phi }_{i/i+1}$ represents the significance (priority) that the i-th criterion has compared to the $(i+1)$-th criterion of the ranking. Note that only $n-1$ comparisons are necessary. Finally, in the third step, the weights are computed, taking into account the fact that they should satisfy the following two conditions:

• The weight ratios should be equal to the comparative priorities, i.e., ${\displaystyle \frac{w_i}{w_{i+1}}}={\phi }_{i/i+1},i=1,\dots ,n-1$.
• The weight should satisfy the condition of mathematical transitivity, i.e., ${\phi }_{i/i+1}\otimes {\phi }_{i+1/i+2}={\phi }_{i/i+2}$ and then ${\displaystyle \frac{w_i}{w_{i+2}}}={\phi }_{i/i+1}\otimes {\phi }_{i+1/i+2}$.

Full consistency is achieved if transitivity is fully respected. In order to meet such conditions, the following constraints are established on weights $|{\displaystyle \frac{w_i}{w_{i+1}}}-{\phi }_{i/i+1}|<\chi$ and $|{\displaystyle \frac{w_i}{w_{i+2}}}-{\phi }_{i/i+1}\otimes {\phi }_{i+1/i+2}|<\chi$, with the minimization of $\chi$ ($\chi =0$ implies full consistency). Then, weights are obtained by solving the minimization optimization problem (where ${\sum }_i{w}_i=1$ is also incorporated as a constraint) with the corresponding degree of consistency, $DFC\left(\chi \right)$.

3.3. Ranking of Differences between the Weights of Consecutive Criteria

The first possibility that we consider consists of providing a ranking of the differences between the weights of consecutive criteria. Given that the DM has established that $w_1\ge w_2\ge ...\ge w_n\ge 0$, we define ${\mathsf{\Delta }}_j=w_j-w_{j-1}$, $j=1,...,n-1$, and a ranking on ${\mathsf{\Delta }}_j$ elements has to be provided by the DM. For instance, the ranking ${\mathsf{\Delta }}_2\ge {\mathsf{\Delta }}_1\ge {\mathsf{\Delta }}_4\ge {\mathsf{\Delta }}_3$ would correspond to the example shown in Figure 2.

In [43], this kind of information is considered in the computation of dominance in additive weighting models, whereas an effective algorithm for an additive utility function and the partial information represented by a set of linear constraints was proposed in [44], including ranking of weights or alternatives, paired comparison of weights or alternatives, and strengths of preferences of weights or alternatives. Ref. [69] extended the work in [44].

The $\mathsf{\Delta }ROC$ rule deals with ordinal information about the position of alternatives and about the difference in values between consecutive alternatives with respect to the different criteria under consideration and ordinal information about weights [70]. The performance of $\mathsf{\Delta }ROC$ is compared with that of DMMs in [7]. Note, however, that the ranking of differences refers to consecutive alternatives rather than weights in $\mathsf{\Delta }ROC$. Thus, it cannot ultimately be classified as a weighting method measuring differences between the weights of consecutive criteria. Therefore, it is not considered further in this study.

More recently, the minimizing squared deviations from extreme points (MSD) was proposed in [45], which extends the ROC weighting method by seeking out surrogate weights adjusted to the barycenter of the weight set. MSD accounts for the strict ranking of weights, ranking with multiples, the ranking of differences in weights, or a mixture of these methods, and it was proven, using simulation techniques, to outperform a linear programming-based weighting method.

The ranking of differences between the weights of consecutive criteria could also be addressed using approaches based on pairwise and absolute dominance, affecting the computation of $U_k$ and $L_k$ in the modification of classical decision rules and of the pairwise dominance matrix in DMMs. Note that linear constraints would be added to the corresponding optimization problems, and a partial, rather than a complete, ranking of differences could be provided, which is less stressful for DMs.

3.4. Semantic Scales to Account for Differences between the Weights of Consecutive Criteria

Another possibility for representing the strength of the differences between the weights of consecutive criteria is to use a semantic scale.

For instance, the following scale was proposed in [71,72]:

• $\ge 0$, equally important;
• $\ge 1$, slightly more important;
• $\ge 2$, more important (clearly more important), and
• $\ge 3$, much more important.

This scale is used in [72] to extend the SR method, leading to the DSR method, whereas the SW methods RS, RR, ROC, and SR are adapted in [71] to account for this scale, leading to the so-called cardinal rank sum (CRS), cardinal rank reciprocal (CRR), cardinal rank-order centroid (CRC), and cardinal sum reciprocal (CSR) methods, respectively. All of these extended methods derive a weight vector in the same way as the original SR and SW methods. Note that the DSR method in [72] is the same as the CSR method in [71]. In [73], the GS method was extended in the same way, leading to the CGS method. In addition, the CAR method proposed in [46] uses the semantic scale to elicit the values of the alternatives under each criterion in a similar way to the weights.

The same scale was used in [74] to propose the partial SWING (P-SWING) method, a refined version of the SWING method [35] that allows for intermediate comparisons, avoiding synthetic constructs to improve understanding. However, these intermediate comparisons refer to the differences in alternative values for different criteria rather than differences between the weights of consecutive criteria. Thus, we will discard this method from further analysis.

3.5. Additional Information on the Basis of Preference Statements

A systematic elicitation process based on an extended difference ranking (EDR) method was proposed in [72], including the DM’s strength of preference order over weights. Specifically, it is stated that the set of attributes can be partitioned as $\{X_I,X_{-I}\}$, where I defines an arbitrary subset of $X=\{X_1,...,X_n\}$ such that $I\cup -I=\{1,2,...,n\}$. If the decision maker asserts $({\overline{x}}_I,{\underline{x}}_{-I})\gtrsim ({\underline{x}}_I,{\overline{x}}_{-I})$ for a specific I, the constraint ${\displaystyle \sum _{i\in I}}{w}_i\ge {\displaystyle \sum _{i\in -I}}{w}_i$ can be added to the set of constraints. The authors propose using these new constraints to refine the weights derived from the DSR method, also proposed in [72], where a semantic scale is used to represent the strength of the differences between the weights of consecutive criteria (see Section 3.4). Unlike the above methods, EDR outputs an interval weight vector.

In the flexible and interactive tradeoff (FITradeoff) method, the DM is required to provide less information than in the standard tradeoff procedure [47]. FITradeoff systematically evaluates the possibility of finding a solution to the problem while the elicitation process is ongoing, using partial information elicited from the DM at any point in the process to solve a linear programming problem in order to reduce the subset of potentially optimal alternatives.

In the first step, the order of the weights $w_i$ is obtained $(w_1>w_2>\cdots >w_n)$, whereas the subsequent steps yield indifference relations in order to find the value of $w_i$. In step i, the DM is asked to provide the value $x_i$ such that they are indifferent regarding the consequences $A=({\underline{x}}_1,\cdots ,{\underline{x}}_{i-1},{\overline{x}}_i,{\underline{x}}_{i+1},\cdots ,{\underline{x}}_n)$ and $B=({\underline{x}}_1,\cdots ,{\underline{x}}_i,x_{i+1}^I,{\underline{x}}_{i+2},\cdots ,{\underline{x}}_n)$, with ${\underline{x}}_i$ and ${\overline{x}}_i$ being the worst and best value in attribute $X_i$, respectively, and thus, ${\underline{x}}_i=0$ and ${\overline{x}}_i=1$. Then, as $v\left(A\right)=v\left(B\right)$, we have $w_1=w_{i+1}v\left(x_{i+1}^I\right)$. Taking into account the fact that ${\sum }_i{w}_i=1$, we would need $n-1$ similar equations to find the values of the weights ($w_i$). If the DM is unable to specify $x_i^I$, they can alternatively provide an interval $x_i^I\in (x_i^{\prime },x_i^{″})$, so that $w_i>w_{i+1}v\left(x_{i+1}^{\prime }\right)$ and $w_i<w_{i+1}v\left(x_{i+1}^{″}\right)$.

Note that, in each iteration, the FITradeoff method solves the corresponding optimization problems with the available weight space to identify non-dominated and potentially optimal alternatives. Dominated alternatives are removed from the analysis, and the method ends if a unique non-dominated or potentially optimal alternative is found. Otherwise, the values for $x_{i+1}^{\prime }$ and $x_{i+1}^{″}$ are elicited in the next step. Thus, the weight space is progressively reduced, and more alternatives can be identified as dominated, thereby possibly reducing the number of DM preference statements.

The use of data visualization in a new method based on FITradeoff is proposed in [75], whereas its implementation in a decision support system (www.fitradeoff.org, accessed on 26 August 2024) is described in [76], accounting for a neuroscience-based decision-making approach in order to modulate changes in the decision-making process and in the design of the DSS.

4. Comparison Analyses of Weight Elicitation Methods

When a new weighting method is proposed, it is usual practice to carry out a performance comparison analysis against other methods in the literature. In addition, papers focused on comparison analyses of weighting methods can also be found in the literature. In this section, we conduct a review of the comparison studies existing in the literature, where different information on weights is provided by the DM, ranging from a simple ranking of the weights to additional information as described above.

We used the Citation Gecko search tool to carry out the review process, taking two seeds corresponding the most recent papers we found in which a weighting method based on ordinal information, or a comparison analysis of such methods, was performed. We completed the search directly using combinations of keywords such as weight, ordinal information, comparison, MAUT, surrogate weights, etc., in ScienceDirect, Scopus, and PubMEd.

Table 2. Comparison analyses in the literature (SWMs) I.

Papers	Data	Compared Methods	Quality Measures
Stillwell et al. (1981) [17]	Empirical	EW, RS, RR, RE	Kendall’s $\tau$
Barron and Barret (1996) [77]	Simulations	EW, RS, RR, ROC	Hit ratio, proportion of
			maximum value range.
Barron and Barret (1996) [18]	Simulations	EW, RS, RR, ROC	Average MAV, empirical
			ranks, average value loss.
Jia et al. (1998) [78]	Simulations	EW, RS, ROC, Ratio weights	Hit ratio
Pöyhönen and Hämäläinen (2001) [79]	Empirical	AHP, direct point allocation	Inconsistencies in preferences,
		SMART, SWING, tradeoff	use of numbers to describe
			preferences, spread of weights,
			similarity of weights, overall
			scores of alternatives.
Bottomley and Doyle (2001) [48]	Both	Direct rating, Max100, Min10	Kendall’s $\tau$, internal consistency,
			convergent validity.
Roberts and Goodwin (2002) [22]	Simulations	RR, RS, ROC, ROD	Hit ratio, average value loss
Noh and Lee (2003) [80]	Empirical	ROC, AHP, a fuzzy method	Correlation, number of pairwise
			comparisons, ease of use.
Ahn and Park (2008) [63]	Simulations	Decision rules (CENT)	Hit ratio, Kendall’s $\tau$
		OUT I, OUT II
		EW, RS, RR, ROC
Sarabando et al. (2009) [11]	Simulations	Decision rules, ROC	Hit ratio, mean position of
			the supposedly best alternative
			in each rule’s ranking, proportion
			of cases where the position was
			1, 2, 3, 4 or higher.
Riabacke et al. (2009) [40]	Empirical	CROC, SMART, direct rating	cognitive effort, practical
			usefulness, consistency.
Ahn (2011) [81]	Simulations	MEOWA, ROC, RR, RS, EW	Hit ratio, Kendall’s $\tau$
Roszkowska (2013) [82]	Empirical	EW, RS, RR, ROC	Theoretical rationale,
			choice accuracy.
Mateos et al. (2014) [32]	Simulations	RS, RR, ROC, EW	Hit ratio, Kendall’s $\tau$
		Decision rules
		OUT I, OUT II, DME1, DME2
Aguayo et al. (2014) [7]	Simulations	OUT I, DME1, DME2, DIM	Hit ratio, Kendall’s $\tau$
Danielson and Ekenberg (2014) [26]	Simulations	ROC, SR, RS, RR, GR	Hit ratio

and

Table 3. Comparison analyses in the literature (SWMs) II.

Papers	Data	Compared Methods	Quality Measures
Larsson et al. (2015) [41]	Empirical	CROC, SMART, direct rating	Ease, consistency
Danielson et al. (2016) [46]	Both	SMART, CAR, AHP	Hit ratio, Kendall’s $\tau$,
			efficiency, ease of use, ease
			of communication, time efficiency,
			cognitive correctness.
Alfares and Duffuaa (2016) [83]	Simulations	RR, RS, ROC, GW, LVS	Mean absolute percentage error,
			least significance distance.
Danielson et al. (2017) [71]	Simulations	ROC, RE, SR, RR, EW	Hit ratio, mean square spread,
		Simos’ methods	mean square variation
Danielson and Ekenberg (2017) [84]	Simulations	ROC, SR	Hit ratio, standard deviation
		CRC, CRS, CRR, CSR
		Simos’ methods
Ahn (2017) [45]	Simulations	MSD, Slack	Hit ratio, Kendall’s $\tau$
de Almeida et al. (2018) [85]	Simulations	EW, RS, RR, ROC	Hit ratio, frequency of cases
			with the perfect consistency.
Kunsch and Ishizaka (2019) [86]	Simulations	ROC, RR, RS, EW	Hit ratio, Kendall’s $\tau$, value loss
Németh et al. (2019) [87]	Simulations	Direct rating, SMARTS, AHP,	Resource requirement, software
		discrete choice experiments	requirement, chance of bias,
		conjoint analysis	general complexity.
Burk and Nering (2023) [23]	Empirical	RS, REF, RR, SR, ERF,	Euclidean distance, mean absolute
		ROC, LVS, ROD	deviation, maximum absolute
			deviation, Kullback-Leibler
			divergence.
Lakmayer et al. (2023) [8]	Simulations	ROC, SR, RR, RS	Hit ratio,
		CRC, CRS, CRR, CSR	standard deviation, mean.
		DIM
Hatefi et al. (2023) [55]	Simulations	EW, RS, RE, RR, ROC,	Steepness, nonlinearity,
		VSL, LSOWA, SR, GRS, ROT,	noncompensatoriness, paretoness,
		IROC, GROC	optimism, utilization, symmetry,
			consistency.
Lakmayer et al. (2023) [73]	Simulations	ROC, SR, RR, RS,	Hit ratio,
		CRC, CRS, CRR, CSR,	standard deviation, mean.
		CGS, GS, LP
Hatefi (2024) [57]	Simulations	ROC, EW, RS, RE	Hit ratio,
	and empirical	VSL, LSOWA, ROL, GRS	Kendall’s $\tau$
		RR, SR, ROT	Steepness, nonlinearity.
Lakmayer et al. (2024) [88]	Simulations	ROC, SR, RR, RS,	Hit ratio,
		GS	Approximate Maximum Hit Ratio.

show a summary of the comparison studies considered, including information about the corresponding authors, the data source (empirical and/or simulation-based), the methods under comparison, and the quality metrics used.

The first paper dates back to 1981, and 27 comparison analyses have been carried out since then, most of which (14) were published in the last decade (2010–2019). In addition, SWMs are clearly the methods that have been most analyzed in these comparisons, either with respect to each other or to other methods, mainly decision rules and dominance measuring methods. However, other weighting methods have sporadically been included in the comparisons, such as the analytic hierarchy process (AHP), tradeoffs, discrete choice experiments, the PAPRIKA methodology, the best worst method, conjoint analysis, and the Slack method (see Table 2 and Table 3).

Regarding the data source, there are two possibilities for empirical-based analyses. The common opinion of a group of DMs is sometimes taken as a sample, whereas other comparison analyses are based on data from previous real-life cases.

On the other hand, Monte Carlo simulation techniques can be used to analyze the performance of weighting methods in different scenarios with different numbers of attributes and alternatives. For this, weight vectors are randomly generated using a probability distribution (uniform, normal, or exponential) and TRUE rankings of alternatives are derived from them using the additive multi-attribute utility function. These random generated weight vectors are then used in the weighting methods under comparison to derive the corresponding rankings, which are compared against the TRUE rankings using the quality measures. The uniform distribution is used as the general case in which the derived weight vectors may have similar or different weight values in magnitude, whereas the normal and exponential distributions function as particular cases in which weights of similar or very different magnitudes will be generated, respectively. Moreover, filtering processes can be incorporated into the generation to discard random vectors that are highly unlikely in real life.

There are significantly fewer papers using empirical than simulated data. This could be explained by the fact that it is hard to assemble enough real-life cases and/or experts. Note that some papers use both empirical and simulation-based information [48,79,87]. Moreover, DMs may be asked their opinion at several stages of an experiment and evaluate methods accordingly [41,48,79,87].

The hit ratio and rank-order correlation are the most used metrics in comparison processes. The hit ratio is the proportion of cases where the method under consideration outputs the same best alternative as in the TRUE ranking, whereas the rank-order correlation (Kendall’s $\tau$, [89]) measures how similar the overall structures ranking alternatives are between the TRUE ranking and the ranking output by the method under consideration. However, other metrics, such as value loss, mean absolute percentage error, Euclidean distance, Kullback–Leiber divergence, the mean absolute deviation, and the maximum absolute deviation, have also been used (see Table 2 and Table 3).

We first review in Section 4.1 the comparison analyses focusing on the situation where no additional information apart from the ranking of importance of attributes is provided, where SWMs, methods based on absolute and pairwise dominance, the SMAA-2 method, and Monte Carlo simulation techniques can be applied. We then look at the comparison analyses focusing on methods in Section 4.2 where additional information is provided by the DM.

4.1. Comparison Analyses When Only a Ranking of Criteria Is Available

Different analyses have been carried out to compare SWMs [17,18,23,26,77,78,82,83,85,86,90] and different dominance intensity measuring methods [7] in order to identify the best one in each category. On the other hand, comparison methods accounting for methods in the different categories have also been carried out [8,11,32,63].

The first comparison analysis on SWMs was conducted by Stillwell based on empirical data from three decision problems, where only the EW, RS, RR, and RM methods were considered [17]. The conclusion was that EW is outperformed by the other methods. Barron and Barret incorporated the ROC method into the comparison analysis, concluding, on the basis of simulation techniques using EMAR software, that the best method is ROC, followed by RR, RS, and EW [18,77]. The same conclusion was reached by de Almeida et al. many years later, taking into account the hit ratio and the frequency of cases with perfect consistency [85]. The EW, RS, and ROC methods were compared against a ratio-based method in [78] based on unbiased judgments of attribute weights but subject to random error, concluding that the comparison favored ROC over the RS and EW methods.

ROC was compared with AHP and a fuzzy method in [80], and it is concluded that the simplicity and ease of use of the ROC method make it a practical method for eliciting weights. In addition, ROC weights have an appealing theoretical rationale and outperform EW, RS, and RR in terms of choice accuracy [82]. Moreover, the ROC method’s performance is very good when applied to real-life cases [91].

The ROD method was proposed in [22] and compared with RR, RS, and ROC using simulation techniques on the basis of the hit ratio and the average value loss. The conclusion was that RS and ROD are the best, followed by ROC and RR. The use of the RS method was recommended since formulae for ROD weights become more complex as the number of attributes increase.

The comparison analysis performed in [81] found that MEOWA weights are perfectly compatible with the ROC weights, which outperform RR, RS, and EW.

The SR and GS methods were proposed in [26], and were also compared to the ROC, RS, and RR methods. An important concept introduced in this comparative analysis is the degree of freedom (DoF) when randomly generating weight data. If the DM stores the criteria preferences in a way similar to a given point sum, then there are $n-1$DoF for n criteria. On the other hand, if the DM stores criteria preferences in a way that places no limit on the total number of allocated points (or mass), then there are n DoF. These two models of DM behavior yield very different results in assessing surrogate weights. Moreover, weak and strong filtering processes were incorporated into the analysis to discard random vectors that are highly unlikely in real life. It was concluded that the GS and SR methods are the most efficient and robust SWMs, with a very good average performance, which was consistent irrespective of differences in DM behavior. Moreover, GS slightly outperforms SR, but is more complex since it incorporates a parameter.

The EW, RE, RR, ROC, and SR methods, along with Simos’ method, were compared in [84] for both $n-1$ and n DoF on the basis of simulation techniques taking into account the hit ratio, the mean square spread and the mean square variation, concluding that SR is clearly the preferred method, followed by ROC, RE, and Simos’ method.

The GW and VLS methods were incorporated to comparison analyses for the first time in [83], together with RS, RR, ROC, and LSD. Simulation techniques varying the number of criteria and weight probability distributions allowed the relative performance of the rank-weighting methods to be compared, accounting for different decision-making situations. Note that in the above comparison analyses based on simulation techniques, the use of a unique distribution favored some weighting methods over others. Specifically, uniform, normal, and exponential distributions defining different priority structures of DMs and two weight normalization rules were used in this analysis. The uniform distribution distributes weights evenly across the whole range of values, whereas the normal distribution implies a greater likelihood of medium weights, avoiding extreme values, and the exponential distribution assigns most weight to a few top-ranked criteria.

It was concluded that the performance of the SWMs under consideration depends on the number of criteria and their weight distribution. The mean absolute percentage error and the least significance distance were used as quality measures. The RS method was best for uniform weights, whereas the VSL and the ROC methods were best for normal and exponential weights, respectively. Thus, if the criteria weight distribution is known in any specific MCDM situation, then the respective best method can be applied; otherwise, the weights can be computed using all three methods and a secondary rule should be applied. Note that the statistical distribution of weights can be elicited from the DM using just a few statements.

Ref. [86] agreed with Ref. [83] that the performance of SWMs is entirely dependent on how the weight simplex is defined, i.e., the probability distribution used to generate the TRUE weights. The EW, RE, RR, and ROC methods were analyzed on the basis of the hit ratio, Kendall’s $\tau$, and the value loss, concluding that ROC outperforms the other methods when a uniform distribution is considered. Note, however, that this is a special case among other elicitation possibilities, such as other non-uniform distributions.

More recently, a comparison analysis was carried out in [23] including a lot more SWMs (RS, RE($p=1.17$), RR, SR, ERF($r=0.716$), ROC, VLS, and ROD). It was stated that it is very difficult to establish what kind of distribution should be used to randomly generate weights and what affects the simulation results. Therefore, a large number of elicited weights from real-world applications (209 sets of weights with the number of attributes ranging from 2 to 12) were collected.

In the comparison analysis, four alternative measures were used to analyze the quality of the SWMs, and the results derived by each measure were different in some cases. Regarding the average Euclidean distance from elicited weights, ERF > SR > RS > ROD > RE > RR > VLS > ROC, whereas the positions of RE and ROD were inverted using the mean absolute deviation, and the same order was derived from the maximum absolute deviation. Finally, the results using the Kullback–Leibler divergence were clearly different: SR > ROD > RE > RS > RR > ERF > ROC > VLS. However, if the four measures are taken into account, SR was always the best or was statistically indistinguishable from the best SWM, whereas VLS and ROC were always the worst SWMs.

In one the first most recent comparisons carried out in [90], the LSOWA, GRS, ROT, IROC, and GROC methods were incorporated into the analysis. A theoretical analysis was performed on the basis of several evaluation measures: steepness (steeper patterns of the weight curves are higher ranked), nonlinearity of the weight curves, non-compensatoriness (number of comparisons in which the weight of a criterion is greater than the sum of weights of the less important criteria), paretoness (80% of outcomes of most phenomena are due to only 20% of relevant causes), optimism (degree of aggregation [52]), utilization of information (making use of weight entropy [50,52]), symmetry, and consistency (accounting for possible rank/weight reversal).

No recommendation about the best SWM was given, since the results differ when different measures are applied, but a five-step selection procedure was proposed. The optimism and utilization of information measures are first used to discard some SWMs by computing the DM’s coefficient of optimism and the degree of certainty about the information, which are compared with the optimism and the utilization levels of the SWMs, respectively. Then, the consistency is analyzed, discarding additional SWMs on the basis of a threshold. Finally, the remaining measures are computed, and the highest-scoring SWM is selected.

The RR, RS, ROC, SR, and GS with different values for the s parameter were compared in [73], where both $n-1$ and n DoF and a filtering process are again used. It was confirmed that the performance of ROC and SR was stronger than that of the classical methods from [17,92], whereas the results for the even newer GS method were mixed. These inconclusive results leave the research field open.

The ROL method was proposed for the first time in [57]. Three types of analysis (theoretical, simulation-based and empirical) were used to compare ROL with EW, RR, RS, ROC, SR, ROT, RE, VSL, LSOWA, and GRS. All three analysis types demonstrated that ROL is as strong as the ROC method.

Finally, RR, RS, ROC, SR, and GS were compared again in [88], using both $n-1$ and nDoF, as well as a filtering process. This comparison is novel in that it considers three different probability distributions (uniform, beta, and log-normal). The results again show that RS and ROC perform better for $n-1$ and n DoFs, respectively, whereas SR is ideal for an equal mixture of DoFs.

Looking at other weighting methods based on the absolute or pairwise dominance notions, different comparison analyses can be also found in the literature.

The first was in [63], comparing the CENT decision rule with SWMs (EW, RS, RR, and ROC), together with the dominance measuring methods (OUT I and OUT II) proposed in that paper. The analysis was performed on the basis of simulation techniques using the hit ratio and Kendall’s $\tau$. It was concluded that ROC was the best method, followed by RR and RS, the dominance measuring methods and the CENT decision rule (identified as the best decision rule).

A similar conclusion was reached in [11], where decision rules were compared with the ROC method using the hit ratio, the mean position of the supposedly best alternative in each rule’s ranking, and the proportion of cases where the position was 1, 2, 3, 4, or higher. The ROC method slightly outperformed the decision rules. Note that dominance measuring methods were not considered in this analysis.

In [93,94], a comparison analysis was carried out accounting for decision rules, the OUT I, OUT II, DME1, and DME2 dominance measuring methods, and the SMAA and SMAA-2 methods. However, weight intervals rather than ordinal information on weights were considered. Later, the SWMs (RS, RR, ROC, and EW), the decision rules, and the DMMs (OUT I, OUT II, DME1, and DME2) were compared in [32]. To achieve this, simulation techniques were used together with the hit ratio and Kendall’s $\tau$ quality measures. The conclusion reached was that DME2 and ROC outperform the other methods.

The DIM method was proposed in [7], which was compared with previous dominance measuring methods (OUT I, DME1, and DME2) and Sarabando and Dias’ method [70]. Ordinal information about the position of alternatives, the difference in values between consecutive alternatives for the different criteria under consideration, and weights were considered, concluding that the DIM and Sarabando and Dias’ methods perform very similarly for a neutral, risk-prone, and risk-averse DM. Both outperform the other dominance-measuring methods.

Finally, the SWMs (RR, RS, SR, and ROC) and dominance-measuring methods were compared in [8]. The effects of the generator on dominance intensity methods was studied for the very first time since the different dimensions of generators were not differentiated in either [32] or [7], affecting both efficiency and robustness analyses. In this paper, criteria weights for DME2 were sampled using $n-1$DoF, and m attribute values were sampled using mDoF, but normalizing the values. The conclusion was that SR outperforms the other methods in terms of robustness, and DME2’s performance is nowhere near as good as that of more classical SWMs. Thus, the added complexity of DIMs is not outweighed by additional performance. Another important conclusion was that the difference in hit ratios for the two weight generation scenarios ($n/n-1$ DoF) does not decrease when the number of criteria is increased.

A similar comparison analysis was performed by the same authors in [73], where SWMs (RR, RS, SR, GS, and ROC) were compared to linear programming, concluding that linear programming performs worse than any of the other analysed methods by a wide margin.

4.2. Comparison Analyses When Additional Information Is Provided by the DM

Methods using precise/imprecise cardinal information or based on ratios have been widely compared in the literature addressing methods that either do (direct rating, Max100, Min10, SMART, and interval SMART/SWING) or do not (direct point allocation, tradeoff method, SWING, AHP, discrete choice experiments, the PAPRIKA methodology, and conjoint analysis) establish a ranking of criteria beforehand. We focus on analyses that include at least one method where a ranking of criteria is established beforehand.

The convergent validity of SMART was compared with that of AHP, direct point allocation, SWING, and tradeoff weighting [95] on the basis of an Internet experiment [79], where weight ratios were used, albeit based on different questions, concluding that the derived weights differ because the DMs are asked to choose their responses from a limited set of numbers. In addition, DMs could select any of the studied methods depending on their personal preferences.

The direct rating (DR) method was compared with the Max100 and Min10 methods on the basis of both empirical data and simulations in [48]. The weights derived from the Max100 method were somewhat more reliable than those derived from DR, followed by the Min10 method. SWARA was also compared to Max100 and pairwise weight elicitation methods in [96], where SWARA is recommended when the DM wants criteria weights to have neither very different nor very close values in relation to each other.

More recently, direct rating, SMARTS, AHP, discrete choice experiments [97], the PAPRIKA methodology [98], and conjoint analysis [99] were compared in [87]. The authors concluded that SMARTS and AHP can reach trade-offs between complexity and the potential for bias.

In addition, two comparison analyses were carried out in [40,41], comparing the CROC method with SMART, SMARTS, and direct rating, using empirical data elicited from several DMs and the cognitive effort and practical usefulness and the consistency quality measures. In both papers, five DMs were engaged in decision making on purchasing a car according to seven criteria on two occasions (one week apart).

In [41], all participants preferred the SMARTS method over direct rating, but considered that it was difficult to explicitly score each criterion using both methods, and preferences between the SMARTS and CROC methods varied. In addition, the CROC method achieved the most consistent results across the two occasions in both comparison analyses, being the more prescriptively useful method. However, CROC has not been compared with SWARA.

Finally, the FUCOM method yields better results than AHP and the best worst method [68], but it has not been compared with CROC and SWARA.

In the case of ranking of differences, MSD was proven to outperform a linear programming-based weighting method (the Slack method) which attempts to minimize the slack of constraints via simulation analysis under different forms of incomplete attribute weights in [45]. However, no comparison analyses have been carried out with the modification of classical decision rules and dominance-measuring methods, which could also have been applied in this situation.

Regarding the use of a semantic scale to express the strength of the differences between the weights of consecutive criteria, a comparison analysis of the CRS, CRR, CRC, and CSR methods together with Simos’ family weighting methods [36,37,38] and SR and ROC SWMs was performed in [71] on the basis of simulation techniques, concluding that CSR outperforms the other methods.

The CRR, CRS, CSR, and CRC methods were again compared to SWMs (RR, RS, SR, and ROC), but also to the DME2 dominance-measuring method in [8]. The conclusion was the same as in [71]. CSR clearly outperforms the other methods with regard to robustness. A similar comparison analysis was performed in [73], incorporating GS and its extension CGS into the analysis. They again concluded that CSR is generally the best-performing method.

The CAR method was proposed in [46] and compared in an empirical experiment with SMART and AHP using an equal combination of $n-1$ and nDoF, concluding that CAR outperforms the other methods in terms of the hit ratio and Kendall’s $\tau$ on the basis of simulated data. An empirical experiment suggests that it also excels in terms of quality measures such as ease of use, amount of time and effort required, and perceived correctness and transparency.

Finally, regarding preference statements, no comparison analysis has been performed accounting for either the extended difference ranking (EDR) method or the flexible and interactive tradeoff (FITradeoff) method.

5. Discussion

The aim of this section is to identify which of the weighting methods described in Section 2 and Section 3 could be recommended depending on the information that the DM is able/willing to provide on the basis of the comparison analyses described in Section 4, or what further comparison analyses should be carried out to arrive at such a recommendation. First, we analyze the situation where the DM provides only a ranking of the criteria under consideration. We then look at the situation where additional information is provided in different ways.

More weighing methods are available for situations where DMs provide only a criteria ranking. These include SWMs, methods based on the notions of absolute and pairwise dominance, and the SMAA-2 method (see Section 2).

As pointed out in Section 4, many comparison analyses are available in the literature concerning these weighting methods. Some focus on comparing SWMs with each other, with decision rules, or with DMMs, whereas other papers compare all of the methods.

Looking at papers focused on the comparison of SWMs, we find that only the EW, RR, RS, and ROC methods were compared in the early comparison analyses, since they were the only SWMs to have been proposed when the analyses were carried out [18,77,78,82,85,91,92]. The conclusion was that ROC outperforms the other SWMs. The EW, RR, and RS methods were also outperformed by other SWMs in subsequent analyses (in [26] or [84] for instance), so they can be discarded for further analysis.

In the comparison analysis performed in [23], the SR method outperformed the VLS, ERF, ROD, ROC, and RE methods. The SR method was also the best weighting method in the comparison analysis in [84] (compared to ROC, RE, RR, EW, and Simos’ methods).

However, although it is more complex since it incorporates a parameter, GS slightly outperforms SR in [26], irrespective of differences in DM behavior, whereas the most recent comparison analysis in [73] is inconclusive, since ROC, GS, and SR outperform each other depending on the DoF of the random number generator and the number of attributes/alternatives under consideration.

What about the remaining SWMs (LSOWA, ROT, GRS, IROC, and GROC)? They were analysed comparatively in [90], but no recommendation about the best was given since the results differed when different measures were applied. Note that this analysis considered neither the SR nor the GS methods.

So, is it possible to arrive at a recommendation about the best SWM when DMs only provide a criteria ranking? The answer is no, because a comparison of ROC, GS, and SR with LSOWA, ROT, GRS, IROC, and GROC is still missing and should be carried out.

Now, what is the best decision rule or the best dominance measuring method, and how do they fare against SWMs and the SMAA-2 method?

In this respect, different comparison analyses were summarized in Section 4.1, concluding that CENT is the best decision rule [63]. However, CENT is outperformed by the ROC method [11,63] and by dominance-measuring methods [32,63]. Thus, decision rules can be safely discarded. In addition, the best dominance measuring method is DIM [7], but it is clearly outperformed by the SR method [8], which also outperforms linear programming.

Thus, we conclude that SWMs (SR) outperform decision rules and dominance measuring methods. However, the SMAA-2 has not been compared with SWMs. This is a missing comparative analysis that should be performed to arrive at a final recommendation.

Let us now discuss weighting methods when different forms of additional information are available.

First, with respect to methods where precise/imprecise cardinal information is used to represent the strength of the differences between the weights of consecutive criteria, the CROC method is more consistent and a more prescriptively useful method than SMARTS and direct rating [40,41].

In addition, although the CROC method itself has not been compared with Simos’ methods, it was demonstrated in [84] that the ROC method outperforms Simos’ methods when only a ranking of weights is available. Meanwhile, the CRC method also outperforms Simos’ methods when the differences between the weights of consecutive criteria are represented by a semantic scale [71]. Therefore, the ROC method and its derivatives designed to account for additional information appear to outperform Simos’ methods. However, CRC method has not been compared to SWARA. Such a comparison analysis would be very useful to identify which method should be recommended in this category.

In the case of ranking of differences, MSD was confirmed to perform better than the Slack method in [45]. However, no comparisons have been made with the modification of classical decision rules and dominance measuring methods, which could also be applied in this situation. Such a comparison analysis would be very useful to identify which method outperforms the others.

Finally, regarding the use of a semantic scale to express the strength of the differences between the weights of consecutive criteria, CSR outperforms CRR, CRC, and Simos’ family of methods [71], but also the SWMs (RR, RS, SR and ROC), and the DME2 dominance-measuring method [8]. Thus, the CSR method is a good recommendation in this case.

6. Conclusions

In this paper, we focused on weighting methods in MAUT/MAVT when ordinal information on criteria is available and the DM can also provide different forms of additional information to express the strength of the differences between the weights of consecutive criteria of such a ranking, including a ranking of differences, a semantic scale, precise/imprecise cardinal information, or ratios.

The aim was to single out the best weighting methods for different situations depending on the available information, or otherwise identify the missing comparison analyses that should be carried out to reach a recommendation. To achieve this, we reviewed the comparative analyses in the literature based on empirical and simulation data and using different quality measures.

After summarizing the weighting methods reported in the literature to address the different types of available information on weights and analyses comparing their performances, we deliberated on the findings of the different authors and arrived at the following conclusions.

First, to be able to provide a recommendation on which method to use when only ordinal information is available, ROC, SR, and GS need to be compared with LSOWA, ROT, GRS, IROC, and GROC. Additionally, a comparison with the SMAA-2 method is also missing. Note that several SWMs outperform decision rules and dominance measuring methods.

If additional information on the ranking of criteria is available in the form of a ranking of differences, a comparison of MSD with the modification of classical decision rules and dominance measuring methods is also missing. If DMs provide additional information in the form of a semantic scale, then the CSR method is recommended. If DMs are able to provide additional precise/imprecise cardinal information, the comparison of CROC against Simos’s methods and SWARA is missing.

Note that methods where the additional information on the ranking of criteria differs (ranking of differences vs. semantic scale vs. imprecise cardinal information) have not been compared as yet, although a broader range of information should provide better results. Moreover, in real decision-making problems, DMs will select which type of information they are able or willing to provide. Thus, the corresponding recommended method would be directly applied. As a future research line, we propose performing the above-mentioned missing comparison analyses aimed at reaching more robust recommendations.

Other future work includes the incorporation of the recommended weighting methods into WEB-MAUT-DSS, a web-based adaptation and improvement of the GMAA decision support system [1,2] that is being implemented. One of the improvements provided by WEB-MAUT-DSS will be the incorporation of new weighting methods on top of the existing direct assignment methods or methods based on trade-offs. These new methods will account not only for ordinal information on criteria but also for additional information in terms of the ranking of differences or a semantic scale. It will be up to DMs to select the type of information that they wish to provide. Other weighting methods will be included in future versions of the system.