Using Pairwise Comparisons to Determine Consumer Preferences in Hotel Selection

Abstract

We consider the problem of evaluating preferences for criteria used by university students when selecting a hotel for accommodation during a professional development program in a foreign country. Input data for analysis come from a survey of 202 respondents, who indicated their age, sex and whether they have previously visited the country. The criteria under evaluation are location, accommodation cost, typical guests, free breakfast, room amenities and courtesy of staff. The respondents assess the criteria both directly by providing estimates of absolute ratings and ranks, and indirectly by relative estimates using ratios of pairwise comparisons. To improve the accuracy of ratings derived from pairwise comparisons, we concurrently apply the principal eigenvector method, the geometric mean method and the method of log-Chebyshev approximation. Then, the results from the direct and indirect evaluation of ratings and ranks are examined together to analyze how the results from pairwise comparisons may differ from each other and from the results of direct assessment by respondents. We apply statistical techniques, such as estimation of means, standard deviations and correlations, to the vectors of ratings and ranks provided directly or indirectly by respondents, and then use the estimates to make accurate assessment of the criteria under study.

1. Introduction

Evaluation of preferences for alternatives based on their pairwise comparisons is a widely accepted approach in decision making, when direct assessment of the preferences is infeasible or impossible [1,2,3,4]. The approach uses the results of pairwise comparisons of alternatives on an appropriate scale, given in the form of a pairwise comparison matrix. Then, various computational methods can be applied to the matrix to make judgment on the preference of each alternatives by evaluating its individual rating (score, priority, weight) and ranking the alternatives according to the ratings.

The methods used to derive ratings from pairwise comparisons may exploit different computational techniques. These methods are mainly based on aggregation (summation) of columns in the pairwise comparison matrix to obtain a vector of ratings of alternatives, or on approximation of the pairwise comparison matrix by a symmetrically reciprocal (consistent) matrix that directly determines the vector of ratings (see, e.g., [5,6]). The key issue in deriving ratings from pairwise comparisons by different techniques is that these techniques may produce rather different or even contradictory results (see, e.g., [7,8,9,10]), which can significantly complicate or even make it impossible to unambiguously assess alternatives when making decisions in practice.

Among other problems arising from the issue indicated above, one has to recognize how much the results of different methods vary in practical problems, and how to make the right decision in case of possible inconsistency of the results. The comparison of results of assessment methods is discussed in many research studies [8,10,11], including statistical analysis of extensive data given by many pairwise comparison matrices. However, in most cases, the source data used in the analysis are obtained by simulation [8,10] and thus are to be considered as an artificial input that may less adequately reflect the usual conditions of practice than authentic results of human evaluation.

In practical situations where the methods used can give different results, a natural but reasonable way to evaluate alternatives more unambiguously is based on the simultaneous application of several methods to make a decision that concurrently considers all the results. If these results differ significantly, the choice of one of them as the basis for making a decision does not seem entirely justified. On the contrary, the closeness and stability of the results can serve as additional arguments in favor of choosing one of them as a solution that can be considered as close to optimal.

The well-known solution approach is the principal eigenvector method [2,3,5], which defines the vector of ratings as a weighted sum of the columns in the pairwise comparison matrix, where the weights are taken proportional to the entries of the vector of ratings itself. This approach leads to a solution calculated as the eigenvector of the pairwise comparison matrix, which corresponds to the maximal eigenvalue. The geometric mean method is another widely used approach, which solves the problem by approximating the pairwise comparison matrix by a consistent matrix in the Euclidean norm on the logarithmic scale [7,12,13]. It provides the solution vector by calculating the geometric mean of the entries in each row of the pairwise comparison matrix.

An approach to the derivation of ratings from pairwise comparison matrices, which applies the Chebyshev approximation on the logarithmic scale (the log-Chebyshev approximation), is developed in [14,15,16,17]. The proposed solution technique is based on methods and results of tropical mathematics which is concerned with the study and application of algebraic systems with idempotent operations [18,19,20,21]. The approximation problem is formulated and solved as an optimization problem in terms of tropical mathematics (a tropical optimization problem), which yields a complete analytical solution given in a parametric form ready for numerical computation and formal analysis.

The problem of evaluating alternatives from their pairwise comparisons often arises in studying consumer preferences in various markets including the market of hotel services. Specifically, for hotel suppliers, it is crucial to identify the most and least important criteria used by consumers in hotel selection. The problem of assessment criteria for hotel selection by various customer groups is studied in many researches (see, e.g., [22,23,24,25,26,27,28,29,30]). These studies mainly concentrate on investigation of preferences of business and (or) leisure travelers, who often constitute the largest segment of hotel guests and thus are of prime interest [23,24,25,26,30], but sometimes more specific customer groups such as university students are under consideration [29].

As a source of information to understand how different factors (hotel attributes), such as location, accommodation cost, free breakfast or amenities, may affect the choice of a hotel, consumer surveys are widely used which can be administered online [26,28] or using a paper-and-pencil format [22,24,27,29,30]. A typical survey can ask respondents for absolute estimates that rate and rank the factors directly or for relative estimates in the form of pairwise comparison ratios for the factors.

The pairwise comparison data are then used to make a final assessment of factors by applying one of the methods of rating alternatives from pairwise comparisons. However, many studies rely on results obtained using only one method, which can lead to inaccurate or wrong conclusions because different methods may produce ambiguous results. To improve the quality of the assessment, it seems necessary to verify the results of one method against the results of other available methods, which makes the development and experimental study of a combined approach, which integrates several solution methods together with procedures of data analysis, a highly relevant issue.

In this paper, we consider the problem of evaluating preferences for criteria used by university students when selecting a hotel for accommodation during a short-term professional development program in a foreign country. As input data for analysis, we are restricted by the results of a survey of 202 respondents who indicated their age, sex and whether they have previous experience of visiting the country. The criteria evaluated by respondents include location, accommodation cost, typical guests, free breakfast, room amenities and courtesy of staff. The respondents assess the criteria both directly by providing estimates of absolute ratings and ranks, and indirectly by relative estimates that are represented as ratios of pairwise comparisons.

The purpose of this applied study is twofold. First, it is to investigate the possibilities of combining different methods of rating and ranking criteria together with statistical procedures to provide more accurate assessment of the criteria, to describe an applied technique intended to improve the quality of assessment, and to demonstrate the implementation of this technique to the problem of evaluating criteria for hotel selection. The second objective is to explore by statistical procedures the results obtained by different methods from pairwise comparisons provided by respondents to gain additional insight and understanding about the similarity and difference between these results.

As the key technique to improve the accuracy of ratings derived from the results of pairwise comparisons, we concurrently apply the methods of principal eigenvector, geometric mean and log-Chebyshev approximation. Then, the results obtained by the direct and indirect assessment of ratings and ranks are examined together to analyze how the results from pairwise comparisons may differ from each other and from the results of direct assessment. The analysis involves statistical techniques, such as estimation of means, standard deviations and correlations, applied to the vectors of ratings and ranks of criteria, which are provided directly or indirectly by respondents.

The paper is organized as follows. We start in Section 2 with a brief overview of the methods used below to derive ratings from pairwise comparisons, including the method based on the log-Chebyshev approximation. Section 3 presents the basic definitions, notation and results of tropical mathematics, which are then used to give direct formulas for calculating vectors of ratings using log-Chebyshev approximation. In Section 4, we formulate the problem of evaluating criteria for hotel selection, which motivates and illustrates the study, and describe the survey data used in the solution.

Section 5 offers results of statistical analysis of ratings and ranks derived from the survey data. First, we investigate whether characteristics of respondents (age, sex, etc.) may affect the degree of difference or similarity between vectors of ratings or ranks obtained for each respondent by different methods. Furthermore, statistical results of the analysis and comparison of the vectors of ratings for the criteria are given including estimates of means, standard deviations and correlations. We conclude the section with the results of comparison of rank vectors that order criteria according to their preferences. In Section 6, we present some discussions, and in Section 7 offer concluding remarks.

2. Evaluation of Alternatives from Pairwise Comparisons

Suppose there are n alternatives ${\mathcal{A}}_1,\dots ,{\mathcal{A}}_n$ for making decisions, which are compared in pairs. The results of the comparisons are represented in the form of a pairwise comparison matrix $\mathit{A}=\left(a_{ij}\right)$ of dimension $n\times n$, where the element $a_{ij}>0$ indicates by how many times the alternative ${\mathcal{A}}_i$ is more preferable than ${\mathcal{A}}_j$. The problem of pairwise comparisons is to find a vector $\mathit{x}=\left(x_i\right)$ of absolute ratings (scores, priorities, weights) of alternatives on the basis of the relative estimates given by pairwise comparisons.

Consider a pairwise comparison matrix $\mathit{A}$ and note that its elements satisfy the condition $a_{ji}=1/a_{ij}$ (and hence $a_{ii}=1$) for all $i,j=1,\dots ,n$, and therefore the matrix $\mathit{A}$ is symmetrically reciprocal. Furthermore, if the transitive property in the form of equality $a_{ij}=a_{ik}{a}_{kj}$ is fulfilled for all $i,j,k=1,\dots ,n$, then the matrix $\mathit{A}$ is called consistent.

The elements of a consistent pairwise comparison matrix $\mathit{A}$ are represented as $a_{ij}=x_i/x_j$ for all $i,j=1,\dots ,n$, where $x_i$ are the components of some positive vector $\mathit{x}$ determined up to a positive factor. Moreover, it directly follows from the equality $a_{ij}=x_i/x_j$ that $\mathit{x}=\left(x_i\right)$ is a solution of the pairwise comparison problem of interest.

In practical applications, the matrices obtained by pairwise comparisons of alternatives are usually not consistent. In this case, a problem arises of finding a consistent matrix that is close to (approximates) the original pairwise comparison matrix. Any vector that determines this consistent matrix is then taken as the vector of absolute ratings. To find an appropriate consistent matrix of pairwise comparisons various heuristic procedures and approximation methods are used (see, e.g., [2,5,6]).

Heuristic methods for solving the problem usually apply techniques of aggregation (summation) of the columns in the pairwise comparison matrix. The columns are added up with coefficients (weights) that are dependent on the technique. The resulting vector, which generates some consistent matrix, serves as a solution to the problem. As examples of the heuristic approach, one can consider the weighted column sum method and the principal eigenvector method.

The most common in practice is the method of principal eigenvector proposed by T. Saaty in the 1970s [2,3,5,31]. The method assumes that the weights of columns for aggregation are set proportional to the components of the solution vector of absolute ratings. This leads to a solution of the problem that takes the form of the principal eigenvector $\mathit{x}$ of the pairwise comparison matrix $\mathit{A}$, which corresponds to the maximum eigenvalue $\lambda$ of the matrix and satisfies the equality

$$\mathit{A}\mathit{x}=\lambda \mathit{x}.$$

The approximation methods solve a problem of minimizing an error in the approximation of the pairwise comparison matrix $\mathit{A}=\left(a_{ij}\right)$ by a consistent matrix $\mathit{X}=(x_i/x_j)$. The application of these methods offers a mathematically justified approach, which however can result in very hard optimization problems. The complexity of the solution essentially depends on the metric and scale used to estimate the approximation error.

To measure the error on the standard linear scale, different metrics may be used including the Euclidean, Manhattan and Chebyshev distance functions. However, minimizing the errors in linear scale usually leads to complex nonlinear multiextremal optimization problems that are difficult to solve [5,32], and hence is not common in practice.

If the approximation error is evaluated in logarithmic scale, then the solution may become less complicated and can sometimes be obtained in analytical form [7,12,13]. Note that when transforming from linear scale to logarithmic scale, the variation between entries in a pairwise comparison matrix $\mathit{A}$ decreases, which reduces the impact on the overall error of large values (for example, $a_{ij}=9$) to the detriment of small ones ($a_{ji}=1/9$). Therefore, the use of the logarithmic scale in the approximation of the pairwise comparison matrices seems to be quite reasonable.

Consider the application of the log-Euclidean metric (the Euclidean metric on the logarithmic scale), in which the distance between the matrices $\mathit{A}=\left(a_{ij}\right)$ and $\mathit{X}=(x_i/x_j)$ is determined using logarithm in a base greater than one by the formula

$$l_2(\mathit{A},\mathit{X})={\left(\sum _{1\le i,j\le n}{\left(log{a}_{ij}-log\frac{x_i}{x_j}\right)}^2\right)}^{1/2}.$$

The minimum distance is immediately found by calculating the derivatives of the squared distance $l_2^2(\mathit{A},\mathit{X})$ with respect to all $x_i$, equating the derivatives to zero, and solving the obtained equations. For a symmetrically reciprocal matrix $\mathit{A}$, the result is a direct solution of the approximation problem, in which the components of the vector $\mathit{x}={(x_1,\dots ,x_n)}^T$ that determines the matrix $\mathit{X}$ are given in the parametric form

$$x_i={\left(\prod _{j=1}^n{a}_{ij}\right)}^{1/n}u,u>0,i=1,\dots ,n.$$

The obtained vector $\mathit{x}$ (usually normalized in rectangular metric, i.e., with respect to the sum of components) is called the solution of the problem of pairwise comparisons by the method of geometric mean [7,12,13].

The distance between the matrices $\mathit{A}$ and $\mathit{X}$ in the Chebyshev metric in logarithmic scale (the log-Chebyshev distance) is defined as

$$l_{\infty }(\mathit{A},\mathit{X})=\underset{1\le i,j\le n}{max}\left|log{a}_{ij}-log\frac{x_i}{x_j}\right|.$$

The problem of approximating a pairwise comparison matrix in the log-Chebyshev metric can be reduced to the following problem of minimizing a function without logarithm (see, e.g., [17,33]):

$$\begin{array}{cccc}\hfill & \underset{\mathit{x}>\mathbf{0}}{min}\hfill & \hfill & \underset{1\le i,j\le n}{max}\frac{a_{ij}{x}_j}{x_i},\hfill \end{array}$$

(1)

which is equivalent to minimizing the maximum relative error [33,34], which is given by

$$\underset{1\le i,j\le n}{max}\left|\frac{a_{ij}-x_i/x_j}{a_{ij}}\right|.$$

In contrast to both methods of the principal eigenvalue and geometric mean, which always lead to a single solution vector (up to a positive factor), the solution based on the log-Chebyshev approximation may be nonunique.

The existence of multiple solutions instead of a single one, on the one hand, can make it difficult to choose the most preferable alternative in practice. On the other hand, the availability of a set of different solutions to the problem extends the possibility of making optimal decisions, for example by taking into account additional constraints to narrow the solutions. In addition, due to the rather approximate nature of the model of pairwise comparisons for which the inconsistency of pairwise judgments is typical, the existence of several solutions to the problem seems to be quite natural.

Suppose that the approximation procedure results in a set $\mathcal{S}$ of optimal vectors, rather than in a single optimal vector of ratings. As the “best” and “worst” solutions to the problem, we can take those two vectors from $\mathcal{S}$ that best and worst differentiate the alternatives with highest and lowest ratings [16,17].

Consider the ratio between the maximum and minimum elements of the vector $\mathit{x}=\left(x_i\right)$, which is called the Hilbert (interval, range) seminorm, and given by

$$\underset{1\le i\le n}{max}{x}_i/\underset{1\le j\le n}{min}{x}_j=\underset{1\le i\le n}{max}{x}_i\times \underset{1\le j\le n}{max}{x}_j^{-1}.$$

The solution vector with the maximum Hilbert seminorm is taken as the best differentiating solution, whereas the vector with the minimum seminorm is as the worst differentiating solution. These vectors are obtained by solving the following problems:

$$\begin{array}{c}\begin{array}{cccc}& \underset{\mathit{x}\in \mathcal{S}}{max}\hfill & & \underset{1\le i\le n}{max}{x}_i\times \underset{1\le j\le n}{max}{x}_j^{-1},\hfill \end{array}\end{array}$$

(2)

$$\begin{array}{c}\begin{array}{cccc}& \underset{\mathit{x}\in \mathcal{S}}{min}\hfill & & \underset{1\le i\le n}{max}{x}_i\times \underset{1\le j\le n}{max}{x}_j^{-1}.\hfill \end{array}\end{array}$$

(3)

Problems (1)–(3), which arise from the log-Chebyshev approximation in the pairwise comparison problem, can be directly solved in analytical form, using the methods and results of tropical mathematics (see, e.g., [14,15,16,17] for further details).

3. Tropical Mathematics Based Solutions

The purpose of this section is to provide a brief overview of the basic concepts, definitions and notation of tropical mathematics, which are used to describe the solution of the pairwise comparison problem. Further information on the theory, methods and applications of tropical mathematics can be found, for example, in [18,19,20,21,35].

3.1. Elements of Tropical Mathematics

Tropical (idempotent) mathematics deals with the theory and applications of algebraic systems with idempotent operations. An operation is idempotent if its application to arguments of the same value yields this value as the result. For example, taking the maximum is an idempotent operation since $max\{x,x\}=x$, whereas the arithmetic addition is not: $x+x=2x$.

Tropical optimization focuses on optimization problems that are formulated and solved in terms of tropical mathematics. Models and methods of tropical optimization make it possible to find new solutions for classical and newly posed problems. Many problems can be solved directly in explicit analytic form; for other problems, only algorithmic techniques that offer solutions in numerical form are known. Applications of tropical optimization include various problems in project scheduling, location analysis, decision making and other areas.

An example of an algebraic system with an idempotent operation is the max-algebra defined on the set of non-negative real numbers ${\mathbb{R}}_{+}=\{x\in \mathbb{R}|x\ge 0\}$. It is closed under addition denoted by ⊕ and defined for all $x,y\in {\mathbb{R}}_{+}$ as maximum: $x\oplus y=max\{x,y\}$, and multiplication defined as usual. The neutral elements with respect to addition and multiplication coincide with the arithmetic zero 0 and one 1.

The tropical addition ⊕ is not invertible (opposite numbers do not exist), and hence a subtraction operation is undefined in max-algebra. The notion and notation of inverse elements with respect to multiplication, and exponents have the usual sense.

Vector and matrix operations are performed according to standard rules, where the arithmetic addition + is replaced by ⊕. Specifically, multiplication of a vector or matrix by a scalar has the same result as in the standard arithmetic. In what follows, all vectors are considered column vectors until otherwise indicated. The zero vector denoted by $\mathbf{0}$, positive vector, zero matrix and identity matrix denoted by $\mathit{I}$ are defined as usual.

For any nonzero column vector $\mathit{x}=\left(x_j\right)$, its multiplicative conjugate transpose is the row vector ${\mathit{x}}^{-}=\left(x_j^{-}\right)$, where $x_j^{-}=x_j^{-1}$ if $x_j\ne 0$, and $x_j^{-}=0$ otherwise. For the vector of all ones, which is denoted as $\mathbf{1}$, the conjugate transpose is given by ${\mathbf{1}}^{-}={\mathbf{1}}^T$. Multiplicative conjugate transposition of a nonzero matrix $\mathit{A}=\left(a_{ij}\right)$ results in the transposed matrix ${\mathit{A}}^{-}=\left(a_{ij}^{-}\right)$, where $a_{ij}^{-}=a_{ji}^{-1}$ if $a_{ji}\ne 0$, and $a_{ij}^{-}=0$ otherwise.

For vectors ${\mathit{a}}_1,\dots ,{\mathit{a}}_n$, its linear combination with nonnegative coefficients $x_1,\dots ,x_n$ is given by $x_1{\mathit{a}}_1\oplus \cdots \oplus x_n{\mathit{a}}_n$. A vector $\mathit{b}$ linearly depends on vectors ${\mathit{a}}_1,\dots ,{\mathit{a}}_n$ if there exist numbers $x_1,\dots ,x_n$ such that $\mathit{b}=x_1{\mathit{a}}_1\oplus \cdots \oplus x_n{\mathit{a}}_n$. The collinearity of two vectors has the usual sense: vectors $\mathit{b}$ is collinear with $\mathit{a}$ if $\mathit{b}=x\mathit{a}$ for some x.

The set of all linear combinations $x_1{\mathit{a}}_1\oplus \cdots \oplus x_n{\mathit{a}}_n$ of a system of vectors ${\mathit{a}}_1,\dots ,{\mathit{a}}_n$ forms a tropical linear space. Any vector $\mathit{y}$ of this space is expressed as the (tropical) product of the matrix $\mathit{A}=({\mathit{a}}_1,\dots ,{\mathit{a}}_n)$, which consists of the vectors in the system taken as columns, and some vector $\mathit{x}={(x_1,\dots ,x_n)}^T$, in the form $\mathit{y}=\mathit{A}\mathit{x}$.

For a square matrix $\mathit{A}=\left(a_{ij}\right)$ of order n, the nonnegative integer powers indicate repeated tropical multiplication of the matrix with itself, and are defined as ${\mathit{A}}^0=\mathit{I}$, ${\mathit{A}}^p={\mathit{A}}^{p-1}\mathit{A}=\mathit{A}{\mathit{A}}^{p-1}$ for all integer $p>0$. The trace of the matrix is given by

$$\mathrm{tr}\mathit{A}=a_{11}\oplus \cdots \oplus a_{nn}=\underset{i=1}{\overset{n}{⨁}}{a}_{ii}.$$

The spectral radius of the matrix $\mathit{A}$ is calculated by the formula

$$\lambda =\mathrm{tr}\mathit{A}\oplus \cdots \oplus {\mathrm{tr}}^{1/n}\left({\mathit{A}}^n\right)=\underset{k=1}{\overset{n}{⨁}}{\mathrm{tr}}^{1/k}\left({\mathit{A}}^k\right).$$

On condition that $\lambda \le 1$, the Kleene operator (the Kleene star) is defined to map the matrix $\mathit{A}$ onto the matrix

$${\mathit{A}}^{\ast }=\mathit{I}\oplus \mathit{A}\oplus \cdots \oplus {\mathit{A}}^{n-1}=\underset{k=0}{\overset{n-1}{⨁}}{\mathit{A}}^k.$$

3.2. Preliminary Results

We start with a formal criterion of linear dependence of vectors. In order to check whether a vector $\mathit{b}$ is dependent on the system of vectors ${\mathit{a}}_1,\dots ,{\mathit{a}}_n$, the following result can be used [36] (see, also [35]).

Lemma 1.

Denote by $\mathit{A}=({\mathit{a}}_1,\dots ,{\mathit{a}}_n)$ a matrix that consists of vectors ${\mathit{a}}_1,\dots ,{\mathit{a}}_n$ as columns. Then, a vector $\mathit{b}$ is linearly dependent on ${\mathit{a}}_1,\dots ,{\mathit{a}}_n$ if and only if the following equality holds:

$${\left(\mathit{A}{\left({\mathit{b}}^{-}\mathit{A}\right)}^{-}\right)}^{-}\mathit{b}=1.$$

We now consider some results in tropical optimization that provide the basis to solve pairwise comparison problems using log-Chebyshev approximation. Suppose that, given an $(n\times n)$-matrix $\mathit{A}$, we need to find positive n-vectors $\mathit{x}$ to solve the problem

$$\begin{array}{cccc}\hfill & \underset{\mathit{x}>\mathbf{0}}{min}\hfill & \hfill & {\mathit{x}}^{-}\mathit{A}\mathit{x}.\hfill \end{array}$$

(4)

A complete solution of the problem is obtained as follows (see, e.g., [37]).

Lemma 2.

Let $\mathit{A}$ be a matrix with spectral radius $\lambda >0$. Then, the minimum in problem (4) is equal to λ, and all positive solutions are given in the parametric form

$$\mathit{x}={\left({\lambda }^{-1}\mathit{A}\right)}^{\ast }\mathit{u},\mathit{u}>\mathbf{0}.$$

Let $\mathit{B}=\left({\mathit{b}}_j\right)$ be a positive $(n\times m)$-matrix with columns ${\mathit{b}}_j=\left(b_{ij}\right)$, and the problem is to obtain positive n-vectors $\mathit{x}$ that provide the maximum

$$\begin{array}{cccc}\hfill & \underset{\mathit{x}>\mathbf{0}}{max}\hfill & \hfill & {\mathbf{1}}^T\mathit{x}{\mathit{x}}^{-}\mathbf{1},\hfill \\ \hfill & \hfill & & \mathit{x}=\mathit{B}\mathit{u},\mathit{u}>\mathbf{0}.\hfill \end{array}$$

(5)

The next result offers a direct solution to the problem [16,17,38].

Lemma 3.

Denote by ${\mathit{B}}_{lk}$ the matrix obtained from $\mathit{B}$ by setting to zero all elements except $b_{lk}$ for some indices k and l. Then, the maximum in problem (5) is equal to $\Delta ={\mathbf{1}}^T\mathit{B}{\mathit{B}}^{-}\mathbf{1}$, and all positive solutions are given in the parametric form

$$\mathit{x}=\mathit{B}(\mathit{I}\oplus {\mathit{B}}_{lk}^{-}\mathit{B})\mathit{u},\mathit{u}>\mathbf{0},$$

where the indices k and l are selected by the conditions

$$k=arg\underset{j}{max}{\mathbf{1}}^T{\mathit{b}}_j{\mathit{b}}_j^{-}\mathbf{1},l=arg\underset{i}{max}{b}_{ik}^{-1}.$$

Finally, suppose that, given an $(n\times n)$-matrix $\mathit{A}$ with spectral radius $\lambda >0$, we need to find positive n-vectors $\mathit{x}$ that attain the minimum

$$\begin{array}{cccc}\hfill & \underset{\mathit{x}>\mathbf{0}}{min}\hfill & \hfill & {\mathbf{1}}^T\mathit{x}{\mathit{x}}^{-}\mathbf{1},\hfill \\ \hfill & \hfill & & \mathit{x}={\left({\lambda }^{-1}\mathit{A}\right)}^{\ast }\mathit{u},\mathit{u}>\mathbf{0}.\hfill \end{array}$$

(6)

A solution to this problem is found as follows [16,17].

Lemma 4.

Let $\mathit{A}$ be a matrix with spectral radius $\lambda >0$. Then, the minimum in problem (6) is equal to $\delta ={\mathbf{1}}^T{\left({\lambda }^{-1}\mathit{A}\right)}^{\ast }\mathbf{1}$, and all positive solutions are given in the parametric form

$$\mathit{x}={({\delta }^{-1}\mathbf{1}{\mathbf{1}}^T\oplus {\lambda }^{-1}\mathit{A})}^{\ast }\mathit{u},\mathit{u}>\mathbf{0}.$$

Finally, note that the parametric form of solutions offered by Lemmas 2–4 defines the set of solutions to problems (4)–(6) as a linear span of columns of corresponding matrices that generate the solutions.

3.3. Application to Pairwise Comparison Problem

Consider the problem of evaluating ratings of alternatives from pairwise comparison, which is solved using the log-Chebyshev approximation of a pairwise comparison matrix. In this case, we need to obtain a solution of problem (1), and then, if the result is not unique, solutions of problems (2) and (3). Below, we use results in [14,15,16,17] to demonstrate how these problems can be analytically solved in explicit form.

We combine the solutions of the above problems into a procedure to handle the pairwise comparison problem under consideration. The procedure includes the following main steps: (i) finding the set of all solution vectors that represent absolute ratings of alternatives; (ii) checking whether the vectors determine a unique solution of the problem; and, in the case of a nonunique solution, (iii) selecting vectors which best and worst differentiate alternatives with the highest and lowest ratings (the best and worth differentiating solutions).

3.3.1. Finding All Solution Vectors

First, we observe that the calculation of all solutions to the pairwise comparison problem with a matrix $\mathit{A}$ by using log-Chebyshev approximation requires solving problem (1). In the tropical algebra setting, the objective function in this problem takes the vector form ${\mathit{x}}^{-}\mathit{A}\mathit{x}$, whereas the problem itself is written as (4) and can be solved by Lemma 2. The application of the lemma starts with the evaluation of the spectral radius for the matrix $\mathit{A}$, given by

$$\lambda =\underset{k=1}{\overset{n}{⨁}}{\mathrm{tr}}^{1/k}\left({\mathit{A}}^k\right).$$

(7)

Furthermore, we construct the matrix ${\lambda }^{-1}\mathit{A}$ and calculate its Kleene star matrix

$$\mathit{B}={\left({\lambda }^{-1}\mathit{A}\right)}^{\ast }=\underset{k=0}{\overset{n-1}{⨁}}{\left({\lambda }^{-1}\mathit{A}\right)}^k.$$

(8)

The set of all solutions of the problem is described using a vector of parameters $\mathit{u}$ as

$$\mathit{x}=\mathit{B}\mathit{u}.\mathit{u}>\mathbf{0}$$

(9)

This result shows that these solutions form a set of linear combinations of columns (are generated by columns) in the Kleene matrix $\mathit{B}={\left({\lambda }^{-1}\mathit{A}\right)}^{\ast }$.

3.3.2. Checking Uniqueness of Solution

In the framework of the pairwise comparison problem, the solution obtained is considered unique if each column in the matrix $\mathit{B}$ generates the same space of vectors, which happens when all columns are collinear. To check the uniqueness of the solution, we apply a refinement procedure based on Lemma 1. The procedure examines the columns of the Kleene matrix $\mathit{B}=\left({\mathit{b}}_j\right)$ sequentially one by one and deletes a column if it is linearly dependent on others. For each column ${\mathit{b}}_j$, we form a matrix ${\mathit{B}}_{\left(j\right)}$ by taking all other columns of $\mathit{B}$, which have not yet been deleted, and then verify the condition

$${\left({\mathit{B}}_{\left(j\right)}{\left({\mathit{b}}_j^{-}{\mathit{B}}_{\left(j\right)}\right)}^{-}\right)}^{-}{\mathit{b}}_j=1.$$

If this condition holds, the column ${\mathit{b}}_j$ is deleted from the matrix $\mathit{B}$, and otherwise retained. The solution is unique if the procedure results in a matrix with a single column that is taken as the solution. In the case when a matrix with several linearly independent columns is obtained to generate a set of different solutions, we turn to finding the best and worst differentiating vectors of ratings among them.

3.3.3. Best Differentiating Solution

Suppose that a solution to the log-Chebyshev approximation problem is obtained in the form $\mathit{x}=\mathit{B}\mathit{u}$, where the matrix $\mathit{B}$ has linearly independent columns. To select the best differentiating solution, we need to solve problem (2). In the framework of tropical algebra, this problem has the objective function written in the form ${\mathbf{1}}^T\mathit{x}{\mathit{x}}^{-}\mathbf{1}$, and can be represented as problem (5), which is solved using Lemma 3.

According to Lemma 3, we need to find indices k and l that satisfy the conditions

$$k=arg\underset{j}{max}{\mathbf{1}}^T{\mathit{b}}_j{\mathit{b}}_j^{-}\mathbf{1},l=arg\underset{i}{max}{b}_{ik}^{-1}.$$

Then, we construct the matrix ${\mathit{B}}_{lk}$ by setting to zero all elements in the matrix $\mathit{B}$ except for the element $b_{lk}$. The best differentiating solution is then given using a vector of parameters ${\mathit{u}}_1$ by

$${\mathit{x}}_1=\mathit{B}(\mathit{I}\oplus {\mathit{B}}_{lk}^{-}\mathit{B}){\mathit{u}}_1,{\mathit{u}}_1>\mathbf{0}.$$

3.3.4. Worst Differentiating Solution

We obtain the worst differentiating solution by solving problem (3), which in terms of tropical algebra, takes the form of (6). The solution of the latter problem by applying Lemma 4 involves evaluating

$$\delta ={\mathbf{1}}^T{\left({\lambda }^{-1}\mathit{A}\right)}^{\ast }\mathbf{1}.$$

The worst differentiating solution is given in parametric form as

$${\mathit{x}}_2={({\delta }^{-1}\mathbf{1}{\mathbf{1}}^T\oplus {\lambda }^{-1}\mathit{A})}^{\ast }{\mathit{u}}_2,{\mathit{u}}_2>\mathbf{0}.$$

4. Evaluation of Criteria for Hotel Selection

In this section, we consider the application of the above-discussed techniques of ratings alternatives from pairwise comparisons to the problem of evaluating consumer preferences in hotel selection. The problem is to identify priorities among criteria used by students in choosing hotels in a foreign country. The research is based on data obtained from 202 students of both sexes aged 17 to 26 years, who are to take a short-term professional development program at an educational institution in the country. Some students have visited this country and may have some experience of living there.

The students are provided with a grant that covers the travel expenses, tuition payments and food expenses during the working day. At the same time, the students must pay for accommodation on their own and choose a hotel by applying information available in the Internet on booking portals and hotel websites as well as their previous experience on visiting the country.

There are six criteria that are used when choosing a hotel: ${\mathcal{C}}_1$—location, ${\mathcal{C}}_2$—accom-modation cost, ${\mathcal{C}}_3$—social environment (typical guests who usually stay at the hotel), ${\mathcal{C}}_4$—free breakfast, ${\mathcal{C}}_5$—amenities (room equipment), ${\mathcal{C}}_6$—courtesy of hotel staff. The aim of the research is to evaluate ratings (scores, priorities, weights) and then obtain ranks of the criteria, based on the results of the survey of the students as respondents.

During the survey process, the respondent directly assesses the degree of importance (the rating) of each criterion, determines the ranks of the criteria, and compares the criteria in pairs. For the direct rating of criteria, the following absolute scale is used: $1/5$—not important, $2/5$—important, $3/5$—fairly important, $4/5$—important, 1—very important. For the pairwise comparison of criteria, the following comparative scale is used: 1—equal importance, 2—moderate importance, 3—strong importance, 4—very strong importance; 5—extreme importance of one criterion over another.

The survey results together with the parameters of respondents (sex, age, etc.) are summarized in a table. For each respondent i, the survey results include

${\mathit{s}}_{\mathbf{R}}\left(i\right)$,

the vector of direct ratings (scores) of criteria by the respondent,

${\mathit{r}}_{\mathbf{R}}\left(i\right)$,

the vector of direct ranks of criteria by the respondent,

${\mathit{C}}_{\mathbf{R}}\left(i\right)$,

the matrix of pairwise comparisons of criteria, provided by the respondent.

Below, the survey results for three respondents with $i=1,2,3$ are given as examples.

Respondent 1, male, age 24, visited the country before, provides the following estimates:

$$\begin{array}{c}{\mathit{s}}_{\mathbf{R}}\left(1\right)=\left(\begin{array}{c}3/5\\ 1\\ 1/5\\ 2/5\\ 1\\ 1/5\end{array}\right)=\left(\begin{array}{c}0.60\\ 1.00\\ 0.20\\ 0.40\\ 1.00\\ 0.20\end{array}\right),{\mathit{r}}_{\mathbf{R}}\left(1\right)=\left(\begin{array}{c}3\\ 2\\ 5\\ 4\\ 1\\ 6\end{array}\right),\\ {\mathit{C}}_{\mathbf{R}}\left(1\right)=\left(\begin{array}{cccccc}1& 1/4& 5& 4& 1/5& 4\\ 4& 1& 5& 5& 1/3& 5\\ 1/5& 1/5& 1& 1/3& 1/5& 2\\ 1/4& 1/5& 3& 1& 1/5& 3\\ 5& 3& 5& 5& 1& 5\\ 1/4& 1/5& 1/2& 1/3& 1/5& 1\end{array}\right);\end{array}$$

Respondent 2, male, age 19, never visited the country before,

$$\begin{array}{c}{\mathit{s}}_{\mathbf{R}}\left(2\right)=\left(\begin{array}{c}1\\ 1\\ 3/4\\ 1/2\\ 3/4\\ 1/2\end{array}\right)=\left(\begin{array}{c}1.00\\ 1.00\\ 0.75\\ 0.50\\ 0.75\\ 0.50\end{array}\right),{\mathit{r}}_{\mathbf{R}}\left(2\right)=\left(\begin{array}{c}2\\ 1\\ 3\\ 4\\ 5\\ 6\end{array}\right),\\ {\mathit{C}}_{\mathbf{R}}\left(2\right)=\left(\begin{array}{cccccc}1& 1& 3& 3& 2& 4\\ 1& 1& 4& 2& 4& 5\\ 1/3& 1/4& 1& 3& 1& 4\\ 1/3& 1/2& 1/3& 1& 1& 3\\ 1/2& 1/4& 1& 1& 1& 3\\ 1/4& 1/5& 1/4& 1/3& 1/3& 1\end{array}\right);\end{array}$$

Respondent 3, female, age 21, never visited the country before,

$$\begin{array}{c}{\mathit{s}}_{\mathbf{R}}\left(3\right)=\left(\begin{array}{c}4/5\\ 1\\ 1/5\\ 3/5\\ 4/5\\ 3/5\end{array}\right)=\left(\begin{array}{c}0.80\\ 1.00\\ 0.20\\ 0.60\\ 0.80\\ 0.60\end{array}\right),{\mathit{r}}_{\mathbf{R}}\left(3\right)=\left(\begin{array}{c}2\\ 1\\ 6\\ 4\\ 3\\ 5\end{array}\right),\\ {\mathit{C}}_{\mathbf{R}}\left(3\right)=\left(\begin{array}{cccccc}1& 1/4& 5& 4& 1/3& 3\\ 4& 1& 5& 5& 3& 5\\ 1/5& 1/5& 1& 1/3& 1/5& 1/3\\ 1/4& 1/5& 3& 1& 1/4& 1\\ 3& 1/3& 5& 4& 1& 5\\ 1/3& 1/5& 3& 1& 1/5& 1\end{array}\right).\end{array}$$

Given the pairwise comparison matrix obtained from each respondent, the absolute ratings of criteria are found based on three computational methods. We obtain vectors of ratings by application of the principal eigenvector method and the method of geometric mean. Furthermore, we solve the problem by using the log-Chebyshev approximation in the framework of tropical algebra, which yields two vectors that are best and worst differentiate the criteria with the highest and lowest ratings (the best and worst log-Chebyshev approximation vectors). The vectors of ratings obtained are then used to rank criteria according to the values of ratings.

As a result, for each respondent i, the following vectors of ratings normalized by dividing by the maximum element, and rank vectors are calculated:

${\mathit{s}}_{\mathbf{R}}\left(i\right)$,

the direct ratings (scores) by the respondent (SR),

${\mathit{r}}_{\mathbf{R}}\left(i\right)$,

the direct ranks by the respondent (RR),

${\mathit{r}}_{{\mathit{s}}_{\mathbf{R}}}\left(i\right)$,

the ranks based on direct ratings by the respondent (RSR),

${\mathit{s}}_{\mathbf{PE}}\left(i\right)$,

the ratings by the method of principal eigenvector (SPE),

${\mathit{r}}_{{\mathit{s}}_{\mathbf{PE}}}\left(i\right)$,

the ranks based on the principal eigenvector ratings (RSPE),

${\mathit{s}}_{\mathbf{GM}}\left(i\right)$,

the ratings by the method of geometric mean (SGM),

${\mathit{r}}_{{\mathit{s}}_{\mathbf{GM}}}\left(i\right)$,

the ranks based on the geometric mean ratings (RSGM),

${\mathit{s}}_{\mathbf{CB}}\left(i\right)$,

the best ratings by the method of log-Chebyshev approximation (SCB),

${\mathit{r}}_{{\mathit{s}}_{\mathbf{CB}}}\left(i\right)$,

the ranks based on the best ratings from the log-Chebyshev approximation (RSCB),

${\mathit{s}}_{\mathbf{CW}}\left(i\right)$,

the worst ratings by the method of log-Chebyshev approximation (SCW),

${\mathit{r}}_{{\mathit{s}}_{\mathbf{CW}}}\left(i\right)$,

the ranks based on the worst ratings from the log-Chebyshev approximation (RSCW).

For respondents $i=1,2,3$ considered above as examples, we have the following results. For respondent 1, the vectors of ratings and ranks take the form

$$\begin{array}{c}{\mathit{r}}_{{\mathit{s}}_{\mathbf{R}}}\left(1\right)=\left(\begin{array}{c}3\\ 1\\ 5\\ 4\\ 2\\ 6\end{array}\right),{\mathit{s}}_{\mathbf{PE}}\left(1\right)=\left(\begin{array}{c}0.3581\\ 0.6526\\ 0.1142\\ 0.1832\\ 1.0000\\ 0.0943\end{array}\right),{\mathit{r}}_{{\mathit{s}}_{\mathbf{PE}}}\left(1\right)=\left(\begin{array}{c}3\\ 2\\ 5\\ 4\\ 1\\ 6\end{array}\right),\\ {\mathit{s}}_{\mathbf{GM}}\left(1\right)=\left(\begin{array}{c}0.3588\\ 0.6680\\ 0.1190\\ 0.1906\\ 1.0000\\ 0.0981\end{array}\right),{\mathit{s}}_{\mathbf{CB}}\left(1\right)=\left(\begin{array}{c}0.3218\\ 0.6551\\ 0.1036\\ 0.1581\\ 1.0000\\ 0.1018\end{array}\right),{\mathit{s}}_{\mathbf{CW}}\left(1\right)=\left(\begin{array}{c}0.3218\\ 0.6551\\ 0.1036\\ 0.1581\\ 1.0000\\ 0.1018\end{array}\right),\\ {\mathit{r}}_{{\mathit{s}}_{\mathbf{GM}}}\left(1\right)=\left(\begin{array}{c}3\\ 2\\ 5\\ 4\\ 1\\ 6\end{array}\right),{\mathit{r}}_{{\mathit{s}}_{\mathbf{CB}}}\left(1\right)=\left(\begin{array}{c}3\\ 2\\ 5\\ 4\\ 1\\ 6\end{array}\right),{\mathit{r}}_{{\mathit{s}}_{\mathbf{CW}}}\left(1\right)=\left(\begin{array}{c}3\\ 2\\ 5\\ 4\\ 1\\ 6\end{array}\right).\end{array}$$

From the results obtained, it follows that the ranks of criteria provided by all methods for respondent 1 coincide and arrange the criteria in the following order of preference with the symbol ≻ used to indicate the preference relation:

$${\mathcal{C}}_5\succ {\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_4\succ {\mathcal{C}}_6.$$

Note that the best and worst differentiating solutions lead the same vector of ratings, and thus the method of log-Chebyshev approximation has a unique solution in this case.

For respondent 2, the calculation results in the following vectors of ratings and ranks:

$$\begin{array}{c}{\mathit{r}}_{{\mathit{s}}_{\mathbf{R}}}\left(2\right)=\left(\begin{array}{c}1\\ 2\\ 3\\ 5\\ 4\\ 6\end{array}\right),{\mathit{s}}_{\mathbf{PE}}\left(2\right)=\left(\begin{array}{c}0.8485\\ 1.0000\\ 0.4462\\ 0.3170\\ 0.3467\\ 0.1392\end{array}\right),{\mathit{r}}_{{\mathit{s}}_{\mathbf{PE}}}\left(2\right)=\left(\begin{array}{c}2\\ 1\\ 3\\ 5\\ 4\\ 6\end{array}\right),\\ {\mathit{s}}_{\mathbf{GM}}\left(2\right)=\left(\begin{array}{c}0.8754\\ 1.0000\\ 0.4292\\ 0.3184\\ 0.3645\\ 0.1434\end{array}\right),{\mathit{s}}_{\mathbf{CB}}\left(2\right)=\left(\begin{array}{c}0.7500\\ 1.0000\\ 0.4543\\ 0.2752\\ 0.2500\\ 0.1101\end{array}\right),{\mathit{s}}_{\mathbf{CW}}\left(2\right)=\left(\begin{array}{c}1.0000\\ 1.0000\\ 0.4543\\ 0.2752\\ 0.4543\\ 0.1667\end{array}\right),\\ {\mathit{r}}_{{\mathit{s}}_{\mathbf{GM}}}\left(2\right)=\left(\begin{array}{c}2\\ 1\\ 3\\ 5\\ 4\\ 6\end{array}\right),{\mathit{r}}_{{\mathit{s}}_{\mathbf{CB}}}\left(2\right)=\left(\begin{array}{c}2\\ 1\\ 3\\ 4\\ 5\\ 6\end{array}\right),{\mathit{r}}_{{\mathit{s}}_{\mathbf{CW}}}\left(2\right)=\left(\begin{array}{c}1\\ 2\\ 3\\ 5\\ 4\\ 6\end{array}\right).\end{array}$$

We observe that the ranks of the criteria obtained by both methods of the principal vector and of geometric mean are the same and determine the order

$${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_3\succ {\mathcal{C}}_5\succ {\mathcal{C}}_4\succ {\mathcal{C}}_6.$$

Furthermore, we may consider that the ratings, which are provided by the worst differentiating vector of log-Chebyshev approximation, lead to the above ranking as well. Indeed, this method gives equal ratings to the first and second criteria to allow these criteria to be ranked in any order, one of which is the same as above.

The best and worst differentiating vectors of ratings obtained as a result of the log-Chebyshev approximation differ in some elements and can be coupled as a vector with interval values in the form

$${\mathit{s}}_{\mathbf{C}}\left(2\right)=\left(\begin{array}{c}0.7500\cdots 1.0000\\ 1.0000\\ 0.4543\\ 0.2752\\ 0.2500\cdots 0.4543\\ 0.1101\cdots 0.1667\end{array}\right).$$

The order provided by these vectors can be represented in combined form as

$${\mathcal{C}}_2⪰{\mathcal{C}}_1\succ {\mathcal{C}}_3⪰{\mathcal{C}}_5\Vert {\mathcal{C}}_4\succ {\mathcal{C}}_6,$$

where the symbol ⪰ denotes the weak preference relation (preferred or indifferent) and ‖ indicates an undetermined preference.

Finally, the results obtained for respondent 3 are given by the vectors

$$\begin{array}{c}{\mathit{r}}_{{\mathit{s}}_{\mathbf{R}}}\left(3\right)=\left(\begin{array}{c}2\\ 1\\ 6\\ 4\\ 3\\ 5\end{array}\right),{\mathit{r}}_{{\mathit{s}}_{\mathbf{PE}}}\left(3\right)=\left(\begin{array}{c}0.3773\\ 1.0000\\ 0.0930\\ 0.1630\\ 0.6213\\ 0.1631\end{array}\right),{\mathit{r}}_{{\mathit{s}}_{\mathbf{PE}}}\left(3\right)=\left(\begin{array}{c}3\\ 1\\ 6\\ 5\\ 2\\ 4\end{array}\right),\\ {\mathit{s}}_{\mathbf{GM}}\left(3\right)=\left(\begin{array}{c}0.3865\\ 1.0000\\ 0.0916\\ 0.1710\\ 0.6368\\ 0.1728\end{array}\right),{\mathit{s}}_{\mathbf{CB}}\left(3\right)=\left(\begin{array}{c}0.3798\\ 1.0000\\ 0.1082\\ 0.1755\\ 0.6163\\ 0.1755\end{array}\right),{\mathit{s}}_{\mathbf{CW}}\left(3\right)=\left(\begin{array}{c}0.3798\\ 1.0000\\ 0.1082\\ 0.1755\\ 0.6163\\ 0.2279\end{array}\right),\\ {\mathit{r}}_{{\mathit{s}}_{\mathbf{GM}}}\left(3\right)=\left(\begin{array}{c}3\\ 1\\ 6\\ 5\\ 2\\ 4\end{array}\right),{\mathit{r}}_{{\mathit{s}}_{\mathbf{CB}}}\left(3\right)=\left(\begin{array}{c}3\\ 1\\ 6\\ 4\\ 2\\ 5\end{array}\right),{\mathit{r}}_{{\mathit{s}}_{\mathbf{CW}}}\left(3\right)=\left(\begin{array}{c}3\\ 1\\ 6\\ 5\\ 2\\ 4\end{array}\right).\end{array}$$

In this case, the solutions provided by the method of principal eigenvector and method of geometric mean as well as the worst differentiating solution by the log-Chebyshev approximation define the same ranks of criteria, which yields the order

$${\mathcal{C}}_2\succ {\mathcal{C}}_5\succ {\mathcal{C}}_1\succ {\mathcal{C}}_6\succ {\mathcal{C}}_4\succ {\mathcal{C}}_3.$$

The best and worst differentiating vectors of ratings differ only by the last element and can be combined as follows:

$${\mathit{s}}_{\mathbf{C}}\left(3\right)=\left(\begin{array}{c}0.3798\\ 1.0000\\ 0.1082\\ 0.1755\\ 0.6163\\ 0.1755\cdots 0.2279\end{array}\right).$$

The order of criteria, which corresponds to these vectors, can be represented as

$${\mathcal{C}}_2\succ {\mathcal{C}}_5\succ {\mathcal{C}}_1\succ {\mathcal{C}}_6⪰{\mathcal{C}}_4\succ {\mathcal{C}}_3.$$

It follows from the above examples that the results of determining the ranks of criteria, which are obtained from their pairwise comparisons by applying different methods may vary. In the next sections, we examine the difference between these results more closely by using appropriate techniques of data analysis.

5. Statistical Analysis of Ratings and Ranks

In this section, we compare ratings and ranks of criteria, obtained by different methods for all respondents participating in the survey. Ratings and ranks that are directly specified by respondents are compared with those which are based on three methods of evaluating criteria from pairwise comparisons. We examine the results of the principal eigenvector method, the geometric mean method, and the method of log-Chebyshev approximation. For the latter method, we consider two solution vectors, which best and worst differentiate the criteria if the solution is nonunique (both vectors coincide when the solution is unique).

To represent the obtained results in table form below, we use the following symbols:

SR, 

the direct ratings (scores) given by respondent (Scores by Respondent),

RR, 

the direct ranks by respondent (Ranks by Respondent),

RSR, 

the ranks from the direct ratings by respondent (Ranks from Scores by Respondent),

SPE, 

the ratings by the method of principal eigenvector (Scores by Principal Eigenvector),

RSPE, 

the ranks from the principal eigenvector ratings (Ranks from Scores by Principal Eigenvector),

SGM, 

the ratings by the method of geometric mean (Scores by Geometric Means),

RSGM, 

the ranks from the geometric mean ratings (Ranks from Scores by Geometric Means),

SCB, 

the best differentiating ratings by the method of log-Chebyshev approximation (Scores by log-Chebyshev, Best),

RSCB, 

the ranks from the best ratings from the log-Chebyshev approximation (Ranks from Scores by log-Chebyshev, Best),

SCW, 

the worst differentiating ratings from the log-Chebyshev approximation (Scores by log-Chebyshev, Worst),

RSCW, 

the ranks from the worst ratings from the log-Chebyshev approximation (Ranks from Scores by log-Chebyshev, Worst).

We start by counting the matches in the rank vectors obtained directly and determined through pairwise comparisons. The numbers of matches between rank vectors for each pair of the ranking methods are presented in

Table 1. Number of matches of rank vectors.

Ranking Method	Ranking Method
	RR	RSR	RSPE	RSGM	RSCB	RSCW
RR	202	28	56	56	59	56
RSR	28	202	20	20	18	23
RSPE	56	20	202	184	124	123
RSGM	56	20	184	202	124	125
RSCB	59	18	124	124	202	130
RSCW	56	23	123	125	130	202

.

The results given by Table 1 show a low level of matches (encountered for 28 out of 202 respondents) between the vectors of direct ranks (RR) and ranks determined from ratings that are specified by respondents (RSR). We can explain this inconsistency by greater variability in determining ratings by respondents, which allows one to assign the same ratings to several criteria. As a result, the ranks obtained from ratings may differ from the direct ranking, when the respondent specifies an individual rank for each criterion to indicate the number (position) of the criterion in the ranking order.

The degree of consistency of the direct ranking (RR) with the ranking on the basis of ratings obtained from pairwise comparisons (RSPE, RSGM, RSCB, RSCW) is better given by 56–59 matches of rank vectors, which are more than a quarter of respondents. Note that the maximum number of matches, which is equal to 59, is provided by the best differentiating solution of log-Chebyshev approximation (RSCB).

We note that the results of ranking criteria from pairwise comparisons demonstrate a higher level of consistency between the methods under consideration. Specifically, the largest number of matches encountered for 184 (more than 90% of) respondents is given by the outcome from the methods of principal eigenvector (RSPE) and of geometric mean (RSGM). The number of matches between the rank vectors produced by these methods and the vectors obtained using log-Chebyshev approximation (RSCB, RSCW) is within the range 123–125 (more than 60% of respondents). Finally, the rank vectors, which are obtained from the best and worst differentiating solutions found by log-Chebyshev approximation, coincide for 130 respondents.

Below, to analyze and compare the results more thoroughly, we use appropriate statistical techniques of parameter estimation and correlation analysis. As preliminary analysis of data, we investigate if parameters of respondents, such as age and sex, may affect the results of the evaluation of criteria. Furthermore, we compare the results of evaluating ratings and ranks obtained from pairwise comparisons by different methods with each other and with the ratings and ranks directly provided by respondents.

5.1. Preliminary Analysis of Data

In the analysis of indirect methods that produce results from pairwise comparison data, it seems reasonable to consider the ratings and ranks of criteria, which are directly specified by respondents, as some basis for comparison. Since these ratings and ranks are known for all respondents, we can try to select those respondents for whom the direct ranks and ranks given by direct ratings appear to be closest to each other. Considering that a part of respondents may demonstrate a higher level of consistency in the direct judgment, one can expect that these respondents are able to assess pairwise comparison of criteria more adequately and definitely.

To examine how the consistency between direct estimates of ratings and ranks may affect the results based on pairwise comparisons, we consider groups of respondents with different degrees of consistency. As a measure of inconsistency (difference) between rank vectors, we apply the Chebyshev metric (the maximum absolute componentwise difference), which is calculated for two n-vectors $\mathit{a}=\left(a_j\right)$ and $\mathit{b}=\left(b_j\right)$ as

$$d(\mathit{a},\mathit{b})=\underset{1\le j\le n}{max}|a_j-b_j|.$$

We consider a series of nested subsets (groups) $R_0\subset R_1\subset \cdots \subset R_5$, where $R_i$ denotes the set of respondents for whom the Chebyshev metric between the vectors of direct ranks and ranks obtained from the direct ratings is less or equal to $i=0,1,\dots ,5$.

Table 2. Groups of respondents according to difference between rank vectors.

Group	Maximum Component-Wise Difference	Number of Respondents in the Group	Percentage of Respondents Who Visited the Country	Male/Female Percentage
$R_0$	0	28	35	33/67
$R_1$	1	95	19	32/68
$R_2$	2	167	18	30/70
$R_3$	3	194	17	28/72
$R_4$	4	201	18	28/72
$R_5$	5	202	19	28/72

shows how the total number of respondents in groups changes with increase in the upper bound for Chebyshev metric, which indicates the maximum degree of difference in groups. For each group, the percentage of respondents who visited the country and male/female percentage are also included.

It follows from the data presented in the table that the direct ranks and the ranks obtained from direct ratings completely coincide for 28 respondents from the group $R_0$, who may be considered as the most accurate evaluators. However, these respondents form a rather small part of all respondents involved in the survey, which makes it unreasonable to take this group as a good representative of the entire sample.

Furthermore, we note that the respondents in the groups $R_1,\dots ,R_5$ with less consistent direct judgments demonstrate almost the same percentage of those who visited the country (17–19%). This apparently indicates that for the most respondents, there is no relationship between visiting the country and accuracy of judgments. At the same time, for the group $R_0$, this percentage increases to 35%, which is in line with the idea that the accuracy of judgments should increase as respondents gain experience of staying in the country. A conclusion about the lack of systematic dependence of the accuracy of judgments on the gender and age of respondents can also be drawn for all groups.

To complete the analysis of the possible influence of the accuracy of direct judgments about ratings and ranks on the other survey results, we estimate correlations between the rank vectors directly provided by respondents and the vectors derived from pairwise comparisons. To measure correlation, we use the Kendall rank correlation coefficient, which is given for two n-vectors $\mathit{a}=\left(a_j\right)$ and $\mathit{b}=\left(b_j\right)$ by the formula

$$\tau (\mathit{a},\mathit{b})=\frac{2}{n(n-1)}\sum _{1\le i<j\le n}\mathrm{sgn}(a_i-a_j)\mathrm{sgn}(b_i-b_j).$$

For each group $R_0$, $R_1$ and $R_5$, we concatenate the vectors of ratings obtained for all respondents in the group to form a single vector of ratings for each method. The estimates of the correlation coefficients between concatenated vectors for the groups $R_0$, $R_1$ and $R_5$ are given in

Table 3. Correlation of rank vectors for the group $R_0$.

Ranking Method	Ranking Method
	RR	RSPE	RSGM	RSCB	RSCW
RR	1.0000	0.8482	0.8482	0.8560	0.8645
RSPE	0.8482	1.0000	1.0000	0.9558	0.9557
RSGM	0.8482	1.0000	1.0000	0.9557	0.9557
RSCB	0.8560	0.9558	0.9558	1.0000	0.9717
RSCW	0.8645	0.9557	0.9557	0.9717	1.0000

,

Table 4. Correlation of rank vectors for the group $R_1$.

Ranking Method	Ranking Method
	RR	RSPE	RSGM	RSCB	RSCW
RR	1.0000	0.8184	0.8234	0.8108	0.8221
RSPE	0.8184	1.0000	0.9943	0.9450	0.9183
RSGM	0.8234	0.9943	1.0000	0.9429	0.9223
RSCB	0.8108	0.9450	0.9429	1.0000	0.9234
RSCW	0.8221	0.9183	0.9223	0.9234	1.0000

and

Table 5. Correlation of rank vectors for the group $R_5$.

Ranking Method	Ranking Method
	RR	RSPE	RSGM	RSCB	RSCW
RR	1.0000	0.7939	0.7940	0.7876	0.7839
RSPE	0.7939	1.0000	0.9862	0.9312	0.9186
RSGM	0.7940	0.9862	1.0000	0.9290	0.9204
RSCB	0.7876	0.9312	0.9290	1.0000	0.9144
RSCW	0.7839	0.9186	0.9204	0.9144	1.0000

(the estimates for groups $R_2$, $R_3$ and $R_4$ do not differ significantly from the results for $R_1$ and $R_5$ and hence are omitted).

The data presented in these tables show that for all groups of respondents, the correlations vary within small limits and maintain the relative order of magnitude for different methods. This allows us to conclude that the change in accuracy of direct estimates does not significantly affect the results of ranking based on pairwise comparisons.

In the subsequent analysis, we take into account the above results and examine all respondents together rather than divide them into groups according to the accuracy of direct judgments or parameters of respondents.

5.2. Comparison of Ratings

We now examine results of evaluating ratings, obtained by different methods from pairwise comparisons and specified directly by respondents. First, we calculate the means and standard deviations of vectors of ratings for each rating method. The mean vectors can be considered as the most representative (typical) vectors over all respondents for each method to give some general idea of the absolute ratings of criteria in hotel selection. The standard deviations describing the spread of the vectors can characterize both the differences in the perception of the relative importance of criteria by respondents and the ability of the methods to reinforce or smooth these differences.

The results of calculating the mean vectors presented in

Table 6. Mean vectors of rating of criteria.

Criterion Number	Rating Method
	SR	SPE	SGM	SCB	SCW
1	0.7480	0.6268	0.6272	0.5693	0.6313
2	0.9428	0.8877	0.8915	0.8504	0.9025
3	0.5784	0.3238	0.3231	0.2861	0.3511
4	0.5588	0.3044	0.3053	0.2735	0.3291
5	0.7331	0.5279	0.5303	0.4910	0.5581
6	0.6149	0.2543	0.2558	0.2252	0.2879

show that on average, criterion 2 is always rated higher than the others, next come criterion 1 and then 5. The other criteria have lower mean ratings, which give the order $(3,4,6)$ for all methods except the direct rating with the order $(6,3,4)$. Examination of the output of individual respondents indicates that the second criterion is evaluated above the others for more than 60% respondents. Specifically, this result encounters in the direct rating (SR) for 165 respondents, in both methods of principal eigenvector (SPE) and geometric mean (SGM) for 125 respondents, and in the best and worst differentiating solutions of log-Chebyshev approximation (SCB and SCW) for 123 and 147 respondents respectively.

It follows from Table 6 that the direct ratings by respondents are sufficiently different from the other results. This can be explained by a rather primitive scale used in the survey, which includes only 5 points. At the same time, the evaluation of mean vectors for all indirect methods of rating criteria gives quite close results. We note that the mean vectors of the log-Chebyshev approximation appear as lower and upper boundaries for the mean vectors for both methods of principal eigenvector and geometric mean.

To measure the overall variability of ratings for all methods, we use the average standard deviation over criteria and a total (vector) standard deviation defined as the square root of the sum of squares of standard deviations for each criterion. The results of calculation are given in

Table 7. Variability estimates for vectors of ratings.

Variability Measure	Rating Method
	SR	SPE	SGM	SCB	SCW
Average deviation	0.2037	0.2202	0.2198	0.2252	0.2339
Total deviation	0.5046	0.5477	0.5478	0.5613	0.5796

and say, in particular, that the total standard deviations for the methods of principal eigenvector and geometric mean almost coincide, and both methods have slightly lower variability of ratings than the log-Chebyshev approximation.

Table 8. Standard deviation of ratings of each criterion.

Criterion Number	Rating Method
	SR	SPE	SGM	SCB	SCW
1	0.2188	0.2578	0.2611	0.2685	0.2633
2	0.1379	0.1837	0.1807	0.2175	0.1879
3	0.2255	0.2300	0.2279	0.2163	0.2426
4	0.2248	0.2052	0.2039	0.1963	0.2150
5	0.1978	0.2769	0.2798	0.2889	0.2914
6	0.2173	0.1676	0.1656	0.1639	0.2032

demonstrates the standard deviation of ratings for each pair of criteria and methods. We note that criteria 2 and 6 most often have the smallest or the next smallest standard deviation for all methods, which suggests that the real assessment of these criteria differs the least from one respondent to another.

Next, we turn to the estimates of correlation between vectors of ratings produced by the direct and indirect assessment methods. For each method, we concatenate the vectors of all respondents into a single vector of ratings, and then estimate the Pearson correlation coefficients between the concatenated vectors. The correlation coefficients calculated for all pairs of methods are given in

Table 9. Correlation of vectors of ratings for all pairs of methods.

Rating Method	Rating Method
	SR	SPE	SGM	SCB	SCW
SR	1.0000	0.6942	0.6918	0.6619	0.6889
SPE	0.6942	1.0000	0.9973	0.9613	0.9622
SGM	0.6918	0.9973	1.0000	0.9631	0.9576
SCB	0.6619	0.9613	0.9631	1.0000	0.9164
SCW	0.6889	0.9622	0.9576	0.9164	1.0000

.

The results presented in this table show that the correlation between ratings obtained from pairwise comparisons is close to one for all pairs of methods, which means that any two concatenated vectors of ratings are close to be linearly dependent (collinear). Specifically, the vectors found by the methods of principal eigenvector and geometric mean correlate at the level of $0.9973$, and thus can be considered poorly distinguishable from the correlation point of view.

We now consider the correlations of ratings produced by all methods for each individual criterion.

Table 10. Correlation of ratings of criterion 1 for all pairs of methods.

Rating Method	Rating Method
	SR	SPE	SGM	SCB	SCW
SR	1.0000	0.5815	0.5639	0.5151	0.5758
SPE	0.5815	1.0000	0.9949	0.9215	0.9408
SGM	0.5639	0.9949	1.0000	0.9250	0.9220
SCB	0.5151	0.9215	0.9250	1.0000	0.8723
SCW	0.5758	0.9408	0.9220	0.8723	1.0000

,

Table 11. Correlation of ratings of criterion 2 for all pairs of methods.

Rating Method	Rating Method
	SR	SPE	SGM	SCB	SCW
SR	1.0000	0.4243	0.4314	0.3778	0.4348
SPE	0.4243	1.0000	0.9959	0.8887	0.9217
SGM	0.4314	0.9959	1.0000	0.8899	0.9239
SCB	0.3778	0.8887	0.8899	1.0000	0.8004
SCW	0.4348	0.9217	0.9239	0.8004	1.0000

and

Table 12. Correlation of ratings of criterion 6 for all pairs of methods.

Rating Method	Rating Method
	SR	SPE	SGM	SCB	SCW
SR	1.0000	0.4926	0.4873	0.4569	0.4842
SPE	0.4926	1.0000	0.9929	0.9301	0.8938
SGM	0.4873	0.9929	1.0000	0.9189	0.8911
SCB	0.4569	0.9301	0.9189	1.0000	0.7776
SCW	0.4842	0.8938	0.8911	0.7776	1.0000

demonstrate the estimates for criteria 1, 2 and 6 (the results for criteria 3, 4 and 5 are similar to those for criterion 1 and thus omitted).

The results obtained for each criterion indicate that there is a high correlation at the level of $0.9929\cdots 0.9959$ between the ratings produced by the method of principal eigenvector and the method of geometric mean.

Furthermore, the ratings provided by the best differentiating solutions of log-Chebyshev approximation are usually more correlated with the ratings by the method of geometric mean ($0.8899\cdots 0.9543$), whereas the ratings from the worst differentiating solutions are more correlated with those from the method of principal eigenvector ($0.8938\cdots 0.9408$). Finally, the correlation between ratings from the best and worst differentiating solutions are somewhat lower and lay in the range $0.7776\cdots 0.8752$.

To summarize the results of the comparison of ratings, we can conclude that for the survey data under study, all indirect methods of rating criteria consistently produce highly correlated vectors of ratings, with the highest correlation very close to one between the results of the methods of principal eigenvector and geometric mean. The good agreement between the results of indirect methods can be explained by the natural origin of the initial data obtained as results of human judgment, which provide rationale for more consistent pairwise comparisons than artificial simulated data.

In conclusion, we note low correlations between direct and indirect ratings, which are within the range $0.3778\cdots 0.5993$. This relative disagreement may serve as an illustration of the known fact that it may be difficult for typical respondents to make a correct judgment when more than two alternatives are simultaneously evaluated, whereas they normally assess pairwise comparisons more accurately.

5.3. Comparison of Ranks

We start with the ranks derived from ratings given by the mean vectors in Table 6. The mean vector of ratings that are directly provided by respondents yields the rank vector $(2,1,5,6,3,4)$, which arranges the criteria in the following order:

$${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_6\succ {\mathcal{C}}_3\succ {\mathcal{C}}_4.$$

To see how frequently this order coincides with the order derived from the vectors of ratings of individual respondents, we calculate the Hamming distance between (the number of different elements in) the corresponding rank vectors. The results of counting the number of rank vectors, which are obtained from vectors of ratings for all methods, within fixed distances from the vector $(2,1,5,6,3,4)$ are given in

Table 13. Number of vectors within fixed distance from $(2,1,5,6,3,4)$.

Maximum Hamming Distance	Rating Method
	RSR	RSPE	RSGM	RSCB	RSCW
0	5	3	3	4	4
1	5	3	3	4	4
2	23	35	32	35	31
3	61	68	65	66	64
4	122	135	130	123	132
5	178	185	186	186	188
6	202	202	202	202	202

.

The mean vectors of ratings derived from pairwise comparisons by different methods imply a common rank vector $(2,1,5,3,4,6)$, which corresponds to the order

$${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_3\succ {\mathcal{C}}_4\succ {\mathcal{C}}_6.$$

The numbers of rank vectors within fixed distances from the vector $(2,1,5,3,4,6)$ are shown in

Table 14. Number of vectors within fixed distance from $(2,1,5,3,4,6)$.

Maximum Hamming Distance	Rating Method
	RSR	RSPE	RSGM	RSCB	RSCW
0	6	4	4	7	5
1	6	4	4	7	5
2	37	40	38	48	41
3	66	64	63	69	63
4	121	133	132	130	129
5	175	180	179	183	184
6	202	202	202	202	202

.

Both Table 13 and Table 14 demonstrate low correspondence between the rank vectors derived from the mean vectors of ratings and the individual rank vectors for each respondent. In this case, we cannot consider the ranks based on the mean vectors as well justified, and need further examination of the individual rank vectors to find those vectors that occur more frequently for each method. The results of determining the five most common rank vectors for each method are presented in

Table 15. Most frequently occurred rank vectors.

Ranking Method	Rank Vector	Order of Criteria	Number of Occurrences
	$(1,2,5,3,4,6)$	${\mathcal{C}}_1\succ {\mathcal{C}}_2\succ {\mathcal{C}}_5\succ {\mathcal{C}}_3\succ {\mathcal{C}}_4\succ {\mathcal{C}}_6$	10
	$(2,1,5,3,6,4)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6\succ {\mathcal{C}}_4$	9
RSR	$(1,2,3,5,6,4)$	${\mathcal{C}}_1\succ {\mathcal{C}}_2\succ {\mathcal{C}}_3\succ {\mathcal{C}}_5\succ {\mathcal{C}}_6\succ {\mathcal{C}}_4$	8
	$(2,5,6,1,3,4)$	${\mathcal{C}}_2\succ {\mathcal{C}}_5\succ {\mathcal{C}}_6\succ {\mathcal{C}}_1\succ {\mathcal{C}}_3\succ {\mathcal{C}}_4$	7
	$(1,2,3,4,5,6)$	${\mathcal{C}}_1\succ {\mathcal{C}}_2\succ {\mathcal{C}}_3\succ {\mathcal{C}}_4\succ {\mathcal{C}}_5\succ {\mathcal{C}}_6$	6
	$(2,1,5,3,6,4)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6\succ {\mathcal{C}}_4$	12
	$(2,1,5,4,6,3)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_4\succ {\mathcal{C}}_6\succ {\mathcal{C}}_3$	8
RSPE	$(2,1,5,4,3,6)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_4\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6$	7
	$(2,5,1,3,6,4)$	${\mathcal{C}}_2\succ {\mathcal{C}}_5\succ {\mathcal{C}}_1\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6\succ {\mathcal{C}}_4$	6
	$(2,1,4,5,6,3)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_4\succ {\mathcal{C}}_5\succ {\mathcal{C}}_6\succ {\mathcal{C}}_3$	6
	$(2,1,5,3,6,4)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6\succ {\mathcal{C}}_4$	12
	$(2,1,5,4,6,3)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_4\succ {\mathcal{C}}_6\succ {\mathcal{C}}_3$	8
RSGM	$(2,1,4,5,6,3)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_4\succ {\mathcal{C}}_5\succ {\mathcal{C}}_6\succ {\mathcal{C}}_3$	7
	$(1,2,5,4,3,6)$	${\mathcal{C}}_1\succ {\mathcal{C}}_2\succ {\mathcal{C}}_5\succ {\mathcal{C}}_4\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6$	7
	$(2,1,3,5,6,4)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_3\succ {\mathcal{C}}_5\succ {\mathcal{C}}_6\succ {\mathcal{C}}_4$	6
	$(2,1,5,3,6,4)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6\succ {\mathcal{C}}_4$	11
	$(2,5,1,3,4,6)$	${\mathcal{C}}_2\succ {\mathcal{C}}_5\succ {\mathcal{C}}_1\succ {\mathcal{C}}_3\succ {\mathcal{C}}_4\succ {\mathcal{C}}_6$	7
RSCB	$(2,1,5,3,4,6)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_3\succ {\mathcal{C}}_4\succ {\mathcal{C}}_6$	7
	$(2,1,5,4,3,6)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_4\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6$	7
	$(1,2,5,4,3,6)$	${\mathcal{C}}_1\succ {\mathcal{C}}_2\succ {\mathcal{C}}_5\succ {\mathcal{C}}_4\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6$	6
	$(2,1,5,3,6,4)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6\succ {\mathcal{C}}_4$	9
	$(2,5,1,3,4,6)$	${\mathcal{C}}_2\succ {\mathcal{C}}_5\succ {\mathcal{C}}_1\succ {\mathcal{C}}_3\succ {\mathcal{C}}_4\succ {\mathcal{C}}_6$	8
RSCW	$(2,1,5,4,3,6)$	${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_4\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6$	8
	$(1,2,5,4,3,6)$	${\mathcal{C}}_1\succ {\mathcal{C}}_2\succ {\mathcal{C}}_5\succ {\mathcal{C}}_4\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6$	8
	$(2,5,1,4,3,6)$	${\mathcal{C}}_2\succ {\mathcal{C}}_5\succ {\mathcal{C}}_1\succ {\mathcal{C}}_4\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6$	6

.

It follows from Table 15 that the most frequently occurred rank vector derived from ratings for all indirect methods and the second most frequent for the direct ratings by respondents is $(2,1,5,3,6,4)$. This rank vector yields the order of criteria defined as

$${\mathcal{C}}_2\succ {\mathcal{C}}_1\succ {\mathcal{C}}_5\succ {\mathcal{C}}_3\succ {\mathcal{C}}_6\succ {\mathcal{C}}_4,$$

and seems to provide a more accurate assessment of the ranks of criteria for the set of respondents participating in the survey.

We conclude with the results in

Table 16. Correlations of rank vectors for all methods.

Ranking Method	Ranking Method
	RR	RSPE	RSGM	RSCB	RSCW
RR	1.0000	0.7939	0.7940	0.7876	0.7839
RSPE	0.7939	1.0000	0.9862	0.9312	0.9186
RSGM	0.7940	0.9862	1.0000	0.9290	0.9204
RSCB	0.7876	0.9312	0.9290	1.0000	0.9144
RSCW	0.7839	0.9186	0.9204	0.9144	1.0000

, which demonstrate the Kendall correlation between concatenated rank vectors obtained by direct and indirect methods (this table actually presents the same results as in Table 5 used in the preliminary analysis of data).

As for the correlation of vectors of ratings obtained from pairwise comparisons, the corresponding rank vectors are highly correlated for all methods. As before, the methods of principal eigenvector and geometric mean have the highest correlation of $0.9863$, which is very close to one. At the same time, the correlation of rank vectors obtained by the log-Chebyshev approximation and by both methods of principal eigenvector and geometric mean increase to $0.9187\cdots 0.9313$. This shows that in terms of rank vectors, the difference between the results of all methods remains quite small, and therefore, the results of one method can be used to validate the results of the other methods.

6. Discussion

In this study, we considered a problem of the assessment of 6 criteria used by university students for hotel selection when attending a short-term educational program in a foreign country. The data available for the assessment procedure include the results of a survey of 202 respondents who report on their age, sex and whether they have previously visited the country, and evaluate the criteria by different ways. In the course of evaluation, each respondent directly estimates ratings and indicate ranks of criteria, and then compares the criteria in pairs to provide an input for subsequent derivation of ratings from the pairwise comparisons by an appropriate computational method.

To provide more accuracy of the assessment, we concurrently applied three methods of rating alternatives from pairwise comparisons, where the results of the most common approaches, the method of principal eigenvector and method of geometric mean, were considered in line with results of log-Chebyshev approximation. Moreover, the ratings derived by these methods and corresponding ranks of criteria were compared with the ratings and ranks provided by respondents directly.

First, we investigated the possible influence of parameters and properties of respondents on the degree of difference and similarity of results provided by different methods of deriving ratings. Specifically, we examined the assumption that the consistency between direct ranks and ratings given by respondents may affect the results of calculating ranks from pairwise comparisons by these methods. As our analysis has shown, for the majority of respondents, the consistency of the direct judgments, as well as parameters of respondents do not have a significant impact on the stability of the results, which made it inappropriate to divide respondents into groups for separate analysis.

Furthermore, we considered ratings provided by respondents directly or indirectly through pairwise comparisons of criteria. Calculation of the mean vectors of ratings has indicated that on average all indirect methods produce very close results. Specifically, the mean vectors of the methods of principal eigenvector and geometric mean appear to be closest to each other, whereas the mean vectors of the best and worst differentiating results of log-Chebyshev approximation form lower and upper bounds for the other two mean vectors. The mean vector of direct ratings by respondents is rather different from the mean vectors of indirect methods, which is due to a rough 5-point scale that narrows the range of assessment values used by respondents.

On average all methods equally determine the highest ratings of criteria in the order $(2,1,5)$. The other criteria are ordered by all indirect methods as $(3,4,6)$, and by the ratings directly provided by respondents as $(6,3,4)$. The standard deviation of ratings of criterion 2 has the smallest or the next smallest value among the criteria, which speaks in favor of a unanimous assessment of this criterion by all respondents. As a result, the analysis of mean vectors of ratings suggests two possible candidates for the optimal rank vector: $(2,1,5,3,4,6)$ and $(2,1,5,6,3,4)$.

To give further insight into the consistency or inconsistency of ratings derived by different methods, we estimated Pearson correlation coefficients between vectors of ratings obtained for all respondents. We have found a high correlation close to one between the results of the methods of principal eigenvector and geometric mean. The correlation between ratings from these two methods and from the log-Chebyshev approximation is somewhat lower, but still close to one. The good agreement of the ratings obtained from pairwise comparisons by all methods can be seen as a result of the natural and meaningful character of the input data that reflect mechanisms of human judgment and hence have a certain level of immanent consistency.

As the next step of the investigation, we examined the vectors of ranks provided by direct and indirect assessments. First, we checked whether the rank vectors $(2,1,5,3,4,6)$ or $(2,1,5,6,3,4)$ derived from the mean vectors of ratings actually occur in the results based on the data provided by individual respondents. We have found that these vectors of ranks are relatively rare as results of all assessment methods, and thus cannot be considered as appropriate solutions. At the same time, calculating the number of the most frequently obtained rank vectors has shown that the most occurred vector is $(2,1,5,3,6,4)$ for all methods except for the ranking from direct ratings by respondents, where this vector is the second most frequent. We note that the last vector of ranks provides the same order for three most important criteria as the two vectors considered above, and differ from them only for the last three least important criteria.

It follows from the analysis of the results obtained that for the given survey data, one can take the rank vector $(2,1,5,3,6,4)$ as the most accurate assessment of criteria for hotel selection. Assuming the respondents are representative of the total population of university students, we can conclude that the typical student ranks the criteria of hotel selection in the following order (from the most important to the least important):

• Accommodation cost, ${\mathcal{C}}_2$;
• Hotel location, ${\mathcal{C}}_1$;
• Room amenities, ${\mathcal{C}}_5$;
• Typical guests, ${\mathcal{C}}_3$;
• Courtesy of staff, ${\mathcal{C}}_6$;
• Free breakfast, ${\mathcal{C}}_4$.

As another less definite but more realistic conclusion, we can group criteria into two levels. The first level consists of the criterion ${\mathcal{C}}_2$ (accommodation cost), which is ranked first most frequently, and then criteria ${\mathcal{C}}_1$ (hotel location) and ${\mathcal{C}}_5$ (room amenities), which can sometimes change the order of each other. On the second level of lower importance are the criteria ${\mathcal{C}}_3$ (typical guests), ${\mathcal{C}}_6$ (courtesy of staff) and ${\mathcal{C}}_4$ (free breakfast), which can go in almost any order.

Finally, we estimated Kendall correlation coefficients for rank vectors derived from pairwise comparisons. As in the case of ratings, the correlations between rank vectors have been found close to one, which shows once again that for the pairwise comparison data provided by respondents, the method of principal eigenvector, the method of geometric mean and the method of log-Chebyshev approximation produce quite consistent outcome. The similarity of assessment results can serve as an additional argument in support of accepting the obtained order of criteria as the optimal solution.

7. Conclusions

The problem of evaluating preferences of criteria for choice is a key component in the multicriteria decision making procedure of assessment alternatives, and it is of independent interest in many applications including marketing research and practice. However, the existing methods to handle the problem, which are based on both direct evaluation of each criterion and indirect evaluation, where the criteria are compared in pairs, are known to give different and even opposite results. In the case when different methods may lead to inconsistent outcomes, a natural approach is to solve the problem by using several most widely used or mathematically justified methods concurrently and then combine the results to provide a more consistent and justified final solution.

In the paper, we have presented an approach that is intended to improve quality of the evaluation of criteria used by students in hotel selection when attending an educational program abroad. The approach is based on implementation of both direct and indirect methods of evaluating criteria, followed by statistical procedures to examine similarity and difference of the results. In addition, we have compared the solutions obtained by three methods of evaluating criteria from pairwise comparisons, including the methods of principal eigenvector, geometric mean and log-Chebyshev approximation.

By applying this technique, we have derived a solution that ranks the criteria of hotel selection in a way, which combines the results of different methods to provide the basis for more accurate assessment of the criteria. The comparison of the solutions produced by the indirect methods under study has shown a fairly high degree of similarity of results. We consider that the combined assessment technique described in the paper may serve as a template for a useful solution approach to handle real world problems of evaluating alternatives, where the accuracy and reliability of results is of prime importance and thus require additional formal and experimental justification.

As directions of future research, one can consider an extension of the approach to incorporate additional methods of evaluating alternatives from pairwise comparisons, including interval, linguistic and fuzzy pairwise comparisons. Further development of the technique into efficient algorithms and procedures to support decision making processes in practical problems presents another promising line of research.