A Comparison of Multi-Criteria Decision Analysis Methods for Sustainability Assessment of District Heating Systems

Abstract

The sustainability of energy systems is increasingly assessed for development of more resilient, greener district heating (DH) systems. That requires compiling technological, environmental, and economic indicators in a social, political, and institutional context. This work investigates DH system sustainability analysis by five frequently applied multi-criteria decision analysis (MCDA) methods—WSM, TOPSIS, PROMETHEE, ELECTRE and DEA. To compare the sustainability assessment results, a selection of 8 criteria describing 12 DH companies (DHC) was examined. Sensitivity analysis was performed to determine the most credible MCDA method. Criteria weights were changed: (1) individually for evaluation of the range of stability for alternatives (score of DHC performance); (2) individually by a fixed value to compare how each criterion weight change affected the average score of a result; and (3) to compare the AHP weighting method to an equal weight scenario. The results of sensitivity analysis along with literature investigation shows that all methods are suitable for sustainability analyses of DH systems while also having differences in the calculation process and in the interpretation of results. The generalized algorithm for sustainability analysis in the energy sector outlined in this study along with the documented features of the main MCDA methods can be used as a guide for future assessment of energy systems by researchers and industry professionals.

1. Introduction

Transition toward sustainable energy systems necessitate the increased proportional use of renewable resources, as well as the modernization and the development of new and existing district heating (DH) networks that are more efficient, to combat climate change. This is supported by the United Nations Sustainable Development Goals (SDGs), in particular Goal 7 on affordable and clean energy and its sub-targets, to increase the share of renewable energy, improve energy efficiency, and facilitate cooperation for research on clean energy and technologies [1]. A sustainability assessment of existing energy systems therefore provides an insight into the current state of DH in a region or sector, and it facilitates decision-making in development and in investment. To impartially perform these evaluations, multi-criteria decision analysis (MCDA) is often employed.

A family of mathematical methods, MCDA can provide decision support for complex cases with opposing objectives, different data types, and high variability [2]. The use of these tools becomes necessary in the selection of heating systems that are hard to compare based on a diverse set of sustainability indicators [3].

Most literature highlight MCDA methods without any justification for their selection. Since the goal of using MCDA is minimization of subjectivity in decision analysis and different MCDA methods can by design lead to different outcomes [4], the method choice should also be justified. No consensus on the best-performing MCDA method exists, so decision-makers often make their selection based on previous experience or recommendations.

In the field of DH, MCDA and other decision-making methods can be used to identify the best technological solution [5], the most sustainable scenario for development of a DH system [6] or DH companies (DHC) [7], the best technological solution [8], and the environmental sustainability of countries [9]. As a result, MCDA is used for decision-making at different levels of DH systems (separate area, DH company, DH system of a city or municipality, national DH policy) (Figure 1). The sustainability assessment of an energy system should refer to the interplay between energy security, energy equity, and environmental sustainability. These three pillars are the energy trilemma, and they are to some extent in competition with each other. This leads to the need for precisely stated questions and a clearly defined purpose for the application of the MCDA. Sustainable and efficient district heating is one of the components of the EU smart city concept [10]. For instance, industrial waste/excess heat integration in district heating can lead to a reduction in CO₂ emissions (decarbonization) and in production costs [11,12]. In one of the projects related to the smart cities concept, cities were also compared using criteria from disciplines such as the smart economy, mobility, the environment, people, living, and governance [13].

The transition towards fourth generation DH (4GDH) is closely related to the introduction of renewable energy sources (RES) into a DH system and considerable reduction of CO₂ emissions [14]. Similarly, fifth generation DH (5GDH) is proposed to lower network temperatures and to develop joint centralized heating and cooling while simultaneously contributing to decarbonization efforts [15]. Each development alternative requires funding and/or application of policy instruments (taxes, subsidies). Thus, MCDA is executed by comprising several criteria: technological, energy, economic, environmental, social, institutional and/or political, and governance [2]. More often, quantitative criteria are selected for the analysis, while qualitative criteria can also be used.

The structure of a DH system includes three main components: heat sources, transmission and distribution pipelines, and heat consumers. The parameters of each DH system component may be included in the initial matrix for the sustainability assessment. For example, increasing the impact of an intelligent grid control system could increase the efficiency of the DH system and reduce emissions, improving the sustainability of the DH system [16]. Therefore, the transmission system parameters can be included in the assessment. An MCDA can be applied to assess the sustainability of the whole system or depending on the selected system’s boundary (Figure 1). For instance, multi-objective optimization of a novel air-source absorption heat pump system as the heat source for a residential DH area has been conducted based on MCDA decision-making [17].

Several systematic literature reviews comprehensively explain the application of different MCDA methods for the integration of RES into the energy supply, considering technological, economic, environmental, social, and institutional criteria (see e.g., [8,11]). However, studies exploring the practical application of different MCDA methods that also perform sensitivity analysis to assess the robustness of results are rare. Implementation of different MCDA methods can often produce different results [3] and the use of multiple methods can therefore be used to gain confidence in results. Meanwhile, after studying 343 MCDA articles that address decision-making in the field of sustainable development. Kandakoglu et al. (2019) found that only 21% of articles used sensitivity analysis to assess weight value uncertainty or overall MCDA model imprecisions [18].

In this study, we aim to look at the most widespread decision-making methods for every step of general sustainability analysis and to identify the main strengths and weaknesses within each one of the methods in the DH sector.

2. Materials and Methods

An algorithm applied in this study for comparing MCDA methods is presented in Figure 2. The first part of our methodology consists of a literature review, selection of MCDA methods, and characterization of the selected case study. In the second part, criteria weighting, data normalization, and alternative ranking via MCDA methods is conducted. The third part is sensitivity analysis for determination of the best-performing and the most credible MCDA method(s) for DH-related studies.

The methods selected for comparison are the analytic hierarchy process (AHP), weighted sum method (WSM), technique for order of preference by similarity to ideal solution (TOPSIS), elimination and choice expressing reality (ELECTRE), preference ranking organization method for enrichment evaluation (PROMETHEE), and data envelopment analysis (DEA), as they are among the most widely used MCDA methods in studies related to DH and in the closely related field of renewable energy [19]. As AHP is more frequently used in criteria weighting than in alternative ranking, we look at it separately from other ranking methods. Meanwhile, TOPSIS along with DEA seem to be the most widely used MCDA methods in the published scientific articles about DH, with PROMETHEE or ELECTRE being less often applied [7,17,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56] (see Supplementary Material).

Each of the selected methods was applied to a case study of 12 Latvian district heating companies (DHC). Accordingly, a set of criteria was selected, and the corresponding weights were assigned based on the AHP method, while ranking of the DHCs was performed with WSM, TOPSIS, ELECTRE, PROMETHEE, and DEA. Sensitivity analysis was conducted to test the robustness of the results and to provide extra insight into each method’s constructed behavior. Finally, the results of an MCDA methods assessment and comparison were interpreted, and their flaws and strengths were summarized to conclude on the use of MCDA for sustainability assessments of DH systems.

2.1. Criteria Selection

Accurate criteria selection is a vital part of sustainability analysis. Criteria must be relevant to the field and encompass technical, economic, and environmental indicators to accurately portray the efficiency, cost, and sustainability of each DH system. Criteria should also be independent of each other to minimize redundancy and to simplify of the model.

The targets set by the European Union for the decarbonization of the energy system are closely linked to the DH system transition toward 4GDH and 5GDH. Society is increasingly involved in the transition process. Consumers engage in heat and power generation to become prosumers [57]. Sometimes the public is not sufficiently informed about the potential of low-emission technology; and, as a result, these non-technical factors become important drivers and constraints on the path to decarbonization [58]. For this reason, social, political, and institutional criteria should be included in DH development models alongside standard technical, economic, and environmental indicators.

The number of selected criteria is an important constituent of the MCDA. Logically, the more criteria there are, the less significant each becomes in the MCDA model. Conversely, having only a couple of criteria cannot fully encompass the characteristics of the various alternatives.

The most widespread method of criteria selection in renewable energy research is a review of the literature, where previous relevant research is assessed to collect a list of the most appropriate criteria for the occasion [19]. That can be done by authors based on their experience and research, or by a set protocol to be more methodical. No widespread methodology for criteria selection based on literature review exists and an intuitive selection of criteria can bring additional subjectivity in the process.

In the field of renewable energy, the Delphi method has been used to determine criteria for MCDA [59,60]. The Delphi method is a method for collection and interpretation of expert judgments through multiple rounds of sequential and evolving questionnaires. The traditional Delphi method consists of multiple iterations of data collection that build upon previously amassed information to achieve consensus among the interviewees [61]. The first round is equivalent to an anonymous brainstorming session. The second questionnaire asks experts to consider and rate every response that has been compiled by researchers from the first round. The third and the following rounds are usually performed until a consensus on rating/ranking is reached. This method requires substantial planning and curating from the researcher’s/consultant’s side. The lengthy multi-step consulting process may also be excessive for some applications for each DHC to quantitatively represent each criterion.

For this study, 8 criteria were selected for comparison of the selected 12 DHCs. The criteria selection was based on a literature review and consultations with a panel of experts. Some criteria were considered as minimizable and some as maximizable (see

Table 1. Decision matrix for 12 Latvian DHC *.

Criterion	Environmental		Economic	Technological				Institutional
	CO2 Emissions, kgCO2/MWh	Share of RES, %	Heat Tariff, €/MWh	Installed Capacity Utilization Rate, %	Heat Losses, %	Average Supply Water Temperature, °C	Produced Heat in Cogene-Ration, %	Heat Consumption of Buildings, kWh/m2
Criterion Type	min	max	min	max	min	min	max	min
DHC 1	223.71	0.04	52.14	21.74	17.14	120.00	22.39	189.02
DHC 2	209.32	1.44	50.53	10.23	16.46	120.00	27.84	176.28
DHC 3	181.82	0.13	58.94	15.58	10.00	90.00	21.41	168.97
DHC 4	144.33	30.83	44.39	16.66	9.55	120.00	18.65	162.34
DHC 5	84.88	61.97	54.95	15.18	15.92	90.00	46.62	136.00
DHC 6	44.41	79.38	60.70	4.85	14.40	100.00	1.36	151.40
DHC 7	55.08	80.00	55.24	23.73	8.76	63.68	0.00	156.85
DHC 8	45.86	80.04	51.73	28.27	16.83	75.95	97.37	143.79
DHC 9	49.25	87.10	54.90	19.91	9.20	90.00	5.70	120.18
DHC 10	3.13	99.00	49.80	30.48	18.54	100.00	49.09	129.03
DHC 11	0.00	100.00	61.55	93.88	13.09	90.00	0.00	162.47
DHC 12	0.00	100.00	53.49	92.59	24.78	90.00	100.00	168.87
Criterion weight, %	14.54	11.59	17.06	14.12	16.90	7.18	6.00	12.60

* Green value—alternative with the highest value for the given criterion; red value—alternative with the lowest value for the given criterion.

). The type of a criterion (min or max) depended on the assessment’s target [62].

The goal of applying MCDA was to determine the sustainability rating of the 12 Latvian DHC based on the concept of the 4GDH by considering technological, economic, environmental, and institutional criteria. The study includes two environmental criteria (CO₂ emissions and the share of RES), one economic criterion (heat tariff that characterizes the economic efficiency of DHS operation), four technological criteria (installed capacity utilization rate, heat losses in DH transmission system, average supply water temperature, produced heat in cogeneration), and one institutional criterion (heat consumption of buildings). The technological criteria partially overlap with or relate to economic and environmental performance of a DH company. Low-temperature DH is one of the 4GDH components as lower heat carrier supply temperature reduces heat losses in networks, which in turn impacts fuel consumption and, as a result, the CO₂ emissions of the DH system. The installed capacity utilization rate demonstrates how efficiently the installed capacity is used and it ultimately determines the cost-effectiveness of the DH system. The heat tariff expresses, how efficient and affordable the DH system is for heat consumers. Meanwhile, the institutional criterion—heat consumption of buildings—is one of the most important indicators, defined in Directive 2012/27/EU on energy efficiency [63]. Lower heat consumption should ensure decarbonization of the building stock by 2050. The social and political aspects of the model overlap with the economic and the institutional criteria, and they are therefore not partitioned.

2.2. Criteria Weighting

Criteria weighting is an integral part of any MCDA method. During the weighting procedure, values are assigned to the selected criteria to represent their relative importance. The simplest weighting method requires no additional input from the decision-makers as every criterion is assigned the same weight. That can be done in cases where all criteria are assumed to have very similar importance or when there is no input from decision-makers and not enough information available [2].

In case of differing relative importance of each criterion, rank-order weighting methods are used. The entropy weight method is an objective rank-order weighting technique that draws from information theory to calculate each criterions’ entropy—the uncertainty of it with respect to input data. That has the advantage of incorporating extra information in the model, as the stochasticity of the input data is a source of uncertainty in MCDA models [64].

An objective rank-order weighting can also be done by assigning weights based on statistical data in a way that adds information to the model without the subjectivity of experts [62].

The AHP method uses pair-wise comparisons to determine the importance of each criterion, i.e., its relative weight. The comparisons are quantified and formed into a matrix from which the criteria weights are calculated. Traditionally, a numerical scale ranging from 1 to 9 is used to appraise the comparison [65]. Experts’ comparisons are collected via questionnaires and then aggregated to come to a set of criteria weights. A consistency ratio is also calculated to assess the credibility of the results.

Additionally, the AHP procedure was applied to the selected case study of 12 DHCs. The weights were based on a survey of DH system’s stakeholders, and they reflect the stakeholders’ opinion about the importance of the criteria toward sustainability of the DH system. The obtained weights are presented in Table 1.

Pairwise comparisons of the 8 selected criteria were quantified via the 1–9 Saaty scale. The procedure along with the resulting criteria ranking and weighting was performed on a free-access software (AHP OS) [66].

The criterion weights must be normalized before application by Equation (1):

$${{\displaystyle \sum }}_{j=1}^n{\omega }_j=1,$$

(1)

where w_j—weight of criterion j; j = 1, 2, …, n. Thus, the sum of all weights must be equal to 1 or to 100%.

2.3. Data Normalization

Data can come in many forms. Normalization of data is a prerequisite for some MCDA since it allows for comparison outside the bounds of each criterion. Moreover, data normalization allows for comparison of criteria with various units, as well as for using both qualitative and quantitative criteria [67].

Several normalization techniques exist, including linear (‘Max’, ‘Max-Min’, and ‘Sum’) normalization, vector normalization, logarithmic normalization, and fuzzification normalization [68]. The choice of normalization method can impact the final ranking of alternatives; therefore, it needs to be selected carefully in parallel to the choice of weighting method and ranking MCDA method [69].

Several studies have proven that vector normalization is the most suitable technique for the TOPSIS [68,70] and the WSM [71] methods. Meanwhile, the linear ‘Max’ method is shown to be effective (second-best option) in the TOPSIS method when the number of criteria is below 5, while the linear ‘Max-Min’ normalization technique is shown to be more effective when the number of criteria is larger (Equations (3) and (4)) [72].

In this study, we have used the linear ‘Max’ normalization technique for all compared MCDA methods. In the linear ‘Max’ technique, the normalized minimizable and the normalized maximizable values of criteria are obtained by Equations (2) and (3), respectively.

$$r_{ij}=\frac{max{x}_{ij}-x_{ij}}{max{x}_{ij}-min{x}_{ij}},$$

(2)

$$r_{ij}=\frac{x_{ij}-min{x}_{ij}}{max{x}_{ij}-min{x}_{ij}},$$

(3)

where r_ij—the normalized value of criterion x_ij; maxx_ij—maximal criterion value; minx_ij—minimal criterion value; x_ij—criterion value; i—number of alternatives; j—number of criteria.

2.4. Ranking of Alternatives

Ranking of alternatives is a critical stage of the MCDA approach. The selected criteria, their weights, and data corresponding to the alternatives are used as inputs to the MCDA algorithm. The ranking from best to worst performing alternative is the desired output of these algorithms, as it facilitates comparison of alternatives with respect to the selected goal.

A simplified course of action for each MCDA method applied in this study is presented in Figure 3. The first step of any MCDA method is to define the goal of assessment, i.e., with what aim is the MCDA applied. Examples of goal definition include finding the most efficient pathway for reaching a sustainable DH system [30], identifying the most efficient catalyst for a chemical conversion process [73], comparing RES for electricity production [74], and identifying the ecological vulnerability of regions [75]. After the goal is defined, a set of alternatives is selected for comparison. The first two steps can also be done in a reverse order, e.g., a set of alternatives is given and only then is the goal of the assessment defined. The third step is criteria selection (described in Section 2.1), followed by data collection representing each alternative according to the selected criteria. Further, the criteria are weighted (see Section 2.2) and criterion values—normalized (Section 2.3) by building the normalized decision matrix. Finally, all alternatives are assessed by applying specific calculation steps according to the selected MCDA methodology and they are ranked with respect to the defined goal of the assessment. The following sub-sections shortly describe each of the MCDA methods selected for this study.

2.4.1. WSM

Also known as Simple Additive Weighting (SAW), WSM is a simple MCDA method that can evaluate alternatives with a minimal number of calculation steps. The normalized matrix values are weighted and summed to get a score ($S_i$) for each alternative (Equation (4)).

$$S_i={{\displaystyle \sum }}_{j=1}^n{w}_j{r}_{ij},$$

(4)

where r_ij—the normalized value of criterion x_ij; w_j—weight of criterion j; i—number of alternatives; j—number of criteria.

When decision-makers need an easy-to-understand MCDA method, WSM is often used. Since WSM can give similar results to more complex MCDA methods [4], the simplicity of it does not compromise the results of the model. It has been applied for sustainability assessment [76].

One of WSM’s requirements is that all selected criteria values are maximizable and positive [77]. This requirement can be addressed by carefully selecting and defining the criteria or by transforming the minimizing criteria to maximizing, which can be done in the normalization step before MCDA.

2.4.2. TOPSIS

The TOPSIS technique sets out to find the alternative closest to the positive ideal solution and furthest from the negative ideal solution [62]. The alternative that has the highest closeness value (that incorporates distances from ideal positive and negative solutions) is ranked the highest of all alternatives. The criteria values are assumed to be rising and falling monotonically, therefore a normalization must be done that adjusts minimizable criteria values by reciprocalization or another method.

The multiplication of each normalized decision matrix’s element r_ij with the assigned weight coefficient w_j results in weighted normalized decision matrix, where v_ij represents the weighted normalized value calculated as:

$$v_{ij}=w_j{r}_{ij},$$

(5)

Next, the positive ideal and the negative ideal solutions for each alternative are determined by finding the alternative’s distance from the matching positive ($S_i^{+}$) and negative ($S_i^{-}$) ideal solution:

$$S_i^{+}={[{{\displaystyle \sum }}_{j=1}^n(v_{\mathit{ij}}-v_j^{+})]}^{1/2},$$

(6)

$$S_i^{-}={[{{\displaystyle \sum }}_{j=1}^n(v_{\mathit{ij}}-v_j^{-})]}^{1/2},$$

(7)

where $v_{ij}$—weighted normalized value of alternative i with respect to criterion j; $v_j^{+}$—maximal normalized value with respect to criterion j; $v_j^{-}$—minimal normalized value with respect to criterion j.

Each alternative’s closeness to the ideal solution ($C_i^{*}$) is then determined (Equation (8)) serving as the value for the ranking of alternatives.

$$C_i^{*}=\frac{S_i^{+}}{S_i^{+}+S_i^{-}},$$

(8)

2.4.3. ELECTRE

The ELECTRE family of outranking MCDA methods use pairwise criteria comparison. Every alternative is set against all others by comparing criterion values. An outranking relation is then determined by these comparisons. Most criteria must be in favor of the outranking in an ELECTRE model. Similarly, the rest of the criteria cannot strongly oppose the ranking. These validations by concordance and non-discordance allow for calculation of the net superior and the net inferior values for alternatives, respectively.

A normalized decision matrix is weighted as in the TOPSIS method. The next steps follow a modified ELECTRE-I methodology (see Appendix A) by a paper performing an illustrative case study [78]. This method provides a net superior and a net inferior value for each alternative. That can be used to create two rankings, but to get one final ranking the two values are averaged (Equation (9)) as described in a paper by Chatterjee et al. [79].

$$x_j=\left(c_j+d_j\right)/2,$$

(9)

where $x_j$—final net value of alternative j; $c_j$—net superior value of alternative j; $d_j$—net inferior value of alternative j.

The final ranking value is called the Net value in ELECTRE.

2.4.4. PROMETHEE

PROMETHEE is a MCDA technique that also uses pairwise comparisons to rank the provided alternatives. Yet, unlike ELECTRE, PROMETHEE includes preference functions to measure exact differences between two alternatives regarding the criterion. Multiple preference functions can be used depending on the criteria properties. The result is one last outranking flow for each alternative based on which they are ranked.

A procedure of PROMETHEE-II described by Behzadian et al. [80] is followed to obtain the final outranking flows for every alternative to be used to construct the ranking (see Appendix B). In PROMETHEE, the final ranking value is called the phi value.

2.4.5. DEA

Data Envelopment Analysis (DEA) is an MCDA method that benchmarks alternatives to calculate their efficiency. It does so by constructing an efficiency frontier on top of the alternatives’ criterion value data set. The method does not require data normalization since function inputs (minimizable criteria values) are evaluated differently from function outputs (maximizable criteria values) within the model. Also, weights are calculated by the model and directly included therefore alleviating decision-makers from weight determination as a separate step. In this study, a free software EMS (Efficient Measurement System) for solving DEA problems was used [81]. In DEA, alternatives are ranked according to their efficiency score.

2.5. Sensitivity Analysis

Sensitivity analysis was conducted to interpret the importance of each criterion and its respective weight to the MCDA model rankings in WSM, TOPSIS, ELECTRE and PROMETHEE. DEA uses internalized weight calculations hence it was left out of this analysis. In the models, the range of stability of each criterion was calculated by applying three different weight change methods—individual weight change, fixed weight change and equal weight method. The total amount of step-by-step weight change calculations for the three models was 702. Without some automation these calculations would be way too impractical as the calculations described in the following sub-sections would have to be repeated hundreds of times in a row.

2.5.1. Individual Weight Change

To determine the range of weight inputs that keep the MCDA model unchanged, an approach presented by Li et al. was used in this study [82].

Each individual weight disturbance is defined as follows:

$$w_j^{*}=w_j{\gamma }_j,$$

(10)

where $w_j$ is the initial weight, $w_j^{*}$ is the new weight, and ${\gamma }_j$ is the initial variation ratio. The rest of the weight values therefore must be changed so their sum stays equal to 1. The resulting weights after the change are calculated:

$${w_j}^{\prime }=w_j{\beta }_j,$$

(11)

$${w_n}^{\prime }=\frac{w_n}{w_1+w_2+\cdots w_j^{*}+\cdots w_n},$$

(12)

where ${w_j}^{\prime }$ is the target weight and ${w_n}^{\prime }$ describes all other weights, ${\beta }_j$ is the ratio that marks the final relation between our target weight before and after disturbance, and n denotes the number of criteria.

The following relationship between the initial variation ratio ${\gamma }_j$ and the final ratio ${\beta }_j$ is considered:

$${\gamma }_j=\frac{{\beta }_j-w_j{\beta }_j}{1-w_j{\beta }_j},$$

(13)

To check the stability of the MCDA models, each weight was increased and lowered by 1% at a time (${\beta }_j$ = …, 0.98, 0.99, 1, 1.01, 1.02, …) until the ranking of an alternative changed (see Figure 4). The range of ${\beta }_j$ values that produced a stable ranking was noted for every weight, and it was expressed as one value—the range of stability (${\beta }_{jmax}-{\beta }_{jmin}$).

2.5.2. Fixed Weight Change

The analysis of method-specific scores instead of ranking changes can explain the correlation between weight changes and the behavior of models more precisely. A weight change is expected to alter the alternative’s scores uniformly, since the registered changes can be incremental. A ranking change, on the other hand, is not quantifiable in such a particular way.

Based on the method described in Section 2.5.1, a fixed weight change was performed on the MCDA models. Weights were increased or lowered by 10, 20 and 30% (${\beta }_j$ = 0.9, 1.1; 0.8, 1.2; 0.7, 1.3) and alternatives’ scores were registered (see Figure 5). These scores were the WSM score, closeness to the ideal solution value in the TOPSIS, net value in the ELECTRE, and phi value in the PROMETHEE model. The range of score change was registered for every alternative. The average range of score was calculated next for every weight respectively:

$$AV{R}_{c_j^{*}}=\left({{\displaystyle \sum }}_{n=1}^{k}C{d}_{jn}^{*}\right)/8,$$

(14)

where $AV{R}_{c_1^{*}}$ is the average criteria change for criteria 1 in the TOPSIS model, j is the respective criteria, and n denotes the alternatives.

These average values should positively correlate with the weight size—the bigger the weight, the more MCDA model changes.

2.5.3. Equal Weight Method

To justify weight determination via AHP (or another complex method), equal weight models were constructed. The differences between AHP and equal weight alternative rankings were assessed to derive which MCDA method made greater use of extra input information and employed it in the ranking process.

3. Results and Discussion

3.1. The Ranking Results

The results of the calculated final rankings of the selected alternatives—12 DHCs—are presented in

Table 2. The list of rankings obtained from each MCDA method.

WSM	DHC 10 > DHC 9 > DHC 8 > DHC 12 > DHC 11 > DHC 7	>DHC 5 > DHC 4 > DHC 6 > DHC 3 > DHC 2 > DHC 1
TOPSIS	DHC 10 > DHC 9 > DHC 8 > DHC 7 > DHC 11 > DHC 12	>DHC 4 > DHC 5 > DHC 6 > DHC 3 > DHC 2 > DHC 1
ELECTRE	DHC 9 > DHC 10 > DHC 8 > DHC 11 > DHC 7 > DHC 12	>DHC 4 > DHC 5 > DHC 6 > DHC 3 > DHC 2 > DHC 1
PROMETHEE	DHC 10 > DHC 9 > DHC 8 > DHC 12 > DHC 11 > DHC 7	>DHC 5 > DHC 4 > DHC 6 > DHC 3 > DHC 2 > DHC 1
DEA	DHC 12 > DHC 11 > DHC 8 > DHC 10 > DHC 9 > DHC 7	>DHC 6 > DHC 5 > DHC 4 > DHC 3 > DHC 1 > DHC 2

Green background—top six performing DHCs, red background—bottom six performing DHCs.

. According to the results of the WSM, TOPSIS, PROMETHEE and ELECTRE models, the two most sustainably operated companies are DHC 10 and DHC 9. Although DHC 10 does not have the highest value by any of the criteria (see Table 1), it has the second highest value with respect to four criteria (CO₂ emissions, share of RES, heat tariff, and heat consumption of buildings), the total weight of which is 55.79%, and it is presented as most sustainable by the WSM, TOPSIS, and PROMETHEE methods. The results of the ELECTRE method show DHC 9 as a leader, for which one criterion (heat consumption of buildings) has the best value, while the others are average. The DEA model highlights DHC 12 and DHC 11 as the two most sustainably operated alternatives. In WSM and PROMETHEE, the alternatives are listed in the same preference order, while other models provide slightly different orders.

In an outranking model (ELECTRE, PROMETHEE), a good value in one criterion can compensate for a bad value in another criteria. Any alternative with one or multiple values that are the worst in the group will most certainly not obtain the highest ranking in these models. Although DHC 11 and DHC 12 have at least one criterion with the lowest value among all alternatives, they are not at the bottom of the ranking because they simultaneously have three maximum values in other criteria. In the DEA model, the efficiency frontier of DHC 11 and DHC 12 is perhaps pushed further than by any other alternatives, and one minimum score does not diminish it enough.

One thing that is common for all the methods is that a clear division exists between the top six alternatives (DHC 7 to DHC 12) and the bottom six alternatives (DHC 1 to DHC 6). In the DEA model, the efficiency values of these top six alternatives exceed 100% and therefore those alternatives can be branded as efficient. The bottom six score below 100%, thus they are not considered efficient.

Table 3. The ranking values in each MCDA model *.

MCDA Method	WSM		TOPSIS		ELECTRE		PROMETHEE		DEA
MCDA Result	Score	%	Closeness Value	%	Net Value	%	Phi Value	%	Efficiency	%
DHC 1	0.21	0	0.30	0	−11.78	0	−0.31	0	33	1
DHC 2	0.26	10	0.34	12	−8.73	15	−0.26	10	31	0
DHC 3	0.31	23	0.38	26	−8.02	19	−0.20	23	48	4
DHC 4	0.50	66	0.54	78	2.22	70	0.00	66	53	5
DHC 5	0.50	66	0.49	62	−2.65	46	0.01	66	71	9
DHC 6	0.42	48	0.45	47	−7.11	23	−0.08	48	77	11
DHC 7	0.59	88	0.58	88	4.97	84	0.11	88	116	20
DHC 8	0.62	95	0.58	91	5.75	88	0.14	95	143	26
DHC 9	0.64	98	0.61	99	8.10	100	0.15	98	124	22
DHC 10	0.65	100	0.61	100	5.92	89	0.17	100	126	22
DHC 11	0.61	92	0.57	86	5.48	87	0.13	92	192	38
DHC 12	0.62	93	0.55	79	2.66	73	0.13	93	460	100

* Green—highest value among all alternatives; red—the lowest value among all alternatives; blue—the lowest value among the six best alternatives; green background—top six performing DHCs, red background—bottom six performing DHCs.

shows the ranking values of all alternatives obtained in the MCDA models. To distinguish among the rankings obtained in each method, the specific final ranking scores are calculated showing the distance to the score of the best alternative by percentage.

Models that display a higher distance between the specific final scores of top alternatives provide a more stable ranking. The two leaders determined by DEA have a stark difference in their final specific scores (100% and 38%). The leaders of ELECTRE have a specific final score difference of 12%. The leading alternative scores of other methods are much more similar. A small change in input data or criteria weights is likely to change the currant ranking of these methods, while ELECTRE and DEA scores are less likely to be disturbed.

The specific final ranking values of alternatives are the same for the WSM and the PROMETHEE methods (see Table 3), explained by the aggregated preference function selected for the PROMETHEE method. It mirrors the weighting and the summation steps in the WSM method. The other transformations in the PROMETHEE method contribute to the calculation of the phi values of each alternative that differ from the WSM scores yet they are still the same after normalization.

3.2. Results of Sensitivity Analysis

3.2.1. Individual Weight Change

The individual weight change sensitivity analysis was performed by changing the weight of a single criterion until a change in the ranking of alternatives was observed. The weight range at which the ranking remained unchanged produced the range of stability for each criterion. The higher the range of stability, the more the weight of a criterion must be changed to alter the ranking of alternatives. The results of the sensitivity analysis are presented in Figure 6a–d.

Dividing the range of stability between the highest and the lowest values into two equal parts, the relatively stable and unstable areas can be identified. The criteria with higher weights are in the unstable area. It can be seen in Figure 6a,c,d, that in the WSM, ELECTRE and PROMETHEE methods respectively, six out of eight criteria are relatively unstable, including the criterion with the smallest initial weight. When using these methods for sustainability assessment, it should be considered that even a criterion with the lowest initial weight may affect the ranking results. Still, in our case study, the criterion with the smallest initial weight did not change the leading alternative of the ranking (DHC 10 in WSM, TOPSIS, PROMETHEE, and DHC 9 in ELECTRE). In addition, the obtained results show that in the TOPSIS model, the range of stability has a clear negative correlation to the weight value—the criteria with larger weights have to be disturbed the least to change the ranking as they have the most importance or the highest impact in the model (Figure 6b). Similarly, it is expected that smaller weights have less impact on a model’s stability. Yet, the results show that the outranking models are much less stable than expected with respect to criterion with the smallest weight (produced heat energy in cogeneration).

A more detailed analysis of the impact of changes in each criterion on the sustainability performance of DHC should be performed. The results of such an analysis could be included in the development strategy for each DHC; MCDA is an appropriate tool for such a task, but it is not the objective of this article.

3.2.2. Fixed Weight Change

The results of the fixed weight change (+/−10, 20, 30%) with respect to the initial weight of criteria are presented in Figure 7. The values on the y axis are averaged from the 12 alternatives responding to the weight change of one criterion.

All models exhibit similar correlation between the weight values and the ranking score values. The correlation between the fixed weight change in the range between +/−10% and +/−30% and the initial weight in the applied MCDA models is nonlinear. The change in the three criteria with the smallest initial weight show a relatively flat line of correlation indicating their flexibility against such weight changes. In contrast, the alternatives’ values increase sharply when changes are made to the criteria with the initial criterion weight above 0.125. When the initial weights are increased, the alternatives’ value changes more significantly in the TOPSIS method than in other methods. For example, a weight change of +/−30% of the most significant criterion, results in an alternatives’ value that is 6.1 times higher than the alternatives’ value when the least significant criterion is changed by +/−30%. Meanwhile, for the ELECTRE method this ratio is 3.7, and for the WSM and the PROMETHEE methods the ratio is 2.4. Therefore, it is concluded that the TOPSIS method is the most sensitive to changes in weight, while other methods are more flexible to weight change.

3.2.3. Equal Weight Method

Each of the eight criteria was assigned a weight of 12.5% (100% divided by 8 criteria). The obtained ranking lists were compared to the ranking lists obtained by using the AHP weights. The results of the comparison are presented in

Table 4. The results of the alternatives’ ranking lists when using equal weight method and AHP method *.

WSM		TOPSIS		ELECTRE		PROMETHEE
Equal Weights	AHP Weights	Equal Weights	AHP Weights	Equal Weights	AHP Weights	Equal Weights	AHP Weights
DHC 12	DHC 12	DHC 12	DHC 12	DHC 12	DHC 12	DHC 12	DHC 12
DHC 11	DHC 11	DHC 11	DHC 11	DHC 11	DHC 11	DHC 11	DHC 11
DHC 10	DHC 10	DHC 10	DHC 10	DHC 10	DHC 10	DHC 10	DHC 10
DHC 8	DHC 8	DHC 8	DHC 7	DHC 7	DHC 7	DHC 8	DHC 8
DHC 7	DHC 7	DHC 7	DHC 8	DHC 8	DHC 8	DHC 7	DHC 7
DHC 9	DHC 9	DHC 9	DHC 9	DHC 9	DHC 9	DHC 9	DHC 9
DHC 6	DHC 6	DHC 5	DHC 4	DHC 6	DHC 5	DHC 6	DHC 6
DHC 1	DHC 3	DHC 1	DHC 3	DHC 1	DHC 3	DHC 1	DHC 3
DHC 4	DHC 2	DHC 4	DHC 2	DHC 4	DHC 1	DHC 4	DHC 2
DHC 3	DHC 1	DHC 3	DHC 1	DHC 2	DHC 2	DHC 3	DHC 1
DHC 5	DHC 5	DHC 6	DHC 5	DHC 5	DHC 4	DHC 5	DHC 5
DHC 2	DHC 4	DHC 2	DHC 6	DHC 3	DHC 6	DHC 2	DHC 4

* Red—the differences in ranking.

.

The results show that in the WSM, ELECTRE, and PROMETHEE methods, assigning equal weight to all assessment criteria does not change the ranking of the top six alternatives. Meanwhile, the TOPSIS results are more affected and only the ranking of the top three and the sixth alternative remain unchanged.

3.3. Results of MCDA Methods’ Comparative Assessment

All methods used in this study were assessed and compared for their simplicity of use, result interpretation, result robustness and other properties. The results of the comparative assessment are summarized in

Table 5. Summary of each methods’ properties.

	WSM	TOPSIS	ELECTRE	PROMETHEE	DEA
Simplicity of calculations
Weighting	Required	Required	Required	Required	Not required
Normalization	Required	Required	Required	Required	Not required
Number of steps	3	4	11	8	Not assessed
Ease of automation for sensitivity analysis	Easy	Easy	High difficulty	Medium difficulty	High difficulty
Overall simplicity	Very easy	Easy	High difficulty	Medium difficulty	High difficulty
Results
Result interpretation difficulty	High difficulty	High difficulty	Easy (negative and positive values)	Easy (negative and positive values)	High difficulty
Robustness
Can ranks reverse if an alternative is deleted?	No, by design	No, by design	Yes, by design	Yes, by design	Yes, by design
Relationship between low and high criteria values
Does a low criteria value get compensated by a high criteria value?	Yes, by increasing the alternatives’ final value	Yes, by increasing the alternatives’ final value	Yes, by outranking principles	Yes, by outranking principles	Yes, by increasing the alternatives’ final value
Can an alternative with one lowest criterion value be the leader?	Yes	Yes	No	No	Yes
Availability of free and documented software	Not needed	Yes (e.g., DecernsMCDA [83])	Yes (e.g., Decision Deck [84])	Yes (e.g., Visual PROMETHEE [85])	Yes (e.g., EMS [81])
Popularity
In the field of DH (number of papers)	Not widely used (2)	Most popular (12)	Least popular (1)	Not widely used (3)	Fairly popular (7)
Additional properties		Rank reversals when the number of criteria is low [86]	Rank reversals when the number of criteria is high [86]; arbitrary definition of threshold values	Different preference functions can lead to different outcomes; arbitrary definition of threshold values	Does not fare well with imprecise data

.

Of the MCDA methods assessed, all but DEA require a weighted normalized decision matrix for the ranking of alternatives. Those methods therefore necessitate an additional step—weighting of criteria. Even though mistakes in criteria weighting are unavoidable, the determination of the relative importance of the criteria is a worthwhile step [2] that is well documented in the MCDA literature. The normalization technique should be considered alongside the MCDA method and other parameters as the number of criteria [69]. Both add to the number of steps a decision-maker must take while performing MCDA, so the appeal of methods like DEA that forgo these steps is logical.

The main flaws of any MCDA method lay in the normalization procedure and in the use of additive formulas that disrupt the initial associations between the alternatives (and alter the rankings) [87]. Both WSM and TOPSIS have the simplest calculations and the fewest steps, while PROMETHEE and ELECTRE are conversely tougher to perform. That is because of the differing number of steps involved and the unique challenges in the automation process for sensitivity analysis without a dedicated software. Altogether WSM seems to be the simplest MCDA method to work with as per its least number of calculation steps. Although it does not need prior weighting and normalization, DEA is the most difficult to calculate (DEA calculations were not manually performed in this study). It also requires input data values to be precise, as ordinal types of data are not sufficient for traditional DEA.

Even though WSM and TOPSIS are the simplest MCDA methods, the results can be difficult to interpret. The ELECTRE and the PROMETHEE methods result in positive and negative net sum and phi values that automatically divide the alternatives into best and worst. The WSM scores and the TOPSIS C* values are all positive. Also, efficiencies calculated by DEA have no negative values, but they are divided into two groups—efficient (above 100%) and non-efficient (bellow 100%).

The robustness of an MCDA method can be affected when new alternatives are added or existing—omitted from the model. By design, the outranking methods (ELECTRE and PROMETHEE) can provide different rankings if an alternative is taken out, since their values are calculated with respect to other alternatives [88]. Alternatives in DEA are evaluated based on their closeness to the efficiency frontier which is comprised of input data. An added alternative could therefore change the efficiency scores of others. This property of ELECTRE, PROMETHEE, and DEA methods should be considered by decision-makers. The WSM scores and the TOPSIS C* values are calculated individually and therefore cannot change in this way.

All MCDA methods compensate low criterion values with high values in some way. Yet, the outranking methods do not tolerate extremely low values by design. In ELECTRE and PROMETHEE models, an alternative with the lowest criterion value cannot be the leader—the non-discordance principle.

The most popular MCDA methods have free software available (all but WSM assessed in this study). In the context of DH, TOPSIS and DEA seem to be the most popular with 12 and 7 papers accordingly in the Scopus database [7,17,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56] (see Supplementary Material). It has been recorded that the TOPSIS method experiences fewer rank reversals when the number of criteria is lower. The opposite was found to be true for ELECTRE [86].

The outranking methods have arbitrary definitions for key concepts like the c or the d critical values in the PROMETHEE model. Furthermore, the aggregated preference functions should be chosen accordingly, when dealing with a decision-making problem.

4. Conclusions

In this study, five MCDA methods were selected and assessed for their application in a sustainability assessment of DHC. The robustness of results obtained in four of the methods was assessed by applying sensitivity analysis of criteria weights. Three different approaches were used for the sensitivity analysis. The results of all three approaches distinguish TOPSIS from the other MCDA methods used. It is concluded that the TOPSIS method is the most sensitive to changes in criteria weights, i.e., the more a criterion is changed the higher the alteration of the ranking results. Moreover, TOPSIS displays a clear negative correlation between the criteria weights and the weight range at which the alternatives’ ranking remains unchanged. Thus, a criterion with larger weight has higher impact to the ranking change compered to criteria with smaller weight. It is also concluded that a careful weight assessment, e.g., by applying the AHP method, is needed when using TOPSIS.

Meanwhile, other MCDA methods, namely WSM, ELECTRE, and PROMETHEE are more flexible to criteria weight change. The results of the equal weight method show that the ranking of the top six alternatives is not affected when equal (12.5%) or differentiated (AHP) weights are used. Hence, at least at the initial phase of a WSM, ELECTRE, or PROMETHEE study, the same (equal) weight can be assigned to all criteria, leaving the robustness tests of results for the end phase of the study.

A comparative assessment of the MCDA methods indicates that in many cases, TOPSIS could be preferred due to its simplicity and clear interpretation of the results. Yet, in balancing the simplicity of a method with the accuracy of the results, the proper weighing method should be selected. The AHP method is a relatively simple and sufficiently accurate method to consolidate various views of involved experts.

Criteria weights directly influence the obtained results of DHC performance evaluation. The AHP method for weighing is preferable because of its comprehensibility and its simplicity in application. Additionally, attention should be paid to both the choice and the number of criteria, depending on the aim of the application of an MCDA method.

The ELECTRE and the PROMETHEE methods allow for a very detailed assessment, however the computation process by these methods is quite complex, therefore the methods are more suitable for MCDA experts. A case-specific choice of an aggregated preference function in a PROMETHEE procedure could refine the results by tailoring the procedure to the input data and the properties of the criteria. The c and the d critical values of the ELECTRE method should also be looked at critically; simpler methods without threshold values from the ELECTRE family exist, and they should be assessed. Moreover, since the method technically produces two distinct rankings for alternatives, a further study of the accuracy of these results should be done.

Overall, depending on the aim and the level of detail of the assessment, any of the methods tested in this study can be used for a sustainability assessment of DHC, DH systems, or energy systems at large. Equally, the MCDA approach can be used for a sustainability assessment of DHC’s transition toward 4GDH and implementation of a smart thermal grid concept or the 5GDH considering the point of view of different stakeholders.