Comparing Multi-Criteria Decision Making Models for Evaluating Environmental Education Programs

Abstract

Educators in the field of Environmental Education often have difficulty identifying and selecting programs that have the potential to best maximize needed resources to implement and achieve desired outcomes. This difficulty is, in part, due to their lack of expertise in evaluation knowledge and practice. The use of multi-criteria decision-making models in evaluating environmental education programs is new and, as a result, not many models have been used and tested in the specific domain. Comparisons of multi-criteria decision-making models have been implemented in various domains but not for environmental education programs’ evaluation. Therefore, we investigate the comparative performance of the SAW, WPM, TOPSIS, and PROMETHEE II models in evaluating and selecting the most appropriate environmental education program. The main objective of this paper is on presenting the different steps of the comparative analysis of multi-criteria decision-making models and on making conclusions on the suitability and robustness of the SAW, WPM, TOPSIS, and PROMETHEE II models in evaluating environmental education programs.

1. Introduction

The evaluation of the EE programs before their implementation can save time and effort [1]. The importance, as well as the difficulty of evaluations of environmental education (EE) programs, have been highlighted by many researchers [2,3,4,5,6,7,8,9,10,11,12,13,14]. Due to this difficulty, educators omit this stage and end up in many cases at implementing EE programs that do not fulfill their goals, objectives, and/or requirements.

An automated evaluation of EE programs would be very useful for many researchers and educators. Prior automated evaluations of EE programs such as the one proposed by Zint [12] mainly assisted the evaluation of one EE program after its implementation and not before. In Kabassi et al. [15] an automated system is designed to evaluate EE programs based on a multi-criteria decision making (MCDM) approach. In that approach, a set of criteria has been formed and the combination of Analytical Hierarchy Process (AHP) [16] with the Technique for the Order of Preference by Similarity to an Ideal Solution (TOPSIS) [17] has been applied to evaluate EE programs prior to their implementation, comparing them and selecting the one that seems more appropriate.

Generally, there are many MCDM methods available. However, finding the best MCDM model to apply is not easy. In MCDM, no single method has been considered as the most suitable for all types of decision-making situations [18,19,20]. Different methods may lead to different rankings of the evaluated objects [18,21]. A solution to the problem may be given by comparing the MCDM models [22]. Different comparative analyses of MCDM methods have been implemented [18,22,23,24,25,26] but none of these analyses concern evaluation of EE programs.

In light of the importance of usage of MCDM in evaluating EE programs and the need for comparison of MCDM approaches for each different domain, we perform a comparative analysis of four MCDM models for evaluating EE programs. For this purpose, we have used AHP for estimating the weights of the criteria as this theory has a well-defined way of forming the set of criteria and estimating the weights of the criteria based on pair-wise comparisons. Then, we apply four different MCDM models for processing the results of the evaluation. For this purpose, we apply SAW (Simple Additive Weighting) [17,27], WPM (Weighted Product Model) [28], TOPSIS (Technique for Order of Preference by Similarity to Ideal Solution) [17], and PROMETHEE II model (Preference Ranking Organization METHod for Enrichment Evaluations II) [29,30]. The main reason for the selection of those four models is that they are among the most popular and widely used in the literature in general.

There is no formal way of comparing MCDM models as has been proved by the different comparison experiments [25,26,31,32,33,34,35,36,37,38,39,40]. Indeed, Salabun et al. [41] concluded in their paper that almost every combination of the method and its parameters may bring us a different result. Therefore, the comparison of MCDM models in the context of EE projects may be of particular interest. Furthermore, TOPSIS, SAW, WPM, and PROMETHEE II have not been compared before for the EE programs’ evaluation. For the comparison of the MCDM theories, we use the data from the EE programs in Greece. More specifically, we have used the data of the 52 EE programs for paths.

After the comparison of the MCDM models, we performed a sensitivity analysis. Sensitivity analysis examines the degree of change in the overall ranking of the alternatives when input data are slightly modified. More specifically, the scheme of weights of criteria changes to examine the robustness of the models and the alternatives. Although sensitivity analysis of the models has been implemented before in different domains and for different MCDM models [40,42,43,44,45], it is the first time that it is implemented for estimating the consistency of the MCDM models in the domain of EE programs.

The rest of the paper is organized as follows: Section 2 presents comparisons of the MCDM models described in the particular paper in other domains. Section 3 describes in detail the steps for the application of AHP for the criteria and their weights. Then, in Section 4, the four MCDM models that are compared are described. In Section 5 and Section 6, those models are applied for the evaluation of EE programs and the results are analyzed and compared. In Section 7, a sensitivity analysis is performed to evaluate the robustness of the four different MCDM models and the conclusions drawn by this study are discussed in Section 8.

2. Bibliographic Review

Quality environmental education involves many partners and stakeholders who collaborate in a research-implementation space where science, decision making, and local culture and environment intersect [46]; environmental education evaluation and assessment often struggle in these productive, yet complex, spaces [3]. Regular program evaluation is needed to better understand the linkages between the issues programs are designed to address, the metrics or indicators used to describe effectiveness, and the actual measured outcomes [6].

Many researchers have referred to the need for evaluation of EE projects [2,3,4,5,6,7,8,9,10,11,12,13,14] while most of them also emphasize to the lack of experiments due to the difficulty in implementing an evaluation experiment. For example, in the studies implemented by Norris & Jacobson [9] and O’Neil [46] reviewing 56 and 37 EE projects, correspondingly, it is concluded that less than one-third of the programs report evaluations. Since then more evaluation experiments and reviews have been done on EE activities [47], EE projects [6,48], or EE Centers [49].

Although progress had been made in terms of environmental education program evaluation practices and use, more emphasis is needed on the subject [48]. The first and most commonly addressed concerns the capacity of educators to evaluate [50]. For this purpose, in order to help stakeholders in EE, automated systems for evaluation have been developed [12]. However, this system mainly assisted the evaluation of one EE program after its implementation.

Indeed, most of the experiments of evaluation of EE projects involve surveys after the implementation of the project and/or observation during the implementation [2]. This means that an EE project has to be implemented before one can say if it is worth being selected and that costs time, effort, and, probably, money. Therefore, an evaluation of the EE projects prior to their implementation is essential.

In Kabassi et al. [15] an automated system has also been designed to evaluate EE programs based on a multi-criteria decision making (MCDM) approach. In this approach, the combination of the MCDM models proved rather effective for evaluating the EE projects prior to their implementation. However, a major concern with decision-making is that different MCDM methods can provide different results for the same problem. For this purpose, comparisons of MCDM models have been made in different domains and concern different MCDM theories [32,36,40,43,46,51,52,53,54,55,56,57]. The MCDM models compared in the particular paper for the domain of the evaluation of EE programs are SAW, WPM, TOPSIS, and PROMETHEE II. These models have been compared in the past in pairs and the comparison experiments are presented in

Table 1. Comparison studies that involve SAW, WPM, TOPSIS, and VIKOR.

	SAW	WPM	TOPSIS	PROMETHEE II
SAW	-	[40] [51]	[40] [51] [43] [36] [52] [53]	[36] [55]
WPM		-	[40] [51]
TOPSIS			-	[32] [36] [54] [55] [56] [57]
PROMETHEE II				-

.

From Table 1, one can easily observe that these four models have never been part of the same comparison. Furthermore, there is no previous experiment that tries to compare MCDM models in the domain of EE programs’ evaluation. As a result, the particular comparison may provide interesting insights and conclusions regarding the evaluation of EE programs.

3. AHP for the First Steps

Analytic Hierarchy Process [16] is one of the most popular MCDM techniques. AHP aims to analyze a qualitative problem through a quantitative method and seems to be very appropriate for implementing the first steps of any MCDM problem. As a result, it has been combined with many other MCDM models [58,59,60].

The application of AHP revealed the set of criteria as this was formed by a group of experts and is described in detail in [15]:

• uc1-Adaptivity: This criterion reveals how flexible the program is and how adaptable it is to each age group of participants.
• uc2-Completeness: This criterion shows if the available description of the program covers the topic and to what extent.
• uc3-Pedagogy: This criterion shows whether the EE program is based on a pedagogical theory or if it uses a particular pedagogical method.
• uc4-Clarity: This criterion represents whether or not the objectives of this program are explicitly expressed or stated.
• uc5-Effectiveness: This criterion shows the overall impact, depending on the programming and available support material.
• uc6-Knowledge: This criterion refers to the quantity and quality of a cognitive object offered to students
• uc7-Skills: This criterion reveals if skills are cultivated through activities involving active student participation
• uc8-Behaviors: This criterion reveals the change in the student’s intentions and behavior through the program.
• uc9-Enjoyment: This criterion shows the enjoyment of the trainees throughout the EE project
• uc10-Multimodality: This criterion represents whether the EE project provides many different kinds of activities, interventions, and methods.

Then AHP was applied to estimate the weights of the criteria. The estimation of the weights is presented in detail in [15] and the values of the weights were estimated as follows: ${\mathrm{w}}_{\mathrm{uc}1}=0.072$, ${\mathrm{w}}_{\mathrm{uc}2}=0.071$, ${\mathrm{w}}_{\mathrm{uc}3}=0.036$, ${\mathrm{w}}_{\mathrm{uc}4}=0.129$, ${\mathrm{w}}_{\mathrm{uc}5}=0.133$, ${\mathrm{w}}_{\mathrm{uc}6}=0.127$, ${\mathrm{w}}_{\mathrm{uc}7}=0.171$, ${\mathrm{w}}_{\mathrm{uc}8}=0.099$,${\mathrm{w}}_{\mathrm{uc}9}=0.111,$ ${\mathrm{w}}_{\mathrm{uc}10}=0.051$. The analysis of the weights revealed that the most important criterion while evaluating EE programs is ‘Skills’ while ‘Effectiveness’, ‘Clarity’ and ‘Knowledge’ were also considered rather important.

4. MCDM Models

The different MCDM models selected to be combined with AHP are SAW, WPM, TOPSIS, and PROMETHEE II. Using each model, the aim was to determine the value of EE programs by combining the values of the criteria. The different models differ in their basic principles and the way they combine the criteria values. However, the first three steps of the MCDM models are identical for all the four models implemented:

Forming the set of alternative EE programs: The set of alternative EE projects was set after running a study on the EE projects in Greece [15]. More specifically, we collected the 553 programs that had been implemented in the past by environmental education centers in Greece. After collecting all this information, we chose only the EE projects that related to the subject of Environmental Paths. These projects had been implemented by environmental education centers in different parts of Greece. The specific set was selected due to its eligible number of programs (52) and its characteristics.

Forming a set of evaluators: The group of evaluators comprised only of expert users. More specifically, three users participated in the experiment, all experts in EE programs.

Calculating the values of the criteria: In this step, the evaluators studied the 52 EE programs for paths in Greece and provided values to the 10 criteria for each program. Those values were on the nine-number scale. As soon as all the values of the three decision-makers were collected, the geometric mean was calculated for the corresponding values of each criterion for each EE program.

Application of the MCDM model to estimate the final value of each EE program and ranking of the alternatives: To implement this step and calculate a final value $U(EE{p}_j)$ for each EE program, four different MCDM models have been applied: SAW, WPM, TOPSIS, and PROMETHEE II.

4.1. SAW

SAW model consists of translating a decision problem into the optimization of some multi-attribute utility function $U$ defined on $A$. The decision-maker estimates the value of the function $U(EE{p}_j)$ for every alternative $EE{p}_j$ and selects the one with the highest value. The multi-attribute utility function $U$ can be calculated in the SAW method as a linear combination of the values of the n attributes:

$$U(EE{p}_j)={\displaystyle \sum _{i=1}^{10}{w}_{uci}}u{c}_{ij},$$

(1)

where $EE{p}_j$ is one alternative and $u{c}_{ij}$ is the value of the $c_i$ criterion for the $EE{p}_j$ alternative. The higher is the value, the more desired is the alternative. The values of $U(EE{p}_j)$ using SAW are presented in

Table 2. The values assigned to each one of the alternative EE programs by the four MCDM models.

EE Program	SAW	WPM	TOPSIS	PROMETHEE II
EEp1	5.380	5.057	0.625	−0.201
EEp2	5.015	4.866	0.568	−0.383
EEp3	4.601	4.311	0.434	−0.481
EEp4	2.633	2.260	0.120	−0.789
EEp5	2.633	2.260	0.120	−0.789
EEp6	4.073	3.889	0.327	−0.609
EEp7	4.073	3.889	0.327	−0.609
EEp8	3.203	2.918	0.187	−0.743
EEp9	4.627	4.537	0.414	−0.488
EEp10	2.819	2.341	0.126	−0.732
EEp11	7.257	7.430	0.923	0.459
EEp12	7.202	7.299	0.897	0.448
EEp13	6.984	7.057	0.886	0.373
EEp14	6.882	6.835	0.881	0.343
EEp15	6.882	6.835	0.881	0.343
EEp16	6.882	6.835	0.881	0.343
EEp17	6.882	6.835	0.881	0.343
EEp18	6.882	6.835	0.881	0.343
EEp19	6.882	6.835	0.881	0.343
EEp20	8.066	8.166	0.971	0.798
EEp21	7.657	7.493	0.972	0.692
EEp22	7.863	7.723	0.979	0.714
EEp23	6.018	5.849	0.763	0.031
EEp24	6.018	5.849	0.763	0.031
EEp25	6.018	5.849	0.763	0.031
EEp26	7.771	7.715	0.970	0.666
EEp27	7.106	6.990	0.935	0.420
EEp28	7.106	6.990	0.935	0.420
EEp29	6.836	6.717	0.904	0.324
EEp30	5.719	5.379	0.655	−0.046
EEp31	4.941	4.020	0.455	−0.282
EEp32	1.000	1.000	0.000	−0.974
EEp33	3.363	2.695	0.218	−0.588
EEp34	6.242	5.973	0.786	0.064
EEp35	5.927	5.653	0.753	−0.041
EEp36	6.677	6.358	0.870	0.209
EEp37	4.103	3.929	0.317	−0.618
EEp38	4.103	3.929	0.317	−0.546
EEp39	7.234	7.365	0.909	0.456
EEp40	6.041	6.028	0.758	−0.022
EEp41	5.554	5.464	0.646	−0.157
EEp42	6.313	6.250	0.836	0.112
EEp43	5.421	5.307	0.676	−0.153
EEp44	6.832	6.852	0.886	0.286
EEp45	5.750	5.504	0.679	−0.114
EEp46	7.245	7.119	0.900	0.468
EEp47	7.245	7.119	0.900	0.468
EEp48	4.645	4.510	0.448	−0.484
EEp49	4.957	4.991	0.509	−0.399
EEp50	6.059	5.570	0.720	−0.035
EEp51	6.994	6.911	0.889	0.363
EEp52	7.066	6.975	0.897	0.394

.

4.2. WPM

Since the classic model of WPM is considered complicated in the case of pairwise comparison as it compares alternatives in pairs by calculating a ratio $U(EE{p}_K/EE{p}_L)$. However, in our case, the alternatives are 52 and this pair-wise comparison would be complicated and time-consuming. Therefore, we used an alternative application of WPM, which is proposed by Triantafyllou [21]. In this alternative approach of WPM, the decision-maker use only products without ratios. Therefore, for each alternative, the following value was calculated:

$$U(EE{p}_j)={\displaystyle \prod _{i=1}^{10}{(u{c}_{ij})}^{w_{uci}}},\mathrm{for}j=1,\dots ,52$$

(2)

The term $U(EE{p}_j)$ denoted the total performance value of the alternative $EE{p}_j$ (Table 2). Similar to SAW, the alternative with the highest $U(EE{p}_j)$ is ranked first.

4.3. TOPSIS

The central principle in TOPSIS model is that the best alternative should have the shortest distance from the ideal solution while the farthest distance from the negative-ideal solution.

Calculating weighted ratings: The weighted value is calculated as: $v_{ij}=w_{uci}\cdot u{c}_{ij}$, where $w_{uci}$ is the weight and $u{c}_{ij}$ is the value of the criterion $c_i$.

Identify positive-ideal and negative-ideal solutions: The positive-ideal solution is the composite of all the best attribute ratings attainable and is denoted:

$$EE{p}^{*}=\left\{v_1^{*},v_2^{*},\dots ,v_i^{*},\dots ,v_{10}^{*}\right\}$$

(3)

Calculate the separation measure from the positive-ideal and negative-ideal alternative: In this step, the system calculates the n-dimensional Euclidean distance from the positive ideal solution $S_j=\sqrt{{\displaystyle \sum _{i=1}^n{(v_{ij}-v_i)}^2}}$ and the negative-ideal alternative $S_j^{-}=\sqrt{{\displaystyle \sum _{i=1}^n{(v_{ij}-v_i^{-})}^2}}$ for each $j$ alternative.

Calculate similarity indices and ranking EE projects: The similarity index represents the similarity to the positive-ideal solution for alternative j, which is finally given by $U_j^{*}=\frac{S_j^{-}}{S_j+S_j^{-}}$. The alternative EE projects are then ranked according to $U_j^{*}$, in descending order, and the one with the higher value is selected as the most desired (Table 2).

4.4. PROMETHEE II

The PROMETHEE methods belong to the family of the outranking methods. The steps of PROMETHEE II after having defined criteria and their weights of importance as well as the values of the criteria for all EE programs are:

Making comparisons and calculate preference degree: This step computes for each pair of possible EE programs and for each criterion, the value of the preference degree. Let $g_j(EE{p}_i)$ be the value of a criterion j for a EE program $EE{p}_i$. We note $d_j(EE{p}_i,b)$, the difference in the value of a criterion j for two EE programs a and b.

$$d_j(EE{p}_i,b)=g_j(EE{p}_i)-g_j(b)$$

(4)

$P_j(EE{p}_i,b)$ is the value of the preference degree of a criterion j for two EE programs $EE{p}_i$ and b. The preference functions used to compute these preference degrees are defined such as:

$$P_j(EE{p}_i,b)=0,d_j(EE{p}_i,b)<0$$

(5)

$$P_j(EE{p}_i,b)=d_j(EE{p}_i,b),\mathrm{if}{d}_j(EE{p}_i,b)>0$$

(6)

Aggregating the preference degrees of all criteria for pair-wise EE programs: This step consists in aggregating the preference degrees of all criteria for each pair of possible EE programs. For each pair of possible EE programs, we compute a global preference index. Let C be the set of considered criteria and $w_j$ the weight associated with criterion j. The global preference index for a pair of possible EE programs $EE{p}_i$ and b is computed as follows:

$$\pi (EE{p}_i,b)=[{\displaystyle \sum _{j=1}^n{w}_j{P}_j(EE{p}_i,b)}]/{\displaystyle \sum _{j=1}^n{w}_j}$$

(7)

Calculate positive and negative outranking flow: This step, which is the first that concerns the ranking of the possible EE programs, consists in computing the outranking flows. For each possible EE program $EE{p}_i$, we compute the positive outranking flow ${\varphi }^{+}(EE{p}_i)$ and the negative outranking flow ${\varphi }^{-}(EE{p}_i)$. Let A be the set of possible EE programs and n the number of possible EE programs. The positive outranking flow of a possible EE program $EE{p}_i$ is computed by the following formulae:

$${\varphi }^{+}(EE{p}_i)=\frac{1}{m-1}{\displaystyle \sum _{b=1}^m\pi (EE{p}_i,b)},EE{p}_i\ne b$$

(8)

The negative outranking flow of a possible EE program $EE{p}_i$ is computed by the following formulae:

$$U^{-}(EE{p}_i)=\frac{1}{m-1}{\displaystyle \sum _{b=1}^m\pi (b,EE{p}_i)},EE{p}_i\ne b$$

(9)

Calculate the net outranking flow: The last step of the application of PROMETHEE II consists in using the outranking flows to establish a complete ranking between the possible EE programs. The ranking is based on the net outranking flows. These are computed for each possible EE program from the positive and negative outranking flows. The net outranking flow $U(EE{p}_i)$ of a possible EE program $EE{p}_i$ is computed as follows:

$$U(EE{p}_i)={\varphi }^{+}(EE{p}_i)-{\varphi }^{-}(EE{p}_i)$$

(10)

Ranking EE programs: The ranking of EE programs is done according to the value $U(EE{p}_i)$.

5. Application of MCDM Models

As soon as all the MCDM models have been applied, the final values of all EE programs using each model are calculated (Table 2).

After calculating the values of the different MCDM models, we estimated the ranking of each EE program (

Table 3. The ranking of the alternative EE program obtained by the four MCDM models.

EE Program	SAW	WPM	TOPSIS	PROMETHEE II
EEp1	36	36	36	36
EEp2	37	38	37	38
EEp3	42	41	41	40
EEp4	50	50	50	50
EEp5	50	50	50	50
EEp6	45	45	43	45
EEp7	45	45	43	45
EEp8	48	47	48	49
EEp9	41	39	42	42
EEp10	49	49	49	48
EEp11	5	5	7	7
EEp12	9	7	12	9
EEp13	14	10	15	13
EEp14	15	16	17	15
EEp15	15	16	17	15
EEp16	15	16	17	15
EEp17	15	16	17	15
EEp18	15	16	17	15
EEp19	15	16	17	15
EEp20	1	1	3	1
EEp21	4	4	2	3
EEp22	2	2	1	2
EEp23	28	27	26	26
EEp24	28	27	26	26
EEp25	28	27	26	26
EEp26	3	3	4	4
EEp27	10	11	5	10
EEp28	10	11	5	10
EEp29	21	22	9	21
EEp30	33	34	34	32
EEp31	39	42	39	37
EEp32	52	52	52	52
EEp33	47	48	47	44
EEp34	25	26	25	25
EEp35	31	30	30	31
EEp36	23	23	23	23
EEp37	43	43	45	47
EEp38	43	43	45	43
EEp39	8	6	8	8
EEp40	27	25	29	29
EEp41	34	33	35	35
EEp42	24	24	24	24
EEp43	35	35	32	34
EEp44	22	15	15	22
EEp45	32	32	32	33
EEp46	6	8	10	5
EEp47	6	8	10	5
EEp48	40	40	40	41
EEp49	38	37	38	39
EEp50	26	31	31	30
EEp51	13	14	14	14
EEp52	12	13	12	12

(The similar rankings are highlighted in different shades of gray).

).

The similar rankings are highlighted in gray in the table. The similarity in those rankings can be further demonstrated by an analysis of pairwise correlation, which is presented in the next section.

6. Comparison of the Models

The main steps for comparing the MCDM models are:

• Implementing pairwise comparisons of the values of the models by calculating the Pearson correlation coefficient.
• Implementing pairwise comparison of the rankings by calculating the Spearman’s rho correlation
• Estimating the Cohen’s kappa for testing the inter-rater comparability, using MCDM models as raters.
• Performing a sensitivity analysis to evaluate the robustness of those models.

In order to estimate the correlation of the four MCDM models being evaluated, we have calculated the Pearson correlation coefficient, for all pairs of MCDM models using the values of Table 2. The values of the Pearson correlation coefficient are presented in

Table 4. Pairwise correlation of the four MCDM models using Pearson correlation coefficient.

	SAW	WPM	TOPSIS	PROMETHEE II
SAW	1	0.995	0.987	0.975
WPM	-	1	0.986	0.973
TOPSIS	-	-	1	0.969
PROMETHEE II	-	-	-	1

, which revealed that all four methods perform very similarly (Pearson correlation coefficient of 0.969 to 0.995, which is very high).

After confirming the correlation of the MCDM models using the values of the Pearson correlation coefficient, we aim at implementing a pairwise comparison of the rankings produced by the MCDM models. This comparison is performed by estimating Spearman’s rho correlation coefficient. More specifically, the rankings of Table 3 are used for calculating Spearman’s rho correlation, for all pairs of MCDM models. The Spearman’s rho correlation is estimated by:

$$R=1-\frac{6{{\displaystyle \sum }}_{i=1}^n{d}_i^2}{n\left(n^2-1\right)}$$

where $d_i$ is the rank different at position i and n is the number of ranks.

The results are presented in

Table 5. Pairwise correlation of the four MCDM models using Spearman’s rho correlation.

	SAW	WPM	TOPSIS	PROMETHEE II
SAW	1	0.994	0.983	0.997
WPM	-	1	0.983	0.991
TOPSIS	-	-	1	0.984
PROMETHEE II	-	-	-	1

and the results are remarkable as all four methods perform very similarly (Spearman’s rho correlation of 0.983 to 0.995, which is very high).

Generally, the values of correlations are very high for all pairs of MCDM models. According to the values of the Pearson correlation coefficient, the highest correlation was between SAW and WPM, which was quite expected since their reasoning is very similar, and the lowest correlation was between TOPSIS and PROMETHEE II. The correlation of the rankings of the different alternative EE programs confirmed the high correlation of SAW and WPM but slightly higher was the correlation of SAW with PROMETHEE II.

In order to further analyze the comparison of the four MCDM models, we decided to check the effectiveness of the models in categorizing EE projects in groups. The EE programs were categorized into five different groups. The groups were five taking into account the five-Likert scale and are presented in

Table 6. Grouping of the alternative EE program obtained by the four MCDM models.

Action	SAW	WPM	TOPSIS	PROMETHEE II
EEp1	4	4	4	4
EEp2	4	4	4	4
EEp3	5	5	5	5
EEp4	5	5	5	5
EEp5	5	5	5	5
EEp6	5	5	5	5
EEp7	5	5	5	5
EEp8	5	5	5	5
EEp9	5	4	5	5
EEp10	5	5	5	5
EEp11	1	1	1	1
EEp12	1	1	2	1
EEp13	2	2	2	2
EEp14	2	2	2	2
EEp15	2	2	2	2
EEp16	2	2	2	2
EEp17	2	2	2	2
EEp18	2	2	2	2
EEp19	2	2	2	2
EEp20	1	1	1	1
EEp21	1	1	1	1
EEp22	1	1	1	1
EEp23	3	3	3	3
EEp24	3	3	3	3
EEp25	3	3	3	3
EEp26	1	1	1	1
EEp27	2	2	1	2
EEp28	2	2	1	2
EEp29	3	3	1	3
EEp30	4	4	4	4
EEp31	4	5	4	4
EEp32	5	5	5	5
EEp33	5	5	5	5
EEp34	3	3	3	3
EEp35	4	4	4	4
EEp36	3	3	3	3
EEp37	5	5	5	5
EEp38	5	5	5	5
EEp39	1	1	1	1
EEp40	3	3	3	3
EEp41	4	4	4	4
EEp42	3	3	3	3
EEp43	4	4	4	4
EEp44	3	2	2	3
EEp45	4	4	4	4
EEp46	1	1	2	1
EEp47	1	1	2	1
EEp48	5	5	5	5
EEp49	4	4	4	4
EEp50	3	4	4	4
EEp51	2	2	2	2
EEp52	2	2	2	2

(The similar rankings are highlighted in different shades of gray).

(1-very good, 2- good, 3-mediocre, 4-not good, 5- bad).

The 52 EE programs have been classified into five mutually exclusive categories; we use Cohen’s kappa to test the reliability of the MCDM models being investigated. Indeed, Cohen’s kappa measures the agreement between two raters, who each classify N items into C mutually exclusive categories. The raters in the particular decision problems are the MCDM models that rate the EE programs and we make pairwise comparisons.

The values of Cohen’s kappa for pairwise comparing the agreement of the MCDM models are presented in

Table 7. The values of Cohen’s kappa for pairwise comparing the agreement of the MCDM models.

	SAW	WPM	TOPSIS	PROMETHEE II
SAW	1	0.903	0.807	0.976
WPM	-	1	0.806	0.927
TOPSIS	-	-	1	0.831
PROMETHEE II	-	-	-	1

. A value of Cohen’s kappa above 0.6 is quite good. Thus the values of the Cohen’s kappa of the particular experiment (0.806–0.976) confirm the reliability of the four MCDM models being compared for the particular domain.

The raters’ resemblance according to Cohen’s kappa is quite high in general (above 0.806 for all pairs of MCDM models). This fact was also confirmed by the Pearson correlation coefficient and Spearman’s rho correlation presented in Table 4 and Table 5. Cohen’s kappa revealed the highest correlation was between PROMETHEE II and SAW, which is in line with the results of Spearman’s rho correlation. Both Spearman’s rho correlation and Cohen’s kappa revealed that the correlation of PROMETHEE II with WPM is also very high. Even though some theories seem to correlate more than others, the general correlation of the four methods is very high. A reasonable disagreement can be observed among the methods, but this does not affect their reliability.

7. Sensitivity Analysis

A sensitivity analysis was performed to investigate the robustness of each MCDM method compared in this paper. A way of performing sensitivity analysis is changing one by one the weights of the criteria. Another way of performing this analysis is using a weighting scheme, which assigns equal weight to each one of the criteria [40,61]. The values of the criteria are not modified. Since there are 10 criteria, the weight for each criterion was determined to be 0.1.

Table 8. The values of the alternative EE programs using the two weighting schemes.

Action	SAW-Scheme 1	SAW-Scheme 2	WPM-Scheme 1	WPM-Scheme 2	TOPSIS-Scheme 1	TOPSIS-Scheme 2	PROMETHEE II-Scheme 1	PROMETHEE II-Scheme 2
EEp1	5.38	5.100	5.057	4.599	0.625	0.535	−0.201	−0.236
EEp2	5.015	4.700	4.866	4.547	0.568	0.452	−0.383	−0.439
EEp3	4.601	4.400	4.311	4.205	0.434	0.378	−0.481	−0.486
EEp4	2.633	2.500	2.26	2.140	0.12	0.104	−0.789	−0.802
EEp5	2.633	2.500	2.26	2.140	0.12	0.104	−0.789	−0.802
EEp6	4.073	3.900	3.889	3.797	0.327	0.272	−0.609	−0.626
EEp7	4.073	3.900	3.889	3.797	0.327	0.272	−0.609	−0.626
EEp8	3.203	3.000	2.918	2.713	0.187	0.137	−0.743	−0.769
EEp9	4.627	4.600	4.537	4.447	0.414	0.426	−0.488	−0.465
EEp10	2.819	2.900	2.341	2.421	0.126	0.148	−0.732	−0.702
EEp11	7.257	7.300	7.43	7.218	0.923	0.921	0.459	0.468
EEp12	7.202	7.300	7.299	7.218	0.897	0.921	0.448	0.498
EEp13	6.984	7.100	7.057	7.050	0.886	0.911	0.373	0.432
EEp14	6.882	6.900	6.835	6.850	0.881	0.888	0.343	0.373
EEp15	6.882	6.900	6.835	6.850	0.881	0.888	0.343	0.373
EEp16	6.882	6.900	6.835	6.850	0.881	0.888	0.343	0.373
EEp17	6.882	6.900	6.835	6.850	0.881	0.888	0.343	0.373
EEp18	6.882	6.900	6.835	6.850	0.881	0.888	0.343	0.373
EEp19	6.882	6.900	6.835	6.850	0.881	0.888	0.343	0.373
EEp20	8.066	8.200	8.166	8.152	0.971	0.981	0.798	0.821
EEp21	7.657	7.500	7.493	7.468	0.972	0.949	0.692	0.651
EEp22	7.863	7.600	7.723	7.544	0.979	0.953	0.714	0.653
EEp23	6.018	5.800	5.849	5.678	0.763	0.697	0.031	−0.014
EEp24	6.018	5.800	5.849	5.678	0.763	0.697	0.031	−0.014
EEp25	6.018	5.800	5.849	5.678	0.763	0.697	0.031	−0.014
EEp26	7.771	7.800	7.715	7.749	0.97	0.971	0.666	0.688
EEp27	7.106	6.900	6.99	6.850	0.935	0.896	0.42	0.362
EEp28	7.106	6.900	6.99	6.850	0.935	0.896	0.42	0.362
EEp29	6.836	6.700	6.717	6.670	0.904	0.875	0.324	0.284
EEp30	5.719	5.600	5.379	5.383	0.655	0.645	−0.046	−0.061
EEp31	4.941	5.100	4.02	4.494	0.455	0.533	−0.282	−0.292
EEp32	1	1.000	1	1.000	0	0.000	−0.974	−0.967
EEp33	3.363	3.200	2.695	2.731	0.218	0.195	−0.588	−0.527
EEp34	6.242	6.200	5.973	5.988	0.786	0.771	0.064	0.054
EEp35	5.927	5.800	5.653	5.662	0.753	0.707	−0.041	−0.075
EEp36	6.677	6.500	6.358	6.413	0.87	0.829	0.209	0.13
EEp37	4.103	4.000	3.929	3.949	0.317	0.285	−0.618	−0.61
EEp38	4.103	4.000	3.929	3.949	0.317	0.285	−0.546	−0.468
EEp39	7.234	7.400	7.365	7.342	0.909	0.937	0.456	0.476
EEp40	6.041	6.000	6.028	5.880	0.758	0.750	−0.022	−0.01
EEp41	5.554	5.600	5.464	5.502	0.646	0.657	−0.157	−0.111
EEp42	6.313	6.100	6.25	6.017	0.836	0.764	0.112	0.041
EEp43	5.421	5.100	5.307	4.981	0.676	0.545	−0.153	−0.253
EEp44	6.832	6.700	6.852	6.624	0.886	0.855	0.286	0.238
EEp45	5.75	5.800	5.504	5.741	0.679	0.704	−0.114	−0.070
EEp46	7.245	7.400	7.119	7.331	0.9	0.933	0.468	0.53
EEp47	7.245	7.400	7.119	7.331	0.9	0.933	0.468	0.53
EEp48	4.645	4.600	4.51	4.573	0.448	0.423	−0.484	−0.467
EEp49	4.957	5.000	4.991	4.959	0.509	0.526	−0.399	−0.355
EEp50	6.059	6.100	5.57	5.940	0.72	0.745	−0.035	0.004
EEp51	6.994	7.000	6.911	6.926	0.889	0.894	0.363	0.379
EEp52	7.066	7.100	6.975	7.054	0.897	0.911	0.394	0.422

presents the values assigned to each one of the alternative EE programs being evaluated using the four MCDM models. First, the values calculated by MCDM models using the weighting scheme 1 is presented. This scheme uses the weights as these were estimated by AHP, and then for each model, the weighting scheme 2 is used in which equal weights are given to all criteria. Using the two different schemes of weights by the four different MCDM, the rankings have been estimated for each alternative EE program. As a result,

Table 9. The ranking of the alternative EE programs using the two weighting schemes.

Action	SAW-Scheme 1	SAW-Scheme 2	WPM-Scheme 1	WPM-Scheme 2	TOPSIS-Scheme 1	TOPSIS-Scheme 2	PROMETHEE II-Scheme 1	PROMETHEE II-Scheme 2
EEp1	36	35	36	37	36	36	36	35
EEp2	37	39	38	39	37	39	38	39
EEp3	42	42	41	42	41	42	40	43
EEp4	50	50	50	50	50	50	50	50
EEp5	50	50	50	50	50	50	50	50
EEp6	45	45	45	45	43	45	45	46
EEp7	45	45	45	45	43	45	45	46
EEp8	48	48	47	48	48	49	49	49
EEp9	41	40	39	41	42	40	42	40
EEp10	49	49	49	49	49	48	48	48
EEp11	5	8	5	8	7	8	7	9
EEp12	9	8	7	8	12	8	9	7
EEp13	14	10	10	11	15	10	13	10
EEp14	15	13	16	13	17	15	15	13
EEp15	15	13	16	13	17	15	15	13
EEp16	15	13	16	13	17	15	15	13
EEp17	15	13	16	13	17	15	15	13
EEp18	15	13	16	13	17	15	15	13
EEp19	15	13	16	13	17	15	15	13
EEp20	1	1	1	1	3	1	1	1
EEp21	4	4	4	4	2	4	3	4
EEp22	2	3	2	3	1	3	2	3
EEp23	28	28	27	29	26	30	26	28
EEp24	28	28	27	30	26	30	26	28
EEp25	28	28	27	31	26	30	26	28
EEp26	3	2	3	2	4	2	4	2
EEp27	10	13	11	13	5	12	10	19
EEp28	10	13	11	13	5	12	10	19
EEp29	21	21	22	21	9	21	21	21
EEp30	33	33	34	34	34	34	32	31
EEp31	39	35	42	40	39	37	37	37
EEp32	52	52	52	52	52	52	52	52
EEp33	47	47	48	47	47	47	44	44
EEp34	25	24	26	25	25	24	25	24
EEp35	31	28	30	32	30	28	31	33
EEp36	23	23	23	23	23	23	23	23
EEp37	43	43	43	43	45	43	47	45
EEp38	43	43	43	43	45	43	43	42
EEp39	8	5	6	5	8	5	8	8
EEp40	27	27	25	27	29	26	29	27
EEp41	34	33	33	33	35	33	35	34
EEp42	24	25	24	24	24	25	24	25
EEp43	35	35	35	35	32	35	34	36
EEp44	22	21	15	22	15	22	22	22
EEp45	32	28	32	28	32	29	33	32
EEp46	6	5	8	6	10	6	5	5
EEp47	6	5	8	6	10	6	5	5
EEp48	40	40	40	38	40	41	41	41
EEp49	38	35	37	36	38	38	39	38
EEp50	26	25	31	26	31	27	30	26
EEp51	13	10	14	12	14	14	14	12
EEp52	12	10	13	10	12	10	12	11

presents the ranking of the alternative EE programs using the four different MCDM models and the two schemes of weights.

The sensitivity analysis aims to check the consistency of the results and to evaluate the robustness of the ranking produced by the method as the scheme of weights changes. For this purpose, we are going to check the Pearson correlation coefficient for the values generated by each MCDM using the two different weighting schemes. The values of the Pearson correlation coefficient are very high and are presented in

Table 10. The statistical analysis of the comparison.

	Pearson Correlation Coefficient	Percentage of Identical Rankings	Spearman’s Rho Correlation
SAW	0.997	42%	0.994
WPM	0.995	27%	0.989
TOPSIS	0.990	13%	0.977
PROMETHEE II	0.996	29%	0.988

. The highest correlation was found between the schemes using PROMETHEE II.

However, what plays the most important role in sensitivity analysis is the consistency of rankings. The consistency is low if the ranking of the alternatives is completely modified after a slight variation of the weights. We compared the rankings of EE programs using the two different schemes by

• Checking how many identical rankings were among the rankings of each model using the different schemes.
• Estimating the Spearman’s rho correlation for each model using the two schemes of weights.

In Table 10 the percentage of identical rankings is presented for the four MCDM models and from these values, it is derived that SAW is less affected by the change in the weights of the criteria. Spearman’s rho correlation also confirmed the high correlation of rankings estimated by SAW using the two different schemes of weights (Table 10). The MCDM model that seems to be less affected by the choice of the set of weights is SAW, while the MCDM model that is most affected is TOPSIS as it has the lowest values of Pearson correlation coefficient, identical rankings, and Spearman’s rho correlation coefficient.

8. Conclusions

The front-end evaluation of EE programs prior to the application is very important because the application of an EE program costs time, money, and effort, and, therefore, one would like to implement only a program that is worth being implemented. Furthermore, the evaluation of EE programs can help the reuse of EE programs that require a lot of effort to be designed and have proven to have complete instructions and confirmed success in the application.

In view of the above, the contribution of this paper is on presenting the use of MCDM models for the front-end evaluation of EE programs and the comparison of the different models. Conclusions have been drawn regarding the application of different MCDM models. The implementation of the different MCDM models and their comparison can be found very useful by Environmental Education Centers, educators and/or generally stakeholders in EE that want to evaluate EE projects prior to their implementation and select the one that seems the most appropriate.

In this paper, we show that AHP has been used for defining the set of criteria as well as their weights. Then we combine AHP with four different MCDM models and compare them. In order to implement the MCDM models and run the comparison test between the model for an initial comparison of the EE projects prior to their implementation, we run a simulation using 52 EE projects of the EE Centers in Greece that involve paths. The data of the 52 EE projects that involved paths were incorporated by SAW, WPM TOPSIS, and PROMETHEE II, which are various MCDM models with different computational mechanisms as an additive or multiplicative combination, similarity to an ideal solution, etc.

The application of the different theories revealed the characteristics of each method as well as their advantages and disadvantages. The application of SAW is very easy, but the values of criteria must be positive to provide valuable results. WPM has the same advantages and disadvantages as SAW and provides similar results as it is proved by the values of Spearman’s rho and Cohen’s kappa. Both SAW and WPM have the ability to compensate among different criteria. Simplicity is also the main advantage of TOPSIS. An additional advantage of TOPSIS is its ability to maintain the same amount of steps regardless of problem size has allowed it to be utilized quickly to review other methods or to stand on its own as a decision-making tool [55]. However, the use of Euclidean Distance does not consider easy for the correlation of attributes but keeps the consistency of judgment. PROMETHEE II is also quite easy and additionally, it does not require the assumption that criteria are proportionate. However, all four models have the disadvantage of not having a clear methodology for estimating weights. For this purpose, in the current paper, AHP is used for estimating weights and combined in turn with SAW, WPM, TOPSIS, and PROMETHEE II.

The application of the MCDM models for the assessment of EE programs has revealed that the MCDM may prove rather effective. However, according to Mulliner et al. [18], different MCDM models can yield different results when applied to the same decision problem. This fact was also confirmed in this particular study. Therefore, we have used the Pearson correlation coefficient, Spearman’s rho correlation, and Cohen’s kappa for pairwise comparison of the different models and checking the inter-MCDM models’ reliability.

The sensitivity analysis that was performed in order to evaluate the robustness of the four different MCDM models in evaluating EE programs was implemented by applying a different scheme of weights and comparing the results of each model using the two different weighting schemes. The comparison involved estimating the Pearson correlation coefficient, identical rankings, and Spearman’s rho correlation. The results of the sensitivity test revealed that all models were quite robust. The MCDM model that proved to be more robust and less affected by the choice of the set of weights is SAW, while the MCDM that is most sensitive is TOPSIS.

The high values of correlation between the different MCDM models revealed that the ranking results mainly depend on the nature and the values of the criteria and less on the model selected. The reasonable disagreement that was observed among the methods did not affect their reliability. As a result, MCDM models proved generally very effective for EE programs before their implementation and selecting the best ones.

However, a possible limitation of this work is that this comparison has been made using only a set of alternative EE projects and it would provide safer conclusions if more sets were involved. Furthermore, the comparison could also involve more MCDM models in order to confirm that the results would not change. Therefore, it is among our future plans to extend the experiment with more MCDM models, such as ELECTRE, Delphi etc. Furthermore, we aim at re-implementing the experiment with other sets of EE projects with different characteristics in order to confirm that the set of projects does not affect the conclusions drawn by this study.