Science, Technology and Innovation Policy Indicators and Comparisons of Countries through a Hybrid Model of Data Mining and MCDM Methods

Abstract

Science, technology and innovation (STI) policies are of great importance for countries to reach their sustainable development goals. Numerous global databases have many indicators that measure and compare the performance of STI policies of countries. However, many problems arise regarding how to identify, classify and systematically analyze these indicators in order to measure, monitor and improve the performance of STI. The study includes a literature review on global problems and new trends in STI policies, while mentioning the necessity of an internationally comparable STI indicator set, current STI indicator studies and efforts, and studies for each continent. In light of these, all the indicators selected are introduced in detail. The strengths and weaknesses of the countries in the study in terms of evaluation indicator values are indicated. After determining the indicator weights objectively with the entropy method, 40 countries are compared with TOPSIS, VIKOR, PROMETHEE I-II, ARAS, COPRAS, MULTIMOORA, ELECTRE, SAW and MAUT methods. In addition, countries that show similarities with each other are evaluated by cluster analysis, which is one of the data mining classification methods. This study offers a new and original approach with MCDM methods on this subject. Considering all the results obtained in the study together, these rankings are compared among themselves and with the rankings specified in the Global Innovation (2019) and Global Competitiveness (2019) indices, and it is seen that the results are consistent. In addition, it is possible to update and publish this study every year with updated data.

1. Introduction

Developments in science and technology and strategies based on innovation have become the basic elements of productivity increase and competition at both country and company level. Science, technology and innovation (STI) are very important for all countries because of their sustainable growth effect and solutions to the energy, food security and climate change. Therefore, STI is a significant topic for both sustainable growth and the achievement of political goals. STI has been on the agenda of both developed countries and developing countries in recent years. Some of the main findings of the 2018 Industrial R&D Investment Scoreboard, published by the European Commission on 17 December 2018, are shown in Figure 1 [1].

As it can be understood from Figure 1, the biggest difference that separates developed countries from developing countries is the knowledge gap between them. As long as the knowledge gaps are eliminated, it is possible to close the development and income gap. Figure 2 shows some key figures about growth rates based on R&D [1]. With the structural change trends in production brought about by technological developments, companies are now strong as long as they can adapt to global value chains, and also countries have competitive power as long as they have companies with global value chains.

Today, science and technology are rapidly globalizing, open innovation systems and collaborations are becoming widespread, and new technologies increase the speed of knowledge dissemination. In a world where competition is rapidly increasing and STI is the most decisive actor, countries need to shape their policies accordingly. Therefore, STI indicators are very important to evaluate the current status of countries and to compare them with each other in terms of STI. Therefore, large global databases such as OECD (Organization for Economic Development and Cooperation), UNDP (United Nations Development Program), ITU (International Telecom Union), Eurostat, the World Bank and the statistical offices of the countries have many indicators that measure and compare the Science, Technology, and Innovation Policy (STIP) of countries. Such efforts produce indicator systems to reveal the state of the capabilities and capacities of nations and update existing ones. STI indicators and regional and international comparisons are considered among the important guides of governments in policy formulation on issues such as economy, welfare, and development. The developing competitive environment increases the need for companies and countries to manage the complexity surrounding the STI policy process. However, many problems arise regarding how to determine the importance of these indicators, how to classify them, and how they should be analyzed systematically in order to measure, monitor, and improve the performance of STIP. Therefore, this is an issue that attracts the attention of experts, policymakers, academics, and investors. Designing strong STI indicators and frameworks that can be used in regional and international comparisons should be seen as a priority for governments to form national and international policies [2,3].

This study includes all the topics that are outside of governments’ policies, whether they are aware of it or not, and that is essential for a sustainable STI policy, by presenting a holistic system approach beyond what countries directly implement with each policy and strategy. Thus, it recommends a sustainable system approach to managers who are decision-makers in policies in order to develop holistic solutions and strategies while working on complex issues that they have to deal with on a global scale. By making use of this STI policy proposal, which consists of multi-dimensional factors, decision-makers can see their strengths and weaknesses, and have the opportunity to see and evaluate the status of their competitors.

In the literature, there are studies that many researchers compare and rank countries in terms of their performance in various subjects with MCDM and multivariate statistical methods. These studies evaluated countries in terms of R&D, innovation, technology, regional development, competitive advantage, trade and macroeconomic indicators. The main criteria used in comparing countries in terms of R&D and STI performance in the literature are as follows [4]: Patent applications made by non-residents (number/year); patent applications made by residents (units/year); trademark applications made directly by non-residents (number/year); trademark applications made directly by residents (number/year); trademark applications made by non-residents (number/year); trademark applications made by residents (number/year); total trademark applications (units/year); number of researchers in R&D (per million people); the ratio of R&D expenditures in GDP (%); high technology export amount (USD); high technology exports (percentage of manufacturing products exports); ICT goods exports (percentage of total goods exports); and the number of articles in scientific and technical journals.

Some similar studies on this subject are as follows. Lin, Shyu [5] used a descriptive analysis with descriptive statistics under the innovation policy framework proposed by Rothwell and Zegveld. This study also informed a comparative policy analysis across China and Taiwan. Chaurasia and Bhikajee [6] aimed to analyze India’s economic growth performance, STI investment and health improvements in comparison with Brazil, China and Singapore. Sun and Cao [7] analyzed the dynamics of China’s science, technology and innovation (STI) work, emphasizing the importance of studying science, technology and innovation (STI) activities in China in understanding international competitiveness in the knowledge-based economy. Erdin and Özkaya [8] assessed the Association of Southeast Asian Nations (ASEAN) countries in terms of the criteria of sustainable development index. They used the TOPSIS method to compare and rank them. Salam, Hafeez [9] compared low-middle-income countries while evaluating the dynamic relationship between technology adaptation, innovation, human capital and economy. Blažek and Kadlec [10] examined the interrelationships and problems between the knowledge bases, R&D structure and innovation performance of European regions, and made regional assessments about their situations. Canbolat, Chelst [11] used the MAUT method to evaluate Mexico, Czech Republic, Poland, South Korea and South Africa in terms of global competitiveness and survival, research and development, government regulations and economic factors for establishing a production facility on a global level. They tried to decide which country was more suitable. Kang, Jang [12] compared the national innovation system between the US, Japan and Finland to improve the Korean Deliberation Organization for national science and technology policy.

Manyuchi [13] studied the use of innovation indicators in South Africa’s science, technology and innovation policy making and introduced the institutions that support them. Özbek and Demirkol [14] assessed the European countries using AHP, ARAS, COPRAS, and GRA (Gray Relational Analysis) methods with macroeconomic indicators. The best performing country in the evaluation was Germany, while Greece was in the last place in the ranking.

In the first revision meeting of the OECD’s Science, Technology and Innovation Policy index, two common issues agreed upon by the delegates and committee members working on this issue are the inability to be sure of the method to analyze the indicators in the most accurate way and whether collecting information only with surveys produces a result consistent with the facts. Therefore, this study aims to create an appropriate comparison framework by taking advantage of the most up-to-date existing STI policy indices and to compare countries using these indicator values with multi-criteria decision-making methods (MCDM) and cluster analysis. Thus, a comprehensive MCDM approach is presented to the STI policy comparisons and evaluations, which has not been done yet in the field. Therefore, the study is expected to add an important novelty to the literature.

In the study, the weights of the indicators are determined with the Entropy method, which is one of the MCDM methods. Then 40 countries whose all indicator values are available are compared with TOPSIS, VIKOR, PROMETHEE I-II, ARAS, COPRAS, ELECTRE, SAW, MAUT, and MULTIMOORA methods. In addition, countries that show similarities with each other are evaluated by cluster analysis, which is one of the data mining classification methods. The selected countries are evaluated within the framework of 10 dimensions and 115 criteria. These criteria were determined as a result of the evaluation of the OECD, the World Bank, the Global Competitiveness Index, and the Global Innovation Indices, which conduct studies and publish reports on science, technology, and innovation. Criteria, indicators, descriptive information about indicators and sources from which data are obtained (SCImago [15], Indexmundi [16], OECD and Group [17], Unesco [18], WorldBank [19], TradingEconomics [20], Schwab [21], Dutta, Lanvin [22], ITU [23], IMF [24], ILO [25] and Numbeo [26]) are shown in Appendix A.

The rest of the study is organized as follows: Section 2 explains the proposed methods. Section 3 presents the obtained results. Section 4 presents the discussion, and Section 5 presents the conclusion.

2. Methodology and Data

All the data obtained have been verified in more than one source and evaluated by comparing them among themselves. The sources and values of all indicators used in the application are included in the appendices of the study.

While the list of the evaluated countries with some descriptive information is presented in

Table 1. Countries and some descriptive specific features.

No	Country	Income	Region	Population (mn)	GDP PPP$	GDP Per Capita, PPP$
1	Australia	High	South East Asia, East Asia, and Oceania	24.8	1386.6	52,375.5
2	Austria	High	Europe	8.8	464.0	52,137.4
3	Belgium	High	Europe	11.5	549.7	48,244.7
4	Brazil	Upper middle	Latin America and the Caribbean	210.9	3370.6	16,154.3
5	Canada	High	Northern America	37	1852.5	49,651.2
6	China	Upper middle	South East Asia, East Asia, and Oceania	1415.0	25,313.3	18,109.8
7	Czech Republic	High	Europe	10.6	396.4	37,371.0
8	Denmark	High	Europe	5.8	300.3	52,120.5
9	Finland	High	Europe	5.5	257.2	46,429.5
10	France	High	Europe	65.2	2968.5	45,775.1
11	Germany	High	Europe	82.3	4379.1	52,558.7
12	Greece	High	Europe	11.1	312.5	29,123.0
13	Hungary	High	Europe	9.7	308.2	31,902.7
14	Iceland	High	Europe	0.3	19.3	55,917.3
15	India	Lower middle	Central and Southern Asia	1354.1	10,401.4	7873.7
16	Indonesia	Lower middle	South East Asia, East Asia, and Oceania	266.8	3495.9	13,229.5
17	Ireland	High	Europe	4.8	378.5	78,784.8
18	Israel	High	Northern Africa and Western Asia	8.5	336.1	37,972.0
19	Italy	High	Europe	59.3	2398.2	39,637.0
20	Japan	High	South East Asia, East Asia, and Oceania	127.2	5632.5	44,227.2
21	Malaysia	Upper middle	South East Asia, East Asia, and Oceania	32.0	999.8	30,859.9
22	Mexico	Upper middle	Latin America and The Caribbean	130.8	2575.2	20,601.7
23	Netherlands	High	Europe	17.1	972.5	56,383.2
24	Norway	High	Europe	5.4	398.3	74,356.1
25	Poland	High	Europe	38.1	1201.9	31,938.7
26	Portugal	High	Europe	10.3	328.8	32,006.4
27	Qatar	High	Northern Africa and Western Asia	2.7	356.7	130,475.1
28	Russian Federation	Upper middle	Europe	144.0	4179.6	29,266.9
29	Singapore	High	South East Asia, East Asia, and Oceania	5.8	556.2	100,344.7
30	Slovakia	High	Europe	5.4	191.1	35,129.8
31	South Africa	Upper middle	Sub-Saharan Africa	57.4	790.9	13,675.3
32	South Korea	High	South East Asia, East Asia, and Oceania	51.2	2139.7	41,350.6
33	Spain	High	Europe	46.4	1867.9	40,138.8
34	Sweden	High	Europe	10.0	542.8	52,984.1
35	Switzerland	High	Europe	8.5	551.4	64,649.1
36	Thailand	Upper middle	South East Asia, East Asia, and Oceania	69.2	1323.2	19,476.5
37	Turkey	Upper middle	Europe	82.9	2314.4	27,956.1
38	United Arab Emirates	High	Northern Africa and Western Asia	9.5	732.9	69,381.7
39	United Kingdom	High	Europe	66.6	3033.7	45,704.6
40	United States	High	Northern America	326.8	20,513.0	62,605.6

Source: Created by author by using the Global Innovation Index (2019) values.

, the definitions of indicators and dimensions is included in Appendix A.

In this study, the Entropy method is proposed in order to avoid subjective evaluations while weighting of indicators to be used in the evaluation of STI policy performance. After determining the indicator weights, the data of the indicators of the countries are analyzed with TOPSIS, VIKOR, PROMETHEE I-II, ARAS, COPRAS, MULTIMOORA, SAW, MAUT and ELECTRE methods and the performances of 40 countries are evaluated and are compared with each other. Also, the countries classified by using the data mining cluster analysis. All results are evaluated among themselves for consistency.

Existing multi-criteria decision-making approaches can cause confusion of users from time to time due to the complex calculation steps and the solutions they produce. Each of these methods has its own strengths and weaknesses. Due to these differences in the structures of MCDM approaches, the problem of obtaining different rankings with different methods arises in the evaluation of the same problem, also known as inconsistent problem ordering. This is the biggest criticism of these techniques. The main reasons for the occurrence of these differences are as follows: use of different weights, using a different approach to determine the best alternative, the effort to measure goals, and using parameters that may affect the result differently. Currently, there is no specific standard rule for the application of multi-criteria assessment methods and the interpretation of the results obtained.

Table 2. Multi-criteria decision making methods and comparisons according to their specific features.

MCDM Methods	Calculation Time	Simplicity	Mathematical Operations	Reliability	Data Type
AHP	Too long	Complex	Maximum	Weak	Mixed
TOPSIS	Intermediate	Simple	Intermediate	Middle	Quantitative
VIKOR	Intermediate	Simple	Intermediate	Middle	Quantitative
MULTIMOORA	Long	Intermediate	Intermediate	Good	Quantitative
ARAS	Intermediate	Simple	Intermediate	Middle	Quantitative
ELECTRE	Long	Complex	Maximum	Middle	Mixed
PROMETHEE	Intermediate	Complex	Maximum	Middle	Mixed
SAW	Intermediate	Simple	Minimum	Middle	Quantitative
GRA	Intermediate	Intermediate	Intermediate	Middle	Quantitative
COPRAS	Intermediate	Simple	Minimum	Middle	Quantitative
ENTROPİ	Intermediate	Simple	Intermediate	Middle	Quantitative
MAUT	Intermediate	Simple	Minimum	Middle	Quantitative

Source: Compiled by the authors, based on Brauers and Zavadskas [27].

was created by the author based on Brauers and Zavadskas [27].

The methodology used in this study combines two approaches to estimate STI performances of selected countries. These approaches are Clustering method and MCDM methods. The proposed methodology in this study includes three phases as stated: Data understanding and collection from indexes; Data preprocessing; Modeling and data analyzing.

Data preparation and data normalization were included in data preprocessing phase. In the data preparation step, some uncompleted data have been omitted. Next, a normalization process is required to put the fields into comparable scales. This process is due to the different scales of STI inputs. In this paper, a min-max approach was used which recalled all record values in the range between 0 to 1. Then, weights of STI variables were calculated by Entropy method and the normalized data of STI have been weighted by these weights. Then, based on weighted STI values countries were clustered by K means clustering. Finally, the clusters of countries were ranked using MCDM methods. Afterwards, the clusters formed in the clustering analysis and the rankings obtained from the MCDM results are evaluated together and the results are compared. When a similar analysis is made for the countries that are not included here, by considering the countries in the study, it can be determined to which cluster a country belongs. Research framework of the study is shown in Figure 3.

2.1. Shannon Entropy and Objective Weights

Two different weight methods are used in MCDM methods, namely objective (objective) and subjective (subjective). Subjective weights are obtained by directly benefiting from the opinions of decision-makers like other MCDM processes. In objective weighting, it makes use of the quantitative features of the criteria. Entropy method, as one of these objective weighting methods, can be applied under conditions where decision matrix values are known [28]. Shannon and Weaver [29] proposed the concept of entropy, a measure of uncertainty in information formulated in terms of probability theory. The concept of entropy is a suitable option for our purpose, as it enables the measurement of relative contrast densities of criteria representing the original information conveyed to the decision-maker [30]. This method has been used in many areas such as spectral analysis [31], language modeling [32], and economics [33].

Shannon has developed an H measure that provides the following properties for all pi in the estimated common probability distribution (P) [34]:

H is a positive continuous function, If all p_i are equal ($p_i=\frac{1}{n}$), then H must be a monotonic incremental function of n.

For all n ≥ 2,

$$H\left(p_1, p_2, \dots , p_n\right) =h\left(p_1+p_2 , p_3, \dots , p_n\right)+\left(p_1+p_2\right) H \left(\frac{p_1}{p_1+p_2}, \frac{p_2}{p_1+p_2}\right)$$

(1)

Shannon showed that the only function that meets these properties is as follows:

$$H_{Shannon}=-{{\displaystyle \sum }}_i{p}_i\log \left(p_i\right)$$

(2)

Shannon’s entropy method is explained as a weighting calculation method with the following process steps [35,36,37,38]:

Step 1: Creating the Decision Matrix

In the first step of the entropy method, the decision matrix is first created similar to other multi-criteria decision making methods.

$$X=\left[\begin{array}{ccc}{x}_{11}& \cdots & x_{1n}\\ ⋮& \ddots & ⋮\\ x_{m1}& \cdots & x_{mn}\end{array}\right]$$

(3)

Step 2: Obtaining the Normalized Decision Matrix

In order to convert the criterion scores into common units, the criteria are normalized according to their benefit or cost characteristics. In this step, Equation (4) is used as follows:

$$\begin{gathered} r_{ij}=x_{ij}/{\max }_{ij}\left(i=1,\dots ,m;j=1,\dots ,n\right) \\ {r}_{ij}=x_{ij}/{\min }_{ij}\left(i=1,\dots ,m;j=1,\dots ,n\right) \end{gathered}$$

(4)

In this equation, $i=\mathrm{alternatives}$; $j=\mathrm{criteria}$; $r_{ij}=\mathrm{normalized} \mathrm{values}$; $x_{ij}=\mathrm{benefit} \mathrm{values} \mathrm{of} \mathrm{the} \mathrm{i}.\mathrm{alternative} \mathrm{for} \mathrm{j}.$

With the normalization process specified in the second equation, the normalized decision matrix defined by the Equation (5) is formed:

$$P_{ij}=\frac{a_{ij}}{{{\displaystyle \sum }}_{i=1}^m{a}_{ij}}; {\forall }_j$$

(5)

Here, P_ij represents normalized values while a is benefit values.

Step 3: Calculate the entropy measure of each indicator using the following equation:

$$E_j=\ast k{{\displaystyle \sum }}_{i=1}^m\left[P_{ij}\ln P_{ij}\right];{\forall }_j$$

(6)

$k=entropy coefficient \{{(\ln \left(n\right))}^{-1}\}$; $P_{ij}=normalized values$; $E_j=entropy value$.

Step 4: (d_j) uncertainty value is calculated by Equation (7)

$$d_j=1-E_j; {\forall }_j$$

(7)

Step 5: w_j weights are calculated as the importance of j criterion by using the Equation (8),

$$w_j=\frac{d_j}{{{\displaystyle \sum }}_{j=1}^n{d}_j}; {\forall }_j$$

(8)

The sum of entropy probability values is always equal to 1.

$$w_1+w_2+w_j+\cdots +w_n=1$$

(9)

2.2. Ranking of Countries Based on MCDM Methods

In the rest of this section, the steps of the methods used in the study are explained mathematically.

2.2.1. TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) Method

TOPSIS (Technique for Order Preference by Similarity to Ideal Solution), proposed by Hwang and Yoon [39] and developed by Lai, Liu [40], is a method that aims to reach the most ideal solution among the available options [41,42]. The positive ideal solution increases the utility metrics while decreasing the cost metrics. The negative ideal solution does the opposite. TOPSIS is generally defined in five steps [43]:

Step 1. First of all, normalization is done to the decision matrix. Using the r_ij values calculated here, the R matrix is obtained:

$$r_{ij}=\frac{x_{ij}}{\sqrt{{{\displaystyle \sum }}_{k=1}^m{x}_{kj}^2}}, i=1,\dots , m;j=1,\dots ,n$$

(10)

Step 2. By applying the weighting process stated below to the matrix in the first step, the v_ij matrix is obtained with the v_ij weighted normal values. w_j represents the weight of the J-th criterion or indicator.

$$v_{ij}=w_j{r}_{ij}, {\displaystyle \sum }_{j=1}^n{w}_j=1$$

(11)

Step 3. In this step, positive ideal (A*) and negative ideal (A⁻) solutions are determined:

$$A^{\ast }=\left\{\left(\underset{i}{\max } v_{ij}\left|j\in C_b),(\underset{i}{\min }{v}_{ij}\right|j\in C_c\right)\right\}=\left\{\left.v_j^{\ast }\right| j=1,2,\dots ,m\right\}$$

(12)

$$A^{-}=\left\{\left(\underset{i}{\min } v_{ij}\left|j\in C_b),(\underset{i}{\max }{v}_{ij}\right|j\in C_c\right)\right\}=\left\{\left.v_j^{-}\right| j=1,2,\dots ,m\right\}$$

(13)

When indicator j is a benefit indicator:

$$v_j^{+}=\max \left\{v_{ij}, i=1,\dots , m\right\}, v_j^{-}=\min \left(v_{ij} , i=1,\dots , m\right)$$

(14)

When indicator j is a cost indicator:

$$v_j^{-}=\max \left\{v_{ij}, i=1,\dots , m\right\}, v_j^{+}=\min \left(v_{ij} , i=1,\dots , m\right)$$

(15)

Step 4. The deviations of all alternatives from positive and negative solutions (discrimination criteria) are obtained individually using the following equations using the m-dimensional Euclidean distance:

$$S_i^{\ast }=\sqrt{{\displaystyle \sum }_{j=1}^m{\left(v_{ij}-v_j^{\ast }\right)}^2, j=1,2, \dots , m}$$

(16)

$$S_i^{-}=\sqrt{{\displaystyle \sum }_{j=1}^m{\left(v_{ij}-v_j^{-}\right)}^2, j=1,2, \dots , m}$$

(17)

Step 5. In this step, the relative proximity to the ideal solution is determined. The relative proximity of the Ai alternative with respect to A* is defined by the following equation. Then, sort results in descending RC_i.

$$R{C}_i^{\ast }=\frac{S_i^{-}}{S_i^{\ast }+S_i^{-}}, i=1,\dots , m$$

(18)

2.2.2. VIKOR (Vise Kriterijumska Optimizacija I Kompromisno Resenje) Multi-Criteria Optimization and Compromise Solution Method

VIKOR is a method proposed by Opricović [44], which aims to reach a compromise order and a consensus result within the framework of determined weights. The term consensus in this definition refers to a consensus of decision makers between options, that is, the determination of a joint decision. The process steps of the VIKOR method are as follows [45]:

Step 1. Calculating the positive ideal solution f_i* and negative ideal solution f_i⁻. I₁ is a benefit indicator, I₂ is a cost indicator.

$$\begin{gathered} f_i^{\ast }=\left[\left(\underset{j}{\max }{f}_{ij}|i\in I_1\right), \left(\underset{j}{\min }{f}_{ij}|i\in I_2\right)\right],{\forall }_i \\ {f}_i^{-}=\left[\left(\underset{j}{\min }{f}_{ij}|i\in I_1\right), \left(\underset{j}{\max }{f}_{ij}|i\in I_2\right)\right],{\forall }_i \end{gathered}$$

(19)

Step 2. Calculate the S_j and R_j of the scheme. W_i represents the weight of index i.

$$S_j={\displaystyle \sum }_i^n \frac{w_i\left(f_i^{\ast }-f_{ij}\right)}{\left(f_i^{\ast }-f_i^{-}\right)},{\forall }_j$$

(20)

$$R_j={\max }_i\left[\frac{w_i\left(f_i^{\ast }-f_{ij}\right)}{\left(f_i^{\ast }-f_i^{-}\right)}\right], {\forall }_j$$

(21)

Step 3. Calculate Q of each scheme.

$$Q_j=\frac{v\left(S_j-S^{\ast }\right)}{\left(S^{-}-S^{\ast }\right)}+\frac{\left(1-v\right)\left(R_j-R^{\ast }\right)}{\left(R^{-}-R^{\ast }\right)}, {\forall }_j$$

(22)

$$S^{\ast }=\underset{j}{\min }{S}_j;S^{-}= \underset{j}{\max }{S}_j;R^{\ast }=\underset{j}{\min }{R}_j;R^{-}= \underset{j}{\max }{R}_j$$

(23)

While v in the equation of Q_j represents the relative importance of the majority of the criteria, namely the maximum group benefit, the value of 1 − v indicates the relative value of the opponents’ minimum regret, ie the weight. By ordering the values of S, R and Q ascending, the order between the alternatives is obtained. In this evaluation process, three rankings are obtained. Two conditions must be met for the result to be valid.

Condition 1 (C1)—(Acceptable advantage):

There must be a distinct difference between the best alternative and the closest alternative. A1 has the smallest Q value i.e., the first best alternative, while A₂ is the second best alternative. The acceptable advantage in this case is shown as follows;

Q(A₂) − Q(A₁) ≥ DQ

(24)

DQ = 1/(m − 1) (m is the number of alternatives)

(25)

Condition 2 (C2)—(Acceptable stability):

In order to ensure the stability condition for the compromised solution found; A1 alternative with the highest Q value must have received the highest value from at least one of the S and R values.

Step 4. Only after these conditions are met, the alternative with the smallest Q value can be considered as the best option.

2.2.3. PROMETHEE

PROMETHEE method is a multi-criteria decision making (MCDM) method that enables the analysis of alternatives to be evaluated using preference functions selected according to the criteria. This assessment for alternatives is obtained by performing paired comparisons [46]. PROMETHEE I method, which was introduced for the first time by Mareschal, Brans [47], performs partial ordering, while the PROMETHEE II method performs full ordering. In addition, later Mareschal and Brans [48] proposed the GAIA (Geometrical Analysis for Interactive Aid) method in 1988, which supports the PROMETHEE method and can obtain graphic presentations. PROMETHEE method consists of 4 steps as follows [49,50,51]:

Step 1. Evaluate the n solutions (a1, a2, …, an) in A under m criteria C_k, and get the decision matrix X = (x_ik) (i = 1, 2, …, n; k = 1, 2, …, m). When G_k(d_ij) = 0, there is no difference between scheme a_i and scheme a_j. When G_k(d_ij) = 1, scheme a_i has definite priority over scheme a_j.

$$G_k\left(d_{ij}\right)=P_k\left(a_i, a_j\right)\in \left[0,1\right]$$

(26)

Step 2. Based on the weight (W) provided by the decision maker, a multi-criteria preference ranking index (H) is calculated.

$$H\left(a_i, a_j\right)={\displaystyle \sum }_{k=1}^m{W}_k{P}_k\left(a_i, a_j\right)$$

(27)

Step 3. The positive and negative directions of the order of A_i’s preference are defined by Φ⁺(a_i) and Φ⁻(a_i), respectively.

$${\mathsf{\Phi }}^{+}\left({\mathrm{a}}_{\mathrm{i}}\right)={{\displaystyle \sum }}_{\mathrm{j}=1}^{\mathrm{n}}\mathrm{H}\left({\mathrm{a}}_{\mathrm{i}},{\mathrm{a}}_{\mathrm{j}}\right) \mathrm{and} {\mathsf{\Phi }}^{-}\left({\mathrm{a}}_{\mathrm{i}}\right)={{\displaystyle \sum }}_{\mathrm{j}=1}^{\mathrm{n}}\mathrm{H}\left({\mathrm{a}}_{\mathrm{j}},{\mathrm{a}}_{\mathrm{i}}\right)$$

(28)

$$\mathsf{\Phi }\left({\mathrm{a}}_{\mathrm{i}}\right)={\mathsf{\Phi }}^{+}\left({\mathrm{a}}_{\mathrm{i}}\right)-{\mathsf{\Phi }}^{-}\left({\mathrm{a}}_{\mathrm{i}}\right)$$

(29)

Step 4. The exact ranking of the alternatives is obtained according to the values of Φ(a_i).

2.2.4. ELECTRE (Elimination and Choice Translating Reality English) Method

ELECTRE (Elimination and Choice Translating Reality English) method is an MCDM method proposed by Benayoun, Roy [52]. ELECTRE is a method that sorts and selects alternatives according to their paired comparison advantages in terms of each of the evaluation criteria. It has an eight-step process [53]. The steps of the ELECTRE evaluation process are as follows:

Step 1. This process converts the elements of the decision matrix into dimensionless comparable elements by applying Equation (30)

$$x_{ij}=\frac{a_{ij}}{\sqrt{{{\displaystyle \sum }}_{k=1}^{m}x{a}_{kj}^2}}$$

(30)

Thus, the normalized matrix X is shown as

$$X=\left[\begin{array}{ccc}{x}_{11}& \cdots & x_{1n}\\ ⋮& \ddots & ⋮\\ x_{m1}& \cdots & x_{mn}\end{array}\right]$$

(31)

where m presents the number of alternatives, n shows the number of criteria, and x_ij is the normalized preference measure of the i-th alternative with regard to the j-th criterion.

Step 2. Construction of weighted standard decision matrix (Y): The importance of evaluation factors may be different for each decision-maker. In order to reflect these significant differences to the ELECTRE solution, the Y matrix is calculated. The decision-maker must first determine the weights (w_i) of the evaluation factors. 0 ≤ w₁, w₂, …, w_n ≤ 1 and the correlation coefficients of normalized interval numbers are between 0 and 1.

$${\displaystyle \sum }_{i=1}^n{w}_i=1$$

(32)

Then the elements in each column of the X matrix are multiplied by the corresponding w_i value to form the Y matrix. Therefore, the weighted matrix which is derived from the normalized matrix is shown in Equation (33):

$$Y_{ij}=\left[\begin{array}{ccc}{w}_1{x}_{11}& \cdots & w_n{x}_{1n}\\ ⋮& \ddots & ⋮\\ w_1{x}_{m1}& \cdots & w_n{x}_{mn}\end{array}\right]$$

(33)

Step 3. Determining the set of concordance (C_kl) and discordance (D_kl).

The Y matrix is used to determine the fit sets. The decision points are compared with each other in terms of evaluation factors and the sets are determined by the relationship shown in the formula:

$$C_{kl}=\left\{\left.j\right|y_{kj}\ge y_{lj}\right\}$$

(34)

The formula is based on the comparison of the superiority of the row elements relative to each other. The number of concordance sets in a multiple decision problem is (m.m−m). The k ≠ l condition should be provided for k and l indices when creating concordance sets. The number of elements in a set of concordance can be the maximum number of evaluation factors (n).

For example, in order to be able to decide the C concordance set for k = 1 and l = 2, the elements of row 1 and 2 of the Y matrix are mutually compared with each other. When there are four evaluation factors, the C₁₂ concordance set will have, at most, four elements. For instance, if the comparison results of rows 1 and 2 are as follows: y₁₁ > y₂₁, y₁₂ < y₂₂, y₁₃ < y₂₃ and y₁₄ = y₂₄. The condition in formula Equation (5) will fit for the values of j = 1 and j = 4, and the C₁₂ concordance set will be defined as C₁₂ = {1,4}. The ELECTRE method has a discordance set (D_kl) which is complementary to each concordance set (C_kl). In other words, there are as many discordance sets as the number of concordance sets. The discordance set elements consist of j values that do not belong to the complementary concordance set. In the example, concordance set is C₁₂ = {1,4} therefore discordance set is D₁₂ = {2,3}.

Step 4. Construction of concordance (C) and discordance (D) matrix.

The concordance index c_kl is the sum of the weights related with the criteria included in the concordance set. ance matrices (D)

Concordance sets are used to create the concordance matrix (C). The matrix C is a mxm matrix and does not have a value for k = l. The elements of the C matrix are calculated by the relationship shown in the formula:

$$C_{kl}={\displaystyle \sum }_{j\in C_{kl}}{w}_{j }for j=1, 2, 3, \dots , n.$$

(35)

The discordance matrix (D) shows the degree that a particular alternative A_k is worse than a competing alternative A_l. The elements of the discordance matrix (D) are calculated by Equation (36):

$$d_{kl}=\frac{\underset{j\in D_{kl}}{\max }\left|y_{kj}-y_{lj}\right|}{\underset{j}{\max }\left|y_{kj}-y_{lj}\right|}$$

(36)

Moreover, both of these two mxm matrices are not symmetric.

Step 5. Determine the concordance and discordance dominance matrices. The concordance dominance matrix (F) is a mxm matrix and the elements of the matrix are obtained from the comparison of the concordance threshold (c) with the elements (c_kl) of the concordance matrix. The concordance threshold value (c) is obtained by the formula

$$c=\frac{1}{m\left(m-1\right)}{\displaystyle \sum }_{k=1}^m{\displaystyle \sum }_{l=1}^m{c}_{kl}$$

(37)

m shows the number of decision points in the formula. More specifically, the value of c is equal to the product of the total value of the elements of C matrix and $\frac{1}{m\left(m-1\right)}$.

Based on the threshold value, the elements of the concordance dominance matrix F are decided by

$$c_{kl}\ge c\Rightarrow f_{kl}=1, c_{kl}<c\Rightarrow f_{kl}=0$$

(38)

it also shows the same decision points on the diagonal of the matrix, so it has no value. In a similar way, the discordance dominance matrix G is described by using a threshold value d, where d could be explained as

$$\begin{gathered} d=\frac{1}{m\left(m-1\right)}{\displaystyle \sum }_{k=1}^m{\displaystyle \sum }_{l=1}^m{d}_{kl} \\ {d}_{kl}\ge d\Rightarrow g_{kl}=1, d_{kl}<d\Rightarrow g_{kl}=0 \end{gathered}$$

(39)

Step 6. Construction of the aggregate dominance matrix (E). Here, the E is a mxm matrix depending on the C and D matrices and it consists of 1 or 0 values.

$$e_{kl}=f_{kl}\times g_{kl}$$

(40)

Step 7. Determining the order of importance for decision points. The rows and columns of the E matrix represent the decision points. For example, if the matrix E is calculated as

$$E=\left[\begin{array}{ccc}-& 0& 0\\ 1& -& 0\\ 1& 1& -\end{array}\right]$$

$$e_{21}=1, e_{31}=1, and e_{32}=1$$

This indicates that the second alternative is preferred to the first alternative, the third alternative is preferred to the first alternative, and the third alternative is preferred to the second alternative by using both the concordance and discordance criteria. In this case, if the decision points are expressed with the symbol A_i (i = 1, 2,…, m) the order of importance for the decision points will be in the form of A₃, A₂, and A₁.

2.2.5. COPRAS (Complex Proportional Assesment) Method

COPRAS (Complex Proportional Assessment) is an MCDM method used to evaluate and rank the alternatives [54]. The evaluation steps of the approach are briefly listed below [55,56,57]:

Variables used in the COPRAS method; Ai: i-th alternative I = 1, 2, …., m; C_j: j-th criterion j = 1, 2, …., n; w_j: significance weight of the j-th criterion j = 1, 2, …, n; x_ij: j-th level of evaluation criterion j = 1, 2, …, n.

Step 1. The decision matrix formed by the x_ij values is obtained.

$$D=\begin{array}{c}{A}_1\\ A_2\\ A_3\\ .\\ A_m\end{array}\left[\begin{array}{ccccc}{x}_{11}& x_{12}& x_{13}& .& x_{1n}\\ x_{21}& x_{22}& x_{23}& .& x_{2n}\\ x_{31}& x_{32}& x_{33}& .& x_{3n}\\ .& .& .& .& .\\ x_{m1}& x_{m2}& x_{m3}& .& x_{mn}\end{array}\right]$$

(41)

Step 2. Normalized values are obtained by dividing each value in the decision matrix by the sum of the column to which it belongs.

$$X_{ij}^{\ast }=\frac{x_{ij}}{{{\displaystyle \sum }}_{i=1}^m{x}_{ij}}, {\forall }_j=1, 2, \dots , n$$

(42)

Step 3. The weighted normalized decision matrix D ‘consisting of d_ij elements calculated by multiplying the weight value (w_j) of each evaluation criterion with the normalized decision matrix values is obtained.

$$D^{\prime }=d_{ij}=x_{ij}^{\ast }\times w_j$$

(43)

Step 4. The sum of the weighted normalized decision matrix values of the benefit and cost criteria is calculated. S_i⁺ represents the sum of values in the i weighted normalized decision matrix of the utility criteria, while S_i⁻ represents the total value of the cost criteria. The formulas for calculating these values are shown in Equations (44) and (45).

$$S_{i+}={\displaystyle \sum }_{j=1}^k{d}_{ij}, j=1, 2, \dots , k$$

(44)

$$S_{i-}={\displaystyle \sum }_{j=k+1}^n{d}_{ij}, j=k+1,k+2, \dots , n$$

(45)

Step 5. In this step, the relative importance value (Q_i) of each alternative is calculated.

$$Q_i=S_{i+}+\frac{{{\displaystyle \sum }}_{i=1}^m{S}_{i-}}{S_{i-}\times {{\displaystyle \sum }}_{i=1}^m\frac{1}{S_{i-}}}$$

(46)

Step 6. The highest relative priority value is determined.

$$Q_{max}=max\left\{Q_i\right\}, {\forall }_i=1, 2, \dots , n$$

(47)

Step 7. The performance index (P_i) value of each alternative is obtained.

$$P_i=\frac{Q_i}{Q_{max}}\times \%100$$

(48)

Performance index value (P_i) which is equal to 100 is determined as the best alternative in terms of alternative evaluation criteria. The COPRAS ranking table is obtained by ranking the performance index value of each alternative in descending order.

2.2.6. ARAS (A New Additive Ratio Assessment) Method

A New Additive Ratio Assessment (ARAS) is a method suggested by Zavadskas and Turskis [58] in order to solve MCDM problems. The ARAS method compares the utility function value of each alternative with the utility function value of the optimal alternative [59]. The process of the ARAS method consists of 4 steps [58]. The first three steps of the method are the same as the COPRAS method. In the last step of the ARAS method, the optimality function value of each alternative is calculated, and thus it is possible to evaluate the alternatives.

Si represents the optimality function value of the i-th alternative. It is equal to the sum of all criterion values for each alternative.

$$S_i={\displaystyle \sum }_{j=1}^n{\widehat{x}}_{ij}, i=0, 1, \dots , m$$

(49)

The alternative with the largest S_i value is defined as the most efficient alternative. Also in this step, K_i utility degrees are obtained by dividing each S_i value by S₀ optimal function value.

$$K_i=\frac{S_i}{S_0}, i=0, 1, \dots , m$$

(50)

The relative efficiency of the utility function values (K_i) of each alternative is determined with K_i, which takes values in the range of [0,1]. An ARAS ranking table from the best alternative to the worst ranked alternative in terms of criteria is obtained by ordering the Ki values in descending order.

2.2.7. Multimoora (The Multi-Objective Optimization by Ratio Analysis) Method

The MOORA method was developed and proposed by [60]. In the literature, there are many MOORA methods, including MOORA-Ratio Method, MOORA Reference Point Approach, MOORA-Significance Coefficient, The full multiplicative form of MOORA, and MULTIMOORA method [61].

MOORA-Ratio Method. In the ratio method, the initial decision matrix values of the alternatives are normalized based on each criterion. As stated in the formula below, each data is divided by the square root of the sum of the squares of the values in the criteria to which it belongs. In this process; x_ij: the value of alternative j in terms of criterion i; j = 1, 2,…, m; m is the number of alternatives included in the analysis; i = 1, 2,…, n; n is the number of criteria included in the analysis; x_ij*: represents the normalized value of alternative j in terms of i criterion [62].

$$X_{ij}^{\ast }=\frac{x_{ij}}{\sqrt{{{\displaystyle \sum }}_{i=1}^m{x}_{ij}^2}}$$

(51)

According to the optimization approach of the method, normalized values are summed up in maximization, subtracted in minimization as expressed in the formula [62];

$$y_j^{\ast }={\displaystyle \sum }_{i=1}^g{x}_{ij}^{\ast }-{\displaystyle \sum }_{i=g+1}^{i=n}{x}_{ij}^{\ast }$$

(52)

i = 1, 2,..., g, values are utility criteria whose values are desired to be large; If i = g + 1, g + 2,..., n are the (cost) criteria whose values are desired to be small. j = 1, 2,..., m shows alternatives. y_j*; It is the total ranking value of alternative j. While ranking the alternatives by using these values, the alternative with the highest value is determined as the best alternative. On the other hand, the alternative with the lowest y_j* value is determined as the worst alternative in terms of evaluation criteria [62].

Reference Point Approach. In the MOORA Reference point approach, the best available criterion values for each criterion are determined and these values are used as reference points in the evaluation. r represents the reference value of the ith criterion. d_ij represents the distance from the reference point of the criterion to which each weighted normalized value calculated in the previous analysis belongs. The deviations of the normalized values given in the decision matrix from the reference series are calculated according to the formulation given in the equation. Then, the maximum values of these distances for each criterion are determined. The alternative with the smallest of these largest values is determined as the best alternative in terms of the criteria evaluated. Meanwhile, the final evaluation score of the ith alternative is represented by P_i [62].

$$d_{ij}=\left|r_i-x_{ij}^{\ast }\right|$$

(53)

$$P_i=\underset{\left(i\right)}{\mathrm{Min}}{(\mathrm{Max}\left|r_i{x}_{ij}^{\ast }\right.|)}_{\left(j\right)}$$

(54)

The full multiplicative form of MOORA. Brauers and Zavadskas describe this approach for MOORA analysis in the following Equation [63]:

$$U_i=\frac{A_i}{B_i}$$

(55)

$$A_i={\displaystyle \prod }_{j=1}^g{x}_{ij}^{\ast }, B_i={\displaystyle \prod }_{j=g+1}^n{x}_{ij}^{\ast }$$

(56)

U_i represents the degree of use of the ith alternative. As seen in the equation, the product of the values of the benefit criteria of the relevant alternative forms the numerator, while the product of the values of the cost criteria forms the denominator [63].

Multi-MOORA Approach. Multi-MOORA is a new MOORA approach proposed by Brauers and Zavadskas in 2010. Multi-MOORA is a method in which the MOORA ratio, reference point and full multiplicative approaches are evaluated by making a dominance comparison [27].

Absolute dominance means achieving the same rank in all MOORA approaches applied. The Multi-MOORA order, which can be an example of the concept of absolute dominance, is (1-1-1).

In general dominance expression, it is expressed as the dominance of two of the three approaches applied. Assuming an order such as a < b < c < d: (d-a-a) to (c-b-b); it is possible to evaluate that (a-d-a) has a general dominance over (b-c-b) and (a-a-d) over (b-b-c) [64].

2.2.8. SAW (Simple Additive Weighting) Method

This MCDM method proposed by Churchman and Ackoff [65] is also known as the Weighted Sum Model in the literature [66]. Compared to many other MCDM methods, it is a frequently preferred method because it has a simpler calculation process [67]. The equations used in the SAW approach are as follows [68,69,70]:

Step 1. Normalizing decision matrix values

Normalization process differs depending on whether the criteria are benefit (maximization) or cost (minimization) criteria. The formulas applied in this step are listed below [66]:

$$r_{ij}=\left\{\begin{array}{c}\frac{x_{ij}}{\max X_{ij}} i=1, \dots , m;j=1, \dots , n for benefit criteria\\ \frac{\min X_{ij}}{x_{ij}} i=1, \dots , m;j=1, \dots , n for costcriteria\end{array}\right.$$

(57)

Step 2. Determination of preference values for each alternative.

The total preference value (S_j) for each alternative is calculated by multiplying the criteria weight for each criterion with the normalized values of the relevant criterion obtained in the first step.

$$S_j={\displaystyle \sum }_{j=1}^m{w}_j{r}_{ij} i=1, \dots , m$$

(58)

w_ij: Weight of the relevant criterion.

The large value indicates that the relevant alternative should be preferred more. The relative value (S_j%) of each alternative is obtained by proportioning the S_j value of the relevant alternative to the total S_j of all alternatives.

$$S_j^{\%}=\frac{S_j}{{{\displaystyle \sum }}_{j=1}^n{S}_j}$$

(59)

The alternative with the largest S_j% value is the first alternative in the SAW ranking table and is identified as the best alternative among alternatives.

2.2.9. MAUT (Multi-Attribute Utility Theory) Method

Multi-attribute Utility Theory (MAUT) approach is one of the MCDM methods that enables the determination of the best alternative in terms of criteria by allowing qualitative and quantitative criteria to be evaluated together [71,72]. The operation process of the MAUT approach consists of two steps [73]. In the first step, the decision matrix elements are normalized.

Step 1. In the normalization process, the values of each criterion are first converted so that the best value is one (1) and the worst value is zero (0). Thus, all values must be in the range [0, 1]. This transformation is done using the following Equation [73]:

$$u_i\left(x_i\right)=\frac{x-x_i^{-}}{x_i^{+}-x_i^{-}}$$

(60)

Definitions of variables in this formula are shown below:

• $x_i^{+}:$ The largest value of the relevant criterion.
• $x_i^{-}$: The smallest value of the relevant criterion.
• $x:$ Current value of the cell under calculation.

Step 2. In the second step after normalization process, the utility values of each alternative are calculated. The formula used in the calculation of these benefit values and the definitions of the variables used are given below [73]:

$$U\left(x\right)={\displaystyle \sum }_1^m(u_i\left(x_i\right)\times w_i)$$

(61)

• U(x): Benefit value of the relevant alternative.
• u_i(x_i): The utility value of the alternative in terms of the relevant criteria.
• W_i: weight value of the relevant criterion.

2.3. K-Means Clustering Algorithm

K-means algorithm was developed by MacQueen [74] which aims to find the cluster centers, (c₁,..., c_K), in order to minimize the sum of the squared distances (Distortion, D) of each data point (xi) to its nearest cluster centre (c_k), as shown in Equation below where d is some distance function. Typically, d is chosen as the Euclidean distance. The steps of K-means algorithm are shown as follows [75]:

(1)

Initialize K centre locations (c₁, ..., c_K).

(2)

Assign each x_i to its nearest cluster centre c_k.

(3)

Update each cluster centre c_k as the mean of all x_i that have been assigned as closest to it.

(4)

Calculate $D={\displaystyle \sum }_{i=1}^n{[\underset{k=\left(1\dots k\right)}{\min }d(x_i,c_i)]}^2$.

(5)

If the value of D has converged, then return (c₁,..., c_K); else go to Step 2.

3. Results

Figure 4 shows the weights of the criteria of the STI framework obtained by the Entropy method. As a result of entropy calculations, while the criterion with the highest importance is the management criterion with a value of 14.583%, the dimensions that follow this dimension in order are the development of human capital: education (13.743%); financial and market sophistication (12.538%); economy (11.785%); R&D investment and research workforce (11.53%); energy, mining and green technology infrastructure (9.828%); information and communication technology (ICT) (7.775%); creative outputs (7.337%); institutions (7.32%); scientific publications and citations (3.561%).

If evaluated specifically in terms of indicators,

Table 3. Entropy weights of STI framework indicators.

Indicators		Weights	Indicators		Weights
C61	The intensity of local competition	0.01038	C37	Enrollment in higher education	0.00977
C51	ICT and business model building	0.01037	C107	Employment in knowledge-intensive services	0.00975
C59	Trade, competition and market scale	0.01036	C109	R&D studies financed by commercial enterprises	0.00974
C60	Work environment	0.01036	C01	Citations per publication	0.00965
C62	Scope of the market	0.01036	C110	Women’s employment	0.00965
C38	PISA scales in reading, mathematics and science	0.01035	C23	Energy density level of primary energy	0.00951
C53	ICT and organizational model building	0.01035	C47	State spending per student, at the tertiary level	0.00948
C63	Foreign market size	0.01035	C19	GDP per unit energy use	0.0094
C79	Government online service	0.01034	C17	High technology import	0.00939
C80	E-participation	0.01034	C40	Quacquarelli Symonds (QS) university rank	0.00929
C115	Scientists and engineers	0.01034	C104	R&D expenses of the top three global companies	0.00917
C35	Reading time expectation	0.01033	C02	The productivity and impact of a scientist or publication	0.00915
C44	Accessibility to the latest technologies	0.01033	C29	Fuel import	0.0091
C52	ICT related laws	0.01033	C06	Trade	0.00905
C95	Value chain width	0.01032	C67	Real GDP growth	0.00902
C111	Staff training scope	0.01032	C55	ICT services import	0.00897
C43	Internet access in schools	0.01031	C106	Total gross R&D expenditure	0.00896
C45	Local availability of customized education services	0.01031	C65	Exports of goods and services	0.00891
C113	Innovation capacity	0.01031	C68	Average monthly net income	0.00865
C97	Quality of scientific research institutions	0.0103	C105	Researchers	0.00862
C14	Production process development	0.0103	C08	Value-added of the services industry	0.00859
C12	Innovation	0.0103	C66	GDP per capita	0.00847
C49	Access to Information and Communication Technologies	0.01029	C31	CO2 emissions	0.00846
C64	Labour force participation, female	0.01029	C18	Intellectual property payments	0.00844
C96	University-industry cooperation in R&D	0.01029	C89	Trademark application	0.00839
C99	Economic cluster development	0.01027	C76	State Activity	0.00836
C100	Ease of access to credits	0.01027	C16	High technology export except for re-export	0.00816
C20	Environmental performance	0.01026	C21	ISO 14001 Environmental Certificates	0.00811
C56	ICT Development Index (IDI)	0.01025	C69	Unemployment	0.0081
C71	Transparency in government policies	0.01025	C108	R&D studies carried out by commercial enterprises	0.00804
C82	Political environment	0.01025	C93	Cultural and creative service export	0.00779
C98	State supply of high-tech products	0.01025	C77	Participation and Accountability	0.00747
C42	Quality of mathematics and science education	0.01023	C26	Renewable energy consumption	0.00734
C85	Regulatory environment	0.01022	C92	Creative goods export	0.00719
C114	R&D expenditures of companies	0.01022	C25	Renewable electric power	0.00706
C39	Science and engineering graduates	0.01021	C103	Joint venture strategic alliance opportunities	0.00703
C50	Use of Information and Communication Technologies (ICT)	0.0102	C54	ICT services export	0.00699
C81	Effectiveness of law-making institutions	0.01019	C87	Patent applications made by the citizens of the country	0.00681
C15	Competitive advantage	0.01018	C07	Added-value of agriculture, forestry and fisheries sectors	0.00679
C101	Venture capital availability	0.01018	C09	Added-value of the manufacturing sector	0.00673
C112	Country capacity to retain talent	0.01018	C30	Energy import	0.00654
C41	Quality of the education system	0.01017	C27	Alternative and nuclear energy	0.00637
C46	State funding/student, secondary school level	0.01015	C48	Foreign student mobility in higher education	0.00636
C75	Judicial independence	0.01015	C88	International patent applications	0.00602
C36	Education expenses	0.01014	C94	Creating a mobile application	0.00595
C78	Political Stability and Violence/Absence of Terrorism	0.0101	C86	Number of patent families made by nationals of the country	0.00559
C58	Investment	0.01007	C90	Industrial designs	0.00552
C57	Credit	0.01006	C83	Intellectual property usage fees not elsewhere classified	0.00538
C70	Effectiveness of government spending	0.01005	C28	Ore and metal export	0.00518
C72	Nepotism in government decisions	0.01005	C102	Venture capital agreements	0.00461
C13	Industrialization intensity	0.01002	C05	The attributional effect of scientific production	0.00375
C11	The medium and high tech industry	0.01	C04	Scientific and technical journal articles	0.00314
C73	Diversion of public funds	0.00999	C84	Intellectual property usage fees, payments	0.00288
C03	International scientific cooperation	0.00992	C32	Total greenhouse gas emissions	0.00044
C10	Value-added of industry (including construction)	0.0099	C34	Nitrous oxide emissions	0.00043
C24	Fossil fuel energy consumption	0.00985	C33	Methane emissions	0.00013
C74	Public trust in politicians	0.00981	C22	Adjusted savings: energy consumption	0.0001
C91	High technology and medium high technology production	0.00979

shows the order of entropy importance weights of STI indicators in descending order. The most important indicator is the intensity of local competition (1.038%). This is followed by ICT and business model creation (1.037%), trade, competition and market scale (1.037%), business environment (1.036%), market coverage (1.036%), PISA (Programme for International Student Assessment) scales in reading, mathematics and science (1.035%), ICT and organizational modeling (1.035%), foreign market size (1.035%), government online service (1.034%), e-participation (1.034%), scientists and engineers (1.033%) and accessibility to the latest technologies (1.033%).

When the administration, which is determined as the biggest weighted criterion with the entropy weighting, is evaluated, the government officials and politicians who perform poorly in this criterion should make government spending effective, increase transparency in government policies, to be fair in the decisions they make. Because the countries with the worst scores in the analyses performed have very low values in these indicators.

In the SAW analysis, Relative values (S_j^%) are calculated for each country. The ranking table of the SAW method is obtained by ordering these values in descending order. The country with the greatest value is the best country in terms of STI criteria. Relative values and rankings are shown in

Table 4. Relative Preference Values (Sj^%) and Ranking of Countries According to SAW Analysis.

Countries	Relative Values (Sj%)	Countries	Relative Values (Sj%)
Switzerland	0.037089	Malaysia	0.025870
Sweden	0.034867	United Arab Emirates	0.025806
Singapore	0.034675	China	0.023533
Finland	0.034031	Qatar	0.023286
United States of America	0.033255	Portugal	0.022326
Netherlands	0.032690	Czech Republic	0.021717
United Kingdom	0.032164	Spain	0.021691
Denmark	0.032146	Italy	0.019917
Germany	0.032054	Poland	0.018643
Norway	0.031168	Slovakia	0.018085
Japan	0.030770	India	0.017665
Ireland	0.029902	Thailand	0.017556
Canada	0.028795	Hungary	0.017555
France	0.028503	Indonesia	0.016312
Austria	0.028456	Russian Federation	0.015907
Belgium	0.028340	South Africa	0.015479
Israel	0.028328	Mexico	0.015257
Iceland	0.027791	Greece	0.014925
Australia	0.027334	Turkey	0.014682
South Korea	0.027189	Brazil	0.014239

. While Switzerland is the best country in the ranking, the other countries in the top five are Sweden, Singapore, Finland, and the United States of America. On the other hand, the five worst-performing countries are South Africa, Mexico, Greece, Turkey, and Brazil, respectively.

In the TOPSIS analysis, TOPSIS ideal (S_i*), negative ideal (S_i⁻) and relative proximity to Ideal solution (C_i*) values for each country were calculated, and they are shown in

Table 5. Ideal (Si*), Negative Ideal (Si⁻) and Relative Proximity to Ideal Solution (Ci*) Values.

Countries	Si*	Si−	Ci*	Countries	Si*	Si−	Ci*
Australia	0.01901831	0.015675165	0.451818822	Malaysia	0.019324005	0.015415112	0.443739316
Austria	0.01759414	0.015844413	0.473836682	Mexico	0.023845258	0.00963452	0.287771323
Belgium	0.018237038	0.015997457	0.467290579	Netherlands	0.016686914	0.018192205	0.521578684
Brazil	0.024212071	0.009532207	0.282483655	Norway	0.017836806	0.017660883	0.49752205
Canada	0.018166962	0.016290239	0.472767332	Poland	0.022064245	0.010895951	0.330579072
China	0.019285968	0.015817095	0.450590166	Portugal	0.020506789	0.013120239	0.390169449
Czech Republic	0.020290718	0.013073121	0.391835034	Qatar	0.021684527	0.015816368	0.421759747
Denmark	0.01668734	0.017805727	0.516211765	Russian Federation	0.023591792	0.009871702	0.294999141
Finland	0.016430098	0.019160181	0.538354335	Singapore	0.015959386	0.0207648	0.565425739
France	0.017870569	0.01583554	0.469812164	Slovakia	0.021791465	0.011965523	0.354460623
Germany	0.016785058	0.017801046	0.514687806	South Africa	0.023439259	0.009957359	0.298154711
Greece	0.023952375	0.010068459	0.295949799	South Korea	0.018643861	0.016121611	0.463724784
Hungary	0.022220209	0.011218068	0.335485827	Spain	0.020634427	0.012388537	0.375149154
Iceland	0.018674999	0.017218927	0.479717014	Sweden	0.015207588	0.019414981	0.560760844
India	0.022314328	0.012810881	0.364720421	Switzerland	0.014494897	0.020852494	0.589930216
Indonesia	0.023067005	0.01117253	0.326304957	Thailand	0.022229048	0.011108414	0.333211148
Ireland	0.01710535	0.01775962	0.509382914	Turkey	0.023471401	0.009656104	0.291482984
Israel	0.019077047	0.016777852	0.467937506	United Arab Emirates	0.020310685	0.016070065	0.441718904
Italy	0.0217416	0.011836049	0.352497848	United Kingdom	0.017018127	0.017925653	0.512985516
Japan	0.018065226	0.017504338	0.492115619	United States of America	0.017426885	0.019477119	0.527777934

.

Table 6. TOPSIS ranking of countries in terms of STI performances.

Countries	Ci*	Countries	Ci*
Switzerland	0.589930216	China	0.450590166
Singapore	0.565425739	Malaysia	0.443739316
Sweden	0.560760844	United Arab Emirates	0.441718904
Finland	0.538354335	Qatar	0.421759747
United States of America	0.527777934	Czech Republic	0.391835034
Netherlands	0.521578684	Portugal	0.390169449
Denmark	0.516211765	Spain	0.375149154
Germany	0.514687806	India	0.364720421
United Kingdom	0.512985516	Slovakia	0.354460623
Ireland	0.509382914	Italy	0.352497848
Norway	0.49752205	Hungary	0.335485827
Japan	0.492115619	Thailand	0.333211148
Iceland	0.479717014	Poland	0.330579072
Austria	0.473836682	Indonesia	0.326304957
Canada	0.472767332	South Africa	0.298154711
France	0.469812164	Greece	0.295949799
Israel	0.467937506	Russian Federation	0.294999141
Belgium	0.467290579	Turkey	0.291482984
South Korea	0.463724784	Mexico	0.287771323
Australia	0.451818822	Brazil	0.282483655

presents the TOPSIS ranking of countries in terms of STI performances. According to the ranking obtained as a result of TOPSIS analysis, While Switzerland is the best country in the ranking, the other countries in the top five are Singapore, Sweden, Finland and the USA, respectively. On the other hand, the five worst-performing countries, Greece, Russia, Turkey, Mexico and Brazil.

In the VIKOR analysis, the weighted and normalized Manhattan distance (S_i), the weighted and normalized Chebyshev distance (R_i), and the compromise value (Q_i) for each alternative (country) have been calculated. These values are shown in

Table 7. Manhattan distance (S_i), the weighted and normalized Chebyshev distance (R_i), and the compromise value (Q_i) obtained through VIKOR Analysis.

Manhattan Distance (Si)	Weighted and Normalized Chebyshev Distance (Ri)	Compromise Value (Qi)
S*	0.28008	R*	0.00773
S−	0.72360	R−	0.01038
		0	0.25	0.5	0.75	1
Si	Ri	Qi (v = 0)	Qi (v = 0.25)	Qi (v = 0.5)	Qi (v = 0.75)	Qi (v = 1)	Countries
0.469415	0.009050	0.498936	0.480925	0.462915	0.444904	0.426893	Australia
0.447639	0.007726	0.000000	0.094449	0.188898	0.283347	0.377796	Austria
0.449898	0.009079	0.509593	0.477917	0.446240	0.414564	0.382888	Belgium
0.723605	0.010310	0.973622	0.980216	0.986811	0.993405	1.000000	Brazil
0.441070	0.008151	0.160122	0.210837	0.261553	0.312268	0.362984	Canada
0.543208	0.009750	0.762596	0.720265	0.677933	0.635602	0.593270	China
0.578447	0.010340	0.984927	0.906875	0.828824	0.750772	0.672721	Czech Republic
0.376017	0.008625	0.338538	0.307982	0.277425	0.246869	0.216313	Denmark
0.339430	0.008577	0.320633	0.273930	0.227228	0.180525	0.133823	Finland
0.446726	0.009554	0.688793	0.610529	0.532265	0.454001	0.375737	France
0.377794	0.008178	0.170268	0.182781	0.195294	0.207807	0.220320	Germany
0.710298	0.010370	0.996232	0.989673	0.983115	0.976557	0.969998	Greece
0.659243	0.010360	0.992463	0.958069	0.923675	0.889281	0.854887	Hungary
0.460556	0.010360	0.992463	0.846077	0.699691	0.553305	0.406919	Iceland
0.657101	0.010360	0.992463	0.956862	0.921261	0.88566	0.850059	India
0.683373	0.010340	0.984927	0.966018	0.947109	0.928201	0.909292	Indonesia
0.419575	0.008385	0.248163	0.264753	0.281343	0.297932	0.314522	Ireland
0.450124	0.010380	1.000000	0.845849	0.691699	0.537548	0.383397	Israel
0.613393	0.010320	0.977390	0.920920	0.864450	0.807981	0.751511	Italy
0.402726	0.009152	0.537387	0.472173	0.406960	0.341747	0.276533	Japan
0.497835	0.009511	0.672608	0.627199	0.581789	0.53638	0.490970	Malaysia
0.703851	0.009751	0.763146	0.811225	0.859304	0.907383	0.955462	Mexico
0.365462	0.010210	0.935939	0.750083	0.564227	0.378371	0.192515	Netherlands
0.394994	0.009058	0.501843	0.441157	0.380472	0.319786	0.259100	Norway
0.638120	0.009530	0.679774	0.711646	0.743518	0.775389	0.807261	Poland
0.566633	0.009057	0.501447	0.537607	0.573766	0.609925	0.646084	Portugal
0.547996	0.010330	0.981158	0.886885	0.792611	0.698337	0.604064	Qatar
0.691227	0.010370	0.996232	0.978924	0.961616	0.944308	0.927000	Russian Federation
0.326935	0.010140	0.909561	0.708583	0.507606	0.306629	0.105652	Singapore
0.648957	0.010340	0.984927	0.946619	0.908312	0.870004	0.831696	Slovakia
0.699541	0.010350	0.988695	0.977958	0.967220	0.956482	0.945745	South Africa
0.472230	0.008743	0.382994	0.395556	0.408117	0.420679	0.433240	South Korea
0.578955	0.008884	0.436284	0.495679	0.555075	0.61447	0.673866	Spain
0.323195	0.008322	0.224501	0.192681	0.160860	0.12904	0.097219	Sweden
0.280076	0.008029	0.114225	0.085669	0.057113	0.028556	0.000000	Switzerland
0.659229	0.009796	0.779814	0.798574	0.817334	0.836095	0.854855	Thailand
0.715004	0.010350	0.988695	0.986674	0.984652	0.98263	0.980609	Turkey
0.499080	0.010150	0.913329	0.808441	0.703552	0.598664	0.493776	United Arab Emirates
0.375661	0.009424	0.639921	0.533818	0.427716	0.321613	0.215511	United Kingdom
0.354486	0.009273	0.639921	0.582902	0.479119	0.271552	0.167769	United States of America

below.

Then, calculations are made by considering five different maximum group utility (v) values. Here, 1 − v also represents a minimum of individual regret. While applying these strategies, there may be a compromise problem with the value of v = 0.5, where v = (n + 1)/2n (v + 0.5 (n − 1)/n = 1). Because the first criterion related to R is also included in S. S* and S⁻ values represent the maximum and minimum values in the Si column, while the R* and R- values show the maximum and minimum values in the Ri column. The rankings obtained according to each Qi value calculated for five different v values are shown in

Table 8. Ranking of countries according to STI criteria as a result of VIKOR analysis.

Ranking	Countries	Qi (v = 0)	Countries	Qi (v = 0.25)	Countries	Qi (v = 0.5)	Countries	Qi (v = 0.75)	Countries	Qi (v = 1)
1	Austria	0	Australia	0.480925	Switzerland	0.057113	Switzerland	0.028556	Switzerland	0
2	Switzerland	0.114225	Austria	0.094449	Sweden	0.16086	Sweden	0.12904	Sweden	0.097219
3	Canada	0.160122	Belgium	0.477917	Austria	0.188898	Finland	0.180525	Singapore	0.105652
4	Germany	0.170268	Brazil	0.980216	Germany	0.195294	Germany	0.207807	Finland	0.133823
5	Sweden	0.224501	Canada	0.210837	Finland	0.227228	Denmark	0.246869	United States of America	0.167769
6	Ireland	0.248163	China	0.720265	Canada	0.261553	United States of America	0.271552	Netherlands	0.192515
7	Finland	0.320633	Czech Republic	0.906875	Denmark	0.277425	Austria	0.283347	United Kingdom	0.215511
8	Denmark	0.338538	Denmark	0.307982	Ireland	0.281343	Ireland	0.297932	Denmark	0.216313
9	South Korea	0.382994	Finland	0.27393	United States of America	0.375335	Singapore	0.306629	Germany	0.22032
10	Spain	0.436284	France	0.610529	Norway	0.380472	Canada	0.312268	Norway	0.2591
11	Australia	0.498936	Germany	0.182781	Japan	0.40696	Norway	0.319786	Japan	0.276533
12	Portugal	0.501447	Greece	0.989673	South Korea	0.408117	United Kingdom	0.321613	Ireland	0.314522
13	Norway	0.501843	Hungary	0.958069	United Kingdom	0.427716	Japan	0.341747	Canada	0.362984
14	Belgium	0.509593	Iceland	0.846077	Belgium	0.44624	Netherlands	0.378371	France	0.375737
15	Japan	0.537387	India	0.956862	Australia	0.462915	Belgium	0.414564	Austria	0.377796
16	United States of America	0.582902	Indonesia	0.966018	Singapore	0.507606	South Korea	0.420679	Belgium	0.382888
17	United Kingdom	0.639921	Ireland	0.264753	France	0.532265	Australia	0.444904	Israel	0.383397
18	Malaysia	0.672608	Israel	0.845849	Spain	0.555075	France	0.454001	Iceland	0.406919
19	Poland	0.679774	Italy	0.92092	Netherlands	0.564227	Malaysia	0.53638	Australia	0.426893
20	France	0.688793	Japan	0.472173	Portugal	0.573766	Israel	0.537548	South Korea	0.43324
21	China	0.762596	Malaysia	0.627199	Malaysia	0.581789	Iceland	0.553305	Malaysia	0.49097
22	Mexico	0.763146	Mexico	0.811225	China	0.677933	United Arab Emirates	0.598664	United Arab Emirates	0.493776
23	Thailand	0.779814	Netherlands	0.750083	Israel	0.691699	Portugal	0.609925	China	0.59327
24	Singapore	0.909561	Norway	0.441157	Iceland	0.699691	Spain	0.61447	Qatar	0.604064
25	United Arab Emirates	0.913329	Poland	0.711646	United Arab Emirates	0.703552	China	0.635602	Portugal	0.646084
26	Netherlands	0.935939	Portugal	0.537607	Poland	0.743518	Qatar	0.698337	Czech Republic	0.672721
27	Brazil	0.973622	Qatar	0.886885	Qatar	0.792611	Czech Republic	0.750772	Spain	0.673866
28	Italy	0.97739	Russian Federation	0.978924	Thailand	0.817334	Poland	0.775389	Italy	0.751511
29	Qatar	0.981158	Singapore	0.708583	Czech Republic	0.828824	Italy	0.807981	Poland	0.807261
30	Czech Republic	0.984927	Slovakia	0.946619	Mexico	0.859304	Thailand	0.836095	Slovakia	0.831696
31	Indonesia	0.984927	South Africa	0.977958	Italy	0.86445	Slovakia	0.870004	India	0.850059
32	Slovakia	0.984927	South Korea	0.395556	Slovakia	0.908312	India	0.88566	Thailand	0.854855
33	South Africa	0.988695	Spain	0.495679	India	0.921261	Hungary	0.889281	Hungary	0.854887
34	Turkey	0.988695	Sweden	0.192681	Hungary	0.923675	Mexico	0.907383	Indonesia	0.909292
35	Hungary	0.992463	Switzerland	0.085669	Indonesia	0.947109	Indonesia	0.928201	Russian Federation	0.927
36	Iceland	0.992463	Thailand	0.798574	Russian Federation	0.961616	Russian Federation	0.944308	South Africa	0.945745
37	India	0.992463	Turkey	0.986674	South Africa	0.96722	South Africa	0.956482	Mexico	0.955462
38	Greece	0.996232	United Arab Emirates	0.808441	Greece	0.983115	Greece	0.976557	Greece	0.969998
39	Russian Federation	0.996232	United Kingdom	0.533818	Turkey	0.984652	Turkey	0.98263	Turkey	0.980609
40	Israel	1	United States of America	0.479119	Brazil	0.986811	Brazil	0.993405	Brazil	1

.

In the VIKOR analysis, acceptable advantage (1st Condition) and acceptable stability (2nd Condition) conditions are provided only for v = 1. The country rankings were obtained by ranking the Qi values calculated according to this value in ascending order.

In the ARAS method, the priority values (Si) and benefit values (Ki) of all countries were calculated, and they are shown in

Table 9. ARAS optimality function values and country rankings.

	Si	Ki	%Ki	ARAS Method Ranking (% Ki)
Optimal Value	0.059780693
Australia	0.02673	0.44713	44.71	Switzerland	63.19
Austria	0.02826	0.47281	47.28	Sweden	59.40
Belgium	0.02758	0.46129	46.13	Singapore	58.85
Brazil	0.01412	0.23617	23.62	Finland	57.10
Canada	0.02855	0.47763	47.76	United States of America	57.09
China	0.02628	0.43959	43.96	Netherlands	54.28
Czech Republic	0.02193	0.36689	36.69	Denmark	53.40
Denmark	0.03192	0.53396	53.40	United Kingdom	53.22
Finland	0.03413	0.57096	57.10	Germany	53.04
France	0.02844	0.47566	47.57	Ireland	51.39
Germany	0.03171	0.53037	53.04	Norway	50.48
Greece	0.01469	0.24581	24.58	Japan	50.37
Hungary	0.01783	0.29825	29.83	Iceland	48.28
Iceland	0.02886	0.48284	48.28	Canada	47.76
India	0.01890	0.31613	31.61	Israel	47.63
Indonesia	0.01640	0.27433	27.43	France	47.57
Ireland	0.03072	0.51389	51.39	Austria	47.28
Israel	0.02847	0.47625	47.63	South Korea	46.25
Italy	0.01953	0.32672	32.67	Belgium	46.13
Japan	0.03011	0.50369	50.37	Australia	44.71
Malaysia	0.02533	0.42374	42.37	China	43.96
Mexico	0.01451	0.24271	24.27	Malaysia	42.37
Netherlands	0.03245	0.54280	54.28	United Arab Emirates	41.70
Norway	0.03018	0.50478	50.48	Qatar	37.37
Poland	0.01814	0.30352	30.35	Czech Republic	36.69
Portugal	0.02171	0.36312	36.31	Portugal	36.31
Qatar	0.02234	0.37365	37.37	Spain	35.71
Russian Federation	0.01530	0.25596	25.60	Italy	32.67
Singapore	0.03518	0.58845	58.85	India	31.61
Slovakia	0.01873	0.31336	31.34	Slovakia	31.34
South Africa	0.01534	0.25658	25.66	Poland	30.35
South Korea	0.02765	0.46247	46.25	Hungary	29.83
Spain	0.02135	0.35711	35.71	Thailand	29.43
Sweden	0.03551	0.59404	59.40	Indonesia	27.43
Switzerland	0.03778	0.63191	63.19	South Africa	25.66
Thailand	0.01759	0.29430	29.43	Russian Federation	25.60
Turkey	0.01488	0.24889	24.89	Turkey	24.89
United Arab Emirates	0.02493	0.41697	41.70	Greece	24.58
United Kingdom	0.03181	0.53219	53.22	Mexico	24.27
United States of America	0.03413	0.57089	57.09	Brazil	23.62

. When the percentage value (% Ki) of the utility value is ordered in descending order, a table is obtained indicating the order from the best country to the worst country in terms of STI. According to the ranking in Table 9, the top five countries in terms of STI are Switzerland, Sweden, Singapore, Finland, and the United States, respectively. The five countries with the worst scores are the Russian Federation, Turkey, Greece, Mexico, and Brazil.

Ranking of the countries according to the calculated COPRAS benefit degrees is presented in

Table 10. Result matrix of the COPRAS method.

Countries	Sj+	Sj−	Ranking	Countries	Qj	Benefit Degree (Nj)
Australia	0.02673	0	1	Switzerland	0.037776	100.00
Austria	0.028265	0	2	Sweden	0.035512	94.01
Belgium	0.027576	0	3	Singapore	0.035178	93.12
Brazil	0.014118	0	4	Finland	0.034132	90.35
Canada	0.028553	0	5	United States of America	0.034128	90.34
China	0.026279	0	6	Netherlands	0.032449	85.90
Czech Republic	0.021933	0	7	Denmark	0.031921	84.50
Denmark	0.031921	0	8	United Kingdom	0.031815	84.22
Finland	0.034132	0	9	Germany	0.031706	83.93
France	0.028435	0	10	Ireland	0.030721	81.32
Germany	0.031706	0	11	Norway	0.030176	79.88
Greece	0.014695	0	12	Japan	0.030111	79.71
Hungary	0.01783	0	13	Iceland	0.028864	76.41
Iceland	0.028864	0	14	Canada	0.028553	75.59
India	0.018898	0	15	Israel	0.028471	75.37
Indonesia	0.016399	0	16	France	0.028435	75.27
Ireland	0.030721	0	17	Austria	0.028265	74.82
Israel	0.028471	0	18	South Korea	0.027647	73.19
Italy	0.019532	0	19	Belgium	0.027576	73.00
Japan	0.030111	0	20	Australia	0.026730	71
Malaysia	0.025331	0	21	China	0.026279	69.56
Mexico	0.01451	0	22	Malaysia	0.025331	67.06
Netherlands	0.032449	0	23	United Arab Emirates	0.024927	65.99
Norway	0.030176	0	24	Qatar	0.022337	59.13
Poland	0.018145	0	25	Czech Republic	0.021933	58.06
Portugal	0.021708	0	26	Portugal	0.021708	57.46
Qatar	0.022337	0	27	Spain	0.021348	56.51
Russian Federation	0.015302	0	28	Italy	0.019532	51.70
Singapore	0.035178	0	29	India	0.018898	50.03
Slovakia	0.018733	0	30	Slovakia	0.018733	49.59
South Africa	0.015339	0	31	Poland	0.018145	48.03
South Korea	0.027647	0	32	Hungary	0.017830	47.20
Spain	0.021348	0	33	Thailand	0.017593	46.57
Sweden	0.035512	0	34	Indonesia	0.016399	43.41
Switzerland	0.037776	0	35	South Africa	0.015339	40.60
Thailand	0.017593	0	36	Russian Federation	0.015302	40.51
Turkey	0.014879	0	37	Turkey	0.014879	39.39
United Arab Emirates	0.024927	0	38	Greece	0.014695	38.90
United Kingdom	0.031815	0	39	Mexico	0.014510	38.41
United States of America	0.034128	0	40	Brazil	0.014118	37.37
				Qmax	0.037775826

. According to the ranking, the country with the highest benefit rating is Switzerland. Other countries in the top five of the ranking are Sweden, Singapore, Finland, and the United States, respectively. The five countries with the worst performance score are the Russian Federation, Turkey, Greece, Mexico, and Brazil.

Visual PROMETHEE, an easily applicable program, was preferred in the application of PROMETHEE analysis. The program is an important multi-criteria decision support program designed for the implementation of the PROMETHEE method.

The weights used in the analysis are the weights obtained from the entropy analysis. In the PROMETHEE method, if there is no priority among the criteria for decision makers, the first type, namely the usual preference function, is preferred. Therefore, the preference function for all criteria has been determined as the first type (ordinary) function in order to make an evaluation using only the determined Entropy weights, regardless of subjective evaluations (without prioritizing certain value ranges for any criterion). In the analysis, when the ordinary type preference function is preferred, the values part of the parameters q (indifference value), p (absolute preference threshold) and s (intermediate value or standard deviation between p and q) are left blank.

A partial ranking of countries in terms of STI criteria is obtained with the PROMETHEE I method. When the results of PROMETHEE I analysis obtained from the Visual PROMETHEE program are evaluated, it is seen that Switzerland is more dominant than other countries. Sweden and Netherlands follow this country in dominance of other countries, respectively. The final and complete ranking is obtained with the PROMETHEE II method.

The PROMETHEE II method, obtained through the program, performs the full ranking process between countries with the net superiority value (Phi) calculated by using negative (Phi−) and positive superiority (Phi+) values. These PROMETHEE II results show positive advantage value, negative advantage value, net superiority value and the ranking of countries. According to this analysis, Switzerland ranks first among other countries as the country with the highest net Phi value in terms of STI criteria. The top five countries following Switzerland are Sweden, Netherlands, Finland, Singapore and Denmark. According to the PROMETHEE II analysis, the total performance scores of the countries are shown in

Table 11. Positive, negative, net superiority values and complete rankings of countries obtained via PROMETHEE II analysis.

Rank	Action	Phi	Phi+	Phi−	Rank	Action	Phi	Phi+	Phi−
1	Switzerland	0.6202	0.8051	0.1849	21	Malaysia	−0.0676	0.4587	0.5264
2	Sweden	0.5025	0.7446	0.2421	22	UAE	−0.0891	0.4482	0.5373
3	Netherlands	0.4494	0.7182	0.2687	23	Spain	−0.1061	0.4397	0.5458
4	Finland	0.4248	0.708	0.2832	24	Portugal	−0.1524	0.4183	0.5707
5	Singapore	0.4044	0.6975	0.2931	25	China	−0.1552	0.4169	0.5721
6	Denmark	0.3934	0.6906	0.2972	26	Czech Republic	−0.1691	0.4107	0.5798
7	UK	0.3736	0.6802	0.3066	27	Qatar	−0.1748	0.4056	0.5805
8	Germany	0.3717	0.6789	0.3072	28	Italy	−0.1904	0.3994	0.5898
9	USA	0.3632	0.6757	0.3126	29	Hungary	−0.3056	0.3432	0.6487
10	Norway	0.3085	0.6464	0.3379	30	Poland	−0.311	0.3385	0.6495
11	Japan	0.2452	0.6145	0.3693	31	India	−0.3236	0.3335	0.6571
12	Belgium	0.2117	0.6013	0.3897	32	Thailand	−0.347	0.3205	0.6675
13	Austria	0.205	0.5953	0.3903	33	Slovakia	−0.3849	0.3009	0.6859
14	Canada	0.2042	0.5951	0.3908	34	Greece	−0.4105	0.2909	0.7014
15	Iceland	0.2019	0.5972	0.3953	35	Indonesia	−0.4366	0.2752	0.7118
16	France	0.1973	0.5918	0.3945	36	S. Africa	−0.4372	0.2771	0.7143
17	Ireland	0.1829	0.5854	0.4025	37	Russia	−0.4645	0.2613	0.7258
18	Israel	0.1333	0.5603	0.427	38	Turkey	−0.4713	0.2591	0.7304
19	Australia	0.1002	0.5441	0.4438	39	Brazil	−0.4724	0.2587	0.7311
20	S. Korea	0.0512	0.5194	0.4682	40	Mexico	−0.4754	0.2541	0.7295

.

In Figure 5, the representation of PROMETHEE II analysis on GAIA plane is presented.

Alternatives (countries) in the GAIA plane are shown as points, and criteria as vectors. Also, the Decision Stick is indicated by π on the GAIA plane. The distribution of the countries to be listed in the GAIA plane is given in Figure 6. Among the countries that are tried to be listed, Switzerland, Sweden, the Netherlands and Finland are in the direction of the best compromise solution because they are in the direction of the decision stick. Mexico, Brazil and Turkey which are situated in the opposite direction of Switzerland are the countries in the worst position in the analysis. It can be said that countries located close to each other and clustered together on the GAIA plane have similar profiles in terms of STI criteria. Similarly, it can be said that the differences are large between countries that are far apart from each other on the plane. When evaluating in terms of criteria, criteria in the same direction are defined as compatible with each other, while criteria in the opposite direction are considered as opposite criteria. Based on the criteria on the GAIA plane and the positions of the countries, it is possible that decision makers may increase the number of comments made above. Showing the single criteria net flows of the countries together reveals the profiles of the countries. Countries that are close to each other in the plane are presented as examples in Figure 6 to show profile graphics in terms of criteria. It is seen that countries that are close to each other have similar profile graphics.

The ranking of countries according to the MOORA ratio approach in terms of STI criteria is included in the

Table 12. Ranking of countries according to MOORA ratio analysis.

Countries	yi*	Countries	yi*
Switzerland	0.205766	Malaysia	0.140255
Sweden	0.192934	China	0.138836
Singapore	0.192742	United Arab Emirates	0.137898
Finland	0.186021	Qatar	0.124734
United States of America	0.183597	Czech Republic	0.120329
Netherlands	0.17769	Portugal	0.120272
Denmark	0.174975	Spain	0.117603
United Kingdom	0.174363	Italy	0.107544
Germany	0.174087	Slovakia	0.101961
Norway	0.167011	India	0.101767
Ireland	0.166859	Poland	0.100334
Japan	0.165761	Hungary	0.097664
Canada	0.156292	Thailand	0.096918
France	0.155474	Indonesia	0.090221
Austria	0.155393	Russian Federation	0.0848
Iceland	0.155318	South Africa	0.084325
Israel	0.154993	Turkey	0.081553
Belgium	0.152769	Greece	0.081098
South Korea	0.15045	Mexico	0.081051
Australia	0.146962	Brazil	0.077507

. A ranking table is obtained from the best ranking country to the worst by ranking the y_i* values obtained in the previous stage in descending order. According to the ranking, the country with the best condition in terms of STI criteria is Switzerland. The countries that rank among the top five countries and follow Switzerland are Sweden, Singapore, Finland and the United States, respectively. On the other hand, the last five countries are South Africa, Turkey, Greece, Mexico and Brazil.

The final ranking of the countries obtained with the MOORA reference point analysis is shown in the

Table 13. Country rankings according to the MOORA reference point approach.

Countries	dij	Countries	dij
Switzerland	0.002071	Belgium	0.003067
Portugal	0.002082	Denmark	0.003077
South Korea	0.002223	Finland	0.003157
Germany	0.002254	Mexico	0.003215
Czech Republic	0.002283	Russian Federation	0.003252
Canada	0.002287	Hungary	0.003299
Slovakia	0.002406	Japan	0.003316
Sweden	0.002406	Poland	0.003326
Austria	0.002428	Netherlands	0.00333
Spain	0.002433	Norway	0.003604
Turkey	0.002527	India	0.003686
China	0.002559	Thailand	0.003695
Greece	0.002652	United States of America	0.003896
Ireland	0.002736	South Africa	0.003898
Brazil	0.002829	Malaysia	0.003915
France	0.002979	Singapore	0.004367
United Kingdom	0.002979	Indonesia	0.004622
Iceland	0.00299	Israel	0.004663
Italy	0.003016	United Arab Emirates	0.00469
Australia	0.003019	Qatar	0.00481

. According to the MOORA reference point approach, the country with the best score in terms of STI is Switzerland. Along with Switzerland, the top five countries are Portugal, South Korea, Germany and the Czech Republic, respectively. The last five countries on the list are Singapore, Indonesia, Israel, United Arab Emirates and Qatar.

According to the full multiplicative form of MOORA approach, the ranking of the countries and their scores are shown in

Table 14. Ranking of countries according to the full multiplicative form of MOORA approach.

Countries	Ui	Countries	Ui
Switzerland	2.15 × 10−76	Singapore	2.39 × 10−97
Sweden	1.43 × 10−78	Poland	6.87 × 10−100
Netherlands	1.49 × 10−80	Hungary	3.32 × 10−100
Denmark	7.87 × 10−81	Portugal	3.04 × 10−100
United Kingdom	9.88 × 10−82	Czech Republic	5.97 × 10−101
Germany	3.32 × 10−82	Malaysia	9.79 × 10−105
United States of America	3.10 × 10−82	Iceland	5.69 × 10−106
Finland	1.24 × 10−84	India	3.72 × 10−106
Canada	6.07 × 10−86	Greece	9.88 × 10−108
France	1.86 × 10−86	Slovakia	2.17 × 10−108
Austria	1.47 × 10−86	China	7.75 × 10−111
Ireland	1.37 × 10−86	Russian Federation	1.39 × 10−112
Norway	6.37 × 10−87	Brazil	2.06 × 10−113
Australia	1.19 × 10−89	Thailand	2.20 × 10−114
South Korea	3.08 × 10−90	South Africa	1.86 × 10−115
Israel	2.83 × 10−90	Turkey	1.42 × 10−118
Japan	7.13 × 10−91	United Arab Emirates	1.72 × 10−119
Belgium	5.31 × 10−94	Indonesia	1.72 × 10−130
Spain	7.84 × 10−96	Mexico	9.43 × 10−131
Italy	5.59 × 10−97	Qatar	7.18 × 10−155

.

According to the ranking, the country with the best value in terms of STI is Switzerland. Other countries in the top five are Sweden, Netherlands, Denmark and the United Kingdom, respectively. The five countries with the worst score are Turkey, United Arab Emirates, Indonesia, Mexico and Qatar.

Table 15. The MULTIMOORA ranking obtained using the three MOORA methods.

MOORA Ratio Method Ranking (yj*)	MOORA Reference Point Ranking (max dij)	The Full Multiplicative Form of MOORA (Uj) Ranking	MULTIMOORA	Ranking
Switzerland	Switzerland	Switzerland	Switzerland	1
Sweden	Portugal	Sweden	Sweden	2
Singapore	South Korea	Netherlands	Singapore	3
Finland	Germany	Denmark	Finland	4
United States of America	Czech Republic	United Kingdom	United States of America	5
Denmark	Canada	Germany	Denmark	6
Netherlands	Slovakia	United States of America	Netherlands	7
Ireland	Sweden	Finland	Germany	8
Germany	Austria	Canada	United Kingdom	9
United Kingdom	Spain	France	Ireland	10
Norway	Turkey	Austria	Canada	11
Iceland	China	Ireland	Norway	12
Japan	Greece	Norway	Iceland	13
Israel	Ireland	Australia	Japan	14
France	Brazil	South Korea	Israel	15
Austria	France	Israel	South Korea	16
South Korea	United Kingdom	Japan	Austria	17
Canada	Iceland	Belgium	France	18
China	Italy	Spain	Belgium	19
Belgium	Australia	Italy	Australia	20
Australia	Belgium	Singapore	China	21
Malaysia	Denmark	Poland	Malaysia	22
United Arab Emirates	Finland	Hungary	United Arab Emirates	23
Czech Republic	Mexico	Portugal	Portugal	24
Portugal	Russian Federation	Czech Republic	Czech Republic	25
Spain	Hungary	Malaysia	Spain	26
Qatar	Japan	Iceland	Italy	27
Italy	Poland	India	Qatar	28
Slovakia	Netherlands	Greece	Poland	29
Hungary	Norway	Slovakia	Hungary	30
India	India	China	Slovakia	31
Poland	Thailand	Russian Federation	India	32
Thailand	United States of America	Brazil	Thailand	33
Indonesia	South Africa	Thailand	Indonesia	34
Greece	Malaysia	South Africa	Greece	35
Russian Federation	Singapore	Turkey	Russian Federation	36
Brazil	Indonesia	United Arab Emirates	Brazil	37
Turkey	Israel	Indonesia	Turkey	38
South Africa	United Arab Emirates	Mexico	South Africa	39
Mexico	Qatar	Qatar	Mexico	40

y_j* represents the total ranking value of alternative j.

presents the MULTIMOORA ranking obtained using the three MOORA methods.

In the ELECTRE method, firstly, the consistency matrix is obtained. After creating the consistency matrix, the inconsistency matrix is created. Then, the necessary evaluation matrix values for all cells are obtained. In the last step, the dominance values are calculated by collecting the row and column values of each country using the evaluation table. Finally, the values obtained by calculating the row and column difference for each country are listed in descending order. Thus, ELECTRE ranking of countries is obtained [76]. All of these values and the order are presented in

Table 16. Dominance table and ELECTRE ranking.

	Dominance on Line (L)	Dominance in the Column (C)	Difference (L-C)	Countries	Ranking	Countries	Score
A1	23	15	8	Australia	1	Switzerland	34
A2	25	15	10	Austria	2	Sweden	32
A3	23	8	15	Belgium	3	United States of America	30
A4	3	14	−11	Brazil	4	Singapore	29
A5	26	12	14	Canada	5	United Kingdom	29
A6	18	10	8	China	6	Netherlands	26
A7	16	10	6	Czech Republic	7	Japan	24
A8	34	13	21	Denmark	8	Norway	24
A9	35	12	23	Finland	9	Finland	23
A10	26	11	15	France	10	Germany	23
A11	33	10	23	Germany	11	Denmark	21
A12	8	9	−1	Greece	12	Ireland	17
A13	11	9	2	Hungary	13	Iceland	16
A14	22	6	16	Iceland	14	South Korea	16
A15	7	4	3	India	15	Belgium	15
A16	0	9	−9	Indonesia	16	France	15
A17	26	9	17	Ireland	17	Canada	14
A18	21	8	13	Israel	18	Israel	13
A19	13	8	5	Italy	19	Austria	10
A20	32	8	24	Japan	20	Malaysia	10
A21	17	7	10	Malaysia	21	Portugal	9
A22	2	7	−5	Mexico	22	Australia	8
A23	33	7	26	Netherlands	23	China	8
A24	30	6	24	Norway	24	Spain	8
A25	12	5	7	Poland	25	United Arab Emirates	8
A26	14	5	9	Portugal	26	Poland	7
A27	11	4	7	Qatar	27	Qatar	7
A28	8	10	−2	Russian Federation	28	Czech Republic	6
A29	34	5	29	Singapore	29	Italy	5
A30	8	5	3	Slovakia	30	India	3
A31	2	4	−2	South Africa	31	Slovakia	3
A32	20	4	16	South Korea	32	Thailand	3
A33	15	7	8	Spain	33	Hungary	2
A34	36	4	32	Sweden	34	Turkey	1
A35	37	3	34	Switzerland	35	Greece	−1
A36	6	3	3	Thailand	36	Russian Federation	−2
A37	3	2	1	Turkey	37	South Africa	−2
A38	17	9	8	United Arab Emirates	38	Mexico	−5
A39	31	2	29	United Kingdom	39	Indonesia	−9
A40	31	1	30	United States of America	40	Brazil	−11

.

According to the ranking, the country with the best value in terms of STI is Switzerland. Other countries in the top five are Sweden, the USA, Singapore and the United Kingdom, respectively. The five countries with the worst score are Russia, South Africa, Mexico, Indonesia and Brazil.

In the MAUT method, the country with the highest benefit value is the best country in terms of STI criteria. Therefore, country rankings obtained as a result of the MAUT analysis are the same as the rankings of the SAW analysis. Benefit values and country rankings of MAUT analysis are shown in

Table 17. Ranking and benefit values of countries according to MAUT analysis.

Ranking	Countries	MAUT Benefit Values (Ux)	Ranking	Countries	MAUT Benefit Values (Ux)
1	Switzerland	0.719924	21	Malaysia	0.502165
2	Sweden	0.676805	22	United Arab Emirates	0.50092
3	Singapore	0.673065	23	China	0.456792
4	Finland	0.66057	24	Qatar	0.452004
5	United States of America	0.645514	25	Portugal	0.433367
6	Netherlands	0.634538	26	Czech Republic	0.421553
7	United Kingdom	0.624339	27	Spain	0.421045
8	Denmark	0.623983	28	Italy	0.386607
9	Germany	0.622206	29	Poland	0.36188
10	Norway	0.605006	30	Slovakia	0.351043
11	Japan	0.597274	31	India	0.342899
12	Ireland	0.580425	32	Thailand	0.340771
13	Canada	0.55893	33	Hungary	0.340757
14	France	0.553274	34	Indonesia	0.316627
15	Austria	0.552361	35	Russian Federation	0.308773
16	Belgium	0.550102	36	South Africa	0.300459
17	Israel	0.549876	37	Mexico	0.296149
18	Iceland	0.539444	38	Greece	0.289702
19	Australia	0.530585	39	Turkey	0.284996
20	South Korea	0.52777	40	Brazil	0.276395

, which was created by using Table 4.

In the cluster analysis, the dendrogram structure of the countries was reached by analyzing the data belonging to 115 criteria and 10 dimensions by hierarchical clustering analysis which is one of the data mining classification methods. The dendrogram is shown in Figure 7.

With the evaluation of the dendrogram, it was decided to divide the countries into 3 groups with k-means clusters analysis. In this way, we have the opportunity to see countries that are similar in terms of evaluation criteria.

Table 18. Number of cases in each cluster.

Number of Cases in each Cluster
Cluster	1	17.000
	2	11.000
	3	12.000
Valid		40.000
Missing		0.000

shows the number of cases in each cluster.

Table 19. Cluster memberships of countries according to K-means clusters analysis.

Cluster Membership
Case Number	Countries	Cluster	Distance
1	Brazil	1	0.017
2	Czech Republic	1	0.019
3	China	1	0.024
4	Indonesia	1	0.021
5	South Africa	1	0.020
6	India	1	0.025
7	Spain	1	0.015
8	Italy	1	0.018
9	Hungary	1	0.016
10	Mexico	1	0.014
11	Poland	1	0.011
12	Portugal	1	0.019
13	Russian Federation	1	0.016
14	Slovakia	1	0.017
15	Thailand	1	0.015
16	Turkey	1	0.016
17	Greece	1	0.020
18	Germany	2	0.013
19	United States of America	2	0.021
20	United Arab Emirates	2	0.021
21	United Kingdom	2	0.014
22	Netherlands	2	0.014
23	Ireland	2	0.019
24	Switzerland	2	0.020
25	Japan	2	0.017
26	Qatar	2	0.028
27	Malaysia	2	0.020
28	Singapore	2	0.021
29	Australia	3	0.017
30	Austria	3	0.012
31	Belgium	3	0.014
32	Denmark	3	0.011
33	Finland	3	0.016
34	France	3	0.013
35	South Korea	3	0.024
36	Israel	3	0.021
37	Sweden	3	0.014
38	Iceland	3	0.023
39	Canada	3	0.015
40	Norway	3	0.014

presents the cluster memberships of countries according to K-means clusters analysis.

Table 20. Ranking of countries according to Multi-Criteria Decision Making Methods.

TOPSIS	VIKOR	ARAS	COPRAS	MOORA Ratio
Switzerland	Switzerland	Switzerland	Switzerland	Switzerland
Singapore	Sweden	Sweden	Sweden	Sweden
Sweden	Singapore	Singapore	Singapore	Singapore
Finland	Finland	Finland	Finland	Finland
USA	USA	USA	USA	USA
Netherlands	Netherlands	Netherlands	Netherlands	Denmark
Denmark	United Kingdom	Denmark	Denmark	Netherlands
Germany	Denmark	United Kingdom	United Kingdom	Ireland
United Kingdom	Germany	Germany	Germany	Germany
Ireland	Norway	Ireland	Ireland	United Kingdom
Norway	Japan	Norway	Norway	Norway
Japan	Ireland	Japan	Japan	Iceland
Iceland	Canada	Iceland	Iceland	Japan
Austria	France	Canada	Canada	Israel
Canada	Austria	Israel	Israel	France
France	Belgium	France	France	Austria
Israel	Israel	Austria	Austria	South Korea
Belgium	Iceland	South Korea	South Korea	Canada
South Korea	Australia	Belgium	Belgium	China
Australia	South Korea	Australia	Australia	Belgium
China	Malaysia	China	China	Australia
Malaysia	UAE	Malaysia	Malaysia	Malaysia
UAE	China	UAE	UAE	UAE
Qatar	Qatar	Qatar	Qatar	Czechia
Czechia	Portugal	Czechia	Czechia	Portugal
Portugal	Czechia	Portugal	Portugal	Spain
Spain	Spain	Spain	Spain	Qatar
India	Italy	Italy	Italy	Italy
Slovakia	Poland	India	India	Slovakia
Italy	Slovakia	Slovakia	Slovakia	Hungary
Hungary	India	Poland	Poland	India
Thailand	Thailand	Hungary	Hungary	Poland
Poland	Hungary	Thailand	Thailand	Thailand
Indonesia	Indonesia	Indonesia	Indonesia	Indonesia
South Africa	Russia	South Africa	South Africa	Greece
Greece	South Africa	Russia	Russia	Russia
Russia	Mexico	Turkey	Turkey	Brazil
Turkey	Greece	Greece	Greece	Turkey
Mexico	Turkey	Mexico	Mexico	South Africa
Brazil	Brazil	Brazil	Brazil	Mexico
Switzerland	Switzerland	Switzerland	Switzerland	Switzerland
Sweden	Sweden	Sweden	Sweden	Sweden
Singapore	Netherlands	Singapore	Singapore	USA
Finland	Finland	Finland	Finland	Singapore
USA	Singapore	USA	USA	United Kingdom
Denmark	Denmark	Netherlands	Netherlands	Netherlands
Netherlands	United Kingdom	United Kingdom	United Kingdom	Japan
Germany	Germany	Denmark	Denmark	Norway
United Kingdom	USA	Germany	Germany	Finland
Ireland	Norway	Norway	Norway	Germany
Canada	Japan	Japan	Japan	Denmark
Norway	Belgium	Ireland	Ireland	Ireland
Iceland	Austria	Canada	Canada	Iceland
Japan	Canada	France	France	South Korea
Israel	Iceland	Austria	Austria	Belgium
South Korea	France	Belgium	Belgium	France
Austria	Ireland	Israel	Israel	Canada
France	Israel	Iceland	Iceland	Israel
Belgium	Australia	Australia	Australia	Austria
Australia	South Korea	South Korea	South Korea	Malaysia
China	Malaysia	Malaysia	Malaysia	Portugal
Malaysia	UAE	UAE	UAE	Australia
UAE	Spain	China	China	China
Portugal	Portugal	Qatar	Qatar	Spain
Czechia	China	Portugal	Portugal	UAE
Spain	Czechia	Czechia	Czechia	Poland
Italy	Qatar	Spain	Spain	Qatar
Qatar	Italy	Italy	Italy	Czechia
Poland	Hungary	Poland	Poland	Italy
Hungary	Poland	Slovakia	Slovakia	India
Slovakia	India	India	India	Slovakia
India	Thailand	Thailand	Thailand	Thailand
Thailand	Slovakia	Hungary	Hungary	Hungary
Indonesia	Greece	Indonesia	Indonesia	Turkey
Greece	Indonesia	Russia	Russia	Greece
Russia	South Africa	South Africa	South Africa	Russia
Brazil	Russia	Mexico	Mexico	South Africa
Turkey	Turkey	Greece	Greece	Mexico
South Africa	Brazil	Turkey	Turkey	Indonesia
Mexico	Mexico	Brazil	Brazil	Brazil

lists the results of ranking using TOPSIS, VIKOR, PROMETHEE, ELECTRE, ARAS, COPRAS, SAW and MAUT. Although there are some differences in the rankings of MCDM methods, the rankings show significant consistency in general. MOORA reference point and full multiplicative form of MOORA are used to obtain MULTIMOORA ordering. Therefore, only the MOORA ratio and MULTIMOORA rankings are taken into account among the MOORA rankings.

Table 21. Ranking of the countries in the study according to the Global Innovation and the Global Competitiveness Indices.

Global Innovation Index (2019)	Global Competitiveness Index (2019)
Switzerland	Singapore
Sweden	USA
USA	Netherlands
Netherlands	Switzerland
United Kingdom	Japan
Finland	Germany
Denmark	Sweden
Singapore	United Kingdom
Germany	Denmark
Israel	Finland
South Korea	South Korea
Ireland	Canada
China	France
Japan	Australia
France	Norway
Canada	Israel
Norway	Austria
Iceland	Belgium
Austria	Spain
Australia	Ireland
Belgium	United Arab Emirates
Czech Republic	Iceland
Spain	Malaysia
Italy	China
Portugal	Qatar
Hungary	Italy
Malaysia	Czech Republic
United Arab Emirates	Portugal
Slovakia	Poland
Poland	Thailand
Greece	Slovakia
Thailand	Russia
Russia	Hungary
Turkey	Mexico
India	Indonesia
Mexico	Greece
South Africa	South Africa
Qatar	Turkey
Brazil	India
Indonesia	Brazil

shows the rankings according to the Global Innovation and Global Competitiveness Indices. When the rankings in Table 20 and Table 21 are evaluated together, it is seen that the rankings are quite consistent.

Country comparisons and evaluations show that economic growth and the spread of a wealth system based on knowledge, transparency in management, sustainable infrastructure in technology and human resources education create significant differences between countries. The countries at the end of the rankings have very low values in these indicators.

Switzerland ranks first in the rankings for all methods. Sweden ranks second in all rankings except TOPSIS ranking, third in TOPSIS ranking. Singapore is in the top three in all rankings except PROMETHEE II and ELECTRE. It is in the fifth place in the PROMETHEE II ranking.

According to the evaluation results in terms of the performance of STI indicators, the countries in the top ten by achieving the best performance scores are Switzerland, Sweden, Singapore, Finland, USA, Denmark, Netherlands, Germany, United Kingdom and Ireland, respectively. When the countries with low performance scores are evaluated, Brazil ranked last in the rankings obtained from TOPSIS, VIKOR, ARAS, COPRAS, SAW, MAUT and ELECTRE methods; In the PROMETHEE II ranking, it is second to last.

Mexico ranks last in the MOORA ratio approach, MULTIMOORA and PROMETHEE II rankings. When evaluating poor performing countries in general, South Africa, Greece, Turkey, Mexico and Brazil are at the last five in almost all rankings.

Also, the countries in the last ten order show significant consistency. According to the analyses made, countries in the last ten places are India, Thailand, Hungary, Indonesia, South Africa, Russia, Greece, Turkey, Mexico and Brazil. While the order of the ARAS and COPRAS methods is exactly the same, the order of the SAW and MAUT methods is exactly the same. In general, when all the rankings of all methods are evaluated, the country rankings are similar.

The countries with lower scores in the analysis and at the bottom of the lists have very low values in indicators that evaluate the trade volume, innovative policies to improve market share, competitive advantage, industrialization intensity, technological development of production processes, and the added value of all sectors, especially industry, technology exports and intellectual property rights. According to the values of indicators, in these countries, the private sector does not make the expected investment in terms of R&D. Most of the investments made by commercial enterprises also consist of public incentives. Therefore, it is suggested that companies should increase their R&D expenditures.

4. Discussion

The most important limitation of this study is the absence of a comprehensive database that would enable all countries or a significant majority to be evaluated together. In order to solve this problem, it is necessary to determine common indicators and to create a common database. The novelty of the study is that it both tries to show the current situation of 40 countries with relevant indicators and proposes an integrated decision support system to improve this situation.

STI policy indicators include many different areas and are in conflict with each other. This feature shows that multi-criteria decision-making methods should be used in this field. The starting point of this study is the need for the STI policy framework and the need to use new methods in this field. Using MCDM methods and data mining cluster analysis together in this field is a novelty proposed by the study. In general, countries are tried to be evaluated with descriptive statistical methods in studies on a similar subject. It is seen that the results obtained by analyzing the 40 countries determined with 10 criteria and 115 criteria using 10 MCDM methods and cluster analysis are generally consistent. Subjective assessment is not used in any of the calculation processes of these MCDM methods.

When all these results are evaluated together, the results obtained from this study, which was carried out by taking into account more indicators and dimensions, are consistent with the results of the indices that conduct global research and evaluate with their own methodologies.

The PROMETHEE method is a little more advantageous than other methods in terms of visually evaluating both the similarities of countries and similar and different country groups in terms of indicator values. Also, the Multi-MOORA method gets a ranking based on the rankings made by three MOORA methods. Therefore it stands out a bit more than other methods in terms of reliability. In the overall assessment, the rankings of all methods are quite consistent.

Since MCDM methods such as the Analytic Network Process (ANP) and Analytic Hierarchy Process (AHP) are based on subjective evaluations, very different results can be obtained in different analyses for the same indicators. If even the criterion weights are determined by subjective evaluations, the results obtained by the same methods for the same indicators will be different. In this study, the MCDM methods and cluster analysis were used, with the criteria being objectively weighted and not requiring any subjective evaluation, and only processing and evaluating the criteria values. In other words, the evaluations were carried out in a completely objective framework. Due to the computational differences that the methods have, there are some differences in the rankings obtained. However, when the results obtained from all methods are generally evaluated, the results are consistent with each other and the other global evaluation rankings.

Determining and comparing the most developed countries in terms of STI, has been discussed in many studies. In the Introduction section of the study, MCDM studies carried out on this or similar subjects in the last five years were examined. Considering the number of studies, the methods they used, and their applications, it is seen that the application of this study brings a novelty to the field with its scope, methods, and the number of countries it includes. As a future study proposal, different methods may be applied to solve this problem and the outputs may be compared with this article. Also, it is recommended to repeat this study in the following years in order to see the performance of these countries in terms of STI policies.

5. Conclusions

The article evaluates 40 countries, mostly European, based on 115 indicators related to (STI) science, technology and innovation. In addition, while the article aims to demonstrate the current status of these countries, it also aims to offer an integrated decision support framework in order to contribute to this field. Indicators used in the application of study include very important topics in terms of sustainable economy and development of countries. The fact that a country has a bad score in terms of STI also means that the stability and sustainability of that country’s productivity, development, market share, and employment are under threat.

With the evaluation of the dendrogram in cluster analysis, it was decided to divide the countries into 3 groups with k-means clusters analysis. In this way, we have the opportunity to see countries that are similar in terms of evaluation criteria. When the countries in the obtained clusters are evaluated, it is seen that the countries with similar scores in the rankings obtained by MCDM methods are together. Consequently, it is observed that the results of the clustering analysis from the data mining classification method with the MCDM rankings show quite consistency.

Among the top ten countries in the overall assessment in Table 20, only Ireland is not among the top ten countries in the list of the Global Innovation Index. Ireland ranks twelfth on this list. Switzerland and Sweden are again in the first two places of these rankings, and they show consistency with the results of the study. When we compare the Global Competitiveness Index with our analyses results, all countries in the top ten, except Ireland, are among the top ten countries in this index. Japan, which is in the top ten in this index, generally ranks 11th or 12th when the results of the MCDM methods are evaluated. When the countries that show poor performance in terms of STI indicators are evaluated according to the results of the study with the rankings of these methods, all of the ten countries with the worst performance, except Hungary, are also in the last ten in the Global Innovation Index. According to the list of the Global Competitiveness Index, all of these countries except Thailand are among the last ten countries. Slovakia, which is among the last ten countries in the list instead of Thailand, is ranked 11th from the last in the results of the study, while Thailand is ranked 11th from the last in the index ranking.

It is seen that Northern European countries such as Sweden, Finland, Denmark, United Kingdom, Ireland and Norway come to the fore in the rankings based on STI indicators. These countries are generally among the top ten countries in all rankings. Therefore, this region seems to be ahead of other regions in terms of STI performance and policies in the global sense. Also, Switzerland is the leading country in the rankings. The Netherlands and Germany are other European countries that rank in the top ten. In general, the countries in the top ten and not in the European continent are the United States of America and Singapore.

When the results of the study and other global indices are evaluated, it can be said that the European continent is ahead of other regions in terms of STI policies and performances. These countries have very high values compared to other countries, based on the following indicator values used in the study and normalized by compressing between 0–1: the quality of the education system, competitive advantage, environmental performance, innovation capacity, ICT and business model creation, GDP per capita, R&D expenditure of companies, access and use of ICT, regulatory environment, patent applications, political stability and absence of violence/terrorism, participation and accountability, employment in information-intensive services, university-industry cooperation in research and development, accessibility to the latest technologies, R&D expenditures of the first three global companies, quality of mathematics and science education, economic cluster development.

When the last ten countries of the rankings are evaluated, there is no specific region or continent that stands out. However, in the rankings, the same countries are generally in the last rows. These countries are India, Thailand, Hungary, Indonesia, South Africa, Russia, Greece, Turkey, Mexico and Brazil. These countries have very low values in terms of these following indicators used in the study and normalized by compressing between 0–1: creative goods export, unemployment, researchers, total gross R&D expenditure, innovation, competitive advantage, high technology exports, renewable energy consumption, international scientific cooperation, internet access in schools, ICT and business model creation, ICT services export, investment, average monthly net income, participation and accountability, political stability, patent applications, added value of the manufacturing sector, high technology and medium high technology production, university-industry cooperation in R&D, and quality of scientific research institutions. Furthermore, these countries perform very low performance in terms of education. These countries have insufficient education expenditures per student; students obtained low PISA scores in reading, mathematics and science; they do not have the necessary trained workforce in the field of science and engineering; it is observed that the education system and universities’ quality evaluations at the global level cannot meet the necessary criteria and get low scores.

Almost all of the countries at the bottom of the rankings perform poorly on issues such as GDP per capita, real GDP growth, average monthly net income, unemployment, and female labour force participation. Consequently, these countries should produce policies that will give priority to the fields of administration, development of human capital: education, finance and market development, R&D investment and research workforce, which are the criteria with the largest entropy weight in the study. The results of the study shows that societies and management approaches should be more inclusive. The study also covers sustainable environment and renewable energy issues with 16 indicators in the dimension of ‘energy, mining and green technology infrastructure’. It emphasizes the need for a sustainable productivity and efficiency policy by addressing education, creative sectors and R&D. Thus, it has the vision to shape the future with its assessment dimensions.