Methodology for Calculating the European Innovation Scoreboard—Proposition for Modification

Abstract

The primary purpose of this article is to identify determinants affecting the Summary Innovation Index and, consequently, the positions of countries on the European Innovation Scoreboard (EIS). Then, based on the identified determinants, these countries are ranked using the linear ordering method. This article presents the concept of innovation as an unwavering subject of interest for researchers from around the world. Issues relating to measuring innovation, which is necessary for the efficient management of an organization, as well as to the study of innovation in individual countries, are discussed. Special attention is drawn to the methodological aspects of constructing the European Innovation Scoreboard (EIS). The identification of determinants affecting the level of the Summary Innovation Index is performed through stepwise regression. This makes it possible to reduce the number of factors utilized in the EIS ranking procedure from 27 to 22. This was the inspiration to apply an innovative approach and use the linear ordering method, in order to show that it is possible to obtain a ranking that is very similar to the EIS ranking with a reduced number of indicators. These results may be significant, both for units developing this type of ranking and for users, such as decision-makers, using the results to make strategic decisions. In our opinion, this innovative approach—that is, using the linear ordering method and a reduced number of indicators—makes it possible to create a more transparent EIS ranking. This article is of theoretical, methodological, and empirical interest.

1. Introduction

In the face of global competition, rapid technological progress, and resource scarcity, the implementation of innovations has become a condition for development, competitiveness and, frequently, for survival. Therefore, supporting innovation remains the main challenge for managers and those who govern, while the very concept of innovation for enterprises, regions, and countries validly considered an important area of research [1,2,3,4,5], as the influence of innovation on economic development is undeniable. The motivation for increasing interest in issues relating to innovation can be seen, on the one hand, in the heterogeneity and diversity of its understanding and, on the other hand, through its roles as an accelerator of change, improvement, success, and wealth. Aside from being a complex economic category, it is also the most important factor in achieving sustainable growth, based on principles that take into account economic, social, and environmental (ecological) aspects. At the same time, the promotion of sustainable development serves to strengthen the competitiveness of the economy, especially an economy based on knowledge and innovation [6,7]. Furthermore, fostering innovation is one of the aims of the 2030 Agenda for Sustainable Development (Goal 9) [8,9].

Innovations are utilized in every area of the economy and are the main factors of economic growth [10]; they are a source of technological progress and a factor in long-term competition. Although the term was initially assigned to enterprises and entrepreneurs, innovation has also appeared, in the literature, in the context of the economy as a whole [11,12]. The ability to innovate allows for the transformation of knowledge into innovations and the creation of new solutions, contributing to the long-term growth of enterprises and the level of societal prosperity [13,14]. Hence, it is important to concentrate effort into creating and supporting responsible innovations that support enterprises, humanity, and the entire planet. As Wiśniewska and Świadek have observed, the global demand for new products is high and growing exponentially, as new knowledge-based economies (unlike traditional industries) generate high income and higher employment rates. Research conducted in OECD countries has proved that, from the 1970s to the present, more than half of the economic growth of the member states has been the result of innovation and a growing share in the knowledge-based economy [15]. It is important to note that innovations largely depend on a proper market environment (i.e., one that is conducive to innovation) and it should be remembered that improving and increasing the innovativeness of a country, consequently, also means improving the innovativeness of enterprises.

Shaping an innovative environment may be associated with legal regulations facilitating collective innovation [7], decisions on financial support at the level of individual EU countries, leveling out differences between countries, and bottom-up support for the processes, products, foreign investments, and sectors [16] that create the desired innovativeness. It is also about making strategic and operational decisions in the field of commercial activity, such as initiatives for international co-operation in broadly understood innovative activities.

Thus, it is extremely important to have reliable information that allows decision-makers to make appropriate decisions, in this regard. These decisions should be supported by the evaluation and measurement of the existing activities in the field of (broadly understood) innovation, which require the use of appropriate measurement methods. As the scope of innovative activity can be assessed at the macro-, meso-, and micro-levels and within different cross-sections, a variety of measurement tools have emerged, which are of great interest to both theoreticians and practitioners [8,17,18,19,20,21,22]. The main group of methods used to assess the level of innovation are statistical and mathematical methods, based on various indicators and measures. On the other hand, the effects of these applied measurement methods have been presented, in the form of various types of report. At the EU level, such reports include the European Innovation Scoreboard (EIS), the Global Innovation Index (GII), the World Economic Forum, The Global Competitiveness Report, and the Summary Innovation Index.

These reports use a set of innovation indicators that allow—with a better or worse result—for assessment of the position of individual countries, in terms of innovation [23]. Their growing importance results from the corresponding importance of innovation as one of the main policy areas and efforts of EU members, and the need to evaluate decisions made, as well as to evaluate the development of the economy and society. It also seems that the search for answers to questions such as “what is the reason for differences in the level of innovation” and “what mechanisms should be implemented to achieve a sustainable competitive advantage” is not without significance.

It seems important too that such studies can provide an indication of entrepreneurial socio-economic orientation, as well as contributing to shaping so-called behavioral proactivity [3], which can be defined as constantly looking for opportunities, experimenting with potential responses to changing market trends, staying ahead of competitors, or eliminating strategic actions that are in the maturity or decline phase of their life-cycle.

Hence, it is worth considering the credibility and excessive complexity of the methodology of this procedure [24], as well as the low readability of both the process and of the report as final product. These reports do not specify the key selection of indicators for measurement and whether they significantly determine the impact on the examined variable; moreover, they use statistical research methods which are sometimes of low reliability. There are no clear indications as to the justifiability of the number and precision of the indicators used, which may lead to divergent results. It is also not fully known how the large number of variables relating to various areas of the economy affect each other. It also appears that the set of indicators utilized may be limited.

For this reason, the aim of this publication is to identify determinants affecting the Summary Innovation Index result and, consequently, the positions of countries on the European Innovation Scoreboard (EIS). Then, based on the identified determinants, the countries are ranked using the linear ordering method. This is due to the reduction of some indicators that do not affect the value of the Summary Innovation Index, basing the ranking on those indicators that are necessary and important, from the point of view of comparing the level of innovation in European countries. In the first stage of the study, the stepwise regression method was used, which allowed for assessment of the impact of 27 indicators on the ranking and identification of 22 determinants that affect the positions of individual countries on the European Innovation Scoreboard. In the second stage, the linear ordering method was applied and a new ranking was built using the determinants identified in the first stage. In the next stage, the resulting innovation level ranking was compared to the actual ranking from 2020, using the Kendall coefficient of concordance and test, examining its significance; as a result, a very high concordance was obtained between the rankings. In the last step, an attempt was made to divide individual countries into four groups, according to the degree of innovation. For this purpose, we used statistical criteria based on the mean and standard deviation. Finally, the consistency of the structure of these groups with those created on the basis of the European Innovation Scoreboard was examined.

The value of the considerations presented in this article lies in the possibility of reducing the number of indicators used to measure and compare the level of innovation in European countries and in an attempt to use the linear ordering method to create this type of ranking. These results may be important, both for the units developing this type of ranking, as well as for all users, such as decision-makers.

Creating new knowledge requires the recognition of its existing state and reference to previous research [25] as well as the use of differentiated research methods. Therefore, the course of developing this scientific study—which, due to the adopted goal, has a new and unique character—called for the utilization of a classic literature review, a narrative review as an auxiliary tool and starting point for further analysis, a critique of the literature [26] and structural and causal analyses as operational methods, obtaining a product by breaking down the whole into smaller elements [27]. The stage of creating a literature review consisted of the formulation of the problem, data collection, data evaluation, data analysis and interpretation, data presentation, and indication of further research directions [28,29,30]. All calculations were based on statistical data included in the European Innovation Scoreboard 2020—Methodology report [31].

2. Innovation and Its Measurement—Conceptual Background

2.1. Innovations—Theoretical Approach

The issue of innovation and innovativeness has become an unwavering subject of interest for researchers from around the world [2,32,33], a trend which is expected to grow in the foreseeable future [34]. This is due, inter alia, to the fact that, as a growth stimulant [1,12,35] and an initiator of change [36,37,38], innovation allows for the achievement of long-term competitive advantage [39,40,41,42,43,44,45,46] and constitutes a source of organizational effectiveness [47,48,49] However, it should be noted that the study conducted by Lim et al. indicated that implementing reactive innovations can negatively affect the results of companies [50].

As contemporary researchers have emphasized, innovation is conducive to achieving sustainable development [6,8,51,52,53]. As rightly noted in the 2030 Agenda for Sustainable Development, inclusive and sustainable industrialization, supported by innovation and infrastructure, can contribute to increased competitiveness and the finding of solutions to increase the efficiency of resource utilization [9]. Although the problem of innovation is not new [40,54,55] we can still notice the heterogeneity and interpretation difficulties it creates for researchers and, thus, difficulties also in its evaluation. It seems that, due to the nature of innovation, there has not been, and there is not, a definition and measurement method and it will be very difficult to unify these attributes. Innovations are the result of scientific and technical progress; although the present definition goes far beyond the sphere of technology, as innovation takes place when there is the economically successful exploitation of new ideas [56]. In general, innovations must contain an element of novelty and refer to something new and different from existing solutions [48]. From the point of view of the level of novelty, these can be novel for the company, for the market, and/or for the world, including product, process, marketing, organizational, business model, or supply chain innovations [1,56] As Khan rightly pointed out, innovation can be a combination of three different conditions: as an outcome, a process, and a mindset [57]. On the one hand, the search for innovations and their use is very risky and unpredictable but, on the other hand, it provides a chance for companies to maintain or even strengthen their market position [49]. Innovativeness can be defined as the ability and interest of society, entrepreneurs, and scientists to conduct research and search for new solutions, which should lead to the improvement in the competitiveness of enterprises. This, in turn, affects the economic development of regions and countries [10]. Hence, in order for the economy to be competitive, it must be characterized not only by relatively high productivity and efficiency, but should also have such features as flexibility, entrepreneurship, and innovation [58]. Global demand for new products is high and growing rapidly. Economies based on new knowledge generate high incomes and a higher employment rate, as opposed to structures based on traditional industries. Innovations are the source of technological progress. They are currently considered the most important factor in long-term competition and allow for the achievement of a position as a leading world economy [23].

This is one of the reasons why an in-depth analysis of innovation is so important, as it can enable economies to assess their results in terms of making decisions about innovation policy strategies or decisions regarding innovation, technology, and science, for example, in order to achieve the goals of sustainable development.

If the essence of innovation should be based on an interdisciplinary and multidimensional approach, taking into account the cause–effect relationships of the impacts of various phenomena and processes on the development of innovation [59], then the analysis of this problem should also be multidimensional. Moreover, the authors also propose a systemic approach to the analysis of phenomena related to innovations.

2.2. Innovation Measurement Methods

Innovation is a value that can be measured using dedicated statistical and mathematical methods, based on a variety of indicators and measures. Due to the lack of a precisely defined definition and boundaries, this issue is difficult to handle: to date, the proposals made to measure the level of innovation have assessed only certain aspects. The effects of applied measurement methods are given in the form of reports presenting the general situation for a given company, country, or region, as well as a comparative analysis as a reference point adequate for the subject of the research. From the macro, meso, and micro points of view, we can distinguish two groups of methods for measuring innovation:

• The first includes methods examining the innovativeness of a sector or company. In this area, we can distinguish:
  • Subject methods, consisting primarily in measuring the number and nature of innovations that actually exist. This method collects special statistical data from both the technical press and reports created by enterprises;
  • Object methods, which use specially constructed questionnaires to examine enterprises that have introduced innovations;
  • Statistical and mathematical methods, which use both descriptive statistics and statistical inference tools, as well as more advanced techniques, to assess the level of innovation;
  • Organizational innovation measurement tools, for which there has been a significant dispersion with the aim of systematizing the tools used to assess the level of innovation in enterprises. Innovativeness, understood as a feature of an innovative enterprise, does not have a synthetic measure and the problem is not systematized. The most commonly used measures are number and type of introduced innovations, number of patent applications, R&D expenditure, the value of sales of new products per employee, and number of new products. Numerous researchers have indicated the relationships between different dimensions, for example:

    ∘

    number of new products introduced, in relation to competition [60];

    ∘

    innovation as part of entrepreneurial orientation [61] or entrepreneurial capability [62];

    ∘

    financial performance as a measure of innovativeness [63]; and

    ∘

    measures of the activity of innovative companies, taking into account features of innovative activity (innovative potential, innovative processes, effects of innovative processes), quantitative measures, and their descriptive characteristics [64] (p. 185).

    However, the study of relationships between measures has been shown to possess various difficulties. First of all, we are assuming linear relationships, which might not be the case. Secondly, there may be significant relationships between various independent and dependent variables [65].
  • The Community Innovation Survey (CIS) is a study of the innovative activity of enterprises, in order to provide information on the innovation of sectors by type of enterprise, different types of innovation, and different aspects of innovation development, such as objectives, sources of information, public financing, and expenditure [66].
• The second group of methods is comprised of tools for researching innovation in economies. It includes, among others:
  • Statistical and mathematical methods, using both descriptive statistics and statistical inference, as well as more advanced techniques in the field of data mining. These include distribution series and histograms, statistics of a random variable, estimation and testing of statistical hypotheses, analysis of interdependence of features (independence tests, correlation), econometric models, neural networks, rules, and decision trees [67];
  • The Global Innovation Index (GII) uses two groups of indicators: “creating innovation” and “innovative activity”. The index primarily examines five main areas, which have been recognized as the most important domains, thanks to which the innovation of a given country can be determined. These areas include institutions, human resources and research, and infrastructure, as well as market and enterprise development. The sphere of implementation of this index focuses on researching scientific and creative results [68];
  • The Global Competitiveness Index (GCI), developed by the World Economic Forum. The structure of this index is based on 12 pillars. Based on these, the economies of individual countries can be classified into three stages of development, in which the economy is driven by basic requirements (factor-driven economies), factors improving efficiency (efficiency-driven economies), or innovativeness (innovation-driven economies). Economies with the highest level of development are driven by innovations and other conditions of the business environment. Thus, for countries aspiring to the classification of their economy at the highest level of competitiveness, it is crucial to acknowledge the weight of innovativeness [69,70].
  • The Summary Innovation Index (SII) is used to evaluate the innovativeness of European Union countries and is calculated on the basis of 25 partial indicators. This report distinguishes three main areas influencing innovation: elements necessary for the occurrence of innovation, company operations, and results. They constitute the basis for distinguishing eight dimensions of innovation which, in turn, can be measured by means of the 25 partial indicators [71,72].
  • The European Innovation Scoreboard (EIS) provides a comparative analysis of innovation performance in EU countries, other European countries, and their regional neighbors. It assesses relative strengths and weaknesses of national innovation systems and helps countries to identify the areas which they need to address [73].

Due to the purpose of this study, we focus on presenting only the European Innovation Scoreboard (EIS) and identifying theoretical decisions concerning the methodological aspects of building rankings.

The European Innovation Scoreboard (EIS) differentiates four major areas and ten dimensions of innovation, to which detailed criteria are assigned. Altogether, the rankings consist of 27 indicators, which are obtained from different sources: Eurostat, the Scopus database, Data calculated by Science-Metrix as part of a contract with the European Commission (DG Research and Innovation), Community Survey of ICT Usage and E-commerce in Enterprises, Global Entrepreneurship Monitor (GEM), Venture capital data from Invest Europe, GDP data from Eurostat, Community Innovation Survey, Patent data from the OECD, Trademark data from the European Union Intellectual Property Office (EUIPO) and World Intellectual Property Office (WIPO), Design data from the European Union Intellectual Property Office (EUIPO), Eurostat (ComExt) for Member States, and UN ComTrade for non-EU countries.

The structure of the European Innovation Scoreboard (EIS) includes [74]:

• Framework condition: The human resources dimension includes three indicators and measures the availability of a high-skilled and educated workforce. Human resources captures new doctorate graduates, population aged 25–34 with completed tertiary education, and population aged 25–64 involved in education and training (Lifelong learning). Attractive research systems includes three indicators and measures the international competitiveness of the science base by focusing on international scientific co-publications, most cited publications, and foreign doctorate students. Innovation-friendly environment captures the environment in which enterprises operate and includes two indicators—broadband penetration among enterprises and opportunity-driven entrepreneurship—measuring the degree to which individuals pursue entrepreneurial activities as they see new opportunities, for example, resulting from innovation.
• Investments: This aspect captures investments made in both the public and business sectors and differentiates between two innovation dimensions: Finance and support includes two indicators and measures the availability of finance for innovation projects by venture capital expenditures, as well as the support given by governments for research and innovation activities, measured by R&D expenditures in universities and government research organizations. Firm investments includes three indicators of both R&D and non-R&D investments that firms make to generate innovations, as well as the efforts enterprises make to upgrade the ICT skills of their personnel.
• Innovation activities: Innovation activities capture different aspects of innovation in the business sector and differentiate between three dimensions. Innovators includes three indicators: measuring the share of firms that have introduced innovations onto the market or within their organizations covering both product and process innovators, marketing and organizational innovators, and SMEs that innovate in-house. Linkages also includes three indicators, measuring innovation capabilities by looking at collaboration efforts between innovating firms, research collaboration between the private and public sector, and the extent to which the private sector finances public R&D activities. Intellectual assets captures different forms of Intellectual Property Rights (IPR) generated in the innovation process, including PCT (The Patent Cooperation Treaty) patent applications, trademark applications, and design applications.
• Impacts: this captures the effects of firm innovation activities and differentiates between two innovation dimensions. Employment impacts measures the impact of innovation on employment and includes two indicators: employment in knowledge-intensive activities, and employment in fast-growing firms operating in innovative sectors. Sales impacts measures the economic impact of innovation and includes three indicators: measuring exports of medium and high-tech products, exports of knowledge-intensive services, and sales due to innovation activities.

The methodology for calculating the Summary Innovation Index distinguishes between eight different steps [75], as presented in

Table 1. Methodology for calculating the Summary Innovation Index.

Steps	Characteristics
Step 1: Setting reference years	For each indicator, a reference year is identified, for all countries, based on data availability for countries for which data availability is at least 75%. For most indicators, this reference year lags one or two years behind the year to which the EIS refers.
Step 2: Imputing for missing values	Reference year data are then used for “2019”, and so on. If data for a year-in-between are not available, missing values are replaced with the value for the previous year. If data are not available at the beginning of the time-series, missing values are replaced with the next available year. The following examples clarify this step and show how ‘missing’ data are imputed. If data are missing for all years, no data is imputed (i.e., the indicator does not contribute to the Summary Innovation Index).
Step 3: Identifying and replacing outliers	Positive outliers are identified as those country scores which are higher than the mean across all countries for all years plus twice the standard deviation. Negative outliers are identified as those country scores which are lower than the mean across all countries for all years minus twice the standard deviation. These outliers are replaced by the respective maximum and minimum values observed over all the years and all countries
Step 4: Transforming data that have highly skewed distributions across countries	For those indicators where the degree of skewness across the full eight-year period is above one, the data are transformed using a square root transformation (i.e., using the square root of the indicator value, instead of the original value).
Step 5: Determining Maximum and Minimum scores	The maximum score is the highest score found for the eight-year period within all countries (excluding positive outliers). Similarly, the minimum score is the lowest score found for the eight-year period within all countries (excluding negative outliers).
Step 6: Calculating re-scaled scores	Re-scaled country scores (after correcting for outliers and a possible transformation of the data) for all years are calculated by first subtracting the minimum score and then dividing by the difference between the maximum and minimum score. The maximum re-scaled score is thus equal to 1, while the minimum re-scaled score is equal to 0. For positive and negative outliers, the re-scaled score is equal to 1 or 0, respectively.
Step 7: Calculating composite innovation indexes	For each year, a composite Summary Innovation Index is calculated as the unweighted average of the rescaled scores for all indicators, where all indicators receive the same weight (which is 1/27 if data are available for all 27 indicators).
Step 8: Calculating relative-to-EU performance scores	Performance scores, relative to the EU, are then calculated as the Summary Innovation Index (SII) of the respective country divided by the SII of the EU multiplied by 100. Relative performance scores are calculated for the full eight-year period, compared to the performance of the EU in 2012 and, for the latest year, compared to that of the EU in 2019. For the definition of the performance groups, only the performance scores relative to the EU in 2019 are used.

.

Subsequently, classification into four groups is used to determine whether countries belong to the EIS result groups or not:

• Innovation Leaders are all countries with a relative performance in 2019 above 125% of the EU average in 2019.
• Strong Innovators are all countries with a relative performance in 2019 between 95% and 125% of the EU average in 2019.
• Moderate Innovators are all countries with a relative performance in 2019 between 50% and 95% of the EU average in 2019.
• Modest Innovators are all countries with a relative performance in 2019 below 50% of the EU average in 2019.

This adopted division makes it possible to identify the degree of modernity and innovation of individual countries, grouped in terms of their homogeneity.

At this point, it should be clearly emphasized that, in order to properly build, evaluate, and modify the methodology for creating the European Innovation Scoreboard (EIS), apart from the statistical guidelines used in the research (presented later in the publication), theoretical guidelines on methodological aspects should also be taken into account. The authors of this study agree that a properly planned, prepared, and conducted assessment of the level of innovation determines its effectiveness and efficiency.

The European Innovation Scoreboard (EIS), like any other method, should meet the methodological requirements, understood as the correlation of the methodological principle, the approach used, languages and equipment within the framework of procedures used to solve specific problems [76]. On the other hand, the concept of procedure, defined in various ways in the literature on the subject, for the purposes of this publication is defined as the course of research regarding a number of guidelines (i.e., rules, directions, recommendations) that specify the manner of performing the task. Thus, the procedure does not so much define directives regarding how to think or act but, instead, it is a description of these actions; a set of recommendations that should be used in conducting the procedure. It is a chronological list of all conscious and intentional behaviors of its implementers (i.e., their actions), complemented by regulations and practical rules (e.g., who performs the action and under what conditions it takes place, as well as what methods or auxiliary techniques are used) [77].

On the other hand, methodology should be understood as a set of rules, ways of performing a specific job, or achieving a specific goal; that is, detailed standards of conduct appropriate to a given science [78]. It is also a methodologically correct set of directives, indicating the methods of action, methods leading to a given goal [27], and another set of directives indicating methods of action (e.g., principles of the method, practical recommendations) leading to the goal [79]. Therefore, if the methodology specifies a set of rules of conduct, it seems that they should be permanent and set a specific framework of activities in the implementation of the entire process of assessing the level of innovation, such that meeting them should guarantee effectiveness.

Consequently, for the purposes of this study, we assume that, by methodology of assessing the level of innovation within the EIS, we understand the sequence of activities performed in accordance with the adopted method, a set of rules and directives, as well as detailed methods and techniques used in the course of the assessment.

Taking into account the above theoretical assumptions regarding the methodological principles, as well as the presented EIS methodology, it can be noticed that it is not entirely consistent with the theoretical indications. While it is a set of guidelines referring to the individual steps of the procedure, and a set of directives, there are no clear indications as to the quantity and quality of selected indicators for assessment. It is not known what indications and rules determined the selection of specific indicators and whether they are a determinant of or capture the essence of what can be called innovation. Moreover, the 27 indicators adopted in the methodology constitute only a certain part of innovative activity. It also seems that the large number of indicators can cause difficulties in determining mutual dependence and clear correlations between the variables, thus increasing the complexity and lowering the readability of the ranking. As has been emphasized, this may lead to divergent results [80]. There is also no information on the methods and auxiliary techniques used during, for example, the selection of indicators. An additional difficulty may be the change in methodology of preparing reports over the years, which makes it impossible to compare the results, due to data corrections.

If we take into account the remark of Szopik-Depczyńska et al. [22], who stated that the construction of rankings assessing the level of innovativeness of countries using the average values of a selected set of indicators (divided into several categories) may give distorted results, especially when there is significant differentiation within various areas, perhaps it would be worthy to redefine the principles of building this type of ranking.

It also seems that adopting the right definition of innovation should be a key and fundamental step. We are aware that this is very difficult in nature, however: As Geodecki [81] has noticed, adopting an incorrect definition may lead to significant distortions of the level of innovation, as enterprises can report on all implemented novelties, of which only some may be innovations. At any rate, this problem is wider as, at present, there is no single accepted definition and no single coherent system for assessing the innovative activity of enterprises.

In the next part of the study, quantitative methods are used, the aim of which is a statistical evaluation, and further indications are made regarding the simplification of the methodology of building the European Innovation Scoreboard (EIS) ranking.

3. Determinants Affecting the Summary Innovation Index—Research Results

The data used in the analysis came from the European Innovation Scoreboard 2020 [73]. The explanatory variables were taken from Annex C: Current performance. On the other hand, a composite indicator, the Summary Innovation Index (SII), from the table of Composite indicators, was adopted as the dependent variable. The SII summarizes the performance of a range of different indicators and is obtained by taking an unweighted average of the 27 indicators [82].

The goal of the analysis, set in this way, was achieved through the following steps:

• Missing data was estimated using the mice package.
• Stepwise regression was used to identify determinants influencing the Summary Innovation Index results using the mass package, and modelling by AIC in a Stepwise Algorithm was chosen.
• Countries were ranked using the linear ordering method and the Hellwig taxonomic measure using the linearOrdering package.
• Kendall’s coefficient of concordance, implemented in the irr package, was used to assess the compliance of the rankings created by the linear ordering method and the European Innovation Scoreboard.
• The countries were divided into groups, based on statistical criteria using the arithmetic mean and standard deviation of the Hellwig synthetic measure.
• The rand index, implemented in the fossil package, was used to assess the compliance of the cluster of countries into four performance groups, according to Hellwig’s synthetic measure, and the method implemented in the European Innovation Scoreboard.

All calculations were performed with the use of R software. All calculations were performed with the use of R software.

Table 2. Explanatory variables included in the analysis.

Codes	Framework Conditions	Codes	Framework Conditions
A.1	New doctorate graduates	C.1	SMEs with product or process innovations
A.2	Population completed tertiary education	C.2	SMEs with marketing or organizational innovations
A.3	Lifelong learning	C.3	SMEs innovating in-house
A.4	International scientific co-publications	C.4	Innovative SMEs collaborating with others
A.5	Scientific publications among top 10% most cited	C.5	Public–private co-publications
A.6	Foreign doctorate students	C.6	Private co-funding of public R&D expenditures
A.7	Broadband penetration	C.7	PCT patent applications
A.8	Opportunity-driven entrepreneurship	C.8	Trademark applications
B.1	R&D expenditure in the public sector	C.9	Design applications
B.2	Venture capital investments	D.1	Employment in knowledge-intensive activities
B.3	R&D expenditure in the business sector	D.2	Employment in fast-growing firms belonging to innovative sectors
B.4	Non-R&D innovation expenditure	D.3	Medium- & high-tech product exports
B.5	Enterprises providing ICT training	D.4	Knowledge-intensive services exports
		D.5	Sales of new-to-market and new-to-firm innovations

presents the explanatory variables including in the analysis.

4. Results

4.1. Identification of Determinants Influencing the Position on the European Innovation Scoreboard

Data from Annex C: Current performance, which is attached to the European Innovation Scoreboard 2020 report, was used to conduct the analysis. In the first step, the missing data were estimated by means of the mice package, implemented in the R environment. Then, Stepwise regression was used to identify determinants affecting the Summary Innovation Index results. The purpose of this algorithm was to add and remove potential candidates in the models, keeping only those which had a significant impact on the dependent variable. This algorithm is meaningful when the data set contains a large list of predictors. The significance of the variables was determined on the basis of the Akaike criterion (AIC), based on the likelihood function. A model with a minimum AIC value was considered, according to this criterion, to be the best fit for the data [83].

Table 3. Results of the first step of stepwise regression.

Start: AIC = 62.29 SII ~ A.1 + A.2 + A.3 + A.4 + A.5 + A.6 + A.7 + A.8 + B.1 + B.2 + B.3 + B.4 + B.5 + C.1 + C.2 + C.3 + C.4 + C.5 + C.6 + C.7 + C.8 + C.9 + D.1 + D.2 + D.3 + D.4 + D.5
Df	Sum of Sq	RSS	AIC
- C.3	0.006	43.864	60.296
- C.5	0.316	44.174	60.557
- C.7	0.917	44.775	61.057
- A.6	1.769	45.628	61.755
- B.5	2.003	45.861	61.944
<none>		43.858	62.292
- A.5	7.252	51.110	65.954
- C.9	8.466	52.324	66.822
- C.8	9.236	53.094	67.363
- D.1	10.524	54.382	68.249
- A.8	11.923	55.781	69.189
- A.2	11.956	55.815	69.211
- A.4	12.842	56.700	69.794
- A.3	12.932	56.791	69.853
- C.4	13.074	56.932	69.945
- B.1	13.799	57.657	70.413
- A.7	15.148	59.007	71.269
- D.5	16.891	60.749	72.346
- C.6	18.905	62.764	73.553
- D.4	22.482	66.340	75.603
- B.3	31.618	75.476	80.377
- A.1	32.857	76.716	80.980
- C.2	34.014	77.873	81.534
- C.1	37.269	81.127	83.049
- D.2	54.975	98.833	90.353
- B.4	55.262	99.121	90.460
- D.3	77.758	121.616	98.028
- B.2	112.629	156.487	107.356

presents the results of the first step of stepwise regression. The Summary Innovation Index (SII) was adopted as the dependent variable and the measures listed in Table 2 were considered as the dependent variables. The lowest AIC value was achieved by the explanatory variable C.3 (SMEs innovating in-house). This means that the amount of information loss due to removing the variable C.3 is at a low level. In the second step, the variable C.3 was excluded from the model.

Table 4. Results of the second step of stepwise regression.

Step: AIC = 60.3 SII ~ A.1 + A.2 + A.3 + A.4 + A.5 + A.6 + A.7 + A.8 + B.1 + B.2 + B.3 + B.4 + B.5 + C.1 + C.2 + C.4 + C.5 + C.6 + C.7 + C.8 + C.9 + D.1 + D.2 + D.3 + D.4 + D.5
Df	Sum of Sq	RSS	AIC
- C.5	0.310	44.174	58.557
- C.7	0.912	44.776	59.058
- A.6	1.769	45.633	59.759
- B.5	2.001	45.865	59.947
<none>		43.864	60.296
+ C.3	0.006	43.858	62.292
- A.5	7.337	51.201	64.019
- C.9	8.900	52.764	65.131
- C.8	9.347	53.211	65.444
- D.1	10.602	54.466	66.306
- A.8	11.981	55.844	67.231
- A.2	12.784	56.647	67.759
- C.4	13.091	56.955	67.960
- A.4	13.122	56.986	67.980
- A.3	13.271	57.135	68.077
- B.1	13.906	57.770	68.485
- A.7	15.526	59.389	69.508
- D.5	17.023	60.887	70.430
- C.6	19.942	63.806	72.162
- D.4	24.141	68.005	74.520
- B.3	31.616	75.480	78.379
- C.2	34.474	78.338	79.754
- A.1	35.744	79.608	80.349
- D.2	58.536	102.400	89.665
- B.4	58.549	102.412	89.669
- C.1	64.458	108.322	91.745
- D.3	77.778	121.642	96.036
- B.2	121.869	165.732	107.480

presents the results of the second step of stepwise regression. The lowest AIC value was achieved by the explanatory variable C.5 (Public–private co-publications). In the next step, this variable was omitted.

Table 5. Individual steps of stepwise regression.

Steps	AIC Value	Variables with the Lowest AIC Value
Step 1	AIC = 62.29	C.3—SMEs innovating in-house
Step 2	AIC = 60.3	C.5—Public–private co-publications
Step 3	AIC = 58.56	C.7—PCT patent applications
Step 4	AIC = 57.07	B.5—Enterprises providing ICT training
Step 5	AIC = 56.03	A.6—Foreign doctorate students

presents the individual steps of stepwise regression and the variables which obtained the lowest AIC values at each stage. According to the analysis, they were not determinants affecting the level of the Summary Innovation Index (SII). The following variables were omitted from the final model:

• C.3—SMEs innovating in-house
• C.5—Public–private co-publications
• C.7—PCT patent applications
• B.5—Enterprises providing ICT training
• A.6—Foreign doctorate students

In the final model, the AIC value was 54.9.

Table 6. AIC value of the variables included in the model, as estimated by stepwise regression.

Codes	Variables	AIC
A.5	Scientific publications among top 10% most cited	57.082
C.9	Design applications	58.558
A.2	Population completed tertiary education	61.229
C.4	Innovative SMEs collaborating with others	62.729
C.8	Trademark applications	63.336
D.1	Employment in knowledge-intensive activities	63.690
A.3	Lifelong learning	64.401
C.6	Private co-funding of public R&D expenditures	64.698
B.1	R&D expenditure in the public sector	65.572
A.8	Opportunity-driven entrepreneurship	66.114
D.5	Sales of new-to-market and new-to-firm innovations	71.032
A.7	Broadband penetration	72.632
C.2	SMEs with marketing or organizational innovations	73.629
D.4	Knowledge-intensive services exports	77.696
D.2	Employment in fast-growing firms of innovative sectors	83.796
A.1	New doctorate graduates	84.232
B.3	R&D expenditure in the business sector	87.247
B.4	Non-R&D innovation expenditure	90.102
A.4	International scientific co-publications	91.514
D.3	Medium- & high-tech product exports	93.459
C.1	SMEs with product or process innovations	94.639
B.2	Venture capital investments	107.347

presents the variables which were retained in the model.

The R² coefficient, describing the quality of the model, had a value of 0.9974, which indicated a very good fit.

Table 7. Results of estimating model parameters using stepwise regression.

Call: lm(formula = SII ~ A.1 + A.2 + A.3 + A.4 + A.5 + A.7 + A.8 + B.1 + B.2 + B.3 + B.4 + C.1 + C.2 + C.4 + C.6 + C.8 + C.9 + D.1 + D.2 + D.3 + D.4 + D.5, data = analiza_final_annexC) Residuals: Min    1Q     Median  3Q  Max −1.8815   −0.8863   −0.1275  0.9664 2.1918 Coefficients:
	Estimate	Std. Error	t value	Pr(>|t|)
(Intercept)	−1.497138	2.269953	−0.660	0.520248
A.1	0.052394	0.012132	4.319	0.000707	***
A.2	0.017388	0.009248	1.880	0.081064
A.3	0.023798	0.010533	2.259	0.040343	*
A.4	0.053959	0.010633	5.075	0.000169	***
A.5	0.035840	0.027689	1.294	0.216491
A.7	0.014820	0.004719	3.141	0.007224	**
A.8	0.031457	0.012832	2.451	0.027964	*
B.1	0.057414	0.024009	2.391	0.031382	*
B.2	0.049089	0.007161	6.855	7.88 × 10−6	***
B.3	0.079540	0.017185	4.629	0.000391	***
B.4	0.035522	0.007211	4.926	0.000223	***
C.1	0.103440	0.019124	5.409	9.21 × 10−5	***
C.2	0.067355	0.020770	3.243	0.005896	**
C.4	0.018243	0.008839	2.064	0.058065	.
C.6	0.049038	0.021385	2.293	0.037844	*
C.8	0.037790	0.017692	2.136	0.050832	.
C.9	0.029158	0.019171	1.521	0.150529
D.1	0.031610	0.014518	2.177	0.047059	*
D.2	0.039411	0.009221	4.274	0.000771	***
D.3	0.065950	0.012486	5.282	0.000116	***
D.4	0.052524	0.014368	3.656	0.002595	**
D.5	0.043760	0.014707	2.976	0.010026	*
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 1.833 on 14 degrees of freedom Multiple R-squared: 0.999, Adjusted R-squared: 0.9974 F-statistic: 628.9 on 22 and 14 DF, p-value: < 2.2 × 10−16

presents the results of estimating model parameters using stepwise regression.

Based on the results obtained, the 22 variables listed in Table 6 were considered as determinants influencing the results of the Summary Innovation Index and were selected for further analysis.

4.2. Using the Linear Ordering Method to Create a Country Ranking

The next step in the adopted research procedure was to create a ranking of countries, taking into account the determinants influencing the Summary Innovation Index. For this purpose, the method of linear ordering and Hellwig’s synthetic measure were used.

The construction of the synthetic measure (measure of economic development) by Hellwig is defined as:

1.

Normalization of variables

$$z_{ij}=\frac{x_{ij}-{\overline{x}}_j}{s_j}$$

(1)

where

$x_{ij}$ is the observation of the jth variable for object i,

${\overline{x}}_j$ is the arithmetic mean of observations for the jth variable, and

$s_j$ is the standard deviation of the observations for the jth variable.

2.

Pattern co-ordinates

$$z_{0j}=\left\{\begin{array}{c}\underset{i}{\max }\left\{z_{ij}\right\}\mathrm{for}\mathrm{stimulant}\mathrm{variables}\\ \underset{i}{\min }\left\{z_{ij}\right\}\mathrm{for}\mathrm{destimulant}\mathrm{variable}\end{array}\right.$$

(2)

3.

Distance of objects from the pattern

$$d_{i0}=\sqrt{{\displaystyle \sum }_{j=1}^m{(z_{ij}-{\overline{z}}_{0j})}^2}$$

(3)

4.

Value of an aggregate variable

$$q_i=1-\frac{d_{i0}}{d_0}$$

(4)

accepting:

$$d_0=\overline{d_0}+2s_d$$

(5)

$$\overline{{\mathrm{d}}_0}=\frac{1}{\mathrm{n}}{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{d}}_{\mathrm{i}0}$$

(6)

$$s_d=\sqrt{\frac{1}{n}{\displaystyle \sum }_{i=1}^n{(d_{i0}-\overline{d_0})}^2}$$

(7)

where

$q_i$ is the synthetic development index (measure of development),

$d_{i0}$ is the Euclidean distance of object $x_j$ from the reference object

$d_0$, is the critical distance of a given unit from the pattern,

${\overline{d}}_0$ is the arithmetic mean of taxonomic distances, and

$s_d$ is the standard deviation of the distance taxonomy.

The analysis assumed that all variables included were stimulants, which means that an increase in their value indicates an increase in the level of the complex phenomenon. The linearOrdering package, implemented in the R environment, was used to perform the analysis.

Table 8. Comparison of the ranking determined using the linear ordering method with the European innovation scoreboard.

Country	Hellwig’s Synthetic Measure	Position in the Ranking by the Linear Ordering Method	Summary Innovation Index	Position in the European Innovation Scoreboard
Switzerland	0.718	1	0.837	1
Sweden	0.597	2	0.713	2
Finland	0.577	3	0.709	3
Denmark	0.530	4	0.682	4
Netherlands	0.529	5	0.648	5
United Kingdom	0.504	6	0.613	8
Germany	0.502	7	0.608	10
Belgium	0.490	8	0.615	7
Austria	0.470	9	0.596	11
Luxembourg	0.461	10	0.639	6
Israel	0.453	11	0.563	14
Ireland	0.426	12	0.568	13
France	0.420	13	0.530	15
Norway	0.419	14	0.611	9
Estonia	0.401	15	0.502	16
Iceland	0.384	16	0.579	12
Portugal	0.364	17	0.490	17
Spain	0.336	18	0.432	19
Slovenia	0.335	19	0.431	20
Czechia	0.316	20	0.427	21
Italy	0.311	21	0.420	23
Lithuania	0.302	22	0.404	24
Cyprus	0.300	23	0.451	18
Greece	0.275	24	0.389	25
Malta	0.263	25	0.426	22
Latvia	0.215	26	0.320	28
Slovakia	0.214	27	0.338	26
Hungary	0.212	28	0.337	27
Poland	0.206	29	0.299	31
Serbia	0.200	30	0.313	30
Croatia	0.165	31	0.298	32
Turkey	0.163	32	0.316	29
Bulgaria	0.121	33	0.230	33
North Macedonia	0.079	34	0.226	34
Ukraine	0.065	35	0.167	36
Montenegro	0.056	36	0.220	35
Romania	0.029	37	0.160	37

presents the results of the ranking of countries determined using the linear ordering method, compared with the European innovation scoreboard ranking.

Comparing the ranking created with the use of the linear ordering method (taking into account 22 variables) with the European innovation scoreboard ranking (taking into account 27 variables), the biggest losses occurred for Norway and Cyprus (by five places, respectively from places 9 to 14 and 18 to 23), and Luxembourg and Iceland (from 6th to 10th and 12th to 16th positions, respectively). Germany and Italy gained the most in the linear ranking (from 10th to 7th and 14th to 11th, respectively). The linear ordering method did not affect the positions of 10 countries.

In the next step of the analysis, the ranking according to Hellwig’s synthetic measure and the European innovation scoreboard ranking were compared using the Kendall’s coefficient of concordance, implemented in the irr package, which measures the degree of agreement of rankings.

The following research hypotheses were formulated:

$H_0$: There is a lack of agreement between rankings; and

$H_1$: The rankings are consistent.

>kendall(IRR_kendall_annexC, correct = FALSE) Kendall’s coefficient of concordance W Subjects = 37 Raters = 2 W = 0.991 Chisq(36) = 71.3 p-value = 0.000409

By comparing the p-value = 0.000409 with the significance α = 0.05, the rankings were found to be statistically consistent. The strength of this agreement was high and amounted to W = 0.991.

4.3. Division of Countries into Four Performance Groups

In the last stage of the analysis, based on Hellwig’s synthetic measure, the Member States were divided into four performance groups:

• Cluster 1: Innovation Leaders;
• Cluster 2: Strong Innovators;
• Cluster 3: Moderate Innovators; and
• Cluster 4: Modest Innovators.

The division of objects into clusters was carried out on the basis of statistical criteria using the arithmetic mean and standard deviation of the Hellwig’s synthetic measure [83]:

• Cluster 1 (high level):$q_i{\ge \overline{\mathrm{q}}+\mathrm{s}}_q$,
• Cluster 2 (higher average level):${\overline{\mathrm{q}}+\mathrm{s}}_q>q_i\ge \overline{q}$,
• Cluster 3 (lower average level):$\overline{\mathrm{q}}>q_i\ge \overline{q}-{\mathrm{s}}_q$, and
• Cluster 4 (low level):$q_i<\overline{\mathrm{q}}-{\mathrm{s}}_q$,

where

$$q=\frac{1}{n}{\displaystyle \sum }_{i=1}^n{q}_{i,}$$

(8)

$$s_q=\sqrt{\frac{1}{n}{\displaystyle \sum }_{i=1}^n{\left(q_i-\overline{q}\right)}^2}$$

(9)

Table 9. The division of countries into four performance groups.

	Clustered by Hellwig’s Synthetic Measure	Clusters of European Innovation Scoreboard
Cluster 1: Innovation Leaders	Switzerland, Sweden, Finland, Denmark, Netherlands, United Kingdom, Germany	Switzerland, Sweden, Finland, Denmark, Netherlands, Luxembourg
Cluster 2: Strong Innovators	Luxembourg, Belgium, Norway, Austria, Iceland, Ireland, Israel, France, Estonia, Portugal, Spain	Belgium, United Kingdom, Norway, Germany, Austria, Iceland, Ireland, Israel, France, Estonia, Portugal
Cluster 3: Moderate Innovators	Cyprus, Slovenia, Czechia, Malta, Italy, Lithuania, Greece, Slovakia, Hungary, Latvia, Serbia, Poland	Cyprus, Spain, Slovenia, Czechia, Malta, Italy, Lithuania, Greece, Slovakia, Hungary, Latvia, Turkey, Serbia, Poland, Croatia
Cluster 4: Modest Innovators	Turkey, Croatia, Bulgaria, North Macedonia, Montenegro, Ukraine, Romania	Bulgaria, North Macedonia, Montenegro, Ukraine, Romania

shows the results of the division of countries into four performance groups, according to Hellwig’s synthetic measure and the European innovation scoreboard. The similarity between the two data classifications was measured using the Rand index. The Rand index has a value between 0 and 1, with 0 indicating that two data clusterings do not agree on any pair of points and 1 indicating that the data clusterings are exactly the same. The rand index was implemented in the fossil package (R software).

>rand.index(Cluster$Clusters_Hellwig, Cluster$Clusters_EIS) [1] 0.8498498

The Rand index was 0.8498498, which proves the high similarity of the division of countries into particular performance groups.

When analyzing the results summarized in Table 9, it can be noted that countries such as Switzerland, Sweden, Finland, Denmark and The Netherlands were classified as Innovation Leaders both in the group division based on Hellwig’s synthetic measure and in the division according to the European Innovation Scoreboard methodology. In the division of groups based on Hellwig’s synthetic measure, this group was also joined by the United Kingdom and Germany, while Luxembourg was not included in this group and was included in the Strong Innovators group. This group also included Spain which, according to the European Innovation Scoreboard, was included in the Moderate Innovators group of countries. We can also see a difference in the case of Turkey, which was included in the Moderate Innovators group in the European innovation scoreboard while, in the case of Hellwig’s synthetic measure, it was included in the Modest Innovators group.

5. Conclusions

The aim of this article was to identify determinants affecting the Summary Innovation Index result and, consequently, the positions of countries on the European Innovation Scoreboard (EIS). Then, based on the identified determinants, the countries were ranked using the linear ordering method. The authors also attempted to divide countries into performance groups, based on the ranking created, using the linear ordering method and Hellwig’s synthetic measure. The division into groups was made on the basis of statistical criteria (i.e., using the arithmetic mean and standard deviation).

Based on the analyses carried out, the following conclusions could be drawn.

• The use of the stepwise analysis made it possible to select, from among the 27 factors taken into account when creating the European Innovation Scoreboard, those that are not determinants influencing the Summary Innovation Index. These factors were: SMEs innovating in-house, public–private co-publications, PCT patent applications, enterprises providing ICT training, and foreign doctorate students.
• The linear ordering method based on Hellwig’s synthetic measure can be used to create rankings based on measures of innovation.
• Limiting the number of measures of innovation to those that constitute determinants influencing the country’s position in the ranking only slightly changed the way that countries are ranked, as evidenced by the high level of Kendall’s coefficient of concordance (0.991).
• When dividing countries into performance groups using statistical criteria based on the mean and standard deviation, based on Hellwig’s synthetic measure, we obtained a similar division into four performance groups, as evidenced by the high value of the Rand index (0.8498498).

Although the research procedure showed which indicators are not determinants of the EIS ranking, this does not give us sufficient grounds to infer causality; it only implies inadequacy. The PCT patent applications indicator is correlated with the number of patents which, as we know, is often used to assess the degree of innovation. However, it seems that it is more synonymous with inventiveness than innovation. Additionally, patents have differing potential commercial value, which may lower their importance as determinants. Not all innovations are patented and some innovations, in the form of intangible technologies, are protected by copyright, not patents. On the other hand, expenditure in ICT training does not necessarily reflect growing innovation. Rather, it serves to raise the level of digital skills which, of course, may have an impact on the level of innovation in the long run. Numerous studies have indicated the benefits of co-operation in innovative activities but, in the case of public–private co-publications, we saw no impact on the ranking. Practice has shown that the complexity and multifaceted nature of the concept of innovation makes it difficult to define what innovation truly is. This may result in the incorrect identification of innovation by enterprises which, in turn, hinders its measurement and study.

Some limitations of the conducted research should be pointed out. The analysis was carried out for one year and the indicated determinants influencing the SII level related to the EIS ranking published in 2020. In the next stages of the research, it should be verified whether the set of determinants influencing the SII level in previous years was similar and whether the linear ordering method allows for the creation of a ranking with such a high level of similarity to the EIS as the results presented in this publication. Long-term indicator observations could also provide a basis for identifying certain regularities, helping to shape the rankings.

It should be clearly emphasized that, during the study, the usefulness of indicators in measuring the degree of innovation in economies was not assessed; instead, we wished to demonstrate a less complicated methodology for calculating EIS. Moreover, if the indicators used to build the ranking are not free from defects—from the point of view of measuring the degree of innovativeness of economies—then, perhaps, attention should be paid to measuring innovation by increasing the productivity of factors, which more fully reflects the objectives of management and enables international comparison [81]. Another suggestion might be to treat the methodologically improved rankings as basic tools and complement them with research on the internal structure of innovation in socio-economic objects [22].

Undoubtedly, the methods, and the indicators used with them to measure innovativeness, are not perfect, being insufficient, complicated, encumbered with many faults, and laborious. Therefore, an in-depth redefinition of the rules and principles used to construct such rankings might be worthwhile. Hence, the authors believe that research in this area is justified and hope that this study will prove to be useful to scientists, institutions, decision-makers who develop and use such methods to deal with innovation, and others who are interested in these issues.