How Efficient Is the Cohesion Policy in Supporting Small and Mid-Sized Enterprises in the Transition to a Low-Carbon Economy?

Abstract

Funds from the European Union that are devoted to fostering a low-carbon economy are aimed at assisting Member States and regions in implementing the required investments in energy efficiency, renewable energy, and smart distribution electricity grids, and for research and innovation in these areas. In this context, we assessed the implementation of these funds in small and medium-sized enterprises across different beneficiary countries and regions of the European Union. Therefore, this study uses a non-radial slack-based data envelopment analysis model coupled with cluster analysis that covers multiple aspects of evaluation, including two inputs and two outputs, to assess 102 programs from 22 countries. Overall, we were able to ascertain that there are 25 efficient operational programs that remain robustly efficient, whereas 51 remain robustly inefficient for data perturbations of 5 and 10%. Under the current output level, there was almost no input surplus. Therefore, to promote a low-carbon economy, operational program managers should concentrate on solving the problems behind the poor results achieved, both in terms of greenhouse gas emissions reduction and the pace of the programs’ implementation.

1. Introduction

Curbing the impact of climate change represents one of the major concerns of the current European Union (EU) policy agenda. The 2050 climate and energy framework, which comprises EU-wide targets and policy objectives, proposes cuts of at least 55% in greenhouse gas (GHG) emissions (from 1990 levels) and a 40% utilization of renewable sources in the EU’s energy mix, set to be achieved by 2030. Hence, the EU strategy for tackling the impacts of climate change requires transitioning to a low-carbon economy (LCE). Simply put, an LCE is an economy where the structure is supported by activities associated with low levels of carbon dioxide emissions into the atmosphere [1]. In this framework, the EU policies, primarily its cohesion policy, provide support in the form of the promotion of low-carbon investments. The number and quality of low-carbon projects, besides the required structures and capacities of regions, will depend upon coordinated national and regional policies and actions. In addition, the significance of cohesion policy programs is extremely reliant on the global funding that has been assigned to this low-carbon thematic objective. The cohesion policy has backed the transition to an LCE in the past programming period, but the 2014–2020 commitments have increased significantly, especially due to the provision of special funds for the transition to an LCE (20% in more developed regions, 15% in regions of transition, and 12% in less developed regions). Therefore, evaluation assumes a prominent role in cohesion policy formulation because it helps support policy design and implementation, also providing sound evidence regarding the results and impact of the actions taken. The rules governing the planning and implementation of cohesion policies are progressively centered on evaluating their impact. From 2014 to 2020, member states (MS) became obligated (for the first time) to conduct evaluations to appraise the effectiveness, efficiency, and impact of each program’s objective. These rules apply to the European Regional Development Fund (ERDF), the European Social Fund (ESF), and the Cohesion Fund.

Various methods can be employed in the assessment of the cohesion policy, each of them with its own merits and demerits [2]. The two approaches most often used for this purpose are based on macroeconomic and econometric models. The first type of these models enables evaluating the potential impact of EU funds on economic growth but do not encompass the assessment of management failures [3]. Besides, these models neglect the allocation of EU funding across themes and sectors in each country. The mainstream research, which is based on econometric analyses, also reaches quite mixed conclusions [4] and has led several authors to start questioning their use [4,5,6].

An alternative technique considers microeconomic studies, which use control groups for contrasting the outcomes of the beneficiaries of structural funds with those of control groups having similar characteristics. Finally, in terms of the case study evaluation type, a blend of data gathered from surveys, program monitoring reports and quantitative analysis is applied to identify the impacts of structural funds at the regional, program or project level [7].

The previously employed evaluation methodologies do not allow any country/region/program to be compared with its counterparts or to establish the adjustments that need to be implemented to make an inefficient country/region/program efficient [8]. Moreover, these methodologies frequently require the satisfaction of statistical assumptions, such as normality, the absence of multicollinearity and homoscedasticity.

In general, stochastic methods can be used to estimate the frontier production function, but most of them need to consider a single output. An alternative way to establish the efficiency frontier is through the use of non-parametric methods, such as data envelopment analysis (DEA), which can easily handle multiple inputs and outputs. In contrast to stochastic methods, DEA is a non-statistical and non-parametric approach that makes no assumption regarding the distribution of inefficiencies or the functional form of the production function. Instead, it utilizes the factors of evaluation, i.e., inputs (resources) and outputs (results generated), using linear programming methods to compute the production possibility frontier. The efficiency of each unit under evaluation is measured so as to reflect the unit’s efficiency relative to all the others in the dataset. With DEA, it is assumed that the deviations from the efficient frontier are the result of inefficiency. In addition, DEA can help identify the benchmarks of each inefficient unit under evaluation and the potential causes of each unit’s inefficiency, providing decision-makers with information on the necessary adjustments required to tackle them. Based on this reasoning, the authors of [9] used the DEA approach to assess the relative spatial disadvantages of the EU’s Level II regions. Kutan and Yigit [10] employed structural break testing and DEA to evaluate the accession process of five European countries (Portugal, Spain, Finland, Austria, and Sweden). Later, the authors of [11] assessed the pure technical efficiency and global technical efficiency of Objective 1 in the application of structural funds from 2000 to 2006 in various regions, using the rate of employment and productivity as outputs in parametric (stochastic frontier) and non-parametric (DEA) approaches. Recently, the authors of [8] suggested the use of a value-based DEA approach, which combines DEA with multiple criteria decision-aiding, considering the main factors that might influence the efficiency of the implementation of structural funds across different EU beneficiary countries and regions. This study uses data made publicly available by the European Commission regarding finance implementation and expected achievements (targets), under the theme of “competitiveness of small and medium-sized enterprises (SMEs)”.

Although the publications available for evaluating structural funds are prolific (see Section 2), there are still gaps in the literature, particularly when the evaluations take place during the planning periods. When performing this sort of assessment by applying the DEA method, fine-grained information can be obtained by allowing management authorities (MA) to identify the benchmarks of inefficient operational programs (OPs) and provide information on the best practices that these OPs should undertake to become efficient. In this way, and differently from other methodologies and approaches that are specially used in ex-post and ex-ante policy evaluations (e.g., microeconomic studies, which use control groups, case study evaluation, and macroeconomic and econometric impact evaluations), DEA enables monitoring the efficiency of OPs’ implementation by considering the performance framework indicators (reported by the MS) during the programming period that is underway, allowing for the adoption of the required corrective actions to achieve efficiency. The performance framework consists of a group of indicators in each OP according to which the European Commission, in cooperation with the MS, evaluates the performance of the programs in each MS. Through this procedure, called the “performance review”, the achievements of the OPs’ priorities are evaluated according to the information obtainable from the annual implementation report presented by the MS. In the event of failing to achieve the targets set in the programs, the European Commission may apply financial rectifications. Furthermore, DEA may provide actual support in terms of the design of future cohesion policy instruments since it enables understanding the key achievements and shortcomings of the preceding implementation programming cycles, while also offering information on how to overcome the identified shortcomings. Finally, by coupling DEA with cluster analysis, MA can also exploit the differences among the performances of distinct regions’ categories.

Since SMEs are the keystone of the European economy, corresponding to 99% of businesses in the EU and around two-thirds of private-sector employment [12], and their impact on the environment is also relevant, their transition toward low-carbon practices is essential for reaching the EU’s target for reducing carbon emissions. Therefore, we have dedicated our analysis to the programs funded by the ERDF, which represent 70.5% of the European structural and investment (ESI) funds devoted to this topic.

Overall, the main research questions that we want to answer with our work are the following:

(1)

“Which factors require special attention for reaching an efficient implementation of the funds devoted to fostering an LCE in the EU?”

(2)

“Which OPs were more often viewed as benchmarks during the programming period under evaluation?”

(3)

“Were the OPs robustly efficient in the face of potential changes of the performance framework indicators used”?

(4)

“Which type of regions managed to attain higher LCE performance”?

The novelty of our work is threefold: (1) it proposes the use of the slack-based measure (SBM) DEA model, combined with cluster analysis, considering two inputs and two outputs to assess 102 programmes from 22 countries devoted to the promotion of an LCE in SMEs; (2) it allows evaluating these funds during their programming period, enabling countries to facilitate the adoption of policy measures to correct the inefficiencies identified, thus providing additional information that can be used in the intermediate monitoring and process evaluations; (3) it considers the distinct regional characteristics of the OPs, i.e., it accounts for the differences between the regions that have below 75% of the EU’s gross domestic product (GDP), which are eligible for considerably greater financial assistance from the cohesion policy (less developed regions), the regions just above this threshold, i.e., regions of transition (75% ≤ GDP ≤ 90%), and more developed regions (GDP > 90%), which are grouped into three distinct clusters.

This paper is structured as follows. Section 2 presents a literature review on the cohesion policy program evaluations that specifically address LCE concerns, also describing works that have applied DEA in the assessment of LCE efficiency in a broad context. Section 3 clarifies the key definitions and concepts regarding the methodology used to evaluate the implementation of the OPs under scrutiny. Section 4 describes the main assumptions considered in the choice of the input and output factors that are used in the efficiency evaluation of OPs, also providing some statistical information on the data used. Section 5 examines the results obtained and explores potential policy implications. Section 6 presents our conclusions and proposes potential recommendations based on the results obtained.

2. Literature Review

Ever since 2015, more than 1000 assessments have been conducted by EU MS, focusing on distinct funds, themes, and regions, monitoring the progress of implementation, and/or evaluating the impact of interventions, both referring to the 2007–2013 and 2014–2020 programming periods [13].

The number of evaluations conducted by MS differs broadly (see Figure 1). This is the result of considerable disparities in the amount and type of investment funding, the number of programs in each MS, and the methodology suggested in the assessment plans (see Figure 1). Besides, while some countries prefer to conduct a large number of smaller assessments, others prefer to conduct aggregate evaluations.

Figure 1. Evaluations and methodologies employed by country since 2015 referring to both the 2007–2013 and 2014–2020 programming periods. Data available from: https://cohesiondata.ec.europa.eu/2014-2020-Evaluation/Cohesion-policy-programme-evaluations-2015-to-date/iz3t-u7bv (accessed on 23 March 2022). Note: TBIE—theory-based impact evaluation; QUAL—qualitative (other); QUANT—quantitative (other); CBA—cost benefit analysis; CIE—counterfactual impact evaluation; MM—macro modeling.

Most of these assessments are focused on implementation matters and evaluate progress regarding target achievement, being mainly concerned with the alignment of the projects and actions with the programs’ objectives and with the effectiveness and efficiency of their implementation. They also assess whether the existing funding is spent wisely or not and if the targets that were established, particularly those of the performance framework, are achieved. The impact assessments are performed later in the program cycle when most actions have already taken place and have also generated impacts.

The distribution of assessments across thematic objectives (TO) by country is depicted in Figure 2. As can be seen from Figure 2, most of the evaluations focus on social themes (TO 8, TO 9 and TO 10).

Figure 2. Number of 2014–2020 evaluations conducted by country TO. Data available from: https://cohesiondata.ec.europa.eu/2014-2020-Evaluation/Cohesion-policy-programme-evaluations-2015-to-date/iz3t-u7bv (accessed on 23 March 2022). Note: TO 1—Research, technological development and innovation; TO 2—Information and communication technologies; TO 3—Competitiveness of SMEs; TO 4—Shift toward an LCE; TO 5—Climate change adaptation, risk prevention and management; TO 6—Preserving and protecting the environment and promoting resource efficiency; TO 7—Sustainable transport and key network infrastructures; TO 8—Employment and labor mobility; TO 9—Social inclusion, combating poverty and any discrimination; TO 10—Education, training and vocational training for skills and lifelong learning; TO 11—Enhancing the institutional capacity of public authorities and stakeholders, and efficient public administration.

Since the MS have dedicated about EUR 60 billion from European structural and investment (ESI) funds to the investment in LCE during 2014 to 2020 (more than twice the amount spent in this area during the previous funding period), our study will be devoted to the assessment of OPs within the framework of TO 4. In this context, Table A1 (available in Appendix A) lists the cohesion policy program evaluations completed by the EU MS, which specifically mention LCE concerns in their abstract summary from 2015 to the present.

Although an analysis of Table A1 suggests that the MS are using systematic methods in their assessments of TO 4, namely, desk research, monitoring data/data analysis, interviews, focus groups/facilitated workshops, surveys, and case studies, only a few assessments (mainly devoted to impact evaluations) consider more robust methods (such as statistical methods or other techniques), showing that despite the commitment of MS to enhancing the evaluation of cohesion policy, there is still room for improvement, in terms of the methodological approaches that can be employed.

For example, none of the reports reviewed in Table A1 apply the DEA method in their assessments. When performing an efficiency evaluation using the DEA method, MA will be capable of identifying the OP benchmarks, throughout the programming period, in terms of best practices and the necessary adjustments that need to be made regarding the group of indicators of the performance framework that will enable inefficient OPs to become efficient. Besides, the DEA method has also been used in the assessment of LCE efficiency at national, regional and sectoral levels—see Table 1. Most of the studies reviewed in Table 1 use the SBM-DEA model but do not account for the robustness assessment or uncertainty handling of the results obtained. In addition, to the best of our knowledge, the SBM-DEA method has not been employed for the evaluation of the implementation of OPs under the LCE theme. Hence, we will specifically address the efficiency assessment of these sorts of OPs through the application of the SBM-DEA method, combined with cluster analysis, and we will also perform robustness and sensitivity analyses of the results obtained.

Table 1. Studies that use DEA regarding LCE efficiency.

Authors	Main Purpose	Methodologies	Inputs	Outputs
Liu and Liu [14]	Estimate the LCE efficiency in the 20 greatest CO2-emitting countries (2000–2012).	A three-stage approach (1) SBM-DEA model; (2) stochastic frontier approach to eliminate the impacts of external environment variables on these slacks; (3) re-estimation of efficiency with adjusted inputs and outputs.	Energy consumption; capital stock; labor force	GDP; GHG emissions
Zhang et al. [15]	Measure the LCE efficiency and the dynamic LCE efficiency of 30 provinces in mainland China (2005–2012).	Super-SBM model with undesirable outputs, combined with the Malmquist productivity index.	Labor employment; capital stock; energy consumption	GDP; CO2 emissions
Meng et al. [16]	LCE efficiency analysis of China’s provinces (2001–2014).	Range-adjusted measure DEA	Capital stock; labor energy consumption	Gross output value; CO2 emissions
Zha et al. [17]	Efficiency assessment of LCE tourism in the cities of Hubei province in China (2007–2013)	SBM-undesirable model and Luenberger index	Tourism resource endowments; number of employees; fixed assetinvestments	Revenue from tourism CO2 emissions from the tourism industry
Zhang et al. [18]	Measurement of LCE efficiency performance of 115 sample countries (1999–2013).	Super-SBM model with undesirable outputs, combined with the Malmquist productivity index.	Labor force; gross national expenditure; energy consumption	GDP; CO2 emissions
Chen et al. [19]	LCE development index for evaluating LCE development process of 56 Belt and Road Initiatives countries (2005–2015)	SBM-DEA model	Capital stock; labor; energy consumption	GDP; CO2 emissions
Zhang et al. [20]	LCE efficiency of 30 provinces in China	Super-efficiency SBM model	Capital stock; labor; energy consumption	GDP; CO2, SO2 emissions, industrial soot, industrial solid waste, and chemical oxygen demand

3. Methodology

In this work, we use the DEA model proposed by Tone, known as SBM [21]. In comparison to the Charnes-Cooper-Rhodes (CCR) and Banker-Charnes-Cooper (BCC) models (see [22,23]), this model provides a more comprehensive efficiency assessment tool because it is non-radial (assuming that inputs and outputs can vary in a non-proportional manner), and it can be input-, output- and non-oriented. Unlike radial models, which disregard slacks, the SBM model provides information on the enhancements required for the values of each inefficient decision-making unit (DMU)’s input and output, respectively. Differently from the additive model [24], which is also a non-radial efficiency measure model, the SBM model allows computing an efficiency score based on the slacks. Besides this, the SBM model can be coupled with clustering analysis by grouping DMUs according to certain characteristics, to evaluate the DMUs’ efficiency based on the cluster frontier, thus reducing the impact of DMUs’ heterogeneity on efficiency [25,26,27].

3.1. The SBM Output-Oriented Model

Consider the set of $n$ DMUs $\left(DM{U}_1,DM{U}_2,\dots , DM{U}_n\right)$, where X = [x_ij, i = 1, 2, …, m, j = 1, 2, …, n] is the (m × n) matrix of inputs, Y = [y_rj, r = 1, 2, …, s, j = 1, 2, …, n] is the vector of outputs (s × n) and the rows of these matrices, corresponding to the inputs and outputs of DMU_k, are respectively given by ${\mathit{x}}_k^T$ and ${\mathit{y}}_k^T$, with ^T denoting the transposing of a vector. The SBM output-oriented model can be given as [27]:

$$\begin{array}{l}\underset{\lambda ,{\mathit{s}}^{-},{\mathit{s}}^{+}}{Min}\rho =\frac{1}{\rho }=1+\frac{1}{s}{{\displaystyle \sum }}_{r=1}^s\frac{s_r^{+}}{y_{rk}}\\ \mathrm{s}.\mathrm{t}.x_{ik}={{\displaystyle \sum }}_{j=1}^n{x}_{ij}{\lambda }_j+s_i^{-},i=1,\dots ,m,\\ y_{rk}={{\displaystyle \sum }}_{j=1}^n{y}_{rj}{\lambda }_j-s_i^{+},r=1,\dots ,s,\\ {\lambda }_j\ge 0,j=1,\dots ,n,\\ s_i^{-}\ge 0,i=1,\dots ,m,\\ s_i^{+}\ge 0,r=1,\dots ,s.\end{array}$$

(1)

Model (1) employs the constant returns to scale (CRS) assumption. To consider variable returns to scale (VRS), it is only necessary to add the constraint ${\mathit{e}}^T\mathit{\lambda }=1$ to model (1).

In this context, the scale efficiency (σ_k) of DMU_k can be given as [27]:

$${\mathsf{\sigma }}_{\mathrm{k}} =\frac{{\rho }_k^{CRS}}{{\rho }_k^{VRS}},$$

(2)

where ${\rho }_k^{CRS}$ and ${\rho }_k^{VRS}$ correspond to the efficiency scores of DMU_k, based on the SBM model under CRS and VRS, respectively. The value of σ_k ranges between 0 and 1, with a larger value indicating a better scale condition.

Definition 1.

A DMU_k is SBM-output-efficient if${\rho }^{\ast }=1$. This condition is equivalent to${\mathit{s}}_{}^{+\ast }=\mathbf{0}$. However, it is possible that${\mathit{s}}_{}^{-\ast }\ne \mathbf{0}$.

Let $x_i^1=x_{ik}-s_i^{-}$ and $y_r^1$ $=y_{rk}+s_r^{+}$. Tone [28] suggested the following SBM output-oriented model to evaluate the super-efficiency of DMU_k:

$$\begin{array}{l}\underset{\lambda ,{\mathit{s}}^{-},{\mathit{s}}^{+}}{Min}{\rho }_1=1+\frac{1}{s}{{\displaystyle \sum }}_{r=1}^s\frac{y_r^1}{y_{rk}}\\ \mathrm{s}.\mathrm{t}.x_i^1\ge {{\displaystyle \sum }}_{j=1,j\ne k}^n{x}_{ij}{\lambda }_j,i=1,\dots ,m,\\ y_r^1\le {{\displaystyle \sum }}_{j=1,j\ne k}^n{y}_{rj}{\lambda }_j,r=1,\dots ,s,\\ {\lambda }_j\ge 0,j=1,\dots ,n,j\ne k,\\ x_{ik}\le x_i^1,i=1,\dots ,m,\\ 0\le y_r^1\le y_{rk},r=1,\dots ,s.\end{array}$$

(3)

The super-SBM-output-oriented model assesses the efficiency of an efficient DMU regarding the nearest point on the frontier except itself [28,29].

The optimal value of the objective function of model (3) is greater than or equal to 1, i.e., ${\rho }_1^{\ast }$≥ 1. Nevertheless, the super-efficiency for DMU_k will be 1 even if DMU_k is inefficient. Thus, to determine whether DMU_k is inefficient or not, both models (1) and (3) must be solved. If DMU_k is efficient according to model (1), model (3) should be used to obtain its super-efficiency score.

3.2. The SBM Model with Cluster Benchmarking

Traditional DEA models assume that all DMUs have similar characteristics. Therefore, all DMUs are considered to provide the reference set to construct meta-frontiers. In reality, DMUs are not always homogeneous, thus affecting the accuracy of DEA findings [30]. Hence, clustering benchmarking can be useful to deal with heterogeneous DMUs [25,26]. Cluster benchmarking is a technique used for separating a set of DMUs into groups (i.e., clusters) with specific characteristics. The clusters can be defined by using a clustering method (in the field of statistics) applicable to the problem under evaluation or provided exogenously, using experts’ knowledge, or established internally according to the degree of scale efficiency [27]. The DMUs that belong to the same cluster are more alike than those that belong to different clusters [26]. The main goal is to take advantage of the homogeneity of DMUs in the same cluster and the heterogeneity of DMUs in distinct clusters. To perform the efficiency assessment regarding the best practices according to the corresponding clusters, the production frontiers need to be built separately. By comparing the results of the non-grouped DEA model (i.e., by considering all DMUs as a reference set) and the results of self-benchmarking after grouping, the technology gap ratio (TGR) can be obtained. The meta-frontier and the cluster frontiers are obtained considering the output-oriented version of the SBM model (see Model (1)). The TGR_k of DMU_k is then calculated as [31]:

$$TG{R}_k=\frac{{{\mathsf{\rho }}^{\prime }}_k{}_{}^{meta \left(VRS\right)\ast }}{{{\mathsf{\rho }}^{\prime }}_k{}_{}^{cluster \left(VRS\right)\ast }},$$

(4)

where $\mathsf{\rho }{{}_k}_{}^{meta \left(VRS\right)\ast }$ is the SBM-output-efficiency value of DMU_k, computed according to the meta-frontier under VRS, and $\mathsf{\rho }{{}_k}_{}^{cluster \left(VRS\right)\ast }$ is the SBM-output-efficiency value of DMU_k obtained according to the cluster frontier under VRS.

The value of the TGR may identify the gap between the cluster frontier and the meta-frontier. It is used to evaluate the technical efficiency gap of the same DMU, according to distinct frontiers. Besides this, TGR can further indicate the need for separating different groups [31]. The lower the TGR value is, the bigger the requirement of grouping will be, and vice versa. Since ${{\mathsf{\rho }}^{\prime }}_k{}_{}^{cluster \left(VRS\right)\ast }$ ≥ ${{\mathsf{\rho }}^{\prime }}_k{}_{}^{meta \left(VRS\right)\ast }$, the value of TGR_k varies between 0 and 1 [25,26]. A TGR_k closer to 1 suggests that there is a small gap between the meta-frontier and the cluster frontier. The meta-frontier shows the underlying technical level of the entire assessed group of individuals, and the cluster frontier depicts the real technical level of each cluster. For example, if we assume that ${{\mathsf{\rho }}^{\prime }}_k{}_{}^{meta\ast }$ = 0.6 and ${{\mathsf{\rho }}^{\prime }}_k{}_{}^{cluster\ast }$= 0.8, this implies that the maximum output that could be produced by the DMU_k, which belongs to the cluster under analysis, is 75% of the output that is feasible when using the meta-frontier as a benchmark. The higher the value of TGR_k, the smaller the gap between the meta-frontier and the cluster frontier and the smaller the gap between the technology used by the DMU and the technology frontier.

Expression (4) allows us to obtain the following decomposition of the efficiency of DMU_k for a particular input–output combination:

$${{\mathsf{\rho }}^{\prime }}_k{}_{}^{meta \left(VRS\right)\ast }={{\mathsf{\rho }}^{\prime }}_k{}_{}^{cluster \left(VRS\right)\ast } \times TG{R}_k.$$

(5)

Expression (5) shows that the efficiency of DMU_k, measured according to the meta-frontier (expressing the current state of knowledge), can be split into the product of the efficiency obtained for the cluster frontier (following the current state of knowledge and the characteristics of the cluster under analysis) and the TGR for the cluster under analysis (which evaluates how close the cluster frontier is to the meta-frontier). Finally, since both clusters and scale efficiencies might significantly influence the outcomes attained, Expression (5) can be further decomposed into:

$${{\mathsf{\rho }}^{\prime }}_k{}_{}^{meta \left(VRS\right)\ast }={{\mathsf{\rho }}^{\prime }}_k{}_{}^{cluster \left(CRS\right)\ast } \times {\sigma }_k{}_{}^{cluster \ast } \times TG{R}_k.$$

(6)

where ${{\mathsf{\rho }}^{\prime }}_k{}_{}^{cluster \left(CRS\right)\ast }$ is the efficiency value of DMU_k obtained according to the cluster frontier under CRS and ${\sigma }_k{}_{}^{cluster \ast }$ is the scale efficiency found for the cluster under evaluation.

Hence, if ${\sigma }_k{}_{}^{cluster \ast }=1$, then DMU_k has no scale demerits, and its slacks are imputed to itself [27]. For example, if ${\sigma }_k{}_{}^{cluster \ast }$ = 0.25, then 75% of the slacks are ascribed to the DMU’s scale demerits.

4. Data and Assumptions

We involved the stakeholders in the choice of a pre-selected set of indicators (the pre-selected set of indicators presented to the stakeholders was based on the scientific literature and on the set of common indicators used by the European Commission in the evaluation of TO 4), by conducting a facilitated workshop on the topic “Evaluating the Co-financed Intervention Policies in Enterprises” with specific policymakers and MA. Besides this, we performed a separate evaluation of regional and national programs, which were grouped into clusters according to the categories of the regions under analysis (i.e., less developed regions, regions of transition and more developed regions). The values considered are cumulative values from different years released on the 19th of November 2021 since these are the most updated data available for the achievement indicators. In our assessment, we have only studied the programs with no missing information (i.e., programs with missing data were eliminated), leading to the consideration of 22 countries and 102 programs (the DMUs).

The input and output factors considered for performing an efficiency assessment of the implementation of the ESI funds devoted to LCE interventions in SMEs were selected from the list of common indicators that are legally required by the EU [32] and are described below (see also Table 2).

Table 2. Inputs and outputs selected.

	EU Co-Financing	Total Eligible Spending	Eligible Cost Decided	GHG Reduction
Description	Percentage of EU financing (calculated as an average).	Eligible costs validated	Financial resources assigned	Estimated annual decrease of GHG
Type of factor	Input	Output	Input	Output
Unit	%	Euro	Euro	Tons of CO2 Equivalent
Source	(a)	(a)	(a)	(b), (c)
Explanation	Considers concerns with the financial absorption capacity of the country or region	Reflects concerns about the pace of programs’ implementation	Reflects concerns about the pace of programs’ implementation	Reflects concerns on LCE
Reference	[8,33,34,35]	[8,36,37]	[8,36,37]	[38,39]

(a) List of structural funds, in terms of financial implemented data. Available at: https://cohesiondata.ec.europa.eu/2014-2020/ESIF-2014-2020-Finance-Implementation-Details/99js-gm52 (accessed on 19 November 2021); (b) list of common indicators that are legally required and listed in the annexes to the ERDF, Cohesion Fund and ETC regulations. Available at: https://ec.europa.eu/regional_policy/sources/docoffic/2014/working/wd_2014_en.pdf (accessed on 19 November 2021); (c) list of structural funds’ achievement data. Available at: https://cohesiondata.ec.europa.eu/2014-2020/ESIF-2014-2020-Achievement-Details/aesb-873i (accessed on 19 November 2021).

4.1. Financial Absorption Capacity

According to [33], an efficiency assessment of the deployment of EU structural funds should specifically involve consideration of the capacity of absorption of EU funds by a given region/group of regions/countries. The financial absorption capacity corresponds to the capacity of co-financing by the MS [34]. A higher financial absorption capacity means lower dependency on EU co-financing of the country/region. Besides this, a rise in the EU co-financing rate relieves the burden on the national budget by reducing the MS’s initial budgeted contribution to the OP but also diminishes the OP’s total value. This indicates that the scale (number or size) of the interventions will be lowered as well, unless the MS keeps investing the funds originally envisioned at the national level. As a result, this may have a negative effect on the aims and outcomes that the OPs may accomplish, which, in turn, impacts the value that the awarded EU funds can deliver [35]. Therefore, the “percentage of EU co-financing”, which measures the weight of EU financing on the total financing received (including the national financing), is used as a proxy to measure the financial absorption capacity. To maximize this, the percentage of EU co-financing in total financing should be minimized and, therefore, should be considered as an input [8].

4.2. The Pace of the Programs’ Implementation

Since its inception, efforts have been made to evaluate and improve the implementation of the cohesion policy [36]. In fact, the goal of enhancing financial performance, or spending capacity, in MS and regions implementing OPs is fundamental. Although the pace of implementation is only one indicator of effectiveness, the extent to which MS and MA are capable of spending their allocated funding effectively and efficiently provides a basic method for assessing the implementation progress.

The assessment of financial implementation is critical because deferred or irregular financial performance in the first half of the programming period generates pressures on MA that may result in a negative effect on efficient and effective OP implementation and closure [36].

Therefore, the financial execution of structural funds, which is a necessary condition for effective policy implementation, should also be incorporated into the analysis, particularly when addressing the pace of the programs’ implementation [37]. In this framework, costs that are not eligible or in agreement with the applicable eligibility rules cannot be claimed. These must be validated by an accredited controller; this is the body or person accountable for validating, at the national level, that the co-financed products and services have been delivered, that the related expenditures have been paid and that a particular project conforms to the applicable EU program and national rules. On the one hand, the “total eligible spending” refers to those eligible costs reported by the selected projects that have been validated by this controller. Hence, the higher its value, the higher each project’s financial execution, and, thus, it is used herein as an output. On the other hand, the “eligible costs decided” refer to those costs that have financial resources assigned to the projects selected for funding (project pipeline). Therefore, these should be minimized (and perceived as inputs), so that the pace of the programs’ implementation is further enhanced.

4.3. Energy and Climate Change

TO 4 encourages the transition to an LCE. This goal is driven first and foremost by the climate agenda, which seeks to reduce the harmful impacts of anthropogenic GHG emissions. The scope is largely focused on a shift that necessitates both physical changes (lower net emissions from a technical infrastructure) and behavioral changes (energy savings), both of which are supported by new technologies and solutions [39].

The investment priority devoted to SMEs (investment priority 4b) is focused on promoting energy efficiency and renewable use, with two types of intervention [38,39].

For energy efficiency measures, the values are computed according to the amount of primary energy saved through the adoption of the supported operations in a given year (either one year after project completion or the calendar year after project completion) [39]. The saved energy is expected to replace non-renewable energy generation. The MS’s total GHG emission per unit of non-renewable energy output is used to calculate the GHG effect of non-renewable energy [39].

In the case of renewable energy production, the estimated values are computed according to the quantity of primary energy produced by the supported facilities in a particular year (either one year after project completion or the calendar year after project completion) [38]. Renewable energy is expected to be GHG-free [39].

Either way, “GHG reduction” is an indicator that is specifically calculated for interventions directly aiming to increase renewable energy production or to decrease energy consumption through energy-saving measures. This indicator should be maximized and is employed as an output.

Data on these indicators are given in Table A2 in Appendix B.

From an analysis of Table 3, it might be established that the average EU co-financing support was 60.77%, 78.672% for the less developed regions, 61.43% for the regions of transition and 49.10% for the more developed regions, respectively. The overall average financial execution rate (i.e., the ratio between the total eligible spending and the total eligible cost) is low (32.33%) and it is even lower for the regions of transition (31.2%), presenting slightly higher values (but still below 50%) for the more developed regions and less developed regions, with values of 39.6% and 39.2%, respectively. Finally, the highest average decrease of GHG is attained (in decreasing order of magnitude) for the regions of transition, followed by the less developed and more developed regions.

Table 3. Descriptive statistics of the inputs and outputs.

Statistics	EU Co-Financing (%)	Total Eligible Cost (Euros)	Total Eligible Spending (Euros)	Decrease of GHG (Tonnes of CO2 eq.)
Number of observations	102	102	102	102
Number of observations (MD)	46	46	46	46
Number of observations (T)	27	27	27	27
Number of observations (LD)	29	29	29	29
Minimum	22.313	17,878,192.280	5,547,887.690	15.000
Minimum (MD)	26.450	17,878,192.280	7,342,763.330	15.000
Minimum (T)	22.313	32,171,439.920	8,325,490.890	20.640
Minimum (LD)	51.890	151,556,615.010	5,547,887.690	218.000
Maximum	85.000	6,832,205,666.950	2,362,923,936.480	713,649.450
Maximum (MD)	80.000	2,646,766,161.980	1,087,091,636.850	162,937.000
Maximum (T)	82.500	6,832,205,666.950	2,362,923,936.480	713,649.450
Maximum (LD)	85.000	3,453,293,426.710	1,600,903,860.740	200,179.000
Mean	60.771	967,862,322.742	348,487,623.566	31,568.860
Mean (MD)	49.099	530,703,953.070	210,210,041.310	16,274.754
Mean (T)	61.430	1,500,715,173.980	468,385,089.590	53,076.290
Mean (LD)	78.672	1,165,181,564.862	456,195,802.918	35,804.318
Standard deviation	17.905	1,248,435,343.104	450,978,493.120	79,450.970
Standard deviation (MD)	11.668	562,478,016.766	242,828,685.991	28,391.834
Standard deviation (T)	16.934	1,916,458,715.137	635,595,328.666	137,418.013
Standard deviation (LD)	10.379	976,808,877.573	436,320,089.219	50,227.247

Note: T—transition regions; MD—more developed regions; LD—less developed regions.

A requirement of DEA refers to input and output factors that should hold an isotonic relationship, which can be validated by correlation analysis [40]. There is a need to guarantee that the link between inputs and outputs is not inconsistent. Increasing the value of any input whilst holding other factors constant should not diminish any output but should rather lead to an expansion in the value of at least one output. If the correlation between input and output factors is positive (and significant), this indicates that the factors verify an isotonic relationship. In this case, since the normality assumption for the application of the tests for the significance of Pearson’s correlation is not verified, we opted for obtaining the Spearman correlation coefficients and the corresponding significance tests—see Table 4. The results obtained corroborate the isotonic relationship of the inputs and outputs used in the analysis.

Table 4. Correlation matrix.

Variable	EU Co-Financing (%)	Total Eligible Cost (Euros)	Total Eligible Spending (Euros)	Decrease of GHG (Tonnes of CO2 eq.)
EU co-financing	1
Total Eligible Cost	0.256 ** (0.009)	1
Tota Eligible Spending	0.209 * (0.035)	0.932 ** (0.000)	1
Decrease of GHG	0.216 * (0.030)	0.380 ** (0.000)	0.464 ** (0.000)	1

Note: Spearman correlation coefficients are presented in this table; in parentheses are the associated p-values of bi-directional tests. * Significance level of 5%; ** significance level of 1%.

5. Discussion of Results

The results were computed with the MaxDEA 8 Ultra software. The values of the TGR, together with the technical efficiencies (CRS and VRS) obtained for the distinct region categories and the meta-frontier, were computed for all OPs. Basic descriptive statistics for these measures are shown in Table 5.

Table 5. Descriptive statistics of the results obtained for the OPs.

Region Category	Statistics	Technical Efficiency Score (CRS)	Pure Technical Efficiency Score (VRS)	Scale Effect Score	TGR
More Developed	Mean	0.333	0.429	0.859	0.896
	Standard Deviation	0.361	0.410	0.255	0.230
	Minimum	0.001	0.001	0.045	0.036
	Maximum	1.000	1.000	1.000	1.000
	Sum	15.337	19.745	39.493	41.230
	Count	46	46	46	46
Transition	Mean	0.356	0.490	0.835	0.770
	Standard Deviation	0.403	0.449	0.305	0.257
	Minimum	0.000	0.000	0.001	0.052
	Maximum	1.000	1.000	1.000	1.000
	Sum	9.614	13.217	22.558	20.797
	Count	27	27	27	27
Less Developed	Mean	0.536	0.618	0.901	0.392
	Standard Deviation	0.399	0.401	0.242	0.235
	Minimum	0.025	0.030	0.058	0.004
	Maximum	1.000	1.000	1.000	1.000
	Sum	15.538	17.918	26.138	11.375
	Count	29	29	29	29

Figure 3 depicts the efficiency scores based on the meta- and cluster frontiers, respectively.

Figure 3. Efficiency scores of 102 OPs on meta- and cluster frontiers. Note: Efficiency scores are presented from the left to the right for the more developed, transition and less developed regions, respectively.

It is worth noting that the cluster frontiers of more developed regions and transition regions were almost tangential to the meta-frontier. The average values of the TGR vary from 0.392 (for less developed regions) to 0.896 (for more developed regions), suggesting the existence of a huge gap between the two frontiers, particularly in the case of the less developed regions (based on the cluster frontier, the number of efficient DMUs—score equals to 1—increased from 2 to 13). Overall, the number of OPs considered to be efficient increased from 19 (meta-frontier) to 35 (cluster frontier)—see Figure 4.

Figure 4. Number of OPs at different subintervals of super-efficiency scores, based on cluster- and meta-frontiers.

This result is mostly influenced by the TGR reached by LCE OPs in less developed regions, which produce, on average, only about 39.2% of the potential output, given the technology available for this type of OP as a whole (22 countries are represented)—see Table 5. Nevertheless, more developed regions produce, on average, 89.6% of the potential outputs (Table 5), whereas the regions of transition produce, on average, 77% of the potential outputs (Table 5). Figure 4 shows the number of OPs at different subintervals of super-efficiency scores, based on cluster- and meta-frontiers, respectively. According to the cluster frontiers, 35 OPs were relatively efficient, suggesting that 34.31% of these were maximizing their outputs. Overall, more developed regions and regions of transition show higher room-for-improvement potentials of 57% and 51%, respectively, while less developed regions had a smaller room for improvement of 48%. Besides, on average, about 10%, of the adjustments required to make the non-efficient OPs of less developed regions efficient are due to scale demerits, whereas 16% and 14% are attained for the regions of transition and more developed regions, respectively.

From an analysis of Figure 5, it can be ascertained that the difference in the average EU co-financing between efficient and inefficient OPs was not significant. Although the mean funds devoted to the eligible cost of efficient OPs (EUR 1,126,456,826.26) were slightly higher than that of inefficient ones (EUR 881,248,217.61), the mean eligible spending of efficient OPs (EUR 477,458,910.63) was substantially higher than that of inefficient ones (EUR 283,346,601.70). Furthermore, the mean reduction of GHG emissions of efficient OPs was 248% higher than that attained by inefficient OPs.

Figure 5. Average input and output values for efficient and inefficient OPs.

5.1. Potential Improvements

The SBM model allows us to identify in inefficient DMUs the potential improvements that inputs and outputs should undergo to become efficient. These outcomes are illustrated in Figure 6, Figure 7 and Figure 8, sorted both by operational program and by region category. The decrease in GHG emissions has the greatest improvement potential, which could improve by about 514% (i.e., the reduction of GHG emissions can increase on average from 16,719 to 102,577 tonnes of CO₂ eq.) according to the current input levels. Either way, the regions of transition and the more developed regions show the highest room-for-improvement potentials of 641% and 553%, respectively, whereas the lowest improvement potential belongs to the less developed regions, i.e., 278%. The pace of the programs’ implementation also shows an overall room for improvement (27%), particularly for the regions of transition (49%), followed by more developed and less developed regions, with potentials for improvement of 20% and 18%, respectively. Under the current output level, there was almost no input surplus, although more developed regions show better potential for improving their dependence on EU co-financing (−7%) and eligible costs (−8%). Hence, to foster an LCE, OPs managers should focus on solving problems that further enhance both GHG reduction and the pace of the programs’ implementation.

Figure 6. Potential improvement of each input for every inefficient OP.

Figure 7. Potential improvement of each output for every inefficient OP.

Figure 8. Average potential improvement of each input and output for every region category. Note: T—transition regions; MD—more developed regions; LD—less developed regions.

5.2. Robustness Analysis

Traditional DEA methods use crisp values for both the inputs and outputs. Nevertheless, in real-world problems, the values used to instantiate the input and output data are occasionally subject to uncertainty. In this context, imprecise or vague data can be conveyed within interval ranges, as ordinal (rank-order) data or as fuzzy numbers [41]. Out of the approaches available in the literature that include the applications of fuzzy set theory in DEA, the α-level approach is possibly the most popular one. This approach consists of converting the fuzzy DEA model into a pair of parametric programs to find the lower and upper bounds of the α-level of the membership functions of the efficiency scores. The MaxDEA software converts the DEA model that has fuzzy inputs/outputs with bounded intervals into a pair of standard DEA models, so that the lower and upper boundaries of the efficiency scores are computed.

This sort of analysis is particularly important to evaluate the robustness of the results obtained, particularly if the programs are still underway. Hence, it is possible to anticipate the impact that potential changes on the output levels might have on the efficiency scores of OPs, given their current levels of input. In particular, we consider that the perturbations in the value of each factor are within an interval range. This interval is found by applying a common tolerance $\delta$ to the output factors, such that $y_{ij}^L=y_{ij}^{}\left(1-\delta \right)\le y_{ij}^{}\le y_{ij}^{}\left(1+\delta \right)=y_{ij}^U$, where L and U designate the lower and upper boundaries, respectively. Besides this, we simultaneously consider data perturbations according to a worst-case scenario and a best-case scenario. While the former presumes increased outputs for all other DMUs and decreased outputs for the DMU under assessment (i.e., the efficiency of DMU_k declines and the efficiency of all the other DMUs improve), the latter supposes the reverse situation. In this context, a DMU is robust to changes in its output factors if it remains efficient (or inefficient), and it thus can be stated as robustly efficient (or robustly inefficient) for the tolerance considered.

If we consider a potential change of the outputs within the tolerance of δ = 5% with the current inputs, 27 programs remain robustly efficient (9, 10 and 8 belong to more developed, transition, and less developed regions, respectively), 59 remain robustly inefficient and the remaining 16 are potentially efficient—see Figure 9. For a tolerance of δ = 10%, 25 programs remain robustly efficient (8, 10 and 7 belong to more developed, transition, and less developed regions, respectively), whereas 51 become robustly inefficient and 26 become potentially efficient. Overall, if the outputs suffer an increase from 5% to 10%, the potential for improvement increases, particularly for less developed regions, from 3% to 17%, whereas for the regions of transition, it just increases from 9% to 10%. Finally, with this change in outputs, more developed regions also have a high room-for-improvement value from 12% to 33%—see Figure 10. Table 6 provides some characteristics of the robustly efficient OPs for both tolerances. From an analysis of this table, it can be seen that the three OPs that are more often viewed as benchmarks in their corresponding clusters are also those more often viewed as benchmarks in the meta-frontier (these are OPs from Ireland, Italy, and Spain). Besides this, it can also be concluded that less developed regions are the ones that usually prefer to perform more intermediate assessments (see, e.g., the cases of the OPs from Poland and Lithuania). There is only one operational program belonging to a less developed region (see the case of Lithuania) that also serves as a benchmark for the other programs in the meta-frontier (but this is just one time). Finally, it can also be established that despite the variety of the number of evaluations conducted by each MS for each robustly efficient operational program, most of them perform some kind of evaluation. Specifically, the majority of MS perform monitoring (43) and process (34) evaluations, with only a few (6) presenting impact evaluations.

Figure 9. Upper-bound and lower-bound efficiency scores for each OP, according to different tolerances.

Figure 10. Average upper bound and lower bound efficiency scores for each region category. Note: T—transition regions; MD—more developed regions; LD—less developed regions.

Table 6. Some characteristics of the robustly efficient OPs for $\delta$ = 5% and $\delta$ = 10%.

Operational Program	Country 1	Region Category	Nº of Times as Benchmark (Cluster Frontier)	Nº of Times as Benchmark (Meta-Frontier)	Impact Evaluation (nº)	Monitoring Evaluation (nº)	Process Evaluation (nº)
Southern and Eastern Regional Programme—IE—ERDF	IE	MD	23	63	0	1	1
Liguria—ERDF	IT	MD	21	40	0	0	0
Multi-regional Spain—ERDF	ES	T	14	37	0	3	1
Śląskie Voivodeship—ERDF/ESF	PL	LD	13	-	1	6	6
Cantabria—ERDF	ES	MD	11	13	0	3	1
Fostering a competitive and sustainable economy—MT—ERDF/CF	MT	T	11	1	0	0	0
Innovation and Sustainable Growth in Businesses—DK—ERDF	DK	T	10	-	0	1	1
Martinique—ERDF/ESF/YEI	FR	LD	10	-	0	0	0
Pays de la Loire—ERDF/ESF	FR	MD	10	19	0	0	1
Investments in Growth and Employment—AT—ERDF	AT	T	9	14	1	3	4
Nordrhein-Westfalen—ERDF	DE	MD	7	-	1	1	1
EU Structural Funds Investments—LT—ERDF/ESF/CF/YEI	LT	LD	6	2	2	5	6
Abruzzo—ERDF	IT	T	4	-	0	5	1
Łódzkie Voivodeship—ERDF/ESF	PL	LD	4	-	0	2	2
Alsace—ERDF	FR	MD	3	3	0	1	0
Franche-Comté et Jura—ERDF/ESF	FR	T	3	1	1	4	4
Mayotte—ERDF/ESF	FR	LD	3	-	0	0	0
Saarland—ERDF	DE	MD	3	2	0	0	0
Castilla-La Mancha—ERDF	ES	T	2	2	0	2	1
Champagne-Ardenne—ERDF/ESF/YEI	FR	MD	2	-	0	1	0
Picardie—ERDF/ESF/YEI	FR	MD	2	1	0	0	1
Quality of Environment—SK—ERDF/CF	SK	LD	2	-	0	1	0
Melilla—ERDF	ES	T	1	-	0	2	2
Molise—ERDF/ESF	IT	T	1	-	0	1	0
Zachodniomorskie Voivodeship—ERDF/ESF	PL	LD	1	-	0	1	1

¹ The results presented include acronyms for each MS as follows: AT—Austria; DE—Germany; DK—Denmark; ES—Spain; FR—France; IE—Ireland; IT—Italy; LT—Lithuania; MT—Malta; PL—Poland; SK—Slovakia.

5.3. Sensitivity Analysis

Since the SBM-DEA approach is a non-parametric method, a distinct approach is usually used to perform sensitivity analysis. This consists of eliminating one factor of evaluation (input or output) at a time and assessing the level of change attained thereby in terms of efficiency [42].

In addition, because the SBM-DEA model used in the assessment of the OPs is output-oriented, we have considered two simple regression models in which the dependent variables are the scores obtained when omitting one output at a time, and the independent variable is the original score. Through these models, it is possible to obtain the slope and the corresponding coefficient of determination (or r-square). The sensitivity of efficiency to the changes in the variables (i.e., outputs) can thus be identified by the gap between the value 1 and the slope of the simple regression function [25,43], meaning that the sensitivity of efficiency to changes in the outputs increases with this gap.

The results of the sensitivity analysis are presented in Figure 11 and Table 7. The factor that shows higher impacts on efficiency is GHG reduction, since the omission of this variable leads to the highest value of |1-slope|, 0.590, whereas the total eligible spending has a smaller impact on efficiency. These results suggest that the type of interventions targeted for funding are critical for producing desirable results regarding an LCE.

Figure 11. Sensitivity analysis for GHG reduction (a) and total eligible spending (b). The y-axis of each graph represents the recalculated efficiency by omitting one variable at a time, and the x-axis represents the original efficiency score. The solid blue lines represent the lines of the best fit.

Table 7. Sensitivity analysis results.

Variables	Slope	|1-slope|	R2	Classification
GHG reduction	0.410	0.590	0.426	Output
Total eligible spending	0.833	0.167	0.724

5.4. Policy Implications

Although 50% of the OPs remain robustly inefficient for both tolerances, from the analysis of Table 8 (further information is obtainable from Table A3 and Table A4 in Appendix C and Appendix D, respectively), it can be concluded that 73% of the inefficient OPs do not need adjustments in EU co-financing. Nevertheless, there are at least four OPs (from more developed regions) that require a meaningful reduction of EU co-financing (of between 34% and 54%) to become efficient. Particularly in such cases, the management structures should consider reprogramming the OPs to reallocate EU co-funding from more developed regions to less developed regions. In terms of the funding assigned to eligible costs, 88% of inefficient OPs do not require adjustments; however, there are four funds (3 from more developed regions and 1 from a region of transition) that require a reduction of funding of between 20% and 32% (see Table A3 and Table A4). Then again, MA should have enough flexibility to decide how to change the coverage of the funding, thus contemplating other less developed regions and/or other investment priorities, if needed. One of the critical issues that preclude OPs’ efficiency is their implementation pace. There are 11 OPs that need to more than double their performance and 7 that need to increase it by more than 75%. In these cases, it is important to understand the true reasons behind the numbers. Some OPs evaluations suggest that some firms have withdrawn their subsidies, probably because of their difficulties in obtaining bank credit for co-financing. In such situations, the MA should be able to support firms in finding other sources of funding, e.g., venture capital funds, business angels and crowdfunding, also facilitating the conditions for bringing other institutional investors. If none of these possibilities is viable, funding should be transferred to regions or other investment priorities with more capacity to spend. Besides, to accelerate the implementation of the OPs, best practices from other countries need to be considered and bureaucratic requirements for accessing funding should be reduced. Management structures should find ways of making project implementation more efficient by promoting the simplification of procedures for preparing and submitting the application for payment requests. The reduction of the implementation failure rate should also involve additional opportunities for direct consultation and better support and guidance of applicants.

Table 8. Policy implications and suggestions for inefficient OPs.

Factor under Evaluation	Nº of OPs	Average % Change	Policy Implications	Suggestions
EU co-financing (absorption capacity)	49	0%	No adjustments required for EU co-financing	No suggestions
	14	−5%	EU co-financing should be slightly reduced	Allow some flexibility in reallocation of EU co-funding between regions.
	3	−34%	EU co-financing should be reduced	Allow more flexibility in reallocation of EU co-funding between regions.
	1	−54%	EU co-financing should be moderately reduced	Reallocate EU co-funding to other regions.
Eligible cost (funding assigned)	59	0%	No adjustments required for eligible funding	No suggestions
	4	−11%	Eligible funding should be slightly reduced	Allow some flexibility in the reallocation of funding between regions and investment priorities.
	4	−25%	Eligible funding should be reduced	Allow more flexibility in reallocation of funding between regions and Investment Priorities.
Eligible spending (implementation rate)	20	0%	No adjustments required for Implementation	No suggestions
	14	8%	Slightly improve implementation	Promote the simplification of procedures for preparing and submitting applications and for payment requests.
	10	32%	Improve implementation	Better clarify the eligibility and selection criteria for each of the calls in order to reduce the failure rate of implementation.
	5	47%	Moderately improve implementation	Create additional opportunities for direct consultation and increase the support and guidance given to applicants
	7	76%	Strongly improve implementation	Envisage specific training and exchange of best practice for managing bodies to improve implementation.
	11	227%	Shift funding to regions or other investment priorities with more capacity to spend	Shift funding to regions or other investment priorities with more capacity to spend.
GHG Reduction (LCE)	2	7%	Need to finance projects that make a slight contribution to meeting climate-related challenges	Further invest in initiatives to reduce carbon dioxide emissions and mitigate the effects of climate change.
	4	27%	Need to finance projects that make some contribution to meeting climate-related challenges	Focus on reducing current energy consumption of businesses; expand the types of renewable sources eligible for funding and define the rules for the joint use of the ERDF with other subsidies.
	2	78%	Need to finance projects that make a large contribution to meeting climate-related challenges	Promote greater involvement of key stakeholders in the planning phase.
	59	30,847%	Assess and change investment strategy	Envisage specific training and exchange of best practice for managing bodies to improve the choice of the projects selected for funding.

Finally, it can be ascertained that GHG emission reduction is actually the indicator that needs greater attention since 59 OPs need to improve their performance substantially in terms of this factor. Further analysis is required, namely, regarding the possibility of replacing or flanking the MA with a competent alternative body or even offering specific training and the exchange of best practices to improve the choice of the projects selected for funding. OPs should ensure better links between policy objectives, actions undertaken and indicators, and, if needed, promote greater involvement of key stakeholders in the planning phase.

Overall, the MA should monitor closely the execution of OPs, especially those indicators that have implementation delays, and revise the allocation of funding if needed.

6. Conclusions

This work aims to evaluate the efficiency of the implementation of structural funds in different OPs from distinct EU beneficiary countries and regions by using data provided by the European Commission. These data involve financial implementation and expected results (targets) in the field of LCEs for the 2014–2020 ESI funds devoted to SMEs under the ERDF. In addition to representing the latest reported information available, these data are well suited for use with non-parametric methods, such as DEA. The main advantage of using this type of approach is the richness of information that they can offer to MA regarding the inefficiency of the Ops, which are compared against their peers. Through DEA, the benchmarks of inefficient OPs are also identified and pertinent information can be obtained thereby regarding the best practices to follow to achieve greater efficiency.

Although DEA has indisputable advantages over other traditional approaches (e.g., microeconomic studies, which use control groups, and case study evaluation), there is still a lack of scholarly attention regarding its use in the framework of structural funds efficiency assessment. Therefore, one of the novelties of our research consists of using the SBM-DEA model, combined with cluster analysis, to evaluate the efficiency of the implementation of structural funds dedicated to the promotion of an LCE in SMEs.

The use of the SBM measure employed herein, besides identifying the benchmarks and the adjustments required (given by the slacks) to improve the implementation of these funds, also enables exploiting the differences between the distinct regions’ categories in a single model. This approach also involves the use of two approaches in the robustness assessment of the results obtained, something which is not usually performed even in other broader studies within the framework of LCE. This type of analysis is particularly relevant if the programs are still in progress, enabling MA to foresee the impact that potential adjustments on the output levels might have on the efficiency scores of OPs, given their levels of inputs.

Differently from other tools and methods that are specifically used for the ex-post or ex-ante evaluation of cohesion policies, the DEA method also enables us to evaluate the efficiency of OPs’ implementation during the programming period, so that the required policies can be adopted within the time needed for making the necessary adjustments during the time horizon in progress.

DEA models can easily be adjusted to evaluate other thematic objectives, as long as the rule of thumb established by [17] is retained, i.e., the number of DMUs under assessment should be at least the double of the number of input and output factors considered.

Overall, regarding the first research question (“Which factors require special attention for reaching an efficient implementation of the funds devoted to fostering an LCE in the EU?”), we were able to conclude that:

(1)

MA should pay special attention to GHG emission reduction, since 59 OPs need to further improve their performance substantially. In such cases, consideration should be given to the prospect of replacing or supplementing these MAs with a capable alternative organization/group of experts, or perhaps providing particular training and exchanging best practices to improve the selection of projects for funding.

(2)

The other main important factor that precludes OPs’ efficiency is their implementation rate. Specifically, there are 11 OPs that have to more than double their performance and 7 who must enhance it by more than 75%. Either way, it is vital to understand the reasons behind these numbers. There are evaluation reports that suggest that some enterprises have withdrawn their subsidies, most likely due to issues related to bank credit. In this context, MA should be able to assist enterprises in obtaining other funding sources, while also easing the criteria for bringing in other institutional investors. Furthermore, in order to expedite the implementation of the OPs, best practices from other countries should be explored, and bureaucratic barriers to getting financing should be minimized. Management structures should seek ways to improve project implementation by supporting the simplification of procedures for submitting payment requests, also providing enhanced assistance and guidance.

Concerning our second research question (“Which OPs were more often viewed as benchmarks during the programming period under evaluation?”), we were able to establish that:

(1)

The OPs more often viewed as benchmarks (either in the meta- or cluster frontiers) were, by decreasing order of importance, the “Southern and Eastern Regional Programme—IE—ERDF”, “Liguria—ERDF”, “Multi-regional Spain—ERDF”, “Pays de la Loire—ERDF/ESF”, “Investments in Growth and Employment—AT—ERDF and “Cantabria—ERDF”.

(2)

There is only one OP from a less developed region (see the case of Lithuania) that also serves as a benchmark (i.e., as a reference of best practices) for the other OPs in the meta-frontier (but just one time).

In what regards our third research question (“Were the OPs robustly efficient in the face of potential changes of the performance framework indicators used?”), we were able to find that:

(1)

50% of LCE OPs were robustly inefficient for the tolerances considered in the analysis, highlighting the difficulty of further improving the efficiency of these OPs.

In terms of the fourth research question (“Which type of regions managed to attain higher LCE performance”?), we were able to ascertain that:

(1)

Our findings appear to support the apparent paradox of the EU Cohesion Policy, since the OPs which are more often viewed as benchmarks according to the meta-frontier belong to more developed regions. In fact, if we had not grouped our OPs according to the region type, we would only have 2 efficient OPs from the less developed regions (instead of the 13 obtained with the cluster frontier analysis).

(2)

The TGR reached by LCE OPs in less developed regions suggests that these produce, on average, only about 39.2%, of the potential output, given the technology available for this type of OPs in EU countries. Therefore, it appears that there is a positive association between the efficiency of the implementation of the OPs and the more advantageous socioeconomic circumstances of the regions where the OPs are implemented.

Finally, we are also aware that although the performance framework provides a group of common indicators, a match between the data gathered for the achievement indicators and the data from financial implementation is not fully possible. This is particularly true regarding the investment priority devoted to SMEs (investment priority 4b), which is focused on promoting energy efficiency and renewable energies, that has data for the achievement indicators, but it does not have data with that level of detail in the financial implementation. Moreover, the data reported is often lacking, thus leading to the consideration of a smaller number of indicators and OPs in our assessment.

Further research should be conducted to evaluate the efficiency of the implementation of other OPs in different TO, eventually considering other performance framework indicators.