**1. Introduction**

Global warming, as it is called, is simply the heating up of the atmosphere attributed mainly to the greenhouse effect caused by the rising concentrations of carbon dioxide, chlorofluorocarbons (CFCs), and other air pollutants. This extreme phenomenon is the primary reason for the climate change that is considered to be an environmental crisis. During the United Nations Framework Convention on Climate Change, several state parties agreed to pursue efforts to reduce this century's average increase of the Earth's temperature from 2 degrees Celsius to 1.5 degrees Celsius. The consensus entered force last November 2016 and is referred to as the Paris Agreement [1]. One foreseen solution is to slowly transform from conventional energy production based on fossil fuel combustion to renewable energy resources. Fuel combustion is a traditional way to generate electric power through the burning of coal and fossil fuels. Many countries are dependent on this practice. Even some developed ones are major contributors to the presence of CO2 in the atmosphere caused by this process [2].

In the fight against climate change, other nations are now making use of different renewable energy sources. In this way, they can lessen the use of CO2-emitting methods in the production of electricity which is very essential to human living standards. Renewable energies are sources of energy which are continuously being replenished naturally by the Earth itself. These energies can be obtained straight from the Sun (solar power and thermal), wind power, hydroelectric power, tidal or wave energy, and geothermal and biomass, but the transition to these energy sources can be difficult and costly. Though many potential benefits can ensue, there are also some technical limitations that must be considered. Also, instead of providing an additional solution to the climate change problem, the improper utilization of renewable energy may lead to more damage to the environment. These environmental damages may be caused by the disadvantages of using renewable energies such as air pollution caused by biomass burning, corrosion problems when using geothermal energy, risk of flooding in the communities surrounding a hydropower plant, negative impact on marine wildlife in using marine energy, and impact on the environmental landscape of using solar and wind energies [3].

There are international agencies and organizations that aim to provide guidance and assistance to those countries that are making their way to the use of renewable energies. One is the International Renewable Energy Agency (IRENA) which is an association of world governments that provides support to countries as they shift to a future with sustainable energy. They also function as a primary platform for global collaboration and as an archive of policies, resources, technologies, and economic know-how on renewable energy [4]. This study will use data from IRENA, Enerdata, and the World Bank.

To evaluate the relative efficiencies of the renewable energy utilization of seventeen (17) nations belonging from highly industrialized countries group (HIC—Russia, Canada, the United States, Japan, the United Kingdom, Italy, Germany, and France) which are also being referred to as the G8 or the Group of Eight Industrialized Nations [5] and newly industrialized countries group (NIC—Turkey, Thailand, Malaysia, Indonesia, India, China, Brazil, Mexico, and South Africa) [6] is the main goal of this study. These HICs and NICs are expected to have highly developed and developing economies. The HICs and NICs are chosen by the authors to be the subject countries for this study because of their high potential in investing in renewable energies since they have more developed economies. The aim is to identify which of the countries are performing efficiently as they progress in the use of their renewable energy resources. Energy consumption has an essential effect to the country's gross domestic product (GDP) as the ratio between the two factors affects the economic output of several countries since energy is a major input in continuous consumption of goods from energy-demanding sectors such as in production and manufacturing [7].

Three input and two output factors during the six-year periods will be considered for forecasting future values using the grey forecasting method or GM (1,1). The data from the past and the future will then be analyzed using the data envelopment analysis (DEA) undesirable output model.

The combination of these two models makes this study different from other papers, especially the use of the undesirable output model in the energy sector. This model will give consideration to the presence of good and bad outputs, treating the bad or undesired outputs as less important contrary to good outputs. This paper will make use of these two methods to evaluate the past to future efficiencies of seventeen countries.

The whole paper is divided into five sections. Reviews of previous literature related to the study are found in the Section 2. The proposed approach in forecasting future values and evaluation of efficiencies is in the Section 3. The Section 4 presents the interpretations and analyses of data gathered using GM (1,1) and the results of the DEA undesirable model. Concluding statements are described in the Section 5.

### **2. Literature Review**

The energy sector has been a very important aspect of human life and has a strong impact on the economic, social, institutional, and environmental conditions of every country. Cirstea et al. [8] calculated the renewable energy sustainable index (RESI) using the normalization and multivariate analysis which affects the said conditions. The goal of the index is to provide a framework that can be used by the renewable energy sector's potential investors to aid their decision making. Another study conducted by Iddrisu and Bhattacharyya [9] made use of the arithmetic mean of the four normalized indicators for the measurement of the sustainable energy development index (EDI) which was devised to evaluate, rate, and rank countries according to the calculated energy indices. Lee and Zhong [10] by using min-max normalization combined with multivariate analysis were able to draft the renewable energy responsible investment index (RERII) wherein the primary intention is to help energy investors to decide effectively and proactively and also, to establish an investment framework for energy stakeholders in developing or revising current approaches for investing in the renewable energy field. The ecological factor was considered by Schlör et al. [11] to form the sustainable development index (SDI). In their study, the methods of selecting variables, normalization, and weighting to analyze whether the German energy sector is on a sustainable development track even under the pressure of sustainability goals. A general sustainability indicator for the consumption of renewable energy resources was established by Liu [12] using weighing, quantification, and evaluation of theoretical criteria. The framework incorporated a multicriteria decision-making model (MCDM) called the analytic hierarchy process (AHP) to provide a precise measurement of sustainability. The same index model was developed beforehand by Doukas et al. [13] applying a multivariate technique called the principal component analysis (PCA) for the analysis of nine different indicators to quantitatively measure the energy sustainability of rural communities. Due to the integration of MCDM techniques in various efficiency analysis, the method has become popular to evaluate the energy sector. Štreimikiene et al. [ ˙ 14] combined AHP with the additive ratio assessment technique (ARAS) to analyze the environmental impact criteria and rate the electricity generation technologies in Lithuania. Another method popularly known as the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) was merged with AHP to provide a comparison and rank the five low carbon energy resources in China in a study conducted by Ren and Sovacool [15] in which they found that wind and hydroelectric power have the most potential for improvement. Troldborg et al. [16] make use of the Preference Ranking Organization Method of Enrichment Evaluation (PROMETHEE) method to formulate an assessment model of sustainability in a national-level and ranks Scotland's technological capabilities for the renewable energy sector. The complex proportional assessment (COPRAS) technique is used by Yazdani-Chamzini et al. [17] for an effective selection of the most pertinent renewable energy project in comparison to the current available options. Another project selection method for renewable energy programs in Spain was applied by San Cristóbal [18] by using the compromise ranking method which allows the Spanish governmen<sup>t</sup> decision-makers to provide weights of importance to different criteria according to their own preferences. Kabak and Da ˘gdeviren [19] used a hybrid MCDM framework—Benefits, Opportunities, Costs, and Risks (BOCR)—combined with an analytic network process (ANP) method, to aid Turkish policymakers in decision-making in terms of choosing which of five alternative energy sources should be given priority for development.

Aside from the studies which make use of advance statistical methods, several past studies have used DEA as a mathematical method to evaluate the energy production and consumption of several countries. The method is very essential to the assessment since it can provide a comparison of systems with multiple input and output factors.

Halkos and Tzeremes [20] used total capital stock and total labor as input variables and GDP as desirable output and CO2 emissions as undesirable output for the analysis of 110 countries using data from the year 2007. In this study, DEA constant return-to-scale (CRS) scores provided an analytical result showing that only five countries most likely performed efficiently. However, the outcome of the bias-corrected scores are larger than the scores of the standard deviation, so the authors were able to rank the countries accordingly to ge<sup>t</sup> a list of ten countries with the highest and the ten lowest scores. The authors then concluded that the environmental efficiencies of each country have shown positive effects over the first six years after signing the Kyoto Protocol Agreement, but this performance did not last long, as the efficiencies of the countries appeared to decline in the following years. This can be reflected from the countries' avoidance of compliance with the impositions of the agreemen<sup>t</sup> and their inability to accordingly adjust their CO2 emission reduction to a value that is relative to the growth rate of their economies.

Wang et al. [21] make use of combined DEA models, the super slack-based model, and the Malmquist productivity index for evaluation and selection of sustainable logistics providers in the US. Using financial indicators such as the equity, liabilities, operating expense, and assets as input factors and revenue, net income, and earnings-per-share as outputs, their paper established a list of rankings among different decision making units (DMUs). With the combined methods used, the paper was able to determine the nine perfectly efficient logistic providers among the sixteen DMUs.

Oggioni et al. [22] measure the ecological efficiency of the cement industry from 21 countries. Using CCR (Charnes-Cooper-Rhodes) and BCC (Banker-Charnes-Cooper) DEA models, the study was able to determine which of these countries performed efficiently in terms of disposability of undesirable outputs. Two outputs were used, cement production and CO2 emission, first being the desirable one and the second otherwise. These data—including the total labor, installed capacity, energy consumption, and raw materials—were collected from the years 2005 to 2008 as inputs for the application of DEA. Analyzing the scores for every country, the outcome points out that during the period under study there are countries like Canada, Brazil, Turkey, and the United States that performed the very worst in terms of eco-efficiency. This is due to the absence of strong environmental protection regulations. Emerging countries like China and India, at that time, were showing high-efficiency levels in cement production. This is attributed to their investment in more efficient technologies and the production of low-quality cement which emits lower CO2 levels.

Woo et al. [23] applied the DEA-based Malmquist productivity index (MPI) to evaluate the environmental efficiency of renewable energy of 31 country members of the Organisation for Economic Co-operation and Development (OECD). This study makes use of total labor, total capital, and total renewable energy supply as input factors, while renewable electricity generation and GDP are used as output factors. Carbon emissions are also part of the study as they were considered to be the undesirable output for the analysis. The MPI model is deployed to measure the technical efficiency change, frontier change, and productivity of the involved countries with and without consideration of carbon emissions. The results of this study concluded that in the evaluation of efficiencies for the energy sector, the undesirable output must always be considered as they have a significant relationship to energy performance. The paper also encourages policymakers to support the development of technologies related to the use of renewable energy in their own country as it also has a significant impact on the energy market.

The same DEA model, MPI, was used by Zhou et al. [24] to measure the carbon emission performance from the world's top 18 CO2-producing countries. Capital stock, total energy consumption, and total labor force were used as the input variables while GDP and CO2 emissions were the outputs. The MPI scores display the performance of these countries in terms of carbon emission productivity, efficiency change, and technological change over the period of 1997 to 2004. The results showed that there was a 24% increase in carbon emissions during the study period and cites technological progress as the primary reason. Germany turned out to be the number one carbon emitter while Indonesia and China displayed negative productivity.

Wang et al. [25] made a forecast of energy e fficiency or the period from 2018 to 2023 using data from 25 countries. The study method used the DEA Slack-Based Model (SBM) to determine the efficiencies using historical data from 2008 to 2017 and then applied grey forecasting to aid the SBM to produce future e fficiency scores. The countries were chosen according to the su fficiency of data that is available from di fferent sources. Two commonly used economic indicators (labor force and capital stock) and one energy-related factor (energy consumption) were used as inputs. The desirable output is GDP and the undesirable one is CO2 emissions. This combination of variables used for the study is enough to provide appropriate results to achieve the goal of the paper. After the result analysis, the authors found out that during the past period, only eight out of 25 countries performed e fficiently and this performance will be maintained for the future period. This indicates the proper balance in their growing economies while protecting the environment due to a deliberate reduction of CO2 emissions. The findings also sugges<sup>t</sup> that European countries have higher e fficiency scores compared to those in Asia and America. The paper further recommended a substantial governmen<sup>t</sup> policy intervention for every country that should focus on improving the energy production and environmental sectors.

More studies have suggested di fferent considerations for input and output factors to be used for the analysis of countries' energy e fficiency scores, as listed in Table 1 below.


**Table 1.** List of commonly used input and output factors for several past literature.

These studies are proof that DEA is an e ffective way to measure the e fficiencies of the energy sector from di fferent countries using diverse combinations of inputs and outputs. This method has played a very important role in e fficiency analyses since it was introduced by Charnes et al. [31] in 1978. The Charnes, Cooper, Rhodes (CCR) model became the first traditional method to calculate the relative e fficiency scores of several numbers of DMUs which represents the technical e fficiency. Banker et al. [32] in 1984 presented another model called Banker, Charnes, Cooper (BCC) to evaluate efficiencies of the DMUs that are not or not ye<sup>t</sup> operating at an optimum scale in which CCR is incapable of. Another DEA-based model was proposed in 1982 by Caves et al. [33] and is called the Malmquist Productivity Index (MPI). It was later been split into two segments by Fare et al. [34] to

represent catch-up and frontier-shift efficiencies. Furthermore, non-radial DEA types such as the additive (ADD) [35] and slacks-based measure (SBM) [36] models were also introduced.

These developments to DEA involve the introduction of a model that will be able to consider the presence of some bad outputs or unwanted factors. Cooper et al. [37] modified the SBM model to be able to provide a more precise measurement of efficiency and is called the undesirable outputs model.

DEA models are used to measure the efficiency coming from data currently available. There can be no measurement of future efficiencies but only can measure the past up to the present scores. Wang et al. [38] use GM (1,1) along with DEA to measure the future efficiency performances of some Vietnamese ICT firms. The grey forecasting method is a time-series prediction model that does not require enormous amounts of data to be able to produce highly accurate results [39]. This capability of GM (1,1) have made it become a popular forecasting tool for many studies which this paper will also utilize and combine with the DEA undesirable output model.

### **3. Materials and Methods**

### *3.1. Research Process*

To be able to reach the goal of this paper, this study is divided into four stages as shown in Figure 1. This will serve as the guide for the authors in finalizing the study.

**Figure 1.** The research process.

### Stage 1. Collection of Data

Data were collected through IRENA, Enerdata, and World Bank. Based on several pieces of cited literature, the authors selected the appropriate input and output factors suitable for this study. These factors are also commonly used by previous studies related to this paper.

### Stage 2. Grey Forecasting Method

The grey forecasting method is used to predict the values of the input and output factors for the future period. The method uses historical data. The mean absolute percentage error (MAPE) determines whether the predicted value is acceptable or not. Lower values of MAPE means higher accuracy.

### Stage 3. Pearson Correlation

To check if the selected input and output factors and the predicted values are suitable for DEA processing, the calculation of the Pearson coe fficient of correlation is very necessary. This method was widely used in previous studies. It is used to confirm the isotonic relationship between factors and a positive correlation is a requirement for DEA.

### Stage 4. Data Analysis and Conclusion

Since this paper focuses on the e fficiency of renewable energy programs, the presence of carbon emissiosn suggests the use of the DEA undesirable output model. The DEA result will show which countries performed e fficiently and those did not. The ranking will be based on the output e fficiencies. This is to determine which countries among HICs and NICs have better renewable energy capabilities. The conclusions will provide a summary and addresses the objective of the study. The authors will specify some recommendations and information valuable for decision-making by policymakers.

### *3.2. GM (1,1) Grey Prediction Model*

The GM (1,1) grey prediction model is a widely used forecasting method associated with time series and di fferential equations. One advantage of this method is the requirement of few historical data, at least four successive data with intervals that are equally distributed in a timely manner, to generate an acceptable prediction and calculated e fficiently as discussed by Julong [39] and supported by Tseng et al. [40]. The procedure for grey prediction is shown in Figure 2 below.

**Figure 2.** The grey prediction model procedure.

Given the variable primitive series *X*(0) in Equation (1), a more detailed procedure of prediction using the GM (1,1) grey model is discussed:

$$X^{(0)} = \left[ X^{(0)}(1), \; X^{(0)}(2), \dots, \; X^{(0)}(n) \right], \qquad n \ge 4 \tag{1}$$

*Energies* **2020**, *13*, 2629

where *X*(0) is a positive sequence and *n* is the total number of historical observations [39].

One very necessary property of a grey model is the accumulating generation operator (AGO) that is used for elimination of uncertainties from the primitive data. The equation for AGO is presented in Equation (2):

$$X^{(1)} = \left[ X^{(1)}(1), \; X^{(1)}(2), \dots, \; X^{(1)}(n) \right], \qquad n \ge 4 \tag{2}$$

where *X*(1)(1) = *<sup>X</sup>*(0)(1), *X*(1)(*k*) = *ki*=<sup>1</sup> *<sup>X</sup>*(0)(*i*), and *k* = 1, 2, ... , *n* [39]. ThepartialdataseriesisdescribedinEquation(3):

$$Z^{(0)} = \left[ Z^{(1)}(1), Z^{(1)}(2), \dots, Z^{(1)}(n) \right] \tag{3}$$

where *Z*(1)(*k*) is the value of the mean from the adjacent data described in Equation (4):

$$Z^{(1)}(k) = \frac{1}{2} \times \left[ X^{(1)}(k) + X^{(1)}(k-1) \right], \; k = 2, \; 3, \ldots, n,\tag{4}$$

Through *<sup>X</sup>*(1), the first order differential equation *X*(1)(*k*) of grey prediction model can be derived from Equation (5) [39]:

$$\frac{dX^{(1)}(k)}{dk} + aX^{1}(k) = b \tag{5}$$

wherein *a* is the developing coefficient and *b* is the grey input.

In general, Equation (5) does not generate the values for parameters *a* and *b*. The above equation is solved through the least square methods (Equation (6)):

$$X^{(1)}(k+1) = \left(X^{(0)}(1) - \frac{b}{a}\right)e^{-ak} + \frac{b}{a} \tag{6}$$

where *X* ˆ (1)(*k* + 1) depicts the prediction value of *X* at a *k* + *1* point in time. Using the method of ordinary least square (OLS), the values of [*<sup>a</sup>*,*b*]*<sup>T</sup>* can be acquired as described by Equations (7)–(9) [39]:

$$\begin{aligned} [a,b]^T &= \left(B^T B\right)^{-1} B^T Y\\ Y &= \begin{bmatrix} \mathbf{x}^{(0)}(2) \\ \mathbf{x}^{(0)}(3) \\ \vdots & \cdots \dots \\ \mathbf{x}^{(0)}(n) \\ \mathbf{x}^{(0)}(n) \end{bmatrix} \\ B &= \begin{bmatrix} -\mathbf{z}^{(1)}(2) & 1 \\ -\mathbf{z}^{(1)}(3) & 1 \\ \vdots & \cdots & \cdots \\ \mathbf{x}^{(1)}(n) & 1 \end{bmatrix} \end{aligned} \tag{9}$$

wherein [*a*, *b*]*T* is referred to as the parameter series, *Y* is the data series and *B* is the data matrix. Thevaluesfor*X*ˆ(1)(*k*)willbecalculatedby letting*X*ˆ(0) becomethepredictedseries:

$$\hat{X}^{(0)} \stackrel{\textstyle \heartsuit}{X^{(0)}} (1), \hat{X}^{(0)} (2), \ldots, \hat{X}^{(0)} (n) \tag{10}$$

where *X* ˆ (0)(1) = *X*(0)(1)

Equation (11) is obtained through the application of inverse accumulated generation operation [39]:

$$X^{(0)}(k+1) = \left(X^{(0)}(1) - \frac{b}{a}\right) \varepsilon^{\text{-ak}} (1 - \varepsilon^a) \tag{11}$$

The accuracy of the predicted values can be measured using the actual and predicted data. The measurement is called the mean absolute percentage error or *MAPE* and is described in the formula below:

$$MAPE = \frac{1}{n} \sum \left( \frac{\mathbf{x}^{(0)}(k) - \hat{\mathbf{x}}^{(0)}(k)}{\mathbf{x}^{(0)}(k)} \right) \times 100\% \tag{12}$$

The acceptability of the predicted data depends on the values of the *MAPE*. Small values for *MAPE* means a higher rate of accuracy. Ju-long [41] also categorized the reliability classes into four as listed in Table 2.

**Table 2.** Equivalent forecast category for every *MAPE* percentage score.


### *3.3. Data Envelopment Analysis—Undesirable Output Model*

The undesirable output model is one of the many widely used DEA models. One thing that makes this model special and different from the others, is that this model considers the presence of bad output factors in the data set. Cooper et al. [37] modified the slack-based model (SBM) to be able to take account of the undesirable outputs during efficiency analysis. This research contains data involving the presence of a bad output making it more suitable for the study. The undesirable model will be described in the following paragraphs.

The input and output matrix of the DMUs will be denoted as (*<sup>x</sup>*0,*y*0). The output parameters of the matrix *y* will be decomposed into two: the desirable outputs are *Yg* (good matrices) and the undesirable outputs are *Y<sup>b</sup>* (bad matrices). The whole decomposition will become *x*0, *yg*0, *yb*0.

The set for production possibility is described by:

$$P = \left\{ \left( \mathbf{x}, \, \, y^g, \, y^b \right) \mid \mathbf{x} \ge \mathbf{X}\lambda, \, y^g \le \mathbf{Y}^g \lambda, \, y^b \ge \mathbf{Y}^b \lambda, \, L \le e\lambda \le \mathbf{U}, \, \lambda \ge 0 \right\} \tag{13}$$

wherein the intensity factor is λ, *L* is the lower boundary and *U* is the upper boundary for λ.

In a presence of bad output, a DMU is efficient if there is no vector *x*, *yg*, *yb*∈ *P* in such *x*0 ≥ *x*, *yg*0≤ *yg*, *yb*0≥ *yb* having at least one inequality.

 The modification of SBM to attain the objective of the undesirable model is described as:

$$\rho^\* = \min \frac{1 - \frac{1}{m} \sum\_{i=1}^m \frac{s\_{\bar{w}}^-}{x\_{\bar{w}}}}{1 + \frac{1}{s} \left(\sum\_{r=1}^{s\_1} \frac{s\_r^{\bar{r}}}{y\_{\bar{m}}^{\bar{r}}} + \sum\_{r=1}^{s\_2} \frac{s\_r^{\bar{b}}}{y\_{\bar{m}}^{\bar{b}}}\right)} \tag{14}$$

subject to *x*0 = *X*λ + *s*<sup>−</sup>; *yg*0 = *Y*λ − *sg*; *yb*0 = *Y*λ + *sb*; *L* ≤ *e*λ ≤ *U*; *<sup>s</sup>*<sup>−</sup>, *<sup>s</sup>g*, *sb*, λ ≥ 0. The excesses in inputs are denoted by the vector *s*<sup>−</sup> and bad outputs is *<sup>s</sup>b*. In contrast, *sg* denotes the lack of good outputs. *s*1 and *s*2 express the number of elements in *sb*, and *<sup>s</sup>g*, and *s* = *s*1 + *s*2.

According to Cooper et al. [37], the DMU *x*0, *yg*0, *yb*0is efficient even under a condition of any undesirable outputs if ρ∗ = 1. An inefficient DMU, ρ∗ < 1, can be enhanced by removing the excesses in inputs and bad outputs and intensifying the deficiencies in good outputs with the following projection:

$$\begin{array}{l} \mathbf{x}\_{0} \Longleftarrow \mathbf{x}\_{0} - \mathbf{s}^{-\star} \\ y\_{0}^{\mathcal{S}} \Longleftarrow y\_{0}^{\mathcal{S}} - \mathbf{s}^{\mathcal{S}^{\star}} \\ y\_{0}^{\mathcal{S}} \Longleftarrow y\_{0}^{\mathcal{S}} - \mathbf{s}^{\mathcal{S}^{\star}} \end{array} \tag{15}$$

Through the Charnes–Cooper transformation method as described by Tone in 2001 [36], the fractional formula can be converted into a linear program with the following variables for the constant return to scale.

Whereas:

*v*, *<sup>u</sup>g*, *ub L* = 0, *U* = ∞ max*ug ygo* − *v* − *ubybo* (16)

subject to:

$$
\mu^{\mathbb{S}}Y^{\mathbb{S}} - \upsilon X - \mu^b y^b \le 0 \tag{17}
$$

$$v \ge \frac{1}{m} \left[\frac{1}{\chi\_o}\right] \tag{18}$$

$$\mu^{\mathcal{S}} \ge \frac{1 + \mu^{\mathcal{S}} y\_o^{\mathcal{S}} - \upsilon \mathbf{x}\_o - \mu^b y\_o^b}{s} \left[ 1/y\_o^{\mathcal{S}} \right] \tag{19}$$

$$
\mu^b \ge \frac{1 + \mu \xi y\_o^{\mathcal{S}} - \upsilon \mathbf{x}\_o - \mu^b y\_o^b}{s} \ \left[ 1/y\_o^b \right] \tag{20}
$$

The *v* and *ub* variables, are respectively referred to as the values of inputs and bad outputs while *ug* refers to the value of good outputs.

Cooper et al. [37], set out the weights for bad and good outputs to be encode before running the undesirable output model. The weight variables must satisfy the *w*1, *w*2 ≥ 0 conditions for the bad and good outputs so that the calculated relative weights will become *W*1 = *sw*1/(*<sup>w</sup>*1 + *<sup>w</sup>*2) and *W*2 = *sw*2/(*<sup>w</sup>*1 + *<sup>w</sup>*2). The final function will be transformed into:

$$\rho^\* = \min \frac{1 - \frac{1}{m} \sum\_{i=1}^m \frac{s\_{ii}^-}{s\_{ii}^-}}{1 + \frac{1}{s} \left(\mathcal{W}\_1 \sum\_{r=1}^{s\_1} \frac{s\_r^\mathcal{S}}{y\_{rv}^\mathcal{S}} + \mathcal{W}\_2 \sum\_{r=1}^{s\_2} \frac{s\_r^\mathcal{S}}{y\_{rv}^\mathcal{S}}\right)} \tag{21}$$

The default value for *w*1 and *w*2 is 1. To give importance to the degree of emphasis for the evaluation of bad outputs, the value of *w*2 can be larger than *w*1 or vice versa. In this model, the heading (O) refers to good outputs while (OBAD) is for bad outputs.
