Integrative Smart Grids’ Assessment System

Abstract

The COVID-19 pandemic has significantly affected the energy sector. The new behavior of industrial and non-commercial consumers changes the energy consumption model. In addition, the constraints associated with the coronavirus crisis have led to environmental effects from declining economic activity. The research is based on evidence from around the world showing significant reductions in emissions and improved air quality. This situation requires rethinking the energy development strategy, particularly the construction of smart grids as a leading direction of energy development. Evaluating the efficiency of smart grids is a vital tool for disseminating successful experience in improving their management. This paper proposes an approach to a comprehensive assessment of smart grids based on a comparative analysis of existing methods, taking into account the changes that need to be considered after the experience gained from the COVID-19 pandemic. The approach provides an accurate set of efficiency indicators for assessing smart grids to account for the direct and indirect effects of smart grids’ implementation. This evaluation approach can be helpful to policymakers in developing energy efficiency programs and implementing energy policy.

1. Introduction

Restrictions on social mobility and economic activity in response to the spread of COVID-19 have led to many economic and social effects. The economic consequences are negative, associated with reduced production and difficulties in the logistics of goods and services. Estimates of the losses suffered by countries’ economies and the world economy differ, but the losses are disproportionately large compared to previous economic crises. At the same time, the coronavirus unexpectedly brought about positive results. In particular, many researchers, including [1,2], who conducted their studies independently of each other around the world, noted the improved air quality and reduced concentrations of harmful substances [3,4,5,6,7,8,9]. However, the effects of emission reductions in various countries differ. As a result [10], studying this issue could not identify a single trend in the EU. That is due to the differences in the severity of social restrictions among EU countries and the time lag among them and reducing pollutants.

One effect resulting from the pandemic is an increase in clear sky insolation [11,12]. It is an essential effect for the development of solar energy. However, changes in emission levels and concentrations of harmful substances in the air result from processes, primarily production. The energy sector is the core of the modern economy, solving pressing socially essential problems. That is confirmed by analyzing scientific publications in energy, sustainable development, and the environment [13,14,15,16,17,18,19,20,21]. The COVID-19 pandemic has significantly affected the industry. The changes are not limited to energy consumption and production [22,23,24,25,26,27,28,29,30,31,32,33,34]. The structure of energy consumption has changed [35]. Businesses and organizations, producers of goods, and service providers have reduced energy consumption. However, households began to consume more [36,37]. This situation has led to significant changes in the energy market [38,39]. Because of this, it is necessary to take a more balanced approach to forecast energy prices [40]. It is also vital given the non-obvious systemic effects caused by changes in energy consumption, such as the impact of energy consumption on the public [41].

One of the most notable changes in the energy sector is the growing share of renewable energy, as noted in some studies [42,43,44], despite declining aggregate electricity consumption [45]. Moreover, according to the study [46], lockdown rigidity directly impacts this process. The coronavirus pandemic and the restriction of mobility and economic activity have led to changes in the energy system’s functioning. One area that needs to be adjusted is assessing and forecasting in the energy sector. Studies [47,48] propose approaches to adapt existing methods of forecasting electricity demand, based on historical data and meteorological information, to the pandemic conditions. These approaches consider energy consumption profiles [49,50], which are becoming an increasingly important factor in forecasting the parameters of the energy grid. It is a tactical response to the energy impact of coronavirus. At the same time, the COVID-19 pandemic is forcing the consideration of strategic changes that need to be made to energy system development plans [51]. Thus, it is necessary to focus on the approaches to evaluating, which are used in designing new energy networks and old ones. Smart grids can respond quickly to instability in the energy system and energy market. Accordingly, in the pandemic and post-pandemic periods, smart grids should become a priority direction of energy development. However, the development of smart grids must consider the experience gained by humanity from the coronavirus pandemic.

Visions for the smart grids’ development and existing plans need adjusting. However, this requires a comprehensive system for evaluating smart grids, determining the maximum number of direct and indirect effects in order to implement them or their development in a particular area.

Evaluating smart grids is crucial because it allows for the identification of successful practices disseminated to a maximum effect. Since energy is a vital area of the modern economy, developing efficient smart grids allows for many complex problems to be solved. Among such tasks are the enhanced economic attractiveness [52,53,54], reduction in energy efficiency gaps [55,56,57], and increasing energy and economic security [58,59,60,61,62,63]. It is crucial to follow common principles and approaches in evaluating energy-efficient projects. It is vital to obtain comparable results suitable for analysis. However, it is also essential to maintain various assessment methods [64]. Additionally, this creates a dilemma in studying the issue of evaluating smart grids.

This article aims to form a theoretical approach to a comprehensive assessment of smart grids based on a meaningful analysis and comparison of existing systems for a comprehensive assessment of smart grids.

2. Literature Review

The benefits of implementing smart grids can be significantly reduced by the wrong approach to their design, the use of suboptimal technologies, and the lack of a plan to integrate them into the global grid. The evaluation of smart grid projects is necessary to avoid mistakes during the smart grids’ development at the planning stage when the cost of error is the lowest. The evaluation of existing smart grids should be used to correct mistakes and increase the efficiency of smart grids. Assessing a set of smart grids within a given area provides information on achievements in their development and helps to monitor the effectiveness of the energy policy. The listed reasons testify to the need to demand a methodology of smart grid assessment from policymakers, businesses, and public organizations. The approaches developed by scientists and companies working in the energy sector to evaluate smart grids respond to the current need. Smart grid evaluation systems developed by IBM, the U.S. Department of Energy, or the Electric Power Research Institute are widely recognized and used to design and develop smart grids. These systems simultaneously evaluate several smart grid areas and systematically characterize them. Therefore, these systems are considered comprehensive systems for assessing smart grids. In the study [65], a comparative analysis of comprehensive smart grid assessment systems was performed to determine whether there is a universal approach to assessment that takes into account all the significant effects of smart grids. The current study aims to improve and deepen the results obtained in [65]. There are many studies of approaches to evaluating smart grids. A review of comprehensive assessment systems was carried out in the paper [66]. However, in addition to comprehensive assessment systems, many narrow studies focus on one aspect of grid operation [67,68,69]. The methodology usually described in such studies is detailed and of considerable scientific interest, but it is not easy to apply in practice.

Comprehensive smart grid assessment systems not only provide benefits. Sometimes they create difficulties. Smart grids must meet modern requirements for the energy system. This is difficult to achieve in an ever-changing environment. Assessment systems for smart grids should align with trends in the energy sector. That is, comprehensive assessment systems must be flexible and adaptive. It means they need constant improvement to meet the new requirements for the reliability of energy systems or to ensure information security [70,71,72,73]. The evolution of smart grids and their proliferation is increasingly drawing scientists’ attention to the need to optimize the spread of smart grids [74], the creation of microgrids [75], their integration into a higher-level energy grid, and interaction with the non-energy sector. In particular, the role and assessment of the smart grids’ efficiency during the development of smart cities are gaining popularity among scientists [76,77]. Such trends should be reflected in comprehensive smart grid assessment systems to preserve their value as a scientific result and a tool used in practice for territorial development planning [78]. This article aims to form a modern comprehensive approach to evaluating smart grids based on a study carried out in the study [65] of existing comprehensive evaluation systems.

3. Materials and Methods

An integrated approach to evaluating smart grids, which covers the maximum number of effects from their operation, is based on a comparative analysis of smart grids’ comprehensive assessment systems, as carried out by [65].

The comparative analysis of smart grid assessment systems has revealed their strengths and weaknesses. The investigation process made it possible to identify the most effective assessment systems in evaluating future smart grids, already functioning smart grids, and diagnosing the state of smart grids’ development at the local, regional, and national levels. However, it was found that none of the existing assessment systems can effectively take into account the effects of smart grids in all areas: the sustainability of the grid, its information, economic, technical, and communication efficiency, its environmental friendliness and the level of electric transport infrastructure.

As a result, there is a need to develop an assessment system that will systematically and comprehensively assess all significant aspects of a smart grid. To this end, a methodological toolkit for the integrative assessment of smart power grids is proposed. It combines the best valuation approaches into a single smart grid evaluation system.

The method used in this study comprises two stages:

-

Stage 1: shaping the basis of the smart grid assessment system (based on the data of comparative analysis of the existing smart grid assessment systems);

-

Stage 2: designing a smart grid integrative assessment system (shaping a set of indicators of a smart grid’s efficiency covering all directions of its development).

Furthermore, this article provides a methodology for designing an index comprehensive assessment system. Additionally, it is the third step in shaping a tool for smart grids’ assessment that will be helpful for policymakers:

-

Stage 3: designing an integrated system of indicators for evaluating a smart grid.

The optimal system for each smart grid area is selected from the assessment systems in the first stage. To this end, three selection steps are used. First, assessment systems with the maximum score (obtained in the process of comparative analysis and ranking of existing comprehensive assessment systems) are determined by specific smart grids areas (Formula (1)):

$$\{\begin{array}{c}{A}_{ki}=A_{max} \to A_{ki}=A_{opt}\\ A_{ki}<A_{max}\\ A_{ki}\ge A_{mi}\\ k\ne m\end{array},$$

(1)

where A_ki is an evaluation of the i-th smart grid area by smart grid comprehensive assessment systems k; k, m are smart grid comprehensive assessment systems; and A_max is the maximum score for a smart grid in an area.

If more than one system best assesses a particular smart grid area, the next step is to select a comprehensive assessment system to be used as a basis for evaluating this area. The following selection step determines the maximum number of areas fully covered by the smart grid assessment system. As the most acceptable option, the evaluation system with the largest number of excellent estimated smart grid areas is chosen (Formula (2)).

$$\{\begin{array}{c}{t}_{ki}\ge t_{mi}\\ k\ne m\end{array},$$

(2)

where t_ki and t_mi are the most covered smart grid areas by smart grid comprehensive assessment systems k and m.

A situation is possible in which several assessment systems have the maximum score in a specific area of the smart grid and the same number of maximally evaluated smart grid areas. In this case, the assessment system with a higher final score (obtained from a comparative analysis of the existing comprehensive assessment systems) is chosen as the basis.

$${\displaystyle \sum }{A}_k>{\displaystyle \sum }{A}_m$$

(3)

As a result of the selection procedure, the basis for designing an integrated evaluation system of the integrative assessment of smart grids is formed. It has the following formalized form:

$$\left\{S_{ki}^b,I_{ki}^b,E_{ki}^b,T_{ki}^b,E{c}_{ki}^b,C_{ki}^b,E{l}_{ki}^b\right\}$$

(4)

where $S_{ki}^b,I_{ki}^b,E_{ki}^b,T_{ki}^b, E{c}_{ki}^b, C_{ki}^b,E{l}_{ki}^b$ are the indicators of the basic comprehensive assessment systems. (In accordance groups: the grid’s stability, information efficiency, economic efficiency, technical efficiency, environmental friendliness, communication efficiency, availability of electric transport infrastructure).

Since the tasks of smart grid assessment can be different, this can be taken into account when forming a comprehensive system for evaluating smart grids. That can be done by differentiating the indicators according to the criterion of the assessment purpose. It is necessary to divide the set of evaluation indicators into universal indicators, which should be used to evaluate the existing smart grids, and those that are inherent in assessing potential projects.

Based on the basis assessment system for forming an integrative assessment system of smart grids, a final list of indicators is created to evaluate each area in terms of its development.

Suppose the basic assessment system has the maximum score for smart grid area coverage. In that case, this system can be accepted as the final result for this area in the integrative assessment system of the smart grid (Formula (5)).

$$\begin{gathered} \mathrm{if} A_{ki}=A_{maxki}, \\ \{\begin{array}{c}\begin{array}{c}{S}_{Ki }=S_{Ki}^b,\\ I_{Ki }=I_{Ki}^b,\\ E_{Ki }=E_{Ki}^b,\end{array}\\ T_{Ki }=T_{Ki}^b,\\ E{c}_{Ki }=E{c}_{Ki}^b,\\ C_{Ki }=C_{Ki}^b,\\ E{l}_{Ki }=E{l}_{Ki}^b,\end{array} \end{gathered}$$

(5)

where $S_{Ki }, I_{Ki },E_{Ki },T_{Ki }$, $E{c}_{Ki }$, $E{l}_{Ki }$, $C_{Ki }$ are the final assessment indicators (a complete list of indicators is given in

Table 1. Basic comprehensive assessment systems and final indicators for assessing smart grids’ areas.

Indicator Group	Indicator Subgroup	Symbol	Assessment System	Indicators
The stability of the grid	System self-recovery	S1ki	DDD	sDGR, sSSR, sTT *, sRPL *
	System reliability	S2ki	EUA	sIR, sLV, sLTT
	System security	S3ki	DDD	sAR, sNA, sISS *
Information efficiency	Customer monitoring, control, and informatization system	I1ki	DDD	iCSG, iOA *, iRA *
	Energy internet and customer informatization	I2ki	DDD	iSSO, iNI, iCSF, iBN
	ERP systems and decision support	I3ki	DDD	iERP, iLAS, iADM *
Economic efficiency	Capital Investments	E1ki	IBM	ePI, eCA, eMI
	Optimization of asset management	E2ki	IBM	ePS, eAO, eWS
	Forming a business model	E3ki	IBM	eTF, eLM, eASP, eFBM, tECO *
Technical efficiency	Automation	T1ki	DDD	tTM, tSS, tFS, tDDM
	Distributed energy generation	T2ki	DDD	tBM, tDRS, tSDG
	Productivity	T3ki	DDD	tML, tESL, tNP *, tOP *
Environmental Friendliness	Reducing harmful emissions	Ec1ki	TTS	efCO, efE
	Land use	Ec2ki	DDD	efL, efEA
	The use of alternative energy and distributed energy generation	Ec3ki	DDD	efWP, efDE, efUN, efEP
Communication Efficiency	Openness policy	C1ki	DDD	cDD, cIS, cIO
	Interaction with consumers	C2ki	DDD	cSP, cQA, cESC
Availability of electric transport infrastructure	Electric vehicles	El1ki	DDD	elVs, elC, elDC

*—additional evaluation indicators that are not provided by the basic assessment system. IBM, IBM Smart Grid Maturity Model; EUA, EU Smart Grid Assessment Benefits Systems; TTS, “Two Type” grid index system; DDD, Evaluation Model of a Smart Grid Development Level Based on Differentiation of Development Demand.

); A_ki is an evaluation of the i-th smart grid area by smart grid comprehensive assessment systems k; and A_max is the maximum score for a smart grid in an area.

Suppose the basic evaluation system does not have the maximum score for the coverage of this area. In that case, it is accepted as the final result for this area in the integrative assessment system after supplementing it with compatible indicators. This action maximizes the coverage of direct and indirect effects in this area of operation of a smart grid (Formula (6)).

$$\begin{gathered} \mathrm{if} A_{ki}<A_{maxki}, \\ \{\begin{array}{c}{S}_{Ki }=S_{Ki}^b+S_i^{ad},\\ A{s}_{Ki }=A_{ma{x}_i},\\ I_{Ki }=I_{Ki}^b+I_i^{ad},\\ A{i}_{Ki }=A_{ma{x}_i},\\ E_{Ki }=E_{Ki}^b+E_i^{ad},\\ A{e}_{Ki }=A_{ma{x}_i},\\ T_{Ki }=T_{Ki}^b+T_i^{ad},\\ A_T{}_{Ki }=A_{ma{x}_i},\\ E{c}_{Ki }=E{c}_{Ki}^b+E{c}_i^{ad},\\ A_{Ec}{}_{Ki }=A_{ma{x}_i},\\ C_{Ki }=C_{Ki}^b+C_i^{ad},\\ A{c}_{Ki }=A_{ma{x}_i},\\ E{l}_{Ki }=E{l}_{Ki}^b+E{l}_i^{ad},\\ A_{El }{}_{Ki}=A_{ma{x}_i}\end{array} \end{gathered}$$

(6)

where $S_i^{ad},$ $I_i^{ad},$ $E_i^{ad}$, $T_i^{ad}$, $E{c}_i^{ad}$, $C_i^{ad}$, $E{l}_i^{ad}$ are the additional assessment indicators for smart grid areas (a complete list of indicators is given in Table 1); A_ki is an evaluation of the i-th smart grid area by smart grid comprehensive assessment systems k; and A_max is the maximum score for a smart grid in an area.

According to the study by Lyulyov et al. (2021), the maximum coverage assessment (score) by indicators of each smart grid area is the same, despite the different number of subgroups of assessment indicators in each area. It is carried out using equilibrium coefficients. As a result, the importance of all areas of smart grid development is the same.

At this stage, it is advisable to check whether the set of evaluation indicators in each area contains the trends and prospects for developing smart grids identified by researchers analyzing the impact of the COVID-19 pandemic on energy.

The aims of the assessment can be different. Evaluating the existing smart grid or its projects and the development of the smart grid in some regions may require different approaches. Some comprehensive assessment systems propose various indicators for different types of smart grids. However, it means that these approaches lose universality. Using multiple sets of indicators is not recommended because some cases or conditions will limit applying comprehensive assessment systems.

After the above steps, it is possible to form a comprehensive assessment system for smart grids evaluating all areas of its development. In this article, such a system is called integrative.

The smart grid integrative assessment system is described as follows:

$$IA{S}_{int}=\{\begin{array}{c}S=f\left(S_{1ki},S_{2ki},S_{3ki}\right)\to max;\\ I=f\left(I_{1ki},I_{2ki},I_{3ki}\right)\to max;\\ E=f\left(E_{1ki},E_{2ki},E_{3ki}\right)\to max;\\ T=f\left(T_{1ki},T_{2ki},T_{3ki}\right)\to max;\\ Ec=f\left(E{c}_{1ki},E{c}_{2ki},E{c}_{3ki}\right)\to max;\\ El=f\left(E{l}_{1ki}\right)\to max;\\ C=f\left(C_{1ki}, C_{2ki},\right)\to max\end{array}$$

(7)

where IAS_int is a smart grids’ integrative comprehensive assessment system, S is the stability of the grid, I is information efficiency, E is economic efficiency, T is technical efficiency, Ec is environmental friendliness, C is communication efficiency, El is availability of electric transport infrastructure, S_1ki is the system’s self-recovery, S_2ki is the system’s reliability, S_3ki is the system’s security, I_1ki is customer monitoring, control, and informatization system, I_2ki is energy internet and customer informatization, I_3ki is ERP systems and decision support, E_1ki is capital investments, E_2ki is optimization of asset management, E_3ki is a forming business model, T_1ki is automation, T_2ki is distributed energy generation, T_3ki is productivity, Ec_1ki is reducing harmful emissions, Ec_2ki is land use, Ec_3ki is the use of alternative energy and distributed energy generation, K_1ki is openness policy, K_2ki is interaction with consumers, and El_1ki is electric vehicles.

The integrative assessment system maximizes the coverage of all smart grids’ areas. Each group of evaluation indicators for each area contains subgroups that prevent skew in the evaluation in favor of particular areas.

It is possible to use multiple sets of indicators for different purposes in the assessment (the regional development of smart grids or the evaluation of existing smart grids or their projects). In terms of differentiation by the aim of the assessment, the integrated evaluation system has the following formalized form:

$$IA{S}_{int}=\left\{IA{S}_u; IA{S}_e; IA{S}_{fut}\right\}$$

(8)

where IAS_u is a set of universal indicators for assessing smart grids; IAS_e is a set of indicators for assessing existing smart grids; and IAS_fut is a set of indicators for assessing potential smart grids.

The integrative assessment system forms the tools for the system evaluation of smart grid efficiency. However, the implementation and monitoring of policy effectiveness in the economy’s energy sector requires a universal approach to evaluation, which will ensure comparability of individual projects’ evaluation results and the level of effectiveness of implemented projects relative to targets. Such an approach helps develop targeted and strategic programs for the development of the energy sector in identifying and monitoring the achievement of energy policy targets.

Based on an integrative assessment system, it is possible to shape an approach that shows the efficiency of smart grids in the form of an integrated indicator. This approach in the current paper is called the index comprehensive assessment system of smart grids.

The formation of the index comprehensive assessment system is carried out based on an integrative assessment system taking into account the degree of achievement of each efficiency indicator’s desired (regulatory) value.

The aggregate evaluation for each subgroup of performance evaluation indicators is carried out according to Formula (9):

$$I{A}_{ sub}={\displaystyle \sum }_{i=1}^y\frac{{\gamma }_i{X}_{fi}}{X_{ni}},$$

(9)

where IA_sub is an assessment of the subgroup within the area of a smart grid; X_ni is the value of the indicator, calculated based on approved regulatory requirements, or, if these are absent, the maximum value by industry or expert opinion; X_fi is the actual value of the indicator; y is the number of indicators in the subgroup; and γ_i is the weighting factor of the indicator.

Group assessment for each area is calculated according to Formula (10):

$$I{A}_{dir}={\displaystyle \sum }_{j=1}^{v}I{A}_{su{b}_j}\times {\propto }_j$$

(10)

where IA_dir is an assessment of the smart grid area; IA_sub is an assessment of the subgroup within the area of a smart grid; V is the number of subgroups of indicators in the group; α_j is the weight of the subgroup of indicators within the area of a smart grid (for example, the weight of the indicators system self-recovery, system reliability, system security in the area, and the stability of the grid).

The assessment results of each smart grid’s area are used to calculate the index assessment of a smart grid.

A principle is used to calculate the index assessment of a smart grid, as is performed for the evaluation for areas:

$$IA=\frac{{{\displaystyle \sum }}_{k=1}^{n}I{A}_{di{r}_k}\times {\omega }_k}{n}$$

(11)

where IA is the index comprehensive assessment system of smart grid; k is the smart grid area; ω is the weight of the smart grid area (for example, the stability of the grid); and n is the number of smart grid areas.

The weights in Formulas (10) and (11) can be set expertly according to the expected results and goals of developing smart grids, such as developing renewable energy, system automation, or involving customers in energy management. The calculation of weights can be carried out by the method of direct estimation. However, it is proposed to assess all areas as equivalent for the systematic development of smart energy networks.

The methodological approach described above allows designing a smart grid comprehensive integrative assessment system. This system is based on a combination of the best comprehensive assessment systems.

4. Results

Critical Areas for Evaluating the Efficiency of Smart Grids

There is currently a significant number of approaches for evaluating smart grids. The results obtained in this study are based on the analysis of comprehensive assessment systems performed by [65] and are a continuation of that study. In particular, this research is based on the EU Smart Grid Assessment Benefits Systems, “Two Type” grid index system, grid development assessment index system, smart grid pilot project evaluation indicator system, and the evaluation model of a smart grid development level based on differentiation of development demand and assessment systems developed by IBM, the U.S. Department of Energy, and the Electric Power Research Institute.

The existing assessment systems for smart grids determine their efficiency quite thoroughly and are widely used. This conclusion is made in the study [65]. However, the results published in that study indicate the imperfection of each of the available approaches for evaluation. Despite the strengths of each approach in evaluation, none of them can provide a comprehensive assessment of the smart grid efficiency. This means a new evaluation system needs to be developed to take full advantage of each evaluation approach’s strengths and address the existing gaps.

The proposed methodology for forming a comprehensive assessment system of smart grids involves making an evaluation basis grounded on the existing approaches. This basis can be improved if necessary. As a result of analyzing the available comprehensive assessment systems, the best of them are defined to estimate each smart grid area (Table 1).

Directions for evaluating smart grids are shown in Table 1, designed to consider all aspects of the smart grid’s operations. Their combination provides a comprehensive characterization of a smart grid.

Each of the identified groups of indicators could contain a significant number of indicators characterizing the processes that occur during the operation of a smart grid. However, excessive detail complicates the use of the assessment system in practice. Defining a limited list of indicators is necessary to assess a smart grid’s efficiency. At this stage of research, the most relevant indicators have been identified. There is variability in approaches to calculating these indicators. That is, the same indicator can be calculated in several ways. This is not true for all indicators in Table 1, as a significant number of them are quite specific. In any case, this study does not consider the method of calculating individual indicators. The main task is to form a list of indicators that assess all crucial aspects of a smart power grid’s operation.

The indicators of the groups “The stability of the grid” and “Technical efficiency” assess the performance of the smart grid and its technical characteristics. The group of indicators for “The stability of the grid” characterizes the ability of a smart grid to counteract threats and unforeseen situations and to restore its parameters after their elimination. This group of indicators includes three subgroups. System self-recovery is the first subgroup. This subgroup contains the indicators “Distribution grid self-recovery index” (sDGR), ”The speed of self-recovery of the distribution network” (sSSR), ”Average troubleshooting time” (sTT), and “The rate of reduction in peak load” (sRPL). These indicators evaluate the effectiveness of the tools implemented in the smart grid to reduce the negative consequences and losses from adverse events in the grid, which lead to the failure of individual components of the grid or the smart grid as a whole. The subset of indicators for “System reliability” complements the previous assessment area.” Improving reliability” (sIR) characterizes the technical level of equipment used in a smart grid to reduce the frequency of technical systems failures. “Provision limiting voltage” (sLV) and “Increasing the lifetime of transformers” (sLTT) are indicators that reflect the effectiveness of solutions to increase the durability of network equipment. That contributes to achieving both security and economic goals. As security issues are essential in developing and implementing smart grids, this aspect is expected to be assessed through a subset of “System security indicators.” The indicators within this subgroup are structured to characterize the system’s security status through the “Number of accidents” (sNA) indicator and, at the same time, check the adequacy of the tools used to improve the security of a smart grid. The “Application of accident reduction technologies” (sAR) indicator is used for these purposes. As the safety of the energy network significantly depends on the technical parameters and information and communication systems, which are different, it is advisable to use “Indicators of structural safety” (sISS) to identify weaknesses in the energy network.

The indicators of the group “Technical efficiency” characterize the efficiency of equipment and technological solutions used in a smart grid. All comprehensive assessment systems studied in the article [65] have many indicators to represent this component of smart grid operation. In the set of indicators offered in Table 1, it is provided to define the technical efficiency of a network in three areas. These areas are “Productivity,” “Automation’” and “Distributed energy generation.” Productivity is proposed to be considered in a broad sense. This subgroup of indicators includes indicators that characterize the amount of energy produced and delivered to the consumer. An important place is occupied by indicators that assess the additional effects that show the advantage of a smart grid over a traditional one. This subgroup includes indicators such as “Maximum load on the network” (tML), “Share of energy saving lines” (tESL), “The number of new products,” “The amount of energy or its capacity supplied as ancillary” (tNP), and “Operations Performance Index” (tOP). The grid’s automation level is assessed using indicators that mainly characterize the share of automated processes in their totality. The indicators that are proposed to be used are “The proportion of lines that use the technology of monitoring and control” (tTM), “The share of smart substations” (tSS),“Coverage by energy forecasting system” (tFS), and “Distribution network dispatching management” (tDDM).

One of the basic principles applied by smart grids is distributed energy generation. The proposed approach to evaluating smart power grids is considered a technical component of the smart grid functioning and a necessary condition for using alternative energy sources. Thus, the indicators that characterize distributed energy generation are present in both groups of evaluation indicators, but they describe a smart grid differently. The technical component of distributed energy generation is estimated using indicators such as “Bidirectional measurement” (tBM), “The use of distributed energy generation sources and their support facilities” (tDRS), and “Forecast of the speed of distribution of distributed energy generation” (tSDG). Information and communication technologies are not only used in a smart grid to increase security. A smart grid can only exist with information and communication technologies. The efficiency of their application allows realizing the advantages of a smart grid over a traditional one. The groups of indicators “Information efficiency” and “Communication Efficiency” characterize the digitalization level of the smart grid and the involvement of consumers in its management. These two groups of indicators are partially related. However, information efficiency focuses on evaluating applied information technologies, while communication efficiency focuses more on implementing an openness policy using information and communication technologies.

Information efficiency consists of three areas. These areas are (1) customer monitoring, control, and informatization system, (2) energy internet and customer informatization, and (3) ERP systems and decision support. These areas of evaluation need special attention. Information systems are the fastest way to change smart grids. They are the link between the energy system and other areas of innovation. In particular, the development of smart cities is meant. The identified evaluation subgroups cover all critical aspects of using information technology in smart grids, from monitoring and controlling grid processes to optimizing and supporting decision making.

The subgroup “Customer monitoring, control, and informatization system” uses the indicators “The percentage of customers connected to a smart grid” (iCSG), “Online availability of data to consumers, data accumulation through all information channels” (iOA), and “Remote asset monitoring systems” (iRA).

Energy share and customer informatization are proposed to be assessed with the help of “The share of secure operations of the information and communication system” (iSSO), “Number of information events” (iNI), “Coverage of substations with a fiber-optic network, and cable coverage of the highway” (iCSF), and “Bandwidth of the communication network platform” (iBN). Evaluation indicators are designed so that it is possible to trace the relationship between the indicators of different groups.

ERP systems and decision support are proposed to be evaluated using the indicators “Coverage of a smart grid with an ERP system” (iERP), “The level of availability of business systems” (iLAS), and “Automated internal decision making” (iADM).

Communication efficiency consists of two areas. These areas are “Openness policy” and “Interaction with consumers.” Assessing this component of a smart grid is extremely important. After all, communication efficiency determines the prospects for expanding the smart grid, which is necessary for forming a business model for developing a smart grid.

Openness policy is proposed to be evaluated using such indicators as “Depth of information disclosure” (cDD), “Information update speed” (cIS), and “Investments in the openness of the energy business” (cIO). At the same time, the interaction with consumers involves using such indicators as “The scale and proportion of electricity purchases by large consumers” (cSP), “The index assessing the quality of service” (cQA), “Energy savings through consumption management” (cESC).

The impact of COVID-19 on the energy sector and the environment demonstrates what results can be expected from the implementation of environmentally friendly energy technologies. Renewable energy sources are a crucial element in the development of smart grids. However, many analyzed smart grids’ comprehensive assessment systems characterize this aspect unilaterally. As a result, the environmental component of smart grids’ operation is not assessed systematically. It concerns the evaluation of the possibilities and efficiency of using renewable energy sources for the development of the energy system [79,80,81]. Systematic assessment of such effects during the development and operation of smart grids in the proposed approach uses the “Environmental Friendliness” group indicators.

The development of smart grids contributes to achieving the climate policy goals by replacing fossil fuels with renewable energy sources. The integrative assessment system provides a comprehensive assessment of the impact of the energy network on the environment. The environmental friendliness indicator group contains three areas of assessment. These areas are “Reducing harmful emissions,” “Land use,” and “The use of alternative energy and distributed energy generation [82].” Reducing harmful emissions is measured by reducing CO₂ emissions (efCO) and environment protection (efE). It allows the measurement of the actual amount of emissions and their reduction and characterizes the effectiveness of environmental measures implemented in a grid. The value of smart grids is that they use renewable energy as much as possible. Given the diversity of renewable energy sources and the different ways of using them in smart grids, the subset of indicators “The use of alternative energy and distributed energy generation” includes efficiency indicators such as “The speed of development of wind and photovoltaic networks” (efWP), “Share of distributed energy generation and storage” (efDE), “Coefficient of unused wind energy” (efUN), and “Distributed energy permeability” (efEP). Indicators of renewable energy use are applied in close connection with implementing the principle of distributed energy generation, which makes large-scale renewable energy possible in smart grids. “Availability of electric transport infrastructure” is the group of indicators considering electric transport and infrastructure for its application and dissemination as a promising area of smart grids. This group is the smallest in terms of the number of indicators. It contains indicators such as “The number and share of annual sales of hybrid and electric vehicles” (elVs), “The density of the charging stations” (elC), and “The degree of conformity of the charging station” (elDC).

The economic efficiency of smart grids is a determining factor for their development.

Creating a profitable business model for smart grids is a modern challenge for business and government institutions. In addition to technological constraints, economic barriers to the spread of smart grids are a significant challenge. Incentives to invest in this area of the energy sector are ineffective in some countries in the short term, and the immediate economic benefits for the investor are not obvious. COVID-19 forced the need torethink the role of smart grids in times of crisis. After all, compared to the traditional energy network, smart grids’ flexibility allows minimizing economic losses, mainly through creating micro-networks using alternative energy sources compared to traditional ones. Therefore, the economic evaluation of smart grid projects should include a wide range of indicators, including revenue generation and risk minimization.

The complete cost-effectiveness of a smart grid with the available comprehensive assessment systems can be described using the IBM Smart Grid Maturity Model. At the same time, this model focuses on the indicators of the current efficiency of the smart grid and tries to assess its prospects for the formation of a long-term successful business model. Complementing the approach developed by IBM, a set of indicators is obtained that evaluate the prerequisites and opportunities for scaling a smart grid. These indicators belong to the subgroup “Forming business model”. Among the indicators of this subgroup are “Optimized formation of tariffs” (eTF), “Distribution of resources in local markets” (eLM), “Profit from ancillary services” (eASP), “Formation of the business model at the functional level” (eFBM), and “Forming ecosystems” (tECO).

The other two subgroups of economic efficiency assessment contain a set of indicators that characterize the costs of implementing and maintaining a smart grid project. Capital investments are evaluated using the indicators “Pilot investments to support the use of a differentiated resource portfolio” (ePI), “Cost analysis of new systems” (eCA), and “Modeling of investment assets for crucial components based on smart grid data” (eMI). At the same time, another subgroup—"Optimization of asset management”—contains resource efficiency indicators during the operation of a smart grid. Among these indicators are “Developing a strategy for a diversified resource portfolio” (ePS), “Optimizing asset utilization participants in the supply chain” (eAO), and “Development strategy of mobile workforce” (eWS).

Using the proposed set of indicators allows stakeholders to analyze the state and prospects for the development of smart grids more effectively and comprehensively. Particular attention in the context of the experience gained by stakeholders in the energy sector is the issue of pricing and a flexible response to market changes [83,84,85,86,87,88,89,90,91].

Experience gained from the COVID-19 pandemic should be taken into account in the assessment system. This will increase the value of assessment systems. The most important conclusion from the impact of coronavirus on the energy system is the actual confirmation of the higher efficiency of the power grid, where renewable energy is widely used. The analysis of changes in energy consumption patterns shows that grid operators respond more efficiently and quickly to changes in demand, having a significant share of distributed energy generation from renewable sources. The results of research [45,92,93,94] show that the restructuring of the energy system during the COVID-19 pandemic to increase the share of distributed energy generation from renewable sources has made the energy system more adaptable, facilitated the integration of renewable energy into the grid, and reduced CO₂ emissions. In addition, this has been reflected in approaches to forecasting the electricity demand.

Studies on the impact of COVID-19 on the energy system have raised questions about the future development of energy. In particular, the studies [95,96] examine the need to stimulate renewable energy to form an environmentally friendly economy instead of restoring the economy built on fossil fuels.

Considering the results of these studies, the evaluation of smart grids should focus on indicators of renewable energy use, demand forecasting algorithms, and business model formation. These indicators are valuable, so their importance, among other indicators, could be significant.

Table 1 shows that 13 of 18 identified critical areas of smart grids were most fully evaluated by the evaluation model of a smart grid development level based on the differentiation of development demand. However, it cannot be concluded that applying this approach to evaluation solves the problem of the comprehensive assessment in these areas. Forming a system of indicators for the most complete coverage of the effects of the operation of a smart grid in each area is one of the steps in developing a comprehensive system for evaluating a smart grid. At the same time, the results of Table 1 do not indicate that other techniques are ineffective in assessing certain areas of a smart grid. Data from Table 1 should be interpreted so that the indicators given in each smart grid area are optimal in forming such a comprehensive evaluation system that will characterize each direction of smart grid development as fully as possible. According to the research methodology described above, if the indicators of the basic evaluation system, which assess a specific area of the smart grid, are not enough to characterize it, it is necessary to add additional indicators. Indicators in Table 1 are the result of supplementing the existing comprehensive systems for evaluating smart grids with additional indicators. It allows for more accurate and comprehensive assessment of each smart grid area. The detailing of indicators is shown in

Table 2. Comprehensive assessment systems’ indicators.

Indicators	Code	Indicators	Code
Average troubleshooting time	sTT	Land use (savings)	efL
The rate of reduction in peak load	sRPL	Specific indicators of energy per unit area	efEA
Distribution grid self-recovery index	sDGR	Share of distributed energy generation and storage	efDE
The speed of self-recovery of the distribution network	sSSR	The speed of development of wind and photovoltaic networks	efWP
Improving reliability	sIR	Coefficient of unused wind energy	efUN
Provision limiting voltage	sLV	Distributed energy permeability	efEP
Increasing the lifetime of transformers	sLTT	Reduction in CO2 emissions	efCO
Indicators of structural safety	sISS	Environment protection	efE
Application of accident reduction technologies	sAR	The proportion of lines that use the technology of monitoring and control	tTM
Number of accidents	sNA	The share of smart substations	tSS
Online availability of data to consumers, data accumulation through all information channels	iOA	Coverage by energy forecasting system	tFS
Remote asset monitoring systems	iRA	Distribution network dispatching management	tDDM
The percentage of customers connected to a smart grid	iCSG	Bidirectional measurement	tBM
The share of secure operations of the information and communication system	iSSO	The use of distributed energy generation sources and their support facilities	tDRS
Number of information events	iNI	Forecast of the speed of distribution of distributed energy generation	tSDG
Coverage of substations with fiber-optic network and cable coverage of the highway	iCSF	Forming «ecosystems»	tECO
Automated internal decision making	iADM	The number of new products, the amount of energy or its capacity supplied as ancillary	tNP
Bandwidth of the communication network platform	iBN	Maximum load on the network	tML
Coverage of a smart grid with an ERP system	iERP	Operations’ Performance Index	tOP
The level of availability of business systems	iLAS	Investments in the openness of the energy business	cIO
Cost analysis of new systems	eCA	Depth of information disclosure	cDD
Optimizing asset utilization participants in the supply chain	eAO	Information update speed	cIS
Development strategy of mobile workforce	eWS	The scale and proportion of electricity purchases by large consumers	cSP
Pilot investments to support the use of a differentiated resource portfolio	ePI	The index assessing the quality of service	cQA
Modeling of investment assets for key components based on smart grid data	eMI	Energy savings through consumption management	cESC
Developing a strategy for a diversified resource portfolio	ePS	Number and share of annual sales of hybrid and electric vehicles	elVs
Optimized formation of tariffs	eTF	The density of the charging stations	elC
Distribution of resources in local markets	eLM	Degree of conformity of the charging station	elDC
Profit from ancillary services	eASP	Share of energy-saving lines	tESL
Formation of the business model at the functional level	eFBM

.

It should be borne in mind that the evaluation system should be simple to apply. It necessarily should be easy to use for a large number of stakeholders, not just scientists.

Conclusions on the effectiveness of the evaluation of smart grid areas by each of the evaluation systems studied in this article were made in the paper [65].

The smart grid efficiency indicators in Table 1 and Table 2 reflect the processes and phenomena that occur during the operation and development of the grid and characterize its condition. This study is closely related to the results described in [65] and uses the same understanding of the evaluation process. The system of evaluation indicators consists of several levels. The first level characterizes the areas of the smart grid. Level two indicators focus on processes and phenomena within a specific area. The indicators of the third level are the most detailed. These indicators can be calculated using appropriate formulas and describe a smart grid based on measurements and calculations. The results shown in Table 1 are a system of indicators of the second level. It aims to form a vision of what processes and phenomena should be evaluated so that the characteristics of a smart grid as a result of the evaluation are accurate and cover all essential aspects of its operation. However, this system of indicators’ practical and effective application to assessing a smart grid is possible only with third-level indicators.

The set of evaluation indicators for each smart grid development area forms an integrative evaluation system, as it uses the advantages and eliminates the disadvantages of existing comprehensive assessment systems. However, energy policy development and implementation require a tool to obtain comparative results from operating different smart grids or their regional development. It is helpful to identify priority projects and monitor the achievement of planned results. An index assessment of smart grids is valuable for these purposes, i.e., the final indicator. The method of forming such an indicator is described in the section Materials and Methods.

5. Discussion

The results of this study systematize the existing smart grids’ comprehensive assessment systems and integrate their strengths within a single approach to evaluation. However, the results are an intermediate stage in shaping the assessment system, which fully considers the direct and indirect effects of operating a smart grid. The results achieved require further work on the detail of indicators and the evaluation algorithm’s formation because it allows practical testing of the proposed approach.

As the results of this study are intermediate, for the practical approbation of the proposed approach, it is essential to complete the last stage, in which the proposed theoretical approach will be brought to the level of practical methodology that stakeholders in the energy sector can use. After processing the array of information and calculating indicators, the relevance of individual indicators will be determined for evaluation purposes and, if necessary, the set of indicators in each proposed area will be optimized. That will solve one of the critical problems of comprehensive assessment systems, namely the complexity of their practical application by many stakeholders. The analysis of the obtained calculation results will provide the necessary information on the role and value of individual indicators in the evaluation system. Due to this, it will be possible to solve the importance of indicators in the proposed assessment system based on a quantitative data analysis. This will make the evaluation system more objective and avoid the subjective component altogether.

6. Conclusions

Existing comprehensive assessment systems do not fully cover the direct and indirect effects of the smart grids’ operation, which is essential for their characterization and analysis to further develop existing smart grids and create new ones. Existing assessment systems use different approaches to evaluation, resulting in asymmetry in evaluation results. Some areas are well evaluated by one assessment system, others by another. Based on the analysis of existing smart grid assessment systems, the most effective ones for assessing each direction of smart grid development have been identified. Systems such as the IBM Smart Grid Maturity Model, the EU Smart Grid Assessment Benefits Systems, the “Two Type” grid index system were identified, as well as the Evaluation Model of a Smart Grid Development Level Based on Differentiation of Development Demand. These systems are used to form a system of indicators for evaluating smart grids to cover the maximum number of effects caused by their operation. Complementing basic assessment systems with additional indicators from other assessment systems to improve the quality of assessment has formed an integrative assessment system for assessing smart grids. In turn, this system can be used as a basis for creating an index assessment of smart grids. Using the developed approaches for integrative assessment and index assessment of smart grid development can help identify priority projects, develop state and regional sectoral programs in the energy sector, and monitor energy policy effectiveness.