A Comprehensive Review of Recent Advances in Smart Grids: A Sustainable Future with Renewable Energy Resources

Abstract

The smart grid is an unprecedented opportunity to shift the current energy industry into a new era of a modernized network where the power generation, transmission, and distribution are intelligently, responsively, and cooperatively managed through a bi-directional automation system. Although the domains of smart grid applications and technologies vary in functions and forms, they generally share common potentials such as intelligent energy curtailment, efficient integration of Demand Response, Distributed Renewable Generation, and Energy Storage. This paper presents a comprehensive review categorically on the recent advances and previous research developments of the smart grid paradigm over the last two decades. The main intent of the study is to provide an application-focused survey where every category and sub-category herein are thoroughly and independently investigated. The preamble of the paper highlights the concept and the structure of the smart grids. The work presented intensively and extensively reviews the recent advances on the energy data management in smart grids, pricing modalities in a modernized power grid, and the predominant components of the smart grid. The paper thoroughly enumerates the recent advances in the area of network reliability. On the other hand, the reliance on smart cities on advanced communication infrastructure promotes more concerns regarding data integrity. Therefore, the paper dedicates a sub-section to highlight the challenges and the state-of-the-art of cybersecurity. Furthermore, highlighting the emerging developments in the pricing mechanisms concludes the review.

1. Introduction

The stipulation on sustainable modernization of the energy sector primarily brought about the need to establish deregulation in the power industry. Among several novel propositions, the concept of microgrids proves to be the most promising solution [1]. Microgrids (MG) are small-scale electrical distribution networks consisting of distributed generators (renewable and/or non-renewable generators), heterogenous loads, and energy storage devices that operate in grid-connected or islanded modes with suitable interfacial power electronic devices [2]. Descriptively, MGs in the form of autonomic grids are existent in remote communities wherein interconnection to utility networks is techno-economically inconceivable. The power delivered by renewable energy sources in the early stages of integration was infinitesimal in comparison to existing conventional generators and, hence, their impact on the power network was inherently unnoticeable.

However, with recent developments and commitments toward renewable and sustainable power generation, renewable energy sources (RES) such as solar photovoltaics, wind, hydropower, and hydrogen technologies are among the most popular and prioritized integrations to the electrical power network. Therefore, microgrid facilitates flexibility to consolidate appropriate control scheme and power management algorithm to maintain the quality of power supplied by transient RES through power electronic interventions [3]. Power electronic devices primarily provide the flexibility to integrate numerous distinctive forms of RES, energy storage devices, and heterogeneous loads. Furthermore, numerous conceptualized topologies of power electronic converters/inverters are proposed [4,5,6] to establish a point of common coupling (PCC) that plays a key role in the formation of DC, AC, and hybrid AC-DC microgrids for various domains of voltage and frequency levels. In this respect, several power electronic interfacing configurations and topologies have been proposed to not only achieve a diversified power generation framework but also enable MGs to effectively manage the power and energy flow [7,8].

Accordingly, with large-scale integration of RES technologies and subsequent deregulation, the overall operation needs to be carefully monitored accompanied by power management frameworks to support and facilitate controllable power sharing and load sharing [9]. Consequently, the paradigm adaptation of microgrids for large-scale integration of distributed renewable energy sources will allow a structuralized and intensive solution for numerous challenges associated with deregulated power networks that will extensively reduce the need for complex centralized coordination and facilitate the formation and realization of the smart grids.

Smart Grid (SG) does not have a unique definition to precisely describe the phenomena [10]. However, the smart grid can be simply defined as an intelligent network that is automated and able to store, communicate, and make decisions. A similar definition of smart grids is dictated by the US Energy Independence and Security Act 2007 [11]. It defines the smart grid as a modernization to the electrical network such that it monitors, increases grid resiliency to disruptions, and automatically optimizes grid operation of interconnected system components starting from central generating units and distributed generation through transmission networks up to load centers. In addition, the US National Institute of Standards and Technologies (NIST) [12] describes the smart grid as “a modern grid that adapts bi-directional flows of energy” and utilizes two-way communication and control capabilities that lead to a wide range of new functionalities and applications. It adds, unlike today’s grid in which the energy is delivered from generation centers to demand centers, Smart grids allow a two-way flow of energy and data. The definition of the smart grid has been stated by numerous entities and researchers, as given in References [13,14,15,16,17,18,19,20,21]. They, however, are conceptually united on the general framework of the term.

Furthermore, the US-based Electric Power Research Institute (EPRI) [16] defines the smart grid as “the transition from the current grid where the flow of power is permitted from the central generation to load locations into a grid where there is a peer to peer consumer interactions, distributed generation, and control centers”. In the UK, the Department of Energy and Climatic Change emphasizes that with a smarter grid, operators are more aware of supply–demand balance information, which in turn helps to intelligently manage the system and shift the demand from peaking instants to off-peak periods [17]. An initiative is launched in Australia called “SmartGrid SmartCity” between Energy Australia and Ausgrid, and the Australian government echoes EPRI’s definitions, defines the smart grid as a novel and a highly intelligent way of supplying electricity. It incorporates advanced communication infrastructure, innovative sensing, and metering technologies with the electrical network to create a two-way, interactive grid. Smart sensing technologies help to achieve fewer grid disruptions and outages. Moreover, smart metering may help consumers to manage their energy consumption to reduce billing costs effectively. The development and the emerging trends in smart grids have been highlighted in References [22,23,24,25,26,27,28,29,30,31,32,33,34].

The smart grid shifts the current conventional grid into a more modernized grid that can function cooperatively and responsively. Users, generators, and consumers may intelligently be integrated into the grid to provide efficient, secure, and economically feasible supplies [18]. The SG incorporates distributed intelligence, bi-directional-based infrastructure for communications, and power flow to improve system efficiency, reliability, and sustainability. Furthermore, the smart grid is a network that integrates digital computing capabilities and highly automated services into the already existing power system infrastructure. Empowering the transition toward the smart grid enhances the robustness and self-healing capabilities of the system, according to Reference [20].

Table 1. Conventional grid versus smart grid [15].

Conventional Grid	Smart Grid
Mechanically operated	Digitized
Unilateral	Bi-directional
Centralized Power generation	Distributed Generation
Radially connected	Dispersed
Small number of sensors	Many
Less monitoring capabilities	Highly monitored
Manual control	Automated control
Less security issues	Vulnerable to security issues
Slow responsive actions	Fast response

briefly illustrates the major differences between the conventional power grid and the smart grid. SGs require an effective deployment to the information and communication technology to have a successful implementation of this concept. Upgrading the conventional power grid toward an active network of a two-way communication capability is one of the main barriers in smart grids (SGs). In accordance with the diversification of the power network, a smooth transition to incorporate SG technologies has been done with appropriate categorical standardization [35,36,37,38,39,40,41,42,43,44].

The remaining of the paper is structured as follows: Section 2 reviews the structure and the main characteristics of SGs. Furthermore, the essential technologies that may facilitate the transition toward a highly functioned infrastructure are highlighted. From a power system perspective, the main components in smart grids are thoroughly, independently studied in Section 3. Moreover, the recent advances in reliability and resiliency indices in smart grids are presented throughout Section 3. The energy data management and cybersecurity in smart grids are reviewed in Section 4. In Section 5, we thoroughly investigate the pricing mechanisms in smart grids. Figure 1 briefly emphasizes the organization of the paper.

2. Structure of Smart Grids

2.1. Definitions

The National Institute of Standards and Technology (NIST) [12] has classified smart grids into seven categories and (sub-categories) that comprise actors and applications. Actors may include devices (e.g., renewable energy generation, and smart meters), control systems, programs, decision-makers (stakeholders), and telecom stations (data-exchange). The applications are known as the tasks that can be performed by actors within the category (e.g., energy management, site automation, and energy storage) or between different categories. Actors in the same category may interact with actors in other categories, and a specific category may contain components from other categories (i.e., distribution utility may contain actors from operation category such as Distribution Management System and in the customer category such as electric meters. A detailed classification is presented in

Table 2. Classification and elements of smart grids [12,15].

Category	Description and the Actors in Domain
Customer	Where the electricity is consumed, End-users. The sub-category includes domestics and large consumers, such as commercial and industrial loads. Actors can generate, manage, and store.
Market	Where the assets are exchanged. The operator and participants are the actors in electricity markets.
Service Provider	An organization that provides services pertaining to the establishment and secure operation of smart grids as per the requirements of the consumers and utilities.
Operations	The proper operation of the power system is monitored—the managers of the flow of electricity.
Bulk Generation	Where the delivery of bulky electricity to consumers starts, actors are the generators of electricity in bulk quantities; energy can also be stored for later distribution.
Transmission	Where the bulky power is transferred from generation centers to distribution. Actors are the carriers of electricity and may also generate and store electricity.
Distribution	Where Distributed Generation, Distributed Storage, transmission, and consumers’ interconnect. Actors are the entities that distribute electricity to and from customers.

.

2.2. Smart Grid Technologies

Numerous technologies can be implemented to achieve successful control and automation in smart grids [21,45]. Such technologies are imperative to facilitate the transition toward a well-functioned infrastructure from the perspective of grid designers and consumers. These technologies may include Automatic Voltage Regulation (AVR), Energy Management System (EMS), Automatic Generation Control (AGC), Advanced Metering Infrastructure (AMI), Meter Data Management (MDM), Distribution Management System (DMS), Geographical Information System (GIS), Outage Management System (OMS), Wide Area Management System (WAMS), and Demand Side Management (DSM). Some of these technologies are highlighted in

Table 3. Smart grids technologies and applications at a glance [21].

Technique	Ref.	Year	Author (s)	Objective
AVR	[52]	2014	Ting-Chia et al.	(a) Hybrid protocol for AVR (b) Power control using FLC
	[53]	2010	Morris et al.	(a) AVR control based on sensitivity (b) Voltage regulation
GIS	[54]	2012	Schneider Electric	(a) Efficient smart grid strategies based on GIS (b) Data accuracy (c) Time reduction (d) Efficient workflow
	[55]	2009	Esri Inc	(a) Providing the necessary tools for network modeling and component tracing.
OMS	[56]	2013	John Dirkman	(a) OMS for normal operations and emergencies (b) OMS assists in advanced analysis and control of the smart grid. (c) Prediction roadmap
	[45]	2011	Eduardo et al.	(a) Power restoration (b) Automated solutions for reporting problems.
AGC	[53]	2014	Siddaharth et al.	(a) Investigate the impact of data (b) Attack detection
	[57]	2012	Ali et al.	(a) Smart grid intermittency mitigation (b) High-frequency fluctuations control
AMI	[45]	2011	Eduardo et al.	(a) Data Gathering cost reduction (b) Connection and disconnection of end-users
EMS	[58]	2012	Rahman et al.	(a) Backup batteries (b) Efficient operation of EMS

.

AVR helps to keep the voltage profiles within the allowable limit, whereas EMS ensures the reliability and secures the operating points for the Supervisory Control and Data Acquisition (SCADA). It could also be considered as a large-scale optimizer for the entire grid. AGC has a vital role in grid stability as it performs optimal load distribution among the generating units to re-gain stability margins. The smart meters allow two-way communication between the end-users and the service provider [15]. This enables consumers to control their energy usage and ensures more accurate billing. In addition, smart meters can provide power outage notifications and power quality monitoring. For a demand to be controllable, smart meters are often used since they possess a two-way communication that allows system operators or aggregators to effectively control the loads [46,47]. For instance, the authors in Reference [46] investigated the ability of smart meters to control domestic demands for frequency regulation purposes. Due to the slow response of communicating channels during frequency surges, an alternative control scheme was proposed, while in Reference [25], a smart meter based controlled load blocking scheme for domestic loads was proposed for primary frequency control considering end-users’ lifestyles and safety. The advanced metering infrastructure (AMI) is a fundamental step toward grid modernization. AMIs are not a single technology, rather an integration of a wide range of technologies that can intelligently communicate between consumers and system operators. The AMI provides the necessary information for consumers to make intelligent decisions and the ability to execute those decisions that lead to substantial benefits they do not currently have. The AMI and distribution automation facilitates a more modernized grid through transformers and feeders monitoring, outage management, and integration of electric vehicles. MDM supports the decision-making process and assists in managing the flow of data.

DMS performs several applications, such as efficient controlling and monitoring of the distribution network. It acts as a decision support system that helps personnel in taking the counteractions once an outage occurs. It may also help in maintaining the voltage and frequency within their nominal values. DMS can also be used to provide the needed functionality for consumers to control their appliances once required. GIS provides the necessary infrastructure that integrates the data onto geographical maps. GIS visualization is an extremely crucial component in smart grids. OMS helps to restore the functionality of the system post outages. WAMS assists in grid synchronization in a high voltage network. It can be utilized to provide disturbance analysis and verification of Flexible AC Transmission Systems (FACTS) [48]. WAMS can also be used to provide time measurements through Phasor Measurement Units (PMUs) [49,50,51]. DSM offers intelligent energy curtailment and will be thoroughly investigated in the sequel of this paper.

2.3. Characteristics of Smart Grids

The following bullet points briefly summarize the main characteristics of smart grids that have been reported in the literature [59,60,61]:

• Integrable Distributed Resources (DER) that include sustainable energy sources.
• Should perform dynamic optimization of grid operation continuously.
• Should have digitized information and control technologies to enhance the reliability and efficiency of the electric network.
• Should have Demand Side Response (DSR) programs and demand-side resources.
• Integrable smart appliances.
• Are fortified against cyber threats.
• Should have advanced storage devices and peak-shaving technologies, including hybrid and plug-in electric vehicles.

2.4. Benefits of Smart Grid Transformation

Utilities are encouraged to efficiently generate and distribute their electricity with a minimal negative impact on the environment. This practice is not only a matter of good corporate citizenship, but some countries also impose a wide range of regulations to limit carbon emissions, and they offer incentives for those transforming their infrastructures to a modernized network. For instance, in Europe, the well-known principle “Cap and Trade” limits the total amount of greenhouse gas productions from power plants and factories. A utility is free to operate as long as its carbon emission allowance is below the cap level. Moreover, a company may trade its spare allowance to a company that violates its assigned cap level. The principle is applied in European states in addition to Iceland, Norway, and Liechtenstein [62]. On the other hand, In the United States, 40% of greenhouse gas emissions are produced from electricity consumption. By 2030, it is estimated that smart grid-based applications that range from shallower domains like voltage control to broader domains like the integration of renewable energy resources would reduce the nation’s carbon dioxide emissions from 211 to 60 million metric tons annually [11,16]. Although the transformation toward smart grids may unfold over a long period, the incremental progress in that area yields the major pros listed below, which are highlighted in References [63,64,65,66,67,68,69]. The application of a modernized network would:

• Improve the reliability and quality of the power grid.
• Optimize the operation of existed assets to avert the future expansion of backup plants.
• Enhance the overall system efficiency.
• Improve system resiliency.
• Facilitate the incorporation of Distributed Resources.
• Enable predictive maintenance and self-healing capacities.
• Lower greenhouse gaseous emissions.
• Increase consumers’ assortments.
• Increase the opportunities to enhance system security.

3. Smart Grid Predominant Components

3.1. Distributed Generation

The conventional paradigm of the well-known centralized power grid is being reshaped due to the high penetration of Distributed Generation (DG). The advent of distributed generation has disrupted the way power is generated and supplied to the electrical grid. The lack of fossil fuels, increased greenhouse emissions, and the automation that has been evolving in informatics and technologies have contributed to the revival of the DG. Researchers believe that DG has a vital role in forming the future of the electrical grid in addition to the storage technologies and demand response [70,71,72,73]. However, the degree of change in the future power grid is somewhat ambiguous since it mainly relies on the extent of DGs deployment that may reach. Furthermore, the precise role of DGs is still debatable and needs to be thoroughly addressed in order to have a comprehensive visualization of how things might evolve in the future. For instance, the authors of Reference [70] have devoted their article to investigate the role of DGs in shaping the future power grid.

The definition of DGs in the literature is somewhat opaque [70] as there is no exact definition to describe the term adequately. The resources [74,75] define the DGs as small-scale and renewable-based units that are geographically confined near the load centers. DGs may also involve large-scale units that are not friendly-based resources. In Reference [76], the authors defined the DGs as electric power sources that are physically connected to a distribution network or at the meter-side of consumers. In Reference [77], the authors classified the DGs, cogeneration, backup generation, storage capabilities, and microgrids under the category of Distributed Energy Resources (DER). Distributed Energy Resources are relatively small capacity sources that can be deployed to meet regular loads. The integration of such resources may facilitate the transition toward smart grids. The need to replace the fossil fuel generators and the mitigation of growing demands can be met by the deployment of Renewable Energy Sources (RESs). However, processing such aggregated sources require a tremendous amount of data that needs intelligent control capabilities.

However, the most prevailing and cited factor that motivated the expansion of DGs is the greenhouse emission reduction [78,79,80]. Furthermore, the growth in electrical demands [81,82], governmental regulations [83,84], market liberalization [85], and lower capital costs [86,87,88] was reported among the motivating factors to the deployment of DGs. The deployment of DGs may also provide a wide range of technical benefits if their size, location, and operation are properly optimized [70]. Furthermore, DGs may contribute to network losses reduction if installed close to the load centers [89,90,91,92,93], improve voltage profile as reported in References [94,95,96,97], enhance systems’ reliability [98,99,100], and provide computational flexibility for system controllability especially pertaining to generation [101,102,103] and demand-side management [104,105,106].

Due to the advances in technologies and the rapid expansion of sustainable resources and resiliency concerns, DER constitutes the backbone of the future grid infrastructure. According to Reference [107], the DER capacity is expected to rise from 134 GW by the end of 2017 to approximately 528.4 GW in 2026 due to distributed Solar PV, small to medium-scale wind turbines, microturbines, fuel cells, Electric Vehicles (EVs), Distributed Energy Storage (DES), and demand response (DR). For instance, the total installed capacity of DER constituted 2% of the overall installed capacity in the United States [108]. Distributed PV accounted for 12% of those installed in 2016. For example, California plans to integrate 12 GW of DER by 2020. For further details on the progress of DER in the United States, interested readers are referred to Reference [108]. In China, by the end of June 2017, the total installed capacity of distributed PVs was 16.15 GW, which constituted 17% of the overall PV installed capacity [109].

3.2. Reliability Assessment of DGs in Modern Grids

The electrical power grid is very complex and highly integrated. Failure in any part of the system exposes the network to catastrophic consequences. However, there are several terms associated with reliability studies such as vulnerability, resiliency, robustness, and security. The reliability of the power system is commonly used to encompass all these metrics. The authors of References [69,110,111,112,113,114,115,116,117,118,119,120,121] have intensively investigated the reliability assessment for smart grid purposes. The reliability is often measured differently at the transmission, generation system, and distribution levels. That is, the Average Interruption Frequency Index (SAIFI) and the System Average Interruption Duration Index (SAIDI) are used at the distribution level. SAIFI is used to quantify the average frequency of outages per consumer, whereas SAIDI is dedicated to measuring the average duration of outages [122,123,124]. On the other hand, at the bulk generation level, the term Loss of Load Expectation (LOLE) is used to describe the energy demanded that remains unserved over a period of time. Recently, the term resiliency has been frequently used in the literature. EPRI [125] defines resiliency as the ability of the grid to quickly recover after low and high-frequency events. Such events (e.g., cyber attacks or physical attacks, natural events, and severe geomagnetic disturbances) devastate the entire electrical grid.

In order to realize the impact of DGs installment on the reliability indices, the available capacity of the DG unit that assists in restoring the disrupted supply needs to be determined beforehand for reliability analysis [126]. However, the following attributes should be considered when carrying out reliability assessment studies:

• The availability of the DG: Distributed generation units are exposed to failures that might restrict their functionality. Therefore, reliability models should consider the availability of DGs under different contingency scenarios. Such models require probabilistic approaches to deal with such a kind of stochasticity.
• The operating mode of DGs (Islanded and grid-connected): In Islanded mode, the lack of appropriated control, protection capabilities, and communication infrastructure limits this kind of operation [126]. However, numerous studies have been proposed to address the reliability assessment of the distribution network to facilitate the transformation toward the intelligently managed grid. Whereas in grid-connected mode, DGs are installed close to the load centers to improve the reliability of the system by partially relieving the centralized units during heavy loading conditions. It is, therefore, necessary to quantify the power exchange between the feeders under the presence of DGs. A summary of the prevailing techniques within the layout of microgrids is highlighted in

  Table 4. Reliability assessment techniques of distributed generation (DGs) in islanded mode [126,127].

  Ref.	Technique	DG Type	Generation Model	Load Model
  [128,129]	Analytical	Dispatchable	-	-
  [130,131,132]	Monte Carlo Simulation	Dispatchable Non-Dispatchable	3 states Markov Models	Averaging
  [133]	Monte Carlo Simulation	Dispatchable Non-Dispatchable	Probabilistic approach	Probabilistic approach
  [133,134,135]	Monte Carlo Simulation	Dispatchable Non-Dispatchable	Hourly profile	Hourly profile
  [136]	Analytical	Dispatchable Non-Dispatchable	Levels of a typical day	Levels of a typical day
  [132,137,138]	Analytical	Dispatchable Non-Dispatchable	Probabilistic approach	Probabilistic approach
  [139,140,141,142]	Analytical	Dispatchable Non-Dispatchable	Segment of a year	Segment of a year

  and

  Table 5. Reliability assessment techniques for DGs in grid-connected mode of operation [126].

  Ref.	Technique	System Constraints	Power Flow	Transferred Capacity
  [143,144]	Analytical	Voltage and load	Yes	Computed
  [145]	Monte Carlo Simulation	Loading	No	Computed
  [146]	Analytical + Monte Carlo Simulation	Loading	No	Computed

  .
• The energy source (Dispatchable and Non-Dispatchable): in Dispatchable DGs, the generated power is fixed and known in Markov State Models [126,127], whereas in Non-Dispatchable units (e.g., wind and solar) the generated power depends on the availability of the intermittent sources.

3.3. Demand Response

The Federal Energy Regulatory Commission defines Demand Response (DR) as “the changes in electric usage by consumers from their normal consumption behavior in response to new pricing schemes, elevated sense of responsibility, and incentive pricings that are mainly designed to induce lower electricity consumption during high price periods or when system reliability is jeopardized” [147]. In Demand response, consumers are actively involved in grid operations as they can adjust their electricity consumption during peaking hours and may benefit through financial incentives. DR may also provide adequate capacity during the state of contingencies instead of relying on traditional measures such as shedding loads to restore the functionality of the system [126]. In addition to the relevant references highlighted earlier or later in this paper, the pivotal role of Demand Response in smart grids has been thoroughly reviewed in References [148,149,150,151,152,153,154] and the classification of demand response programs is depicted in Figure 2.

Demand response has a vital role in shaping the future power grid in addition to the storage technologies, distributed generation, and communication infrastructure [73,153,155]. In conventional power grids, consumers are effectively passive and have no choice whatsoever to monitor or/and to participate in controlling their adjustable devices intelligently. On the contrary, in smart grids, consumers are more involved in many aspects of the restructured power grid. Demand response can be classified into Price-based programs and Incentive-based programs. In the former, end-users are offered several pricing mechanisms, whereas the latter offers incentives for end-users for performing specific tasks [73,126,155,156]. Furthermore, the latter is also broken down into Direct Control Load (DCL) and Indirect Control Load (ICL). The former allows the utility to adjust the energy consumption for the controllable devices with notification beforehand, while the latter is designed for appliances that can be disrupted over a short period to reduce the peak during heavy loading conditions. The following flowchart summarizes the DR programs.

3.3.1. Reliability of Demand Response

Although demand response causes load curtailment, it is essential to distinguish the DR actions from the natural interruptions (i.e., partial or complete failure of the components) while carrying out traditional reliability assessments [157]. More often, the former is planned beforehand, whereas the latter is unforeseen, and consumers will not be able to take any precautions to alleviate the problem. Moreover, DR is performed on less sensitive loads (i.e., thermostatically controlled devices), thereby such devices may remain energized. The impact of DR on reliability requires additional metrics compared to traditional reliability assessments [126,157]. For further details, the authors of Reference [157] discussed thoroughly different reliability techniques.

Table 6. Distinct propositions and techniques for reliability assessment of DRs in the literature [126].

Ref.	Technique	Operating Mode	DR Program	DR Criteria	ICT Impact
[161]	Analytical	Grid Connected	Incentivized	Interruption cost minimization	No
[113]	SMCS	Grid Connected	Incentivized	Shifting less critical loads	No
[158,162]	Analytical SMCS	Grid Connected	Incentivized	Minimizing interruption cost/payback incentives	Yes
[159]	SMCS	Off-grid	TOU	Conflictive objectives to maximize the profits of the supplier and to minimize the payments of consumers	Yes
[160]	SMCS	Off-grid	Incentivized	Interruption cost minimization	Yes

summarizes different reliability techniques that can be used in DR programs.

The available capacity that might exist once the loads are curtailed needs to be precisely allocated and considered during conducting reliability studies. Besides, load profiles over different time horizons are necessary to model the demand in reliability assessment. Furthermore, Sequential Monte Carlo Simulation (SMCS) is frequently used to assess the reliability as in References [113,158,159,160]. DR programs can be classified into three classes according to the party in-charge as follows:

• Reliability-based (Incentive-based) Programs: in this program, a set of demand curtailment signals are sent to the participants in the form of voluntary requests or mandatory commands.
• Rate-based Programs: the prices are set beforehand over a period. Consumers are obliged to pay higher rates during peaking instants and lower prices during off-peak periods. This program is seen as a Time-of-Use (TOU) structure.
• Demand Reduction Bids: in this program, the participating consumers submit their bids to the demand aggregator or the independent system operator (ISO) offering their available capacity to be curtailed.

3.3.2. Applications of Demand Response

Controllable demands or aggregated demand response (DR) may act as a Virtual Energy Storage Systems (VESS). Intelligent management of the power and energy consumption of DR may result in functions similar to those provided by energy storage devices. Furthermore, the smart utilization of the existing assets will provide the ancillary services required at a much lesser cost. For instance, 1.5 m of aggregated refrigerators with a total capacity of 20 MW would cost £3 m as compared to the VESS, which costs nearly £20–£25 m, [163,164,165]. Furthermore, DR has the potential to reduce the ESS market share by 50% in 2030 [165].

Frequency regulations services can be provided by proper adjustments to the power consumption of the controllable devices [166,167,168,169,170,171,172,173,174]. A control algorithm for dynamically controlled refrigerators was proposed in Reference [166] to provide primary frequency control services in Great Britain. The authors of Reference [167] investigated the ability of dynamically controlled loads to maintain grid frequency within a particular range after a sudden loss of generation. The study indicated a significant delay in frequency fall and less reliance on rapidly deployable backup generators. In Reference [168], the authors used aggregated controlled bitumen tanks to provide sufficient reserve capacity for frequency support applications. The model was proved to be more reliable and faster in response compared to a frequency-sensitive generating units.

The authors of Reference [169] investigated the potential range of average power that can be offered by electric water heaters if adequately controlled for power-balancing applications. The control strategy was achieved by adjusting the setpoint temperature and the drawn hot water. The authors in Reference [170] presented a methodology to support grid frequency and voltage profile by using a controllable load consisting of an electric water heater and electric vehicles. A decentralized controllable protocol was developed to suppress active and reactive power fluctuations of renewables. In addition, Reference [175] proposed aggregated electric vehicles to provide a centralized supplementary frequency regulation through a mediator between the demands and the power system control center considering the charging requirements. The authors in Reference [171] investigated the availability of domestic refrigerators and industrial bitumen tank load to provide frequency response services. A decentralized controller was proposed to adjust load power consumption in proportion to grid frequency. In Reference [172], a two-layer control algorithm for thermostatically controllable loads to effectively participate in fast frequency regulation services due to the scarcity of AGCs in Microgrid that is highly penetrated by RESs has been proposed.

On the other hand, the authors of Reference [173] proposed a collaborative control scheme to improve the stability of islanded Microgrids. Unlike other control schemes, this study deploys all agents in the Microgrid to provide the reserves needed to achieve system stability. The presented algorithm used an MPPT to limit the power output of the solar panels during frequency rise events. In Reference [174], a comprehensive central demand response strategy for frequency regulation support with minimal adjustments to the load was presented. A simulation on the 13-bus IEEE benchmark was carried out to validate the study.

3.4. Energy Storage Technologies

Energy Storage Systems (ESSs) are fundamental parts when it comes to renewable energy resources incorporated in smart grids. The variability of such resources can be mitigated by an efficient operation of the ESSs. Energy systems store the excessive energy during off-peak periods and supply it back during peaking hours. Conventional storing technologies can provide numerous applications in power systems in addition to their primary functions, as depicted in Figure 3 [176]. The application of energy storage is not new, yet the technologies require further developments. Similarly, the operational schemes for the energy-storing devices are designed intensively for respective exclusive applications. Their inherent degraded efficiencies and high capital costs are still questionable despite the rapid developments in other fields, owing to the Electric Vehicles (EVs). Therefore, their operation and size need to be carefully optimized. In this respect, the authors of References [177,178,179,180,181,182,183] have thoroughly reviewed the role of storage systems in modern smart grids in terms of applications, costs, characteristics, optimal operation, sizing, and hybridization of storage technologies. For instance, the applications may include, but not limited to, energy arbitrage [184,185,186,187,188,189], peak shaving [190,191,192], frequency regulation [193,194,195,196] and [197], spinning reserves [198], voltage support [199,200,201], black start capabilities [202], intermittency smoothing [203], congestion mitigation [204], system expansion deferrals [205], multi-agent grid services [206,207,208,209], and load-following applications [210];

Table 7. Technical characteristics of energy storage technologies [220,221].

Energy S. Technology	Energy Density	Power Density	Discharge Time	Lifetime	Capital Cost		Technological Maturity
	Wh/kg	W/kg		(Year)	$/KW	$/KWh
Mechanical Energy Storage
PHES	0.5–1.5	-	1–24 h+	40–60	600–2000	5–100	Matured
CAES	30–60	(0.5–2.0)	1–24 h+	20–40	400–800	2–50	Developed
Flywheel	10–30	400–1500	Millisec–15 min	15	50–300	500–1000	Commercial
Electrochemical Energy Storage
Lead Acid Battery	30–50	75–300	Sec–hrs	5–15	200–300	120–150	Commercial
NiCd Battery	50–75	150–300	Sec–hrs	10–20	500–1500	800–1500	Commercial
Sodium Sulfur (NaS)Battery	150–240	150–230	Sec–hrs	10–15	1000–3000	300–500	Commercial
Lithium Battery (Li-ion)	75–250	150–315	Mins–hrs	5–15	1200–4000	300–1300	Demonstration
VRFB	10–30	--	Sec–10 h	5–10	600–1500	50–1000	Demonstration
Electrical Energy Storage
SuperCapacitor	2.5–15	500–1300	Millisec–60 Min	20+	100–300	300–10,000	Developed
SMES	0.5–5	500–2000	Millisecs–sec	20+	200–300	1000–10,000	Demonstration
Chemical Energy Storage
Hydrogen Fuel Cells	800–10,000	500+	Sec–24 h+	5–15	–	6000–20,000	Developing
SNG	10,000	(0.2–2)	1–24 h+	10–30	-	-	Developing
Thermal Energy Storage
CSP	−43.05	-	Mins–hrs	30	-	3500–7000	Developing

and

Table 8. Associative characteristics of energy storage technologies [220,221].

Energy S. Technology	Power Rating	Storage Duration	Self-Discharge per Day	Cycle Life (cycles)	Round trip Efficiency (%)	Response Time	Class
Mechanical Energy Storage
PHES	100–5000 MW	Hrs–Mons	Very small	–	65–87%	L–2 min	Long term
CAES	5–300 MW	Hrs–Mons	Small	-	50–89%	1–2 min	Long term
Flywheel	0–250 kW	Sec–Mins	100%	-	85–95%	1–2 min	Short term
Electrochemical Energy Storage
Lead Acid Battery	0–20 MW	Mins–days	0.1–0.3%	500–1000	75–80%	Seconds	Long term
NiCd Battery	0–40 MW	Mins–days	0.2–0.6%	2000–2500	85–90%	Seconds	Long term
Sodium Sulfur (NaS) battery	50 kW–8 MW	Sec–Hrs	20%	2500	80–90%	Seconds	Short term
Lithium Battery (Li-ion)	0–100 kW	Mins–days	0.1–0.3%	1000–10,000+	85–90%	Seconds	Long term
VRFB	30 kW–3 MW	Hrs–months	Small	12,000+	85–90%	Seconds	Long term
Electrical Energy Storage Systems
Double Layer Capacitor/super Capacitor	0–300 kW	Sec–hrs	20–40%	100,000+	90–95%	Milliseconds	Short term
SMES	100 kW–10 MW	Sec–hrs	10–15%	100,000+	95–98%	Millisecond	Short term
Chemical Energy Storage Systems
Hydrogen Fuel cell	0–50 MW	Hrs–months	Nearly 0	100+	20–50%	Sec–Mins	Long term
Thermal Energy Storage Systems
CSP	10 kW–20 MW	–	1%	–	<60%	10 min	Long term

highlight the most recent characteristics of storage technologies.

Within the layout of ancillary services, Electric Vehicles (EVs) can be used as an independent energy source. Once the EV is parked, the energy can be retrieved back into the grid (Vehicle to Grid V2G) [211,212,213]. Fixed or roaming energy storage systems can be deployed to provide sufficient reserve capacity for ancillary services purposes [214,215,216,217,218]. For instance, the authors of Reference [212] have proposed a real-time control strategy for a fleet of EVs to provide ancillary services. The proposed controller has considered the bidirectional charging/discharging efficiency in an extended effect during the implementation of Model Predictive Control (MPC). The simulation that has been carried out has revealed a significant improvement in battery lifetime extension, tracking error, and regulation capacity. A planning framework for a virtual investor that aims to provide ancillary services to a wholesale market has been proposed in Reference [213]. Two alternatives have been presented for the investor. The former has used a dedicated battery to achieve the objective, whereas the latter has deployed aggregated EVS to achieve the same task. The objective of the proposed optimization model is to maximize the long-term payoffs for the investor by providing an optimized daily bidding strategy. The proposed study has used an economic measure (i.e., present worth) to compare the alternatives. A sensitivity analysis has been carried out to investigate the impact of planning and operation variables on the feasibility of the two alternatives.

On the other hand, the authors of Reference [214] proposed a Virtual Energy Storage System (VESS) that consists of controllable demands and Flywheel Storage System (FSS) to provide power system ancillary services. Feasibility studies were carried out to validate the benefits of deploying such protocols over conventional measures. A hybrid ultra-capacitor and battery storage systems were proposed in Reference [215] to provide large scale regulation services. The proposed scheme aims to mitigate the abusive utilization of the battery and to increase the profitability of regulation services. Moreover, Reference [216] presented a vanadium-redox flow battery-based energy device model to provide multi-ancillary services focusing on peak-shaving application and frequency support. A cooperative dynamic energy level balancing based on consensus control among distributed energy storage devices offering frequency regulation capabilities in a droop-controlled Microgrid was investigated in Reference [217].

Furthermore, the authors of Reference [222] presented a study that assesses the financial performance of a battery energy storage system (BESS) to provide market-based frequency regulation services. The potential impact on profitability was also highlighted and evaluated. On the other hand, the replacement of bulky conventional generators by renewables may result in an inertial mass reduction, which jeopardizes grid stability. Therefore, the authors of Reference [193] have proposed a coordinated control scheme to effectively operate a hybrid system that is composed of ultra-capacitor and batteries for ancillary services purposes in the electricity market.

4. Data Management

4.1. Energy Data Management in Smart Grids

One of the main challenges toward the transformation into a functional smart grid is the scarcity of resources [223]. A smart city incorporates engineering solutions and informatics capabilities to effectively run a reliable network. It encompasses a wide range of intelligent services such as smart grids, smart transportation, smart education, smart waste management, and smart communication. Among these, the smart grid forms the backbone of the hierarchal structure of the smart city because it supplies sustainable energy to all other parts to guarantee a successful integration. Smart grids should comprise intelligent data systems in order to function appropriately. Capturing and processing such a massive amount of information needs to be carefully addressed. For instance, the References [224,225,226,227,228,229,230,231,232,233,234,235] are dedicated to investigating the progress in the field of big energy data management by enumerating different studies that have been reported to cover several data management aspects that involve data collection, data preprocessing, data integration, data analytics, data storage, visualizing the data, and decision-making.

The traditional grid heavily relies on primitive data that can be collected through limited points within the network, such as demand data, voltage data, and current data. On the contrary, the smart grid captures real-time data through enormous data points within the grid where the process can be referred to as big energy data. It may include smart meter data, dispersed sensors data, weather forecast data, load profile patterns, and may even carry out online operations for a wide range of power system applications. However, once the data are collected, proper analyses are performed to assist decision-makers in making their right decisions. The collected online data may also assist consumers to adjust their electricity consumption during peaking periods. They also help utilities in determining the faulty networks and restore the grid to regular operation. However, the expansion in advanced infrastructure brings new challenges from the data standpoint of view (e.g., data security, data privacy, cost of data storage, and retrieval).

4.1.1. Data Collection

The flow of data management in smart grids is sequentially organized, as depicted in Figure 4. However, the data collection phase is the first step in the data management process, where the information is collected from the data centers [236]. The ultimate source of data is the so-called advanced metering infrastructure (AMI) that retrieves the information from end-users’ premises. The AMI measures and collects the data every 10–15 min. The data are prone to an escalation in size, depending on the population of a given location. For instance, in Australia, Sydney has a population of nearly 4,000,000, where the last report from the Australian Bureau of Statistics indicates that there are a total of 1850 dwellings are located in Greater Sydney [237]. Such a large number of dwellings emphasizes that there are approximately 267,228,432 messages sent to the control center each day. There are, however, several data loggers that can transmit data in smart grids such as sensors data, power metrics data, mobile terminals, metadata, control devices, historical data, and reliability data [238]. Geographic information systems (GIS) may also contribute to smart grids by providing a piece of valuable information that can be utilized to perform specific tasks such as identifying the proper PV farm to be installed, visualization of generation, and distribution facilities [239,240,241].

There has been continuous progress in this area to address some challenges, such as standards, security, privacy, reliability, and scalability [223]. For instance, the authors of References [242,243] investigated the data collection attribute from the security and privacy point of view. On the other hand, the authors of References [244,245] investigated the data collection in a nonhierarchical network. The proposed studies developed a lightweight message authentication framework for smart grid data communication, considering privacy [246]. Other studies were devoted to investigating reducing storage when establishing multiple sessions with source devices/sensors [247]. The authors in Reference [248] proposed a smart meter data reporting strategy based on three novel data collection mechanisms to optimize the performance of the communication network and the performance of TCP-based communication in IEEE 802.11s mesh AMI networks.

4.1.2. Data Preprocessing

In this phase, the data collected undergo several processes before performing any task of analysis. However, the data acquired might be incomplete, inaccurate, and need to be filtered out where this process is known as data cleansing [249]. The data-cleansing phase comprises five steps. That is, once the erroneous data are received, they are defined, identified, corrected, documented, and modified to avoid future faults [250]. For instance, in smart grids, data cleansing helps in forecasting the generation from the PV system to decide the proper dynamic tariff rate [251,252].

The acquired data may even suffer from redundancy and repetition, which require more storage capacity and add more cost. Hence, the redundant data need to be identified and eliminated [253,254,255]. The authors of Reference [256] presumed that the data collected from smart grids facilities are equivalent to the data generated from media platforms. As reported above, data preprocessing may involve data cleaning, data aggregation, redundancy, repetition, and elimination [257,258,259,260]. In particular, the authors of Reference [261] discussed the data aggregation aspects where most of which were devoted to addressing privacy preservation. However, data aggregation helps in keeping them anonymous to preserve privacy.

4.1.3. Data Integration

Before the data get assimilated once originated from scattered data points, the data collected are not uniform and need to be appropriately integrated before the analysis [223,262]. For example, the load profile data and weather condition data are integrated to help utilities optimize and schedule their power production in accordance to the demands considering uncertainties [262,263,264].

4.1.4. Data Storage

In the data storage phase, the data get stored, located, and can be accessed at any instant of time [224]. In traditional grids, the load data profiles are stored for load forecasting purposes, hence they do not require excessive data storage capabilities. On the contrary, in smart grids, a wide range of data needs to be allocated and stored. The speed of processing the input/output information of the data storage is crucial due to the online operations that need to be performed [265]. Smart grids collect a massive amount of diversified data from various spots. The ability to access the stored data needs to be fast enough for real-time operations. There are several approaches reported in the literature for effective data storage mechanisms. For instance, the authors of Reference [266] proposed using the Simplified Mandatory Access Control Kernel (SMACK)-based framework, which combines Kappa and Lambda for data storage in Microgrids, while [267] utilized a graph storage approach to effectively store collected data in smart grids. The authors suggested an automatic migration of a database from the Resource Description Framework (RDF) to a graph storage engine. The schematics representation of the overall data flow is presented in Figure 4.

4.1.5. Data Mining and Data Analytics

In smart grids, a massive amount of data is regularly gathered. Several applications are then performed on the data gathered, such as end-users’ behavior analysis, state analysis, and fault analysis. The analyses that need to be carried out are classified according to the response time required while processing the data. The first category, which does not require high response time, may include nonurgent operations such as long load forecasting [268] and customer analysis. Faults analysis and smart metering data are designated among those applications that need to be analyzed as quickly as possible. For example, the authors of Reference [224] proposed the use of a machine-learning algorithm to process local data to save network bandwidths, reduce time delays. Furthermore, the proposed study deployed a central processor to act as a mediator between local processors. Such a mixing approach contributed to overall cost reduction.

4.1.6. Data Visualization

Data visualization provides a visual representation to help decision-makers for better analysis assimilation [269,270]. The insights provided by data visualization may contain graphs that are more efficient to understand than digesting only numerical data. It provides visual patterns that help in recognizing the sources of concerns and sources of opportunities. There are several visualization tools (e.g., 2D and 3D Vis tools) available in the market that can be used by both consumers and utilities to visualize the end-user power consumption, generation of renewable-based sources, and power quality data. Moreover, GIS software such as ArcGIS, QGIS, MapInfo, and Maptitude are valuable alternatives for visualizing smart grid data on the maps [270].

4.1.7. Online Decision-Making

Smart grids allow a real-time and automated decision-making process where such features are unlikely seen in traditional grids [271]. Important decisions can be taken, such as real-time pricing mechanisms, the capability of providing on-demand renewable generation, and estimating capacity constraints [271,272]. The ability to provide real-time analysis and decisions is impacted on reliability improvement in smart grids and thereby increasing end-user’s confidence in the technology. Based on real-time analysis, it is easier to visualize the faulty sections in the smart grids, isolate them, and take the proper corrective measure to mitigate such issues [273].

4.1.8. Data Management Challenges in Smart Grids

The data gathered from various spots support the functionality of smart grids and helps to attain intelligent decisions once needed. The data collected supports consumers and utilities to manage their energy effectively, helps both participants to optimize their usage or production, and allows an interactive mode among different participants. However, such massive data bring numerous challenges in terms of volumes, reliability, scalability, as well as data security and privacy. Storing such massive data requires enormous storage capacity. However, the current infrastructure may lack the skilled workforce to manage such amount of data. Therefore, active participation from end-users to deal with smart metering data is mandatory to ensure maximum utilization. On the other hand, data reliability and scalability are essential attributes and required to place confidence in the data for a trustworthy decision-making process.

4.2. Cyber Security of Smart Grids

Due to the necessity of data security issues, we devoted this sub-section to highlight the complexity of the widespread problems in cybersecurity. The advanced automation and communication capabilities in smart grids expose the entire system to cyber threats. Although the integration of smart grids empowers electric utilities and end-users and enhances the reliability and availability of the service with the ability to monitor and manage the behavior of the demands continually, it brings various security constraints and vulnerabilities [274,275]. Antagonists may illegally and arbitrarily disrupt the system dynamics or induce a secondary perturbation if no proper security measures are taken [276]. Cybersecurity issues, threats, vulnerabilities, and countermeasures have been extensively investigated in the literature, particularly during the past decade due to the rapid advancements in automation [277,278,279,280,281,282,283,284,285,286,287,288,289]. However, within the analysis framework, it is very critical to find adequate mathematical models that describe the cyber attacks. However, cyber attacks can be classified into Denial of Service (DoS) attacks, reply attacks, and deception attacks whose models are summarized in

Table 9. Mathematical models of cyber attacks [276,290].

Type of Attack	Mathematical Model
DoS attacks	$\overline{{\mathit{y}}_{\mathit{k}}}\in \mathit{\varphi }$$:\mathrm{refers}\mathrm{to}\mathrm{data}\mathrm{failure},\overline{{\mathit{y}}_{\mathit{k}}}$$\mathrm{and}{\mathit{y}}_{\mathit{k}}\mathrm{are}\mathrm{the}\mathrm{received}\mathrm{data}\mathrm{and}\mathrm{the}\mathrm{measured}\mathrm{data},\mathrm{respectively}.$
Reply attacks	$\overline{{\mathit{y}}_{\mathit{k}}}\in \mathit{Y}:\mathrm{refers}\mathrm{to}\mathrm{the}\mathrm{previous}\mathrm{information}.$
Deception attacks	$\overline{{\mathit{y}}_{\mathit{k}}}={\mathit{y}}_{\mathit{k}}^{\mathit{a}}+{\mathit{y}}_{\mathit{k}}$$:\mathrm{where}{\mathit{y}}_{\mathit{k}}^{\mathit{a}}\mathrm{indicates}\mathrm{to}\mathrm{the}\mathrm{erroneous}\mathrm{data}\mathrm{injected}\mathrm{by}\mathrm{intruders}.$

.

DoS refers to the attempts that are carried out by adversaries to disrupt system resources and make them unavailable. Up to date, there are few models reported that can quantify the performance degradation, among which the Queuing models [291,292,293], Bernoulli models [294], and Markov models [295]. In Queuing models, attacks are transformed into a timely delayed system where the traditional analysis can then be performed to investigate the stability [296,297,298]. For example, the authors of Reference [292] proposed a round-trip-based predictive control protocol to neutralize the adverse impact that results from weak DoS attacks. On the other hand, in Replay attacks, antagonists maliciously contaminate the valid data. Such attacks are not easily detectable since they pass the examination of cryptographic keys [276]. Further beyond, attackers may create channels between two terminals acting as a mediator, and they lunch repetitive messages [299]. In Deception attacks, adversaries manipulate the integrity of the transmitted data [300,301]. The objective of cyber attacks is to destroy the performance of the system and immobilize the network temporarily or permanently. The impact of cyber threats is not only limited to the service provided but also extended to affect the economy [276]. Therefore, attack detection techniques are vitally crucial in cyber-physical systems especially in power system applications. Bad data detectors are incorporated to detect the deviated estimates and may provide alarming according to the tolerance that is set beforehand [302,303,304,305]. However, there are two defense approaches against cyber attacks, the former protects the essential components before the attack takes place, whereas the latter identifies the contaminated data that were injected by adversaries afterward [306]. For example, sensitive information can be placed inside normal readings using a wavelet-based steganography technique [307]. In the second approach, the incorrect data are isolated and removed once the attack is detected. So far, there are four detection schemes in cyber system: (1) Bayesian detection [308,309,310]; (2) weighted least square (WLS) [311,312,313,314]; (3) Kalman filters-basedχ2-detector [315,316]; and (4) quasi-FDI (fault detection and isolation) techniques [317,318].

Within the framework of smart grids, it is very imperative to realize the interaction between the physical system and the cyber system. The Supervisory Control and Data Acquisition (SCADA) system, which controls the electrical grid, can be regarded as the primary target for attackers [319]. For instance, the authors of Reference [320] imputed the cyber-vulnerabilities to the data being exchanged by the Wide Area Network (WANs) between various entities. SCADA incorporates various sophisticated units such as AMI, DER, and Distribution Automation (DA) in the distribution system, which increase the level of damages once attacks take place. However, the systems that can be affected by attacks may include SCADA, Programmable Logic Controller (PLCs), and nuclear facilities [321,322]. An assessment layout to evaluate the potential vulnerabilities in the SCADA system was proposed in Reference [323]. The authors proposed a meantime-to-compromise as an index to quantify the vulnerabilities. In addition, the high-level penetration of AMI in smart grids brings more challenges in terms of data security. However, the forms of cyber attacks that may target the AMI units, such as leakage of the customer information, false data injection, and energy theft, have been identified in References [324,325,326,327]. However, the common detection techniques that can be adopted in smart grids are briefly summarized in

Table 10. Detection techniques in smart grids [319].

Range of Protection	Domain-Based	Detection System
SCADA	Networked	[328,329,330]
Substations	Host	[331]
	Networked	[332]
	Integrated	[333,334,335]
Wide area monitoring (WAM)	Host	[274,336]
GPS (PMU)	Host	[337]
Distribution system	Host	[338]
AMI	Host	[339,340,341,342]

.

Intruders may access medium control layers and inject false signals or malware that may lead to a complete failure of the entire network. Therefore, the data need to be encrypted to avoid future threats. For instance, during June of 2017, a catastrophic and unprecedented cyber attack took place against the Ukrainian power system, causing a six hours blackout [343]. This incident was considered a milestone in cybersecurity because it was the first major attack that caused a substantial collapse affecting 80,000 people. Two types of malware were cultivated in the facilities. The perpetrators flooded the company’s call center with calls to make matters worse and prevent the consumers from reporting their outages. However, the severity of the attack raises new concerns and promotes calls for serious countermeasures to avoid cyber attack severe issues.

5. Pricing Mechanisms in Smart Grids

The earliest traditional power grid started as a unidirectional power flow that permits the electricity flow from the generation to distribution centers. The typical structure of the conventional power grid is vertically integrated, where a single entity owns, operates, and controls the generation, transmission, and distribution. Regulated monopolistic utilities are obliged to serve certain regions with a predefined pricing rate [344]. The utilities are allowed to set the prices that recover the operating cost plus “normal” profits where a reviewing process to the prices takes place to mitigate profits reduction [345]. Critics, however, argue that such a policy lacks appropriate pricing mechanisms for effective pricing control and promotes inefficiency, monopoly. They also added that this model had run its course, and the costs of the mistakes that utilities make should not be passed into consumers. Public utilities very often are close to governments where politics may intervene and side with an entity against another [346].

However, the rapid growth in demands, the pressure exerted by critiques led to restructuring the current electricity market. The resulted restructured (Deregulated) market induced the following entities: GENCOs (generating utilities), TRANSCOs (transmission utilities), DISCOs (distribution utilities), RESCOs (Retailers), and ISOs (independent system operators) [347]. International Energy Agency (IEA) indicates that restructuring electricity markets breaks the monopolistic approach exercised by firms and encourages competition among participants. The electric utility has undergone several reforms to lower entry barriers in electricity markets, increase the reliance on market-based output prices, and allow consumers a choice of power suppliers [348]. Such reforms shift the vertically integrated structure toward an open competition in the market, modifying the traditional regulatory regime that somehow protected utilities from the risk effects in terms of cost and demand variations [349]. From the government’s perspective, deregulation brings capitals and encourages competitors to diversify their resources. If companies were granted the opportunity to compete for the provision of electricity freely, the efficiency due to such reforms would be impacted by the consumers. In a perfectly competitive market, when all participants compete as ‘price takers’ (i.e., no big company affects the prices), the most efficient generators would make more profits [350]. Once the market-clearing price is defined, efficient generators are always infra-marginal producers. In a deregulated market, renewable-based generators may bid exactly zero marginal cost since there is no operating cost involved [351]. One might observe why perfect competition encourages efficient electricity production.

5.1. Dynamic Pricing Mechanisms

In contrary to the conventional grids where the demand is relatively a passive participant, demands in smart grids are more involved in various applications. Electric Power Research Institute (EPRI) introduced demand Side management (DSM) in the 1980s. DSM is a global term that may include various activities ranging from load consumption management to improving energy efficiency [352]. DSM can be implemented through energy efficiency or demand response (DR). The dynamic pricing framework is designed to provide economic incentives for consumers to participate in demand management by means of “the participation of demands”. The smart grid infrastructure allows the interactive mode that takes place between the consumers and the utility to accomplish such tasks. However, several dynamic pricing modalities have been proposed for demand response such as Time-of-Use (TOU), Critical Peak Pricing (CPP), Real-Time Pricing (RTP), and Day-Ahead Pricing (DAP) [353]. For instance, the articles [354,355,356,357,358,359] have adopted the TOU as a price modality, whereas the References [360,361,362,363,364,365,366,367,368,369,370] have used the RTP scheme. Furthermore, CPP modality can be efficiently utilized, as reported in References [371,372,373,374,375,376,377], while the authors of References [378,379,380,381,382,383] have proposed a DAP as a pricing modality in a competitive electricity market. However, the complexity and cost issues may constitute the main barriers toward the expansion in adopting different pricing frameworks as a day-to-day tool in the existing grid [384,385].

The objective of demand response programs is to flatten the load pattern either by consumption reduction during peaking hours or by shifting part of the loads to off-peak periods. Demand response programs can be classified into two broader categories, as depicted in Figure 5, that is, dispatchable and non-dispatchable price-driven programs. The ability for a utility to control the load during peaking hours is known as a dispatchable program. The price-driven programs (i.e., non-dispatchable) are price-sensitive programs that provide different pricing options (Dynamic Pricing Schemes) to control the demand [386,387,388].

5.2. Time-of-Use

In the time-of-use (TOU) approach, a set of different tariffs for different instants of time daily or seasonally is predefined beforehand for a given tariff cycle. TOU is an attempt provided by utilities to alleviate the surge in demands during peaking hours by enforcing a tariff structure that charges higher rates for energy consumption during peaking periods [389]. Time-of-Use tariffs are more often used in areas where consumers have been subjected to a flat rate. Ideally, such a pricing scheme should shift the demand from peak hours to off-peak periods. Although the TOU pricing approach suffers from several challenges, it remains a preferred modality for utilities due to its simplicity. Consumers still tend to prefer such a pricing scheme over other approaches due to the fact that: (i) consumers are used to the fixed tariffs and they might feel hesitant to adhere to frequently changing rates (ii) end-users might not feel comfortable with the variability of Real-Time Pricing [390].

TOU remains one of the most important pricing schemes utilized in DR programs. Naturally, the TOU is a static time-dependent pricing scheme. Although the cost of implementing such pricing approaches is relatively low, difficulties may arise when it comes to real implementation in the existing distribution system. The existing infrastructure lacks the facilities to provide real-time data of consumers’ behavior. Therefore, the attempts to replace the current conventional meters with smart meters would significantly impact the expansion of such a pricing policy. Moreover, there are uncertainties involved when it comes to the design of TOU policy due to the constant volatility of generation and demand [355]. Furthermore, different jurisdictions apply different market structures; therefore, they require different TOU policies according to their needs. For instance, the inclusion of space heaters is needed when designing a TOU mechanism in certain territories, whereas, in some places, such a factor is not required depending on the climatic conditions. The presence of DER also needs to be considered when designing an effective TOU scheme.

Given the above, there are several attempts reported in the literature to overcome the challenges usually encountered when applying TOU policy.

Table 11. Recent advances in time-of-use (TOU) modality.

Ref.	Year	Aim
[356,391]	Jan 2019, May 2013	A game-theoretic approach has been proposed to design an optimal TOU pricing by designing utility functions to reach a Nash equilibrium.
[357]	August 2012	The authors proposed variation inequality models to design an optimal TOU pricing policy under different market structures. The proposed study considered the variations in social welfare under different market schemes.
[358,359]	May 2018, 2013	The authors have investigated the impact of the existence of DER on the TOU tariffs.
[392]	December 2013	The authors used a stochastic optimization approach and quadratic programming to design an optimal TOU price, considering the uncertainties in demand/Generation.

summarizes the challenges and the recent developments of the TOU approach.

5.3. Real-Time Pricing Method

The Real-Time Pricing (RTP) principle was proposed in early 1982 [360]. It has been widely recognized as one of the promising approaches that maximize the social welfare between consumers and service providers. RTP is a dynamic pricing approach that reflects the spot price of the wholesale market [63]. Electricity (i.e., Energy) is a commodity that can be transacted through biddings and offers. The transactions are settled and cleared by the system operator or market operator through the market clearing process. Once the price is cleared, a price signal is sent to market participants according to the market timeframe (i.e., a day ahead, an hour ahead). In contrary to the TOU pricing policy that follows a pricing scheme by blocks over a predefined cycle, RTP frequently responds to the spot market, and hence it is constantly changing. In a distribution network, a retailer may reveal the price a day ahead or on an hourly basis so the end-users can adjust their energy consumption accordingly [361,362,363,364,365,366].

However, the challenges that may arise when implementing this scheme need to be considered. Among these, a win-to-win case for both consumers and electricity providers needs to be taken care of when designing the framework. Moreover, consumers’ load profiles and comfort need to be considered. The availability of consumers to take corrective actions once the price signal is received is not guaranteed and, thereby, consumers might be heavily charged during their absence. However, automated load management capabilities may lessen such adverse impacts to some extent. Furthermore, applying such a pricing policy requires advanced infrastructure and intelligent metering technologies where the high penetration of intelligence increases the exposure to cyber threats. The following

Table 12. Recent development in real-time pricing (RTP) pricing policy.

Ref.	Year	Purpose
[367]	Jan. 2019	The authors have proposed a distributed online pricing strategy for demand-side programs under uncertainties and restricted communication links. Performing the optimization process aims to allocate the minimum operating cost for the utility considering time variant DRs. The proposed study has succeeded in reducing the gap between the online algorithm and the offline optimization process.
[368]	Dec. 2010	The authors used a least-square support vector machine to compute short-run tariffs by applying a model predictive control.
[369]	Jun. 2012	An optimal load management strategy for residential consumers that uses the communication capability of a typical smart grid was proposed in Reference [369]. The main objective was to allocate the optimal relationship between the spot price and users’ electrical appliances, including electric vehicles in a typical smart building.
[370]	May 2018	Markov’s decision-based multi-stage optimization algorithm has been proposed in Reference [370] to maximize the social welfare under the RTP pricing scheme. The optimization process has been divided into sub-problems; the former optimizes the problem from the consumers’ perspectives, while the latter is dedicated to the energy supplier that uses a dual-sub gradient convex optimization.
[394]	Jan. 2019	The authors have presented a real-time Energy Management System (EMS) that is suitable for the rooftop PV system integrated with battery storage. The EMS is grid-connected, where the price signal controls the flow of power within the system. The proposed study aims to maximize the revenue over a given cycle using Lagrange multiplier-based optimization algorithm.

highlights some of the recent developments in the RTP structure.

5.4. Critical Peak Pricing

Several obstacles prevent the expansion of adapting TOU pricing schemes and RTP. The former lacks the incentives that may further reduce the peak demand during heavily loading conditions, while the latter is rather complex when it comes to implementation [393]. However, the shortcomings usually encountered in the TOU and RTP mechanisms, along with the fast-evolving infrastructure, have promoted more interest in peak reduction by utilizing dynamic rates. In a Critical Peak Pricing (CPP) mechanism, a penalty is imposed during the operation within the peaking periods that are known beforehand [371]. CPP pricing policy augments a TOU pricing framework with a dispatchable price (i.e., penalized rate) whenever the system is heavily loaded [393]. Although the CPP policy is relatively less dynamic, it is considered a successful approach in reducing demand peaks. CPP pricing structure is a typical TOU policy with the addition of critical events that are adjusted by the utilities due to system constraints [63]. Such critical events are usually announced a day ahead; thereby, the CPP does not precisely reflect the prices of the wholesale market. Furthermore, the utilities should carefully optimize the new tariffs to make profits even though the demands are expected to be curtailed. For instance, if the prices were set very high, users may not shift their demands. On contrary, if the prices were set low, consumers may not respond to the new price signals in order to cut down their energy consumption.

The structure of a successful CPP mechanism needs to be carefully designed since the CPP possesses more constraints compared to RTP. For instance, the authors of Reference [372] developed a CPP framework to maximize the utility’s profits considering end users’ response to dynamic pricing, the total number of critical events, duration cycles, and peak rates. The proposed study utilized a profit index to address the impact of the previous metrics on the profits incurred. The implementation of the CPP approach resulted in a significant reduction in the average consumption of users. In Reference [373], the authors carried out a study to compare a TOU and a CPP framework to investigate the behavior of consumers under both pricing structures. The proposed analysis revealed the success of CPP over TOU in reducing the overall energy consumption. Dynamic pricing approaches may be less attractive in the distribution network where consumers lack the interest to follow price signals regularly. The authors in Reference [384] investigated the impact of applying such approaches on a group of people of different ages. Consumers were allowed to choose either to switch their electrical appliances or leaving homes during the events of higher rates. The study concluded that the dynamic pricing mechanisms have a negative impact on elders due to their restriction on mobility; thereby, they are exposed more likely to high prices.

Table 13. Recent developments in critical peak pricing (CPP).

Ref.	Year	Purpose
[374]	Jun. 2018	The authors have combined pricing framework (RTP and CPP) to allocate the optimal operation cycle for a smart home appliance based on a priority list. The proposed study has used an enhanced differential evolution (EDE) and teacher learning-based optimization (TLBO) to attain the maximum satisfaction of consumers.
[375]	Jan. 2019	A bi-level framework that aims to maximize the profits of a smart distribution company with electric vehicle parking lots has been proposed. The objective is to minimize the cost of energy purchased from the wholesale market for the leader (e.g., distribution company) and to maximize the profits of parking lots owner (e.g., follower) under Critical Pricing Policy.
[376]	Dec. 2018	A mixed-integer linear programming model to allocate the optimal size and optimal planning scheme of Onsite Generation System (OGS) under a Critical Pricing structure has been proposed by Reference [376]. The results have indicated a significant reduction in electricity cost if the OGS is appropriately sized and operated.
[370]	May 2018	The authors have proposed a Markov decision-based multi-stage optimization algorithm to maximize the social welfare under the RTP pricing scheme. The optimization process has been divided into sub-problems; the former optimizes the problem from the consumers’ perspectives, while the latter is dedicated to the energy supplier that uses a dual-sub gradient convex optimization.
[377]	Jun. 2018	A security-constrained program to schedule the supply–demand by designing an optimal pricing scheme has been proposed. The proposed study aims to find the best pricing modality for different demand-side programs (i.e., TOU, CPP, and RTP) that improves efficiency while guaranteeing the environmental restrictions. A multi-objective optimization approach has been used to minimize the ISO operational cost and the greenhouse emissions resulted from the generating units.

summarizes the advancements achieved to optimize the CPP frameworks.

5.5. Day-Ahead Pricing

Due to the growing developments in the industry of smart metering technologies, along with the dire need to increase the price-responsive demands, utilities have proposed and utilized various price modalities to incentivize consumers to reduce their consumption, particularly during peak periods [378]. In addition, recent studies have indicated that the exposure of end-users to hourly real-time pricing mechanisms succeeded in driving consumers to a wiser and efficient energy utilization [395,396,397]. RTP allows consumers to reduce their bills following the price deviation during low-periods and high-periods. However, there are some barriers associated with real-time pricing mechanisms, including regulatory concerns, lack of interest, and implementation issues. Therefore, it is essential to figure out a pricing mechanism that overcomes the barriers introduced earlier and possess advantages for both utilities and consumers. Day-Ahead Pricing (DAP) policy is a time-dependent pricing scheme that is set day ahead. Such a pricing scheme is more attractive to consumers since they have the ability to schedule their energy consumption, and therefore they may benefit from the operation during the off-peak periods [101].

Table 14. Developments in day-ahead pricing mechanism.

Ref.		Purpose
[380]	Jan. 2019	The authors have used a bi-objective optimization model to achieve a minimum electricity cost and to reduce the aggregated peak demands for residential Electric Vehicles (EVs) under a day ahead pricing structure.
[381]	Jan. 2019	The authors have discussed the feasibility of trading electrical power as a commodity in a day-ahead electricity market instead of trading energy. The objective of this study is to overcome the coarse discretization that is commonly seen in wholesale energy markets.
[382]	Jan. 2019	The authors have proposed an optimal hourly configuration and day head-pricing scheme in a smart distribution system considering the operation of protective devices. The authors have used a metaheuristic-based optimization model to minimize the purchasing power from the wholesale market and distributed resources owners, cost of power loss, cost of switching actions, and the cost of implementing the demand response Programs.
[383]	Jan. 2019	A game-theoretic multi-stage optimization model to simultaneously determine the dynamic pricing policy for the Independent System Operator and EV parking lots has been proposed. The study aims to minimize the electricity bills of EV owners while ensuring the profitability of the ISO in a day-ahead pricing framework.

presents numerous strategies and progress achieved to optimize the mechanism of day-ahead pricing.

Day-ahead modality looks promising approach for users and energy providers. However, there might be some challenges that need to be addressed for a successful application. The design of an optimal price beforehand may involve several factors that should be considered, such as load forecasting [398], supply availability, climatic forecasting, and energy price forecasting. Therefore, robust optimization algorithms are needed. Furthermore, since the prices are set and sent day ahead, the utility may incur a financial loss if the peak occurs during low-price periods [399].

6. Conclusions

A thorough review of the advances on the prevailing applications of smart grids has been presented mainly focusing on data management, cybersecurity, different pricing modalities, demand response, renewable power integration, and reliability indices. Since the various components in a smart grid generate a massive amount of data, a complete section is dedicated to investigating the flow, analysis, and management of such a broad set of heterogeneous data. The associated challenges and recent developments have also been presented. Due to the high-level penetration of informatics and communication infrastructure, which exposes the network to security vulnerabilities, the presented study has highlighted the potential sources of threats, the type of cyber-attacks, and the mathematical models that can be used to describe the attacks. The study has also highlighted the advances in detection techniques in the cyber system within the layout of smart grids. The power system is vast and complex, which is geographically spread over a wide area; therefore, increasing the immunity of the system against cyber attackers can also be a significant challenge. The majority of the pricing modalities proposed in this study are either price-based or incentive-based schemes. Therefore, consumers have little choice once the tariff rate is agreed upon. Such deficiency allows different entities to set the prices the way they want during peaking hours; thereby, consumers are obliged to follow the pricing schedule. However, the expansion in the advanced metering infrastructure increases the flexibility for consumers to control their energy consumption more effectively. The presented study has extensively investigated different pricing mechanisms that can be deployed by means of demand response. The state-of-the-art in that area has also been highlighted. Furthermore, the presented work has comprehensively reviewed the role of demand response in leveraging the transition toward the smart grid. Demand response can be regarded as a promising technology to promote the functionality of end-users to be more involved not only in electricity bill management but also in increasing the efficiency of the electricity market. It is pointed out that behavioral science might assist in unlocking the potentials of a successful smart grid transformation. The emerging advances in demand response applications have also been enumerated. The integration of distributed generation (DG) along with energy storage in smart grids is very imperative. We have briefly presented the recent applications in this emerging area as well. The reliability indices associated with the deployment of DGs and demand response (DR) in smart grids have also been summarized.