A Survey on Smart Agent-Based Microgrids for Resilient/Self-Healing Grids

Abstract

This paper presents an overview of our body of work on the application of smart control techniques for the control and management of microgrids (MGs). The main focus here is on the application of distributed multi-agent system (MAS) theory in multi-objective (MO) power management of MGs to find the Pareto-front of the MO power management problem. In addition, the paper presents the application of Nash bargaining solution (NBS) and the MAS theory to directly obtain the NBS on the Pareto-front. The paper also discusses the progress reported on the above issues from the literature. We also present a MG-based power system architecture for enhancing the resilience and self-healing of the system.

1. Introduction

Conceptually, the term “resilience” is defined as the ability to prepare for and adapt to changing conditions, as well as withstand and recover rapidly from disruptions [1,2]. This definition holds for microgrid (MG) and smart grid (SG) systems. A resilient MG or SG is capable of isolating parts of the system affected by catastrophic failures, such as electrical faults or climate events, in a fraction of a second after they occur and rapidly restore the healthy parts of the system through proper communication between these healthy parts [3]. Fault isolation and system restoration are directly connected to the “interoperability” of SGs, which is defined as: “the capability of two or more networks, systems, devices, applications, or components to exchange information between them and use the information exchanged” [4]. This property can only be achieved through proper use of intelligent control techniques and communication media as part of the grid infrastructure [5,6].

Massive power outages in bulk interconnected power systems around the world underscore the vulnerabilities to the current electricity infrastructure. Although uncommon, these major failures (blackouts) generally caused by cascading faults, interrupt large amounts of load, cause significant inconvenience to the people affected, and result in hundreds of millions of dollars in losses.

Table 1. Major blackouts around the world.

Country	Date	Reference
India	2012	[7]
China	2008	[8]
UK	2003	[9]
US	2003	[10]
Australia	2016	[11]
Ukraine	2015	[12]

lists a few of such major blackouts around the world.

Reliable operation of power systems in the SG era has only been made more difficult given the presence of conventional distributed generation (DG) and the high penetration of highly variable renewable generation sources (mainly wind and solar photovoltaic (PV)). Conventional analytical techniques are no longer adequate or are too cumbersome for power system analysis, performance evaluation, control, and providing reliable protective measures for these systems [13]. The reliability of large power systems requires distributed and local/regional control with adequate data processing and communication between neighboring regions. Therefore, this multidisciplinary task requires multidisciplinary knowledge with collaboration between a wide array of fields including power, control, communication, signal processing, computer science, and economics to design, develop, and maintain resilient power systems adequate to meet the challenges of the SG era. This paper reviews a body of work from the authors and from the literature on the application of intelligent multi-agent-system (MAS)-based control to maintain reliable and resilient MGs and proposes an MG-based interconnected power system with the goal of making the system resilient and self-healing.

MGs have the potential for automated local control and demonstrate the feasibility of self-healing power systems. Self-contained MGs are power system blocks in the few kW to tens of MW range, containing controllable and non-controllable loads, and local energy storage (ES). They can be readily integrated into the existing power system infrastructure without significant system-wide modification [13]. Under normal conditions, MGs participate in utility-connected power system operations, by providing power and ancillary services to the system, or purchasing electricity from the power system. Under emergency conditions, they can separate (island) from the utility power system and operate independently, within limits. Besides operating in an autonomous manner, the MG concept allows the local MG operator, owner, or group of participants to establish their own operating objectives and operate their MG assets towards local goals. In this way, MGs are seen as a means to aid the growth of distributed generation (including renewable generation), facilitate implementation of customer-managed demand response (DR), and raise overall system reliability while at the same time allowing the existing power system infrastructure to grow and evolve in a smart, adaptable way.

Self-contained MGs are rapidly being developed all over the world, including and aboard modern ships [14]. Since MGs normally contain more than one generating asset, including renewable generation, controllable load and ES (with individual and overall MG objectives that can be conflicting), multiple-objective (MO) optimization methods are necessary for their efficient and cost-effective operation [15,16,17,18,19]. Furthermore, because of the potential for single point of failure, it has been shown that both single-objective and MO optimization techniques are better suited to reach a solution (or a series of potential solutions) through distributed operation and peer-to-peer negotiation [20]. In this paper we focus on the application of distributed MAS for achieving peer-to-peer negotiation and obtaining a set of trade-off solutions (called the Pareto-front) and finally find a unique solution to the MO optimization problem on the Pareto-front using the concept of Nash bargaining solution (NBS) [21]. While using an agent-based MO approach the Pareto-front of the power management problem of an MG can be tracked, using the concept of NBS a unique and “fair” solution to the MO optimization problem is found via a distributed bargaining framework employing distributed optimization techniques. This tends to decrease the computational burden of the problem per agent and facilitate real-time implementation [19]. The fairness of NBS will be discussed in Section 3.3.

The use of MAS-based control and decision approaches in power system engineering has generated growing interest in recent years. Here we present a brief overview of the progress reported in the area from the literature. In [22], the 1999–2000 California energy crisis is studied using an intelligent MAS platform to identify the causes of market failure [22]. A MAS-based gradient descent continuous actor-critic algorithm is proposed in [23] to model the learning process of different parties participating in day-ahead electricity markets. In [24], a distributed agent-based pricing mechanism has been proposed for optimal power management of SGs using a supply-demand mismatch discovery mechanism. A similar MAS-based approach is proposed in [25] using a consensus-based algorithm considering transmission line losses. Distributed optimization techniques have been gaining scientific interest in recent years. The goal of these methods is to optimize a large-scale global objective function (composed of local objectives) subject to certain system-wide and local constraints on the decision variables. Distributed synchronous and asynchronous subgradient-based methods [26,27,28], Lagrangian-decomposition-based algorithms [29,30,31], a distributed Newton-type approach [32], diffusion strategies [33], and dual averaging [34] are a few of the noteworthy works done in this area. A review of different distributed optimization techniques from a control engineering perspective has been presented in [35]. A MAS-based constrained gradient algorithm is proposed in [36] for optimal thermal generator power management in power systems. An agent-based distributed dynamic programming algorithm is proposed and implemented in [37] to provide solution for online economic dispatching problem. Another class of MAS technique, closely related to the field of distributed optimization, is known as Distributed Constraint Optimization Problems (DCOP) [38,39]. In a DCOP setting, a community of agents cooperatively try to minimize the total cost related to a set of constraints. The goal is to perform distributed constraint satisfaction over a network of autonomous agents. Hence, unlike a distributed optimization problem, in which the global objective function plays a prominent role, in a DCOP formulation the decision problem is translated to a set of constraints. In [40,41], the power management and islanding problem of an MG have been formulated as a DCOP, respectively, and solved using the Distributed Pseudotree-Optimization Procedure (DPOP) algorithm [42].

Game theory tools have also been applied to develop distributed agent-based control and decision methods [43]. Potential games have been used in [44,45] to solve constrained optimization problems through agent-based frameworks. These agent-based frameworks (and variations of them) have been applied to the problem of charge management of electric vehicles [46] and optimal power management of distributed generators in SGs [47]. A distributed method for seeking Nash equilibrium is proposed in [48] using a discrete extremum seeking algorithm. In [49], we employed an agent-based model using the concept of dynamic Bayesian networks within an online learning framework to enable distributed seeking of Nash equilibrium. Population games and population dynamics represent another class of game theoretic methods for distributed decision making [50]. Population-game-based methods, employing MAS-based frameworks, have been used in [51,52] to model economic dispatching problem in MGs, along with DR.

Agent-based techniques have been used for islanding detection and operation. In [53] a fully decentralized approach is proposed to solve the total capacity discovery of an MG to perform successful islanding operations of the MG. Another agent-based island identification method is presented in [54]. Through this scheme the agents are able to detect different viable islanding scenarios prior to disturbances, considering stability, power balance, and voltage constraints. A hierarchical control structure for MGs is proposed in [55] where different computational tasks are distributed among different levels of the hierarchy. The proposed MO energy management problem of the MG is solved by a central control agent at the highest level of the decision structure. Another market-based distributed control mechanism for MG resource allocation is proposed in [56] based on a replicator dynamics strategy. A rule-based MAS decision hierarchy is proposed in [57] to achieve different goals through power management in a distribution system. A hierarchical hybrid (i.e., using both centralized and decentralized approaches) control mechanism is proposed in [58] for power management of an MG. Contract net protocol and multifactor evaluation are employed for coordination of different agents at different levels. Another hierarchical MAS-based control architecture within a market context is proposed in [59] for MG power management. Using an agent-based distributed control technique and droop-based frequency regulation, a cost minimization scheme is proposed for MGs in [60]. A probabilistic MAS-based model (with different agents representing vehicles, aggregators, and retailers) was proposed in [61] to study the charging behavior of electric vehicles and their effects on distribution systems through Monte-Carlo simulations. Using a consensus-based scheme within a MAS distributed control framework a frequency control mechanism is designed in [62] considering system dynamics. In [63] an agent-based DR mechanism is proposed using intelligent control techniques for an MG with ES and an industrial load agent. In [6], we employed a bi-level agent-based framework to implement a market-based DR program in a distribution feeder, using machine learning techniques. These works show that employing a MAS-based framework for analysis, control and power management of both power systems and MGs can be achieved effectively through distributed reasoning and optimization techniques.

In Section 4, we propose to divide an interconnected network into several regions, each of which contains several autonomous MGs. The MGs are connected to the regional grid and export/import power to/from the grid under normal operating conditions, but disconnect from the grid to operate in island mode at full or reduced capacity should there be a fault on the regional grid or in any of the neighboring MGs affecting regional operations. In case the fault is within an MG, the MG will isolate from the grid, while healthy neighboring MGs reconfigure system topology through agent-based negotiation, achieving a new operating point. The faulted MG may rejoin regional operation once it becomes healthy. This autonomous interactive operating principal holds for broader interconnected regions as well. In modular fashion, unhealthy regions may separate from one another, isolating the faulted region, while other regions continue operation under reconfigured operating condition. This procedure can protect power system continuity of operations by limiting the reach of potential cascading faults, in addition to embedding a hierarchical self-healing resilient system.

The rest of the paper is organized as follows. In Section 2, a general overview of different control strategies for MG power management from the literature is provided. In Section 3, MAS-based power management techniques (from authors’ works) are discussed. Section 4 presents an MG-based power system architecture with the aim of improving system resiliency and self-healing capabilities. Conclusions of the paper are given in Section 5.

2. Intelligent Control Strategies for MG Power Management

Proper control of MGs is critical to achieving high system reliability and operational efficiency [64]. Typically, a control (or energy management) system must determine active and reactive output power dispatch for each energy source in the MG while maintaining voltage and frequency within specified limits. Generally, the control structure of such systems can be classified into three categories: centralized, distributed, and hybrid control. If multiple (and at times conflicting) objectives must be met then each energy source may not operate optimally, and a compromised operating decision may be achieved. A brief description of each control category follows.

2.1. Centralized Control Scheme

In a centralized control scheme, the measurement signals of each MG energy source (generation, storage, and load) are sent to a centralized controller through their own controller, as shown in Figure 1 [65].

The centralized controller acts as an energy supervisor [66,67] and makes control action decisions based upon measured signals and objective functions, which are communicated to each local controller [15,16,17,68,69,70]. Objective functions may be conflicting; for example, to minimize system operation and maintenance costs and environmental impact (carbon footprint), while maximizing system efficiency may be competing objectives, complicating the achievement of a solution. Often, MO problems do not have a single solution but rather a set of non-dominated solutions, called a Pareto set, which include alternatives representing potential compromises among the objectives. This creates a range of choices available to decision-makers and provides them with trade-offs between multiple objectives [68]. Control signals are sent to corresponding energy sources for the purpose of developing the proper amount of aggregate output power. This MO energy management system can achieve trade-off optimal solution based on all available information, e.g., objectives and constraints. However, this scheme suffers from a heavy computation burden and is subject to single-point failures. Also, since energy sources have to share sensitive information with the centralized controller, the data privacy cannot be assured.

2.2. Distributed Control Scheme

In a fully-distributed control scheme, measurement signals of each MG energy source are sent to corresponding local controllers, as shown in Figure 2 [71,72,73]. These controllers communicate with one another for the purpose of collaborating. That is, to develop compromises (Pareto) leading to MG-collective operating decisions and the achievement of global optimization. An advantage of this scheme is the ease of “plug-and-play” operation. Also, different parties do not need to share sensitive data with their peers. In this control structure, the computation burden of each controller is greatly reduced, and there are no single-point of failure; its disadvantage, however, is the potential complexity of its communication system. Intelligent (model-free) algorithms, such as fuzzy logic, neural networks, genetic algorithms, and their hybrid combinations are potential tools for solving such problems.

A promising approach for distributed control problems is MAS [74,75]. MAS has been used for power system integration, restoration, reconfiguration, and power management of MGs [24,25,36,37,64,76,77,78,79]. A MAS may be distributed as a coupled network of intelligent hardware and software agents that work together to achieve a global objective.

2.3. Hybrid Centralized and Distributed Control Scheme

Hybrid control, a more practical scheme, combines centralized and distributed control, as shown in Figure 3 [80,81]. Distributed energy sources are grouped within an MG; a centralized control is used within each group, while distributed control is applied to a set of groups. With a hybrid energy management scheme, local optimization is achieved via centralized control within each group, while global coordination is achieved among the centralized controllers of the different groups through distributed control. In this way, the computational burden of each controller is reduced, and single-point of failure problems are mitigated.

Another hybrid control scheme, called a multi-level control framework, is shown in Figure 4 [82]. This scheme is similar to the hybrid control scheme discussed above with an additional supervisory (strategic) control level. At the operational level, basic decisions related to real-time operation are made, and actual control of each energy source is performed very rapidly, e.g., within milliseconds, based on the control objective(s) of the unit. Within a slower time frame, the tactical level aims to make operational decisions for a group of local control units or the entire subsystem, within seconds to minutes. Strategic decisions concerning the overall operation of the system, e.g., system “startup” or “shutdown”, are made at the top level [82]. Two-way communication exists among the different levels to execute decisions.

In the next Section we will explain, from our published body of work, how the AI-based MAS can improve MG intelligence and present a distributed MAS to show the process of negotiation and obtaining the Pareto-front solutions which show the trade-off among different objectives. The aim is to employ optimization techniques to address often conflicting considerations such as: emissions concerns, fuel availability, operational costs, weather conditions, the spot-market price of electricity, etc. This Section will provide a concise description of the simulation results and their interpretation, as well as conclusions.

3. Improving MG Intelligence with a MAS

The primary purpose of the MG power management architecture is to ensure stable delivery of electrical power to its local load customers, while optimizing energy production towards assigned objective(s). Whatever the objectives may be, the interaction of MG assets (local generation, ES, and load) dictates real and reactive power flows on the MG (and to/from the grid to which the MG is connected, if grid-connected.) The benefits of decentralized control methods can become apparent by embedding intelligence at specific nodes within an independent MG, whether grid-connected or in stand-alone mode. In either case, the control strategy dynamically balances multiple assets of the MG (i.e., generation, ES, and load), including the interactions of controllable and non-controllable components.

In a distributed MAS the ability of each agent (to affect the system domain) is limited, but the agents can communicate information about their goals to neighboring agents within the MAS. Cooperation arises as agents propose, accept, reject, or counter-propose courses of action. This cooperation is based on conferring with other agents of the MAS. The core of a distributed MAS (i.e., without a centralized supervisor) is based on system organization and prioritization. In other words, at each instant, the MAS agents evaluate their local situation, determine a local best solution to the problem dictated by the agents’ goals, and communicate their intended action to their neighboring agents and collectively participate in MAS prioritization, and adjust their action based on collective group decision.

A distributed MAS-based MG power management utilizes three basic agent types that facilitate islanded or grid-connected operation without a central supervisor. The agent types can be producer and consumer agents, as well as observer agents for sensing nodal information developed, consistent with classes of MG components: generator, load, and ES assets. Note that ES assets will be assigned both a producer agent (for their discharging mode) and a consumer agent (for their charging mode.) When the MG is interconnected with the utility network, an observer agent is also assigned to the point of interconnection (POI) with the grid to pass information about spot price of electricity and the status of the POI to the agents of the MG assets.

The simulation roadmap and agents’ decision cycle for distributed MAS power management are discussed in [13,18], and illustrated in Figure 5 and Figure 6, respectively.

3.1. MAS Cooperation—A Grid-Connected MG Example

Cooperative behavior amongst the agents arises from a desire to raise the overall performance of the system they are responsible for (in this case a MG.) Although agents act in a primarily self-interested way, however when considering the overall MG objectives agents will need to work together, and this often requires one or more assets to sacrifice operation at their locally “best” point to settle at a trade-off solution. The agents are willing to take such action, as long as the overall system performance is improved. The MAS may incorporate any number of negotiation techniques appropriate to the overall MG system objectives to settle at a solution on the Pareto. Under this strategy, negotiating agents share information about their individual performance with their neighboring agents.

Functional relationships within a decentralized MAS for the power management of a grid-connected MG is shown in Figure 7 [13]. Each asset of the MG has its own agent, producer agent for PV and diesel, consumer agent for load, producer and consumer agents for battery (for discharging and charging, respectively), and observer agents to observe the status of the MG bus and the circuit breaker at the point of connection of the MG with the main grid (PCC). Each agent supervises its assigned asset to best achieve its assigned goals. Each agent’s decisions rely on knowledge of its asset cost and performance (e.g., efficiency), as well as utility market price information passed along by the agent observing the utility interconnection. The producer agents have the ability to ramp generation up or down based on the constraints of the generator and the available resources; the consumer agents may start, stop or control loads, assuming they have the permission of the load centers (i.e., consumers).

Two scenarios are presented to show the capabilities of decentralized MAS management of the MG with distributed control, including how cooperation can arise leading to better system-wide performance.

Scenario 1: Sudden Change in the Market Price of Electricity from the Grid

In this scenario, the MG is initially in stable condition, interconnected with the utility grid. Figure 8 shows the cost function of operating the diesel genset and the genset performance function vs. its operating power, and grid price of electricity ($0.15/kWh). The basically cost-free solar PV generation is not sufficient to supply all the loads. So, between the choices of turning on the diesel genset and buying power from the grid, the agents collectively come to the decision to leave the genset off and purchase the remainder of the power the MG needs to supply its loads and charge the battery from the grid, because the grid price of energy is assumed to be less than the cost of operation of the diesel genset ($0.27/kWh) [13].

The above condition is suddenly disturbed by a large spike in the market price of electricity supplied by the utility (raised from $0.15/kwh to $0.30/kwh) [13]. Given the new price, since the operating cost of the diesel generator is lower than the grid price of electricity, the diesel producer agent senses an opportunity to operate its asset profitably at its maximum capacity, and sell its excess available power to the grid. Figure 9 shows this scenario. The top dashed vertical line shows the point at which the cost function of the diesel genset intersects with the grid price of electricity, and the lower solid vertical line shows the operating point of the genset, which is at its maximum performance point. Therefore, the amount of power produced by the diesel generator that can be sold to the grid is between the vertical dashed lines and the vertical solid lines. Decentralized MAS can also work for power management of islanded MGs [18].

Scenario 2: Sudden Loss of Solar Resource

This scenario shows how distributed agents can cooperate to raise the overall performance of the MG under a sudden loss of solar radiation. Initially, the MG load demand is 43 kW and maximum PV generation is 45 kW (rated). Since the grid price of electricity is still at $0.15/kWh the MG sells a small surplus of power (2 kW) to the utility grid.

The PV power is suddenly lost because of a complete loss of production from the PV array. Immediately following this disturbance, since the grid price did not change and still is low, the agents negotiate and collectively decide to purchase the needed power from the grid; so, the MG shifts from selling its excess 2 kW power to the utility to purchasing power from the grid and discharging a small amount of power from the storage, keeping its SOC near 100%. This scenario is shown in Figure 10 where based on the storage cost function and performance function (shown in Figure 10), and grid price of electricity ($0.15/kWh), storage provides a minimal amount of power (7 kW), and the MG purchasing 36 kW from the grid.

Table 2. Scenario Parameters.

Initial Conditions
Grid	Purchasing 2 kW from MG at $0.15/kWh
Diesel	Off
Load	43 kW of load energized
Storage	100% state of charge
PV	100% of maximum solar resource available, producing 45 kW
Disturbance
PV	Available solar resource drops to 0%, produces 0 kW
Final Conditions
Grid	Selling 36 kW to MG at $0.15/kWh
Diesel	Off
Load	43 kW of load energized
Storage	99% state of charge, discharging 7 kW (about 70 min rate)
PV	0% of maximum solar resource available, producing 0 kW

gives the parameters for the different scenarios discussed above [13].

3.2. MG Power Management: Managing Multiple Objectives with MAS

In the previous Section, the main objective was to minimize cost while using all the available renewable (PV) power. In this Section the cooperation of distributed intelligent agents will be highlighted for decentralized MO decision-making for optimal power management of a simple diesel-battery MG in grid-connected mode, considering the trade-offs associated with MO design [18]. Two objective functions are imposed upon each agent within the system—to minimize operational cost f_c and maximize system performance, f_p. Operational cost is related to fuel consumption of the diesel genset and the cost of battery charge/discharge, respectively. Performance is defined by operational efficiency of the assets and environmental emissions. The MG operator can impose the limitations upon the storage agents by specifying a relative importance between revenue and battery health, i.e., battery SOC. Estimation of battery efficiency, health and operational (charge/discharge) cost will not be covered here; it is reported in [83].

In each cycle of the decentralized MAS decision-making (which occurs by an event such as changes in spot market price, changes in resource availability, emergency conditions, etc.) each agent formulates, negotiates, and executes a solution for the operation of its own asset, similar to the trading cycles of market agents engaged in economic auctions, e.g., [84]. Ultimately, in each decision cycle the agents manage their assigned asset independently and respond to events as they occur, without relying upon a centralized supervisor. Collectively, agents operating autonomously self-organize and collaborate through negotiation during their decision cycles. This concept allows the MAS to be flexible and responsive to events while seeking optimal solutions to assigned goals.

Multi-objective optimization seeks an optimal solution while considering a combination of objective functions to be maximized or minimized within a set of constraints. However, in most cases, a global best of any particular individual objective function may not be a satisfactory solution for the remaining objectives [85]. Because of this, we need to alter our concept of optimality for multi-objective optimization problem at hand and settle at a trade-off solution called a Pareto optimal solution. Each Pareto optimal solution implies that any one objective’s satisfaction cannot be raised any further without lowering another objective’s satisfaction, or vice versa. In other words, many “optimal” solutions exist where the value of each objective function cannot be improved any further. This process is shown in Figure 11, where two decision variables (x₁, x₂) with three constraints (dashed lines), and two objective functions F(x) = (f₁(x), f₂(x))^T are given. A particular solution in the region of feasible solutions, shown on the left side of Figure 11, maps to the objective functions graph, projection shown on the right side of Figure 11. The Pareto front represents the combinational minimization of the two objective functions, shown by the dark line in Figure 11, along which F(x) is at a minimum.

It should be noted that in an MG setting, conditions, such as the amount of available renewable resource, load demand, or market price of electricity, can change very rapidly, therefore changing the constraints and decision variables. This means that the Pareto set of solutions in one time increment may be quite different from the Pareto set in the next increments.

In the MO MAS-based power management, the agent of each MG asset calculates cost and performance functions for its own assets at each computational increment, and determines a performance measure for optimal operation of the asset based on its given objectives. The characteristics of the cost and performance functions vary dynamically depending on the asset and conditions of the asset, such as loading, available renewable resource, and state-of-charge of the battery, etc. The agents share their own cost function f_c($\overrightarrow{x}$) and performance function f_p($\overrightarrow{x}$) information with their neighboring agents, and with cooperation and negotiation among the agents, the first best operating condition (overall objective function) assigned by each agent to its own asset is as follows:

$$F_a(\overrightarrow{x})=w_1{f}_c(\overrightarrow{x})+w_2{f}_p(\overrightarrow{x})+...$$

(1)

where, F_a is the combinational objective function for each agent and w_i are user defined weights [13,18]. Note that the selection of the weights of different objectives depends on the priority that the user or operator assigns to these objective functions. This linear sum of assigned objectives defines the decision-making search space which the agent must solve. Figure 12 shows a simple grid-connected diesel battery MG with fixed load. The MG asset ratings are given in

Table 3. MG Parameters.

Asset	Rating
Diesel Genset	125 kW rated (Fuel price: $0.97/L)
Storage Battery Bank (Lead-Acid)	16 kWh (25 kW @ 10-min rate)
Load Center	100 kW (fixed)

. Agent-based power management of this MG under two distinct scenarios will be presented in the next two subsections.

Grid-Connected Mode: Change in Spot-Market Price

This scenario demonstrates how agents sense and respond to a system disturbance using their asset cost and performance information to guide their decision-making; they also communicate with their neighboring agents to deviate from their local objectives, if necessary, considering the overall cost and performance objectives of the MG. Initially, the MG is in stable operating condition (see

Table 4. Scenario Parameters.

Initial Conditions
Grid	Purchasing 17 kW from MG at $0.31/kWh
Diesel	Supplying 112 kW
Load	100 kW of load energized
Storage	80% state of charge, discharging 5 kW
MG	Operational savings: $0.412/hr System performance: 79.184%
Disturbance
Grid	Selling price goes from $0.31/kWh to $0.38 kWh
Final Conditions
Grid	Selling 3 kW to MG at $0.38/kWh
Diesel	Supplying 92 kW
Load	100 kW of load energized
Storage	80% state of charge, discharging 5 kW
MG	Operational savings: $6.462/hr System performance: 90.737%

) interconnected with the utility and the cost and performance objectives are weighted equally (w₁ = w₂).

The MG is operating the diesel (supplying 112 kW) and discharging the battery (5 kW) considering their costs and performances to power the local load (100 kW). The MG sells the remainder of the produced power (17 kW) to the utility at the grid price of $0.31/kWh. Then, system conditions are disturbed by an increase in the market price of electricity supplied by the utility, from $0.31 to $0.38 per kWh. At this new price, the MG produces what it can within its constraints and purchases the remainder from the grid. The communication between the battery producer and consumer agents yields their decision to continue discharging the battery at the pre-disturbance level (5 kW) since the battery SOC had not changed appreciably, and the diesel producing 92 kW within its constraints of cost and performance. Therefore, the MG purchases 3 kW from the grid to supply its 100 kW load. Ultimately, as grid price rose, the new operating parameters determined by the agents lead to better overall MG performance and more operational savings than prior to the disturbance, as shown in Table 4 [18]. The pre-disturbance and post-disturbance search spaces, showing all possible discrete solutions are shown in Figure 13 and Figure 14, respectively. In each case, each small circle in the Figures displays an incremental solution until the minimum operating solution at each incremental cost is reached. The set of these solutions (on the dark red curve) make the Pareto-front, which are the trade-off solutions between minimizing cost and maximizing performance. Note that in Figure 13 and Figure 14, the vertical axis shows (1-Performance)%, which is to be minimized, indicating Performance% is maximized.

Grid-Connected Mode: Change in Battery Goal

This scenario shows how agents react and respond to a sudden change in goals made by the user (MG operator). The operator changes the relative importance of the MG revenue to the health and longevity of the battery. In other words, the operator influences the agents’ decision-making by conveying goals dynamically.

In this case, the MG conditions are initially identical to the final conditions from the previous case discussed above (

Table 5. Scenario Parameters.

Initial Conditions
Grid	Selling 3 kW to MG at $0.38/kWh
Diesel	Supplying 92 kW
Load	100 kW of load energized
Storage	60% state of charge, discharging 5 kW
MG	Operational profit: $6.462/hr System performance: 80.737%
Disturbance
User	Changes relative importance of Revenue vs. Battery Health to heavily favor battery health.
Final Conditions
Grid	Selling 33 kW to MG at $0.38/kWh
Diesel	Supplying 92 kW
Load	100 kW of load energized
Storage	60% state of charge, charging 25 kW
MG	Operational profit: $3.781/hr System performance: 99.079%

). The grid price is $0.38/kWh, which is below the charging cost of the battery ($.45/kWh), and the battery SOC is assumed to be 60%. The user-defined goal is now changed from equal importance of cost and performance (i.e., w₁ = w₂) to making the battery health and longevity (performance) more important than the MG revenue or cost, i.e., the weights are adjusted to w₂ > w₁. The battery charging and discharging agents negotiate regarding this change in the goal, and ultimately come to the decision to change the initial 5 kW battery discharge rate to charging at 25 kW to bring up the battery SOC. As shown in Table 5, by making this charge, the pre-disturbance profit of $6.46/hr reduces to the post-disturbance profit of $3.78/hr. The screen shots showing the pre-disturbance and post-disturbance cost versus performance characteristics for the charging and discharging performance of the battery are shown in Figure 15 and Figure 16, respectively. They show the agents’ negotiated operating point of 5 kW battery discharging before the disturbance changed to 25 kW battery charging after the disturbance. In the above examples the multiple solutions (Pareto-front) are obtained by changing each decision variable until the other decision variable(s) reach their maximum or minimum limits. In the next Section we propose a real-time approach to power management, based on NBS [21], to directly obtain a “unique” and “fair” solution to the MO power management problem of MGs on the Pareto-front, without the need to track the entire Pareto-front.

3.3. NBS-Based MG Power Management

In short, NBS was proposed as a solution strategy for bargaining games. A bargaining game presents situations of cooperative resource allocation, where several agents with different sets of objective functions negotiate with each other to reach Pareto-efficient solutions. NBS is a unique solution on the Pareto-front of the bargaining game. NBS has three desirable properties [21]:

• Pareto-efficiency: NBS is guaranteed to lie on the Pareto-front of the bargaining gme.
• Fairness: NBS does not discriminate between the objective functions of the agents. This implies that under symmetric and identical conditions, NBS yields identical solutions for different agents. Hence, NBS introduces a fair balance into the bargaining situation.
• Distributed solver: NBS can be formulated as an optimization problem, (2), which can be solved using a distributed optimization technique within an agent-based framework:

$$\mathrm{NBS}=\arg \underset{\mathrm{x}}{\max }{\displaystyle \prod }_{\mathrm{i}=1}^{\mathrm{N}}\left({\mathrm{U}}_{\mathrm{i}}\left(\mathrm{x}\right)-{\mathrm{d}}_{\mathrm{i}}\right)$$

(2)

where, x denotes the decision variable, U_i defines the objective functions and d_i denotes the disagreement points. The disagreement points represent the worst case scenarios for different objective functions. Hence, they embody the threat that negotiations fail. The optimization problem (2) is solved using distributed gradient method [26], as discussed in [19].

In a MG, we can define different sets of objective functions for different agents. In other words, the MO MG power management problem can be formulated as a bargaining game. Therefore, NBS can be applied to obtain a unique and fair solution to the MO problem. In this way, we can avoid calculating the whole Pareto-front at each time instant. Since NBS can be determined through a distributed optimization framework, the bargaining framework has several advantages:

• There is no need for a central power management unit to solve the NBS.
• Different agents only share their estimated optimal solutions with each other. Hence, sensitive cost and constraint data does not need to be shared, and the data privacy of different parties in the MG is respected. Each agent only has acess to its own private cost and constraints. However, global constraints (such as load balance constraint within the MG) need to be considered by all the agents.
• The distributed gradient method is guaranteed to find the optimal solution of the power management problem even when the communcation network is time varying (e.g., when some communcation links are lost temporarily). From a practical point of view, this property of the distributed gradient method contributes to the resiliency of the automation system.
• Another advantage of using NBS is that it eliminates the problem of weight selection, as it is a weighting-free solution approach.

As discussed in [19], this method can be applied to both grid-connected and islanded MGs. To verify that the NBS of the MG-based bargaining process actually lies on the Pareto-front of the MO power management problem is solved for a generic islanded MG (shown in Figure 17 [86]) with three objective functions: profit, efficiency, and the utility of power consumption (all to be maximized). Figure 18 shows the three-dimensional Pareto-front and the NBS solution for the power management problem. In this Figure the Pareto-front is obtained using central weighted-sum method [87]. The NBS, which is determined using the proposed agent-based bargaining framework, is also shown in the Figure. As can be observed, the NBS is located on the Pareto-front of the MO power management problem, as expected. Compared to the weighted-sum method, NBS has a much lower computational burden. As the number of agents and objective functions grow, the distributed bargaining framework is able to converge to the optimal solution much faster than the centralized model. Note that comparison of the centralized and distributed decision models can also be employed as a result verification tool (since bothe of them have to converge to the same result in the end). Hence, NBS defines the optimal operating point of the MG, which is sent as a reference signals to the different controllers of the elements of the MG.

According to the distributed gradient algorithm, each agent (at each iteration) takes the following steps, as discussed in [26,27]:

• Averaging operation: Each agent calculates the average of the received decision variables of its neihghboring agent in the optimization problem (including its own decision variable).
• Gradient operation: Each agent moves the decision variable against the dierction of the gradient of its private objective function.
• Projection operation: Each agent projects the decision variable to its private feasible set. The resulting variable is then submitted to the neighbors, as the agent’s updated decision variable.

3.4. Tools for MAS Modeling

Several computer programming environments have been developed for implementing and testing agent-based decision tools and models. JADE [88] is a Java-based platform for programming MAS, which was employed for MG power management and electrical distribution system control in [13,18,59,89,90]. Another recent agent-based platform is VOLTTRON which was developed at the U.S. Pacific Northwest National Laboratory (PNNL) [91,92]. we have also used MATLAB [93] in our previous work to implement agent-based decision making tools for control and power management of electricity markets and market-based systems [6,19,49].

4. An MG-Based Power System Architecture with Resiliency and Self-Healing Capabilities

In this Section we propose an MG-based power system architecture and address key research challenges for making interconnected power systems resilient and self-healing by using resilient (adaptive/cooperating) MGs at the distribution level. The goal is to provide a general direction for future research for extending the single-microgrid bargaining framework (introduced in previous Sections) to multi-microgrid power systems. We propose to use MAS-based resilient MGs presented in the previous Sections with the goal of enhancing the resilience of interconnected power systems and work toward their self-healing under the occurrence of a fault or catastrophe. Our proposed steps toward reaching this goal follows:

(1)

Develop a systematic way to break an interconnected power system, into a collection of smart regional energy networks (RENs), containing several inter-regional MGs (IRMGs).

(2)

Develop a distributed and hierarchical agent-based power management algorithm for the MGs under normal operating conditions.

(3)

Develop a comprehensive framework power management of cooperative/adaptive MGs to intelligently detect and isolate one or more faulty (or disabled) part(s) of the system and restore the rest of the system fully or partially (i.e., through DR) until the faulty parts are repaired and brought to service.

The goal is for the system to have the ability to make hierarchical control decisions virtually in real-time to achieve the above objectives. Figure 19a shows a power system (the top large circle) with many interconnected RENs, where each REN is a collection of IRMGs, as shown in the lower part of the Figure, zoomed in on one REN. There is an observer agent associated with each REN and each IRMG. The agents are collectively the decision-makers of the hierarchical MAS. Figure 19b shows a collection of feeders (each consisting of several IRMGs) in two interconnected RENs; each entity in Figure 19b has an observer agent associated with it. In each case, the cooperating IRMGs are autonomous and controllable power system entities, which can operate in grid-connected mode under normal operating conditions, and separate from the grid (as might an entire feeder or an entire REN do, depending on the severity of the fault) and operate in islanded mode when necessary. Figure 20 shows islanded IRMGs, feeders, and an islanded REN in a radial distribution system. The control and management of these entities under normal operation and after a fault is easier than controlling the large power system. Under normal operating conditions, the entities cooperate and exchange power to their benefit, as discussed earlier for each individual MG. In islanded mode, given a lack of sufficient generation, consumer agents operate under partial loading (i.e., through DR.) The healthy IRMGs, feeders and RENs continue operation at new operating points, and faulted units may return to service when they meet normal operating criteria (i.e., become healthy).

Figure 21 shows a hierarchical MAS-based power system architecture, consisting of decision-making agents at all levels (i.e., IRMG (lower) level, feeder (mid) level, and REN (upper) level). The smart agents at every level observe the electrical quantities at their level, the direction of real and reactive power exchange with their neighboring agents (solid arrows), and the communication link between them (dashed arrows). The observer agents at all levels (i.e., IRMG, feeder and REN levels) will monitor their main bus voltage and frequency, as well as communicate information about any abnormal condition to their neighboring agents. The agents collectively make proper actions (i.e., cooperating decision-making) to island and isolate faulty units. Agents at different levels can exchange information with their counterpart neighboring agents through wireless communication, secure Internet, or via synchronized phasor measurement units (PMUs), as appropriate. System monitoring and information processing through these media is essentially in real-time with communication latencies on the order of milliseconds. The feasibility of Internet-based communication was investigated in [94].

We have investigated the resiliency and self-healing of a generic networked agent-based and MG-based power system architecture comprising of eight MGs (C1–C8), as shown in Figure 22 [95]. The agents’ decision-making is based on cost and performance functions defined for each MG. Figure 22a shows the example power system under steady-state condition, with all MGs healthy. The percent active generation and load served at each MG are shown with the pi-charts. Energized transmission lines connecting MGs are shown with solid lines and dashed lines represent inactive transmission lines between the MGs.

Figure 22b shows the same system, where MG C4 (shown red) is faulted and completely lost. Therefore, the transmission lines connecting the faulted MG to the neighboring MGs have become de-energized. As a result, the agents of the remaining MGs react to the fault and recover operation at a new operating point. This results in the load served in MGs C1, C3 and C8 to decrease significantly (i.e., through DR) to accommodate for the continuous operation of the healthy MGs. This shows an example of a small resilient self-healed power system. As the size of the system under study and the number of agents increase, the scalability and the convergence rate of the implemented methods should be taken into account. The convergence rate and the scalability of distributed optimization methods highly depend on factors such as the structure of the implemented algorithm, the topology of the communication network, mathematical properties of the problem under study, the choice of the parameters, algorithm design assumptions, and the choice of the initial condition. In [34], a comprehensive convergence analysis and scaling properties of the dual averaging method is provided. The effects of network topology on the rate of convergence of the algorithm is obtained through the concept of spectral gap, which is a parameter related to the structure of the communication network. Different convergence rates are calculated for different types of standard networks. This study was extended in [96] to include distributed dual averaging method with delays in subgradient computation. In [26], the convergence rate of the distributed subgradient method is analyzed. It is shown that under certain assumptions on the structure of communication matrix, connectivity of the interaction graph, and interaction intervals, the algorithm has a geometric rate of convergence. Also, it is demonstrated that there is a trade-off in the algorithm design between the accuracy of the final results and the speed of convergence. This trade-off is reflected in the choice of the step size for the subgradient operation. In [97] the scalability of the projected subgradient algorithm is studied. Based on the convergence properties of the algorithm a stopping rule is obtained for the method. Also, it is shown in [32] that through distributed Newton method the convergence speed of the algorithm can be significantly improved by achieving super-linear convergence rates. In [98] a smoothing technique is used to develop a new decomposition method which significantly improves the efficiency of conventional dual decomposition methods for distributed optimization.

5. Conclusions

This paper presents an extensive review of the related literature on distributed MAS and the application of NBS to directly find a unique solution on the Pareto-front of the MO power management problem of MGs. In addition, the paper presents an overview of our previous works on MAS-based MO power management of MGs, including an algorithm showing agent decision roadmaps and agent dynamic decision time-frames. It shows that MAS-based control and decision-making systems and the application of NBS are capable of efficient management of MGs and handling the challenges faced in emerging energy networks. As a proposed future direction, a hierarchical MAS-based, MG-based power system architecture is also presented for enhancing the resiliency and self-healing capabilities of the system, while taking into account power system reliability.