Cooperative Control of Microgrids: A Review of Theoretical Frameworks, Applications and Recent Developments

Abstract

The development of cooperative control strategies for microgrids has become an area of increasing research interest in recent years, often a result of advances in other areas of control theory such as multi-agent systems and enabled by rapid advances in wireless communications technology and power electronics. Though the basic concept of cooperative action in microgrids is intuitively well-understood, a comprehensive survey of this approach with respect to its limitations and wide range of potential applications has not yet been provided. The objective of this paper is to provide a broad overview of cooperative control theory as applied to microgrids, introduce other possible applications not previously described, and discuss recent advances and open problems in this area of microgrid research.

1. Introduction

The theory and application of microgrids has received much attention over recent years, and they are widely recognized as an enabling concept for future electrical grids. In comparison with isolated distributed energy resources, microgrids offer several advantages. In particular, their ability to operate autonomously from the grid, either in isolation or as “clusters”, is seen as a key advantage. Power electronic converters are increasingly being utilized to interface energy resources to the grid, and allow for highly controlled power flows and flexible operating strategies that are still the focus of much research, though these create other potential issues for the grid. In this regard, research interests have focused on inverter strategies that enable autonomous, or islanded, operation. For example, introducing frequency and voltage droops [1] was an early innovation that emulates the power sharing of synchronous machines without the need for communication links. Based on these early innovations, microgrid control hierarchies have been defined [2], which introduce additional control layers allowing for the management of power quality closer to utility-prescribed norms, which emulate a conventional control hierarchy in large-scale power systems. In a comprehensive review of research on microgrids [3], cooperative control was identified as an important emerging topic. It is common for research articles relating to microgrids to describe a control strategy as “cooperative”, yet the scope of these works is wide-ranging, and often cooperative behavior is not in fact demonstrated in the normally understood sense.

Introducing some level of information exchange, even if only between nearest neighbors, allows for cooperative action to be used, and this is the principle of agent-based control. The application of multi-agent systems (MAS) theory to microgrids is an active research area [4]. Indeed, the use of MAS in microgrids includes system design, simulation, capacity sizing, scheduling, protection, operation, and maintenance. Advances in communications technology, particularly wireless networks, have been rapid and, together with new automation frameworks, are enabling agent-based strategies to be realized in a practical sense [5]. Multi-agent theory provides a comprehensive theoretical framework for cooperative control, including rigorous definitions of stability, performance, and robustness [6,7]. However, power system agents are strongly coupled. The consensus dynamics are complicated by physical and data network couplings, so that the standard agent-based techniques cannot always be utilized.

Intuitively, we associate several desirable characteristics with cooperative behavior, which is normally understood as a team-based, goal-directed activity using social interaction between agents. This can also include game theoretic notions of cooperative action, and consensus-based decision making. In this context, the objective of this paper is to provide an overview of the theoretical frameworks that are the basis of various cooperative schemes described in the microgrid research literature, while highlighting several applications to standard microgrid control problems. The intention is to demonstrate the limitations and scope of cooperative control theory to microgrids, which is broader than commonly understood. Along these lines, several novel applications are described, with discussion of recent developments that will further progress the field.

The scope for cooperative action is broad and includes load- and demand-side responses, power-generating and storage technologies available for implementing microgrids [8,9], and the ICT systems used to communicate between agents. Recent progress in high-speed wireless technology, such as 5G for Machine-to-Machine (M2M) applications using mmWave (24–40 GHz), holds the promise of ultra-low latency networked control and an Industrial Internet-of-Things (IIoT), with widespread potential for smartgrid applications. The increasing use by distribution and transmission network operators of not only distributed or embedded generators but also Phasor Measurements Units (PMUs), FACTS devices, and Wireless Sensor Networks (WSN) requires a reconsideration of conventional power systems’ control hierarchies and communication methods [10]. Industry automation frameworks are also moving in this direction.

Several qualifications are required regarding the scope of the study, including (i) microgrids are understood to be cyber–physical systems—this requires communication links and so decentralized strategies are not included (this is consistent with recent trends in microgrid control); (ii) network performance with respect to errors, packet loss, and time delays is not covered in a rigorous way, though references are provided for the reader; (iii) the focus is cooperative feedback control systems—for example, cooperative Model Predictive Control (MPC), which uses a separate theoretical framework [11], is omitted due to space limitations.

The structure of the paper is as follows: Section 2 is a short primer on graph theory, including the necessary notation, definitions, and background concepts; Section 3 introduces the notion of cooperative inverter control on graphs; Section 4 provides a general overview of modern cooperative control theory and research trends, including selected references to indicate useful microgrid applications; Section 5 discusses wireless networks and MAS in the context of recent developments in grid automation; Section 6 describes several applications of cooperative control to microgrid problems; and Section 7 discusses open problems and future directions for the research.

2. Preliminaries: Graph Theory, Definitions, and Notation

A directed graph, or digraph, $G=\left(V,E,A\right)$ is shown below in Figure 1, where the set of nodes are $V=\left\{v_1,v_2,\cdots ,v_n\right\}$, the set of directed arcs or edges $E$ from $v_i$ to $v_j$ given by $\left(v_i,v_j\right)\in E$, and the weights associated with each edge $0\le a_{ij}\le 1$ described by an adjacency matrix $A=\left[a_{ij}\right]$ for $G$. Each node has an associated set of neighbors $N_i=\left(v_j:\left(v_j,v_i\right)\in E\right)$. The weighted in-degree and out-degree of a node are shown in Equation (1). A balanced node has in-degree equal to its out-degree, and a digraph $G$ is balanced if $d_{in}=d_{out}$ for all nodes. All undirected graphs are balanced. A graph $G$ is strongly connected if there is a continuous path between any pair of nodes, and is said to have a spanning tree if, for a given node, all other nodes are reachable. The diagonal in-degree matrix $D$ is defined by Equation (2), and the graph Laplacian matrix $L$, or using notation $ℒ$ in some cases, is shown in Equation (3).

$$d_{in}=\sum _{j\in N_i}{a}_{ij},d_{out}=\sum _{j\in N_i}{a}_{ji}$$

(1)

$$D=\mathrm{diag}\left\{d_i\right\}$$

(2)

$$L=D-A$$

(3)

The eigenvalues of $L$ are particularly significant since they determine the dynamics of the MAS on graph $G$ [6,7]. Each node on the graph represents an agent with an associated dynamic model. The terms control law, policy, and strategy are used interchangeably, and a control policy is said to be admissible if it is stabilizing in the sense of Lyapunov and has finite performance cost as $t\to \infty$. The Kronecker matrix product is $\otimes$, Euclidian norm $\Vert .\Vert$, and identity matrix $I$. A non-negative matrix $A$ is irreducible if its associated graph is strongly connected, stochastic if each of the rows sum to 1, and primitive if irreducible and stochastic with one eigenvalue having a maximum modulus.

Table 1. Nomenclature.

Symbol	Definition
$i$	Index variable for agent number i
$j$	Index variable for agent number j
$N$	Total number of agents N
$G$	Communication graph
$L$	Laplacian matrix
$J$	Performance cost
$P$	Active power
$Q$	Reactive power
$\delta$	Power angle
$\theta$	Phase angle
$\omega$	Frequency
$\lambda$	Eigenvalue or Lagrange multiplier
$V$	Voltage
$I$	Current
$K$	Gain

below lists frequently used symbols in the article, unless otherwise defined in the body of the text.

Variable subscripts such as $X_i$ are used to index the agent number, variable superscripts $X^0$ denote a specific value for that variable, such as the rated value, while $\overline{X}$ represents the average value. Three-phase voltage and current vectors for device $i$ are shown as $v_i={[{\mathrm{v}}_{ia},{\mathrm{v}}_{ib},{\mathrm{v}}_{ic}]}^T$ and $i_i={[{\mathrm{i}}_{ia},{\mathrm{i}}_{ib},{\mathrm{i}}_{ic}]}^T$. The nonlinear Park transformation of a three-phase voltage or current measurement to a reference frame rotating at $\theta \left(t\right)$ is shown in Equation (4), with the zero-sequence component normally omitted for 3-wire microgrid power systems.

$$\left[\begin{array}{c}{\mathrm{v}}_{id}\\ {\mathrm{v}}_{iq}\\ {\mathrm{v}}_{i0}\end{array}\right]=F_{Parke}\left(\theta \left(t\right),{\mathrm{v}}_{ia},{\mathrm{v}}_{ib},{\mathrm{v}}_{ic}\right)$$

(4)

3. Architectures for Voltage-Source Converter-Based Microgrids

3.1. Conventional Control Hierarchy for Automous Operation of Microgrids

Multi-generator electrical power networks, particularly at large scale, are complex, nonlinear, dynamic systems where control is based on a hierarchical strategy of temporal and spatial decompositions, using engineering experience, robust design concepts, and well-understood features such as diagonal dominance of the system matrices [12]. This is consistent with a single power utility ownership structure based on centralized command and supervisory control with fixed performance objectives and metrics. Primary control is the action of the fast inner loops, which regulate the output. The purpose of secondary control is to eliminate offsets or drift in a small subset of state variables (or outputs) to desirable rms or magnitude norms, and is normally characterized by integral control action, while tertiary control is responsible for ensuring that generation is dispatched to meet aggregate demand over a longer timeframe, normally using constrained least-cost convex solver algorithms. Microgrids often utilize a supervisory energy management system (EMS) to perform the secondary and tertiary control functions. An example control hierarchy is shown in Figure 2.

Power electronic converter controls and architecture for microgrid applications have advanced significantly in recent years. Grid-interfaced power electronic converters are now pervasive within the electricity network. Grid-connected renewable energy converters are predominantly grid-following by design, with no capability until more recently for autonomous operation or some level of grid support function. Increasing deployment of energy storage converters implementing primary control of voltage, frequency, and power sharing capability enables the autonomous operation of microgrids in a practical sense [13]. Primary voltage control is normally fast-acting and reaches steady state within a few cycles. Load sharing of power is based on frequency or voltage droops, with a secondary voltage and frequency reference input to eliminate steady-state errors. Several performance improvements for secondary controller design have been proposed, particularly with regard to droop response [14].

A simple inverter model is given by Equations (5)–(8), based on [15,16,17]. The following variables are shown; for each inverter with index number $i$, ${\delta }_i$ is the power angle, $P_i$ and $Q_i$ are the active and reactive powers, respectively, $V_i$ is the output voltage reference to be synthesized at the output terminals, ${\omega }_{ni}$ and $V_n$ are the nominal frequency and voltage, respectively, $k_p$ and $k_q$ are the droop gains, and ${\omega }_c$ is the low-pass filter time constant, since Equations (6) and (8) describe a first-order low-pass filter. Low-pass filtering of the measured AC output power is used to eliminate transients and switching harmonics.

$${\dot{\delta }}_i={\omega }_{ni}+k_p\left(P_i^0-P_i\right)$$

(5)

$$\dot{P_i}=-{\omega }_c{P}_i+{\omega }_c{P}_{iAC}$$

(6)

$$V_i=V_n+k_q\left(Q_i^0-Q_i\right)$$

(7)

$$\dot{Q_i}=-{\omega }_c{Q}_i+{\omega }_c{Q}_{iAC}$$

(8)

Following the modeling approach in [15,18], the admittance between two nodes $i$ and $j$ is represented using complex notation as $Y_{ij}=G_{ij}+j{B}_{ij}$, where $G_{ij}$ is the conductance and $B_{ij}$ is the inductive susceptance. It is common in microgrid studies to neglect the effect of the capacitive susceptance, and to absorb the inverter filter and coupling transformer admittance into the line admittance. For an arbitrary electrical power network, with certain exceptions regarding capacitor shunt elements and voltage-dependent loads, Kron reduction can be utilized to eliminate interior or intermediate load buses from the nodal equations [19]. The complex load admittance at any node $i$ is $Y_{ii}=G_{ii}+j{B}_{ii}$, where $G_{ii}$ is the shunt conductance and $B_{ii}$ is the shunt inductive susceptance. Using this notation, since the line resistance is often neglected, the real and reactive three-phase power flow equations for the case of purely inductive lines are shown in Equations (9) and (10).

$$P_{iAC}=G_{ii}{V}_i^2+\sum _{\begin{array}{c}j=1\\ j\ne i\end{array}}^N{V}_i{V}_j\left|B_{ij}\right|\sin \left({\delta }_i-{\delta }_j\right)$$

(9)

$$Q_{iAC}=\left|B_{ii}\right|V_i^2-\sum _{\begin{array}{c}j=1\\ j\ne i\end{array}}^N{V}_i{V}_j\left|B_{ij}\right|\cos \left({\delta }_i-{\delta }_j\right)$$

(10)

Referring to the inverter Equations (5)–(8), secondary controls are normally introduced as auxiliary inputs for nominal frequency restoration, voltage restoration, load sharing correction, or other performance improvement, and while a multitude of designs are available in the microgrid literature, for the purposes of this paper, we are interested in a class of distributed strategies using cooperative control. Given inverters have a finite rated output; there is an ancillary secondary control problem for each inverter that relates to load sharing, given that droop gain settings are generally not optimal. Moreover, while a microgrid may operate without secondary controls, disadvantages of conventional droop controls relate to the trade-off between active power sharing accuracy and frequency deviation, as well as dynamic response [20,21].

Definition 1.

Secondary voltage and frequency control.

Given a prescribed nominal frequency and voltage values ${\omega }^0$ and $V^0$, respectively, the objective of secondary control action is to restore the power system to nominal following primary control action resulting from a disturbance, i.e.,

$$\underset{t\to \infty }{\lim }{\omega }_i={\omega }^0,\forall i,j\in N$$

(11)

$$\underset{t\to \infty }{\lim }{V}_i=V^0,\forall i,j\in N$$

(12)

Since restoration of reference norms is normally desirable in finite time, in that case, we require the existence of a $T_0\in \left[0,+\infty \right)$ such that, for all feasible initial states, $\lim t\to \infty$ in Equations (11) and (12) is replaced with $\lim t\to T_0$.

This conventional hierarchical power system model is currently being disrupted as power systems move to an open-access regime, along with the rapid progress of enabling technologies such as embedded intelligent sensing, high-speed wireless communication, and highly controllable power electronic interfaces. Controller designs will need to consider distributed multi-layered performance objectives and dynamic topologies with the aim of ‘just-in-time’ and ‘just-in-place’ delivery of services [12]. Under this scenario, the distinction between what is a primary, secondary, or tertiary control action becomes no longer relevant. In this regard, new energy paradigms are being considered, and new automation frameworks are being developed. Significantly, research into the control and automation of cyber–physical systems informs much of the discussion on future electricity grids. For the purposes of this paper, the relevant area is cooperative control of cyber–physical systems. Information exchange can be introduced to assist agents in reducing the ‘knowledge gradient’ that exists for decentralized agents. In this case, we are interested in the optimality of performance of the cyber–physical system, not simply the physical system.

Introducing some level of information exchange, even if only between nearest neighbors, allows for cooperative action to be used, and this is the principle of agent-based control. The application of MAS theory to microgrids is an active research area [3,4]. For MAS-based control of a microgrid, in addition to coupling through information exchange, the effect of Equations (9) and (10) introduces physical coupling between agents. Notwithstanding the notion that droops are a form of implicit signaling between inverters, the topic here is explicit information exchange. An appropriate starting point for the discussion is the graph-theoretic concept of converter control.

3.2. Control of DC/AC Converters on Graphs

Taking inspiration from the self-organizing behavior observed in certain natural systems, it has been demonstrated in [22,23] that a microgrid power system consisting of inter-connected power electronic DC/AC inverters, implementing power-frequency droops, is dynamically equivalent to coupled oscillators on a graph in terms of synchronization behavior. Utilizing this fact, several authors have proposed inverter control strategies based on a coupled oscillator model [24,25]. This approach enables the tools of graph theory and information flow to be applied to microgrid control problems. In [25], the algebraic connectivity properties of network graphs are used to analyze adaptive droops in microgrids. Graph-theoretic methods can be traced back to the study of nonlinear behavior in coupled electrical circuits and systems based on their structural properties [26,27]. The advantage of such an approach is that, considering the microgrid as a dynamic cyber–physical system, the flow of power or information is indistinguishable in a strictly graph-theoretic sense. Along these lines, the study of consensus and output synchronization of inter-connected systems characterized by both information and physical couplings has received recent attention in the literature [28,29,30,31], including microgrid applications.

Referring to Figure 3, the microgrid electrical network has an associated physical graph $G_P$, which represents the connectivity structure of the Kron-reduced electrical sub-system layer, and an associated communication graph $G_C$ that represents the communication layer. The topic of this paper is not the synchronization and stability of coupled oscillators over graphs, which is a significant research area in itself [26,32], but rather to describe the range of cooperative control strategies over $G_C$, which are useful for microgrid problems.

4. Theoretical Frameworks for Cooperative Control

The purpose of this section is to introduce the theory of cooperative control systems, which will provide the context for a review of cooperative techniques in the microgrid literature. Intuitively, there are several characteristics that are desired in a cooperative system, such as a distributed architecture, team or group decision-making, information exchange, and a capacity for goal-directed behavior. These concepts are further explored in the following sub-sections.

4.1. Decomposition and Coordination Algorithms

Due to the rapid advancement of communications technology and distributed computing, distributed control has become an active research area, with work relating to team problems and the control of large-scale multi-variable and multi-level systems originating with, amongst others, [33,34]. Game-theoretic and mathematical optimization concepts using dynamic programming involving multiple agents were introduced in [35]. More recently, methods for decomposition and coordination were described by [36,37]. Distributed MPC is described in [38,39]. The rationale for much of this work is that it is easier, from a computational perspective, to break down a large central control problem into smaller, independent sub-problems.

The decomposition of a large-scale system into coupled sub-systems, each with a controller agent, is shown in Figure 4. Physical coupling may, for example, be an electrical network, with information flows over a wireless network. Each sub-system produces a desired output and receives an input signal from its controlling agent, to which it sends state information for processing. Importantly, both physical and information couplings have a graph-theoretic interpretation, which provides deeper insights into the collective behavior of the system and forms the theoretical basis for multi-agent theory and cooperative control.

For feedback control of the system in Figure 4, we can make explicit the notion of information exchange and, more importantly, an information constraint. Assume that the aggregate plant transfer function $P\left(s\right)$ exists and, initially, we have a global output feedback matrix $K$ that stabilizes the plant, where the off-diagonal terms of $K$ imply communication of output values between agents. We can define an “interconnection sparsity” space of matrices, by using graph-theoretic notation and the binary matrix $A_{ij}^{\mathrm{Bin}}=1\mathrm{if}\left(v_j,v_i\right)\in E\mathrm{else}{A}_{ij}^{\mathrm{Bin}}=0$, as Equation (13) below.

$$\mathrm{Sparse}\left(A\right)=\left\{B\in {ℂ}^{m\times n}|B_{ij}=0\mathrm{if}{A}_{ij}=0\right\}$$

(13)

Requiring that $K$ is a member of this set introduces an information constraint. We can now define an optimal control problem in Equation (14), which is to find a $K$ that minimizes a particular performance norm using the Lower Fractional Transformation [40]. However, we restrict $K\in \mathrm{Sparse}\left(A\right)$, which is the set of admissable controllers with information constraints. This optimization problem is not easily solvable since it is generally not convex. Let $G=P_{22}$, if we instead restrict $K$ such that $KGK\in \mathrm{Sparse}\left(A\right)\forall K\in \mathrm{Sparse}\left(A\right)$, also known as quadratic invariance of $K$, and the problem in Equation (14) is convex [40]. It will be seen below that, for consensus problems in MAS, the Laplacian matrix $ℒ$ represents relative information exchange in the closed-loop state-space.

$$\begin{gathered} \underset{K}{\min }\Vert P_{11}+P_{12}K{\left(I-P_{22}\right)}^{-1}{P}_{21}\Vert \\ \mathrm{s}.\mathrm{t}.K\mathrm{stabilizes}P \\ K\in \mathrm{Sparse}\left(A\right) \end{gathered}$$

(14)

Decomposition and coordination algorithms have application in the constrained least-cost dispatch of microgrids, energy routing problems, scheduling problems, and cross-layer optimization [41]. For linear quadratic regulator synthesis using distributed algorithms, the reader is referred to [42,43]. The design objective is to find a closed-loop feedback control $u\left(t\right)=K\mathit{x}\left(t\right)$, which minimizes the quadratic cost function Equation (15) subject to the constraint in Equation (16), which is the system dynamics. For a linear interconnected system, it is often the case that the performance costs are additively separable, i.e., $Q$ is a diagonal matrix and so the off-diagonal elements are zero, and the system matrix $A$ has a graph-defined sparsity structure such that, as above, $A_{ij}=0$ if $\left(v_j,v_i\right)\notin E$. The sub-system dynamics can be expressed as Equation (16) with an interaction term as Equation (17), which accounts for the effect of other sub-systems.

$$\min J={\displaystyle \sum }_{i=1}^N{\int }_0^T\frac{1}{2}\left({\mathit{x}}_i^T{Q}_i{\mathit{x}}_i+u_i^{T}R{u}_i\right)dt$$

(15)

$$s.t.{\dot{\mathit{x}}}_i=A_{ii}{\mathit{x}}_i\left(t\right)+B_i{u}_i\left(t\right)+D_i{z}_i\left(t\right),i=1,\cdots ,N$$

(16)

$${\mathbf{z}}_i\left(t\right)={\displaystyle \sum }_{j=1,j\ne i}^N{A}_{ij}{\mathit{x}}_i\left(t\right),i=1,\cdots ,N$$

(17)

Using dual decomposition, the constrained minimization problem of Equations (15)–(17) is transformed into an unconstrained problem as Equations (18) and (19), where $L$ is the Lagrangian, with $v_i$ the Lagrange multiplier. Frequently described as multi-level or hierarchical decomposition, the reason is implicit in Equation (19), which implies that a two-stage solution is needed. The salient point is that given fixed trajectories for $v_i$ and ${\mathit{z}}_i$, Equation (18) breaks into additively separable problems, meaning $N$-independent minimizations [37].

$$L={\displaystyle \sum }_{i=1}^N{{\displaystyle \int }}_{t=0}^{t=T}\left(\frac{1}{2}{\mathit{x}}_i^T{Q}_i{\mathit{x}}_i+\frac{1}{2}{u}_i^{T}R{u}_i+v_i^T{\mathbf{z}}_i-{\displaystyle \sum }_{j=1,j\ne i}^N{v}_i^T{A}_{ij}{\mathit{x}}_i+{\lambda }_i^T\left[-{\dot{\mathit{x}}}_i+A_{ii}{\mathit{x}}_i+B_i{u}_i+D_i{\mathbf{z}}_i\right]\right)dt$$

(18)

$$\min J=\underset{v_i}{\max }\underset{u_i}{\min }\left\{L\right\}$$

(19)

More useful is to formulate a distributed version of this control problem, and provide an iterative process such as gradient descent to calculate the optimal gain values for each sub-controller in a dynamic sense [36]. The decomposition process introduces dual or auxiliary variables, which allow for decoupling of the problem into $N$-independent sub-problems, using the method of Lagrange multipliers. The sub-systems coordinate through exchanging these values using a stepwise iterative algorithm. In this way, the independent sub-problems are forced towards a solution that is optimal for the main problem, without an overall coordinating level.

4.2. Multi-Agent Systems Theory and Consensus Problems

MAS is a conceptual and analytical framework for agent-based systems. Motivated by the collective behavior of large groups observed in nature, how collective decisions use relative information exchange between nearest neighbors, a comprehensive mathematical framework has now been developed. A well-known application is cooperative control of autonomous vehicle formations [44]. For distributed systems, it is the MAS characteristics of resilience, adaptation, and autonomy that have promoted this framework. In general, MAS may be categorized according to a particular group decision problem. The following sub-sections introduce common MAS problems in control theory, which have direct microgrid applications. These problems are normally classified [45] as either:

A.

Cooperative regulator problems (leader-less consensus) or

B.

Cooperative tracking problems (leader-follower consensus).

4.2.1. Information Discovery and Distributed Computation

Distributed computing utilizes the idea that certain problems may be solved through decomposition of the problem, then allocation of sub-problems among distributed nodes, processors, or agents. The early work by [46] on distributed asynchronous optimization has been widely cited. A particular class of this problem involves discovery of the average, weighted average, or other function of the initial state values at each node. This is also known as the ${\rm X}-consensus$ problem [47].

 A. 

Discrete-time consensus problems on graphs.

The discrete-time consensus algorithm is shown as Equation (20) below, with $k$ the iteration number and $\epsilon$ the step size. Using the transformation in Equation (22) and global system vector $\mathit{x}$, the problem can be concisely stated as Equation (21), where $P$ is known as the Perron matrix.

$${\mathit{x}}_i\left(k+1\right)={\mathit{x}}_i\left(k\right)+\epsilon {\displaystyle \sum }_{j\in N_i}{a}_{ij}\left({\mathit{x}}_j\left(k\right)-{\mathit{x}}_i\left(k\right)\right)$$

(20)

$$\mathit{x}\left(k+1\right)=P\mathit{x}\left(k\right)$$

(21)

$$P=I-\epsilon L$$

(22)

The consensus proof relies on the structure of P. Recalling the basic non-negative matrix facts from the Preliminaries, if $G$ is a digraph with maximum in-degree $\mathsf{\Lambda }=\max d_{in}\forall i\in N$, then (i) assuming $\epsilon \in \left(0,1/\mathsf{\Lambda }\right)$, $P$ is row stochastic with all eigenvalues in the unit circle from Gershgorin’s theorem [48], (ii) if $G$ is balanced, then $P$ is doubly stochastic, and (iii) if $G$ is strongly connected, then $P$ is also primitive.

Lemma 1.

Perron–Frobenius [45].

For any primitive nonnegative matrix $P$ with right and left eigenvectors $v$ and $w,$ respectively, normalized such that $v^{T}w=1$, then $\underset{k\to \infty }{\lim }{P}^k=v{w}^T.$

Theorem 1.

Discrete-time consensus.

Given the consensus problem in Equation (21), assuming that $G$ is a strongly connected digraph with $\epsilon \in \left(0,1/\mathsf{\Lambda }\right),$ then (i) consensus is reached for all initial states with the group decision ${\sum }^{}{w}_i{\mathit{x}}_i\left(0\right)$ and ${\sum }^{}{w}_i=1$, with $w=\left[w_1,\cdots ,w_n\right]$ the left eigenvector of $P$, (ii) if $G$ is balanced (or undirected), the consensus value is given by Equation (23), i.e., the average of the initial states.

$$\frac{1}{N}{{\displaystyle \sum }}^{}{\mathit{x}}_i\left(0\right)$$

(23)

With a minor reformulation, Equation (21) becomes $\mathit{x}\left(k\right)=P^k\mathit{x}\left(0\right)$, and the proof uses the above lemma, and facts about the structure of $P$ given above.

The above theorem, while guaranteeing consensus, provides no guidance on selection of the step-size, which depends on $\mathsf{\Lambda }$ and, in turn, selection of edge weights $a_{ij}$. This suggests that the edge weights, or indeed the topology itself, may be changed in an adaptive way to speed up the convergence. An example procedure for selecting edge weights is the metropolis method [49], shown in Equation (24) below, where $n_i$ and $n_j$ are the number of neighbors of nodes $i$ and $j$, respectively.

$$a_{ij}=\{\begin{array}{c}\frac{1}{\left[\max \left(n_i,n_j\right)+1\right]}\mathrm{for}j\in N_i\\ 1-{\displaystyle {\displaystyle \sum }_{j\in N_i}}\frac{1}{\left[\max \left(n_i,n_j\right)+1\right]}\\ 0\mathrm{otherwise}\end{array}\mathrm{for}i=j$$

(24)

 B. 

Synchronous versus asynchronous update.

The standard formulation of the discrete-time consensus algorithm assumes that iterations occur synchronously, which presents a design challenge for any practical implementation, especially using coordination over wireless networks. Practical solutions include (i) update and weight, or (ii) update and continue, as described in [50]. Alternatively, a novel algorithm for fast, asynchronous, distributed averaging among $N$-agents is shown in Equations (25) and (26) [51]. The steady-state consensus value is the average of the initial values ${\mathit{x}}_i\left(0\right)$ as shown in Equation (23). The block diagram implementation is shown in Figure 5. The auxiliary variable ${\delta }_{ij}$ serves as a memory storage element, while ${\theta }_{ij}$ and ${\gamma }_i$ are step-size parameters [51]. This idea is utilized in a scheme for cooperative control of unbalance in [52].

$${\mathit{x}}_i\left(t+1\right)={\mathit{x}}_i\left(t\right)+{\gamma }_i\left[{\displaystyle \sum }_{j\in N_i}{\delta }_{ij}\left(t\right)+{\mathit{x}}_j\left(0\right)-{\mathit{x}}_i\left(t\right)\right]$$

(25)

$${\delta }_{ij}\left(t+1\right)={\delta }_{ij}\left(t\right)+{\theta }_{ij}\left[{\mathit{x}}_j\left(t\right)-{\mathit{x}}_i\left(t\right)\right]$$

(26)

 C. 

Distributed optimization, and the alternating direction multipliers method (ADMM).

Another common application for distributed algorithms is (convex) optimization, and microgrid capacity scheduling and dispatch are described in Section 6.4. These methods utilize Lagrangian dual decomposition, and also include game-theoretic problems as a sub-category based on coupled rather than aggregate costs. It can be shown that distributed optimization and consensus problems are related, and the alternating direction multipliers method (ADMM) is an alternative approach to consensus problems based on convex optimization. A microgrid application of the above is load sharing [53,54]. ADMM is based on convex optimization and Lagrangian dual decomposition [55]. Using this approach, the average consensus problem may be formulated as the minimization problem in Equation (28), with the introduction of an auxiliary variable $x_i$ and a set of equality constraints for each node; the solution then requires consensus across the network, and the nodes converge to the average $\overline{\psi }$.

$$\overline{\psi }=\frac{1}{n}{{\displaystyle \sum }}_{i=1}^n{\psi }_i\left(0\right)$$

(27)

$$\begin{array}{}\overline{\psi }=\underset{\mathit{x}}{\min }{{\displaystyle \sum }}_{i=1}^n{\left(x_i-{\psi }_i\right)}^2\\ \mathrm{s}.\mathrm{t}.x_i-{\psi }_i=0\end{array}$$

(28)

4.2.2. Consensus-Based Feedback Control

 A. 

Synchronization tracking for agents with LTI dynamics.

For the class of linear systems, the dynamics of each agent can be described as

$${\dot{\mathit{x}}}_i=A{\mathit{x}}_i+B{u}_i,{\mathit{y}}_i=C{\mathit{x}}_i,\forall i\in N$$

(29)

where $\mathit{x}$ is the state vector $\mathit{x}=\left[x_1^T,x_2^T,\cdots ,x_N^T\right]$. A leader node is selected with similar dynamics described by

$${\dot{\mathit{x}}}_0=A{\mathit{x}}_0,{\mathbf{y}}_0=C{\mathit{x}}_0$$

(30)

The leader dynamics describe a trajectory to which all nodes should synchronize. The cooperative tracking problem is to design a control input $u_i$ for each agent such that all nodes in the system synchronize to the leader trajectory, i.e., if the global synchronization error, or disagreement vector, is defined as

$$\mathbf{\delta }=\mathit{x}-{\underset{¯}{\mathit{x}}}_0$$

(31)

then the tracking synchronization problem is solved if

$$\underset{t\to \infty }{\lim }\delta \left(t\right)=0$$

(32)

Define the local neighborhood synchronization error as

$${\mathit{e}}_i={\displaystyle \sum }_{jϵN}{a}_{ij}\left({\mathit{x}}_i-{\mathit{x}}_j\right)+g_i\left({\mathit{x}}_i-{\mathit{x}}_o\right)$$

(33)

The objective of tracking synchronization is to select a distributed control for (29) of the form

$$u_i=-cK{\mathit{e}}_i$$

(34)

where $c>0$ is the scalar coupling gain and $K\in {ℝ}^{m\times n}$ is the feedback control gain matrix, and $g_i$ is also known as the pinning gain. It should be noted that the only information available to each node is the local neighborhood error. The conditions for asymptotic stability of this system are that all matrices given by

$$A-c{\lambda }_{i}BK,\forall i\in N$$

(35)

are Hurwitz [7], and the coupling gain satisfies

$$c\ge \frac{1}{2{\lambda }_{min}}$$

(36)

where ${\lambda }_{min}=mi{n}_{iϵN}Re\left({\lambda }_i\right)$ and $Re\left({\lambda }_i\right)$ is the real part of the eigenvalue ${\lambda }_i$ of $L$.

Theorem 2.

Cooperative tracking synchronization.

Select the state variable feedback control gain in Equation (34) as

$$K=R^{-1}{\mathbf{B}}^{T}P$$

(37)

where $Q=Q^T\in {ℝ}^{m\times n}$ and $R=R^T\in {ℝ}^{m\times n}$ are positive definite design matrices and $P$ is the unique positive definite solution to the Algebraic Riccati Equation (ARE)

$${\mathbf{A}}^{T}P+P\mathbf{A}+Q-P\mathbf{B}{R}^{-1}{\mathbf{B}}^{T}P=0$$

(38)

Then, for values of $c$ in Equation (36), the distributed controller in Equation (34) solves the tracking synchronization problem, i.e., ${\mathit{x}}_i\to {\mathit{x}}_o$ for all nodes. It should be noted that by Equations (31)–(33), the consensus convergence problem has been transformed into a stability problem, amenable to conventional control-theoretic techniques. Matrices $Q$ and $R$ in Equation (38) are selected for dynamic performance.

For the class of problems having nonlinear agent dynamics, feedback linearization can be used, or other feasible methods found in the literature. The literature on cooperative systems is extensive, and only the relevant results are included here. Lyapunov stability analysis is usually also performed as proof of asymptotic stability and convergence of the MAS. The discrete time formulation is similar to the above, with some important differences [45]. Observe that selection of a coupling gain in Equation (36) requires a known value of ${\lambda }_{min}$, which in turn requires the structure of $L$, i.e., the topology information, which is generally not available to each agent. As such, [56] proposes a fully distributed control using adaptive coupling gains as Equations (39)–(41), and other authors have proposed similar for output consensus and other problem variations.

$$u_i=c_i{\rho }_i\left({\xi }_i^T{P}^{-1}{\xi }_i\right)K{\xi }_i$$

(39)

$${\dot{c}}_i={\xi }_i^T\mathsf{\Gamma }{\xi }_i,i=1,\cdots ,N$$

(40)

$${\xi }_i={\displaystyle \sum }_{j\in N_i}{a}_{ij}\left(x_i-x_j\right)$$

(41)

Theorem 3.

Fully distributed adaptive synchronization tracking [56].

Convergence and stability of the leader–follower consensus problem in Equations (29) and (30) is achieved with the adaptive control protocol as Equations (39)–(41), using gains selected as $K=-B^T{P}^{-1}$, $\mathsf{\Gamma }=P^{-1}B{B}^T{P}^{-1}$, and ${\rho }_i\left({\xi }_i^T{P}^{-1}{\xi }_i\right)={\left(1+{\xi }_i^T{P}^{-1}{\xi }_i\right)}^3$, where $P$ and $B$ are from the ARE in Equation (38).

The above problem has found particular application in microgrid control, and examples will be provided in Section 6.1.

 B. 

Cooperative regulator with single integrator dynamics.

The simplest case comprises the single integrator consensus problem shown in Equation (42), where the control rule is based on the weighted sum of the relative difference in state values between nodes.

$${\dot{\mathit{x}}}_i\left(t\right)={\displaystyle \sum }_{j\in N_i}{a}_{ij}\left({\mathit{x}}_j\left(t\right)-{\mathit{x}}_i\left(t\right)\right)$$

(42)

Using the global state vector $\mathit{x}={\left[{\mathit{x}}_1^T\cdots {\mathit{x}}_i^T\cdots {\mathit{x}}_j^T\cdots {\mathit{x}}_N^T\right]}^T$, the system dynamics may be summarized as Equation (43). Considering that $L$ has row sums equal to zero, then a steady-state solution to Equation (43) is shown in Equation (44), where the null space $N\left(L\right)$ is a column vector of the constant $c$, which is also the consensus value.

$$\dot{\mathit{x}}=-L\mathit{x}$$

(43)

It can be shown that if $L$ has rank $N-1$, then ${\lambda }_1=0$ is non-repeated, if and only if $G$ has a spanning tree, and therefore the consensus value is unique.

$$\mathbf{0}=L\overline{\mathbf{1}}c$$

(44)

Theorem 4.

Average consensus for single integrator dynamics.

Define the normalized left eigenvector for ${\lambda }_1=0$ as $w_1={\left[p_1\cdots p_N\right]}^T$, then $w_1{v}_1=1$, where $w_1$ and $v_1$ are left and right eigenvalues, respectively. Assuming that the directed graph $G$ has a spanning tree, then all states ${\mathit{x}}_{i}i=1,\cdots ,N$ of the MAS in Equation (42) with initial values $\mathit{x}\left(0\right)$ will asymptotically converge to Equation (45), the consensus value. For an ordering of the eigenvalues of $L$ such that $\left|{\lambda }_1\right|\le \left|{\lambda }_2\right|\le \cdots \le \left|{\lambda }_N\right|$, the second largest eigenvalue ${\lambda }_2$ of $L$ is known as the Fiedler eigenvalue. For connected graphs, ${\lambda }_2$ is strictly positive and it can be shown [57] that the system described by Equation (43) reaches consensus with a time constant $\tau =1/{\lambda }_2$.

$$c={\displaystyle \sum }_{i=1}^N{p}_i{x}_i\left(0\right)$$

(45)

As a corollary of this result, for balanced graphs $c=\left(1/N\right)\sum _{i=1}^N{x}_i\left(0\right),$ i.e., the average of initial state values. Moreover, for undirected graphs, the states sum ${\sum }^{}{\mathit{x}}_i$ is an invariant quantity along the trajectories of the system. The introduction of a consensus-based secondary control layer in DC microgrids has been comprehensively detailed in [29], with structural enhancements introduced in [58].

 C. 

Cooperative regulator with second-order dynamics and PI controller [59].

Frequently, for secondary control problems, integral action rather than proportional control is used since the system is subject to various disturbances. As such, several interesting variations of the cooperative regulator are presented below. Second-order consensus dynamics are often used in modeling cooperative control of autonomous vehicles, where ${\mathit{x}}_i$ is position, ${\mathit{v}}_i$ is velocity, and the input is acceleration. The model is shown in Equations (46) and (47) below, where $b$ and $c$ are global coupling gains and $d_i$ is a disturbance input.

$${\dot{\mathit{x}}}_i={\mathit{v}}_i$$

(46)

$${\dot{\mathit{v}}}_i=d_i+b{\displaystyle \sum }_{j=1}^N{a}_{ij}\left({\mathit{x}}_j\left(t\right)-{\mathit{x}}_i\left(t\right)\right)+c{\displaystyle \sum }_{j=1}^N{a}_{ij}{{\displaystyle \int }}^{}\left({\mathit{x}}_j\left(\tau \right)-{\mathit{x}}_i\left(\tau \right)\right)d\tau -\gamma {\mathit{v}}_i$$

(47)

The global dynamics are expressed concisely in (48) using the stacked vector of agent states $\mathit{x}={\left[{\mathit{x}}_1^T\cdots {\mathit{x}}_i^T\cdots {\mathit{x}}_j^T\cdots {\mathit{x}}_N^T\right]}^T,\mathit{v}={\left[{\mathit{v}}_1^T\cdots {\mathit{v}}_i^T\cdots {\mathit{v}}_j^T\cdots {\mathit{v}}_N^T\right]}^T$, and by introducing the new state $\mathit{z}={\left[{\mathit{z}}_1^T\cdots {\mathit{z}}_i^T\cdots {\mathit{z}}_j^T\cdots {\mathit{z}}_N^T\right]}^T$ on account of the integrator in Equation (47). It is shown in [59] that the consensus value, with $d_i=0$, converges to $\left(1/n\right){\sum }_{i=1}^N{\mathit{x}}_{0,i}+\left(1/\gamma n\right)\sum _{i=1}^N{\mathit{v}}_{0,i}$. The characteristic equation for Equation (48) is $0=\det \left(\left(b+c\mathit{s}\right)ℒ+{\mathit{s}}^2\left(\mathit{s}+\gamma \right)I\right)$, and stability is demonstrated using the Routh–Hurwitz stability criteria, based on the eigenvalues of $ℒ$, which results in constraints on the allowable range of the coupling gains.

$$\left[\begin{array}{c}\dot{\mathit{z}}\\ \mathit{x}\\ \mathit{v}\end{array}\right]=\left[\begin{array}{ccc}\mathit{0}& I_n& \mathit{0}\\ \mathit{0}& \mathit{0}& I_n\\ -bℒ& -cℒ& -\gamma I_n\end{array}\right]\left[\begin{array}{c}\mathit{z}\\ \mathit{x}\\ \mathit{v}\end{array}\right]$$

(48)

 D. 

Finite-time consensus.

As remarked for secondary control in Section 3.1 above, it is normally preferable for consensus to be achieved during a finite time period. The input protocol in Equation (49) below has been suggested in this regard [60], with the binary function $\mathrm{sgn}(.)$ defined in Equation (50). Finite-time consensus protocols are based on the observation of fast synchronizing behavior in complex networks where the agents’ decisions are binary, i.e., either ON or OFF; for example, large swarms of fireflies appear to collectively switch their lights to the same state rapidly in unison [61].

$$u_i=-c_i{\displaystyle \sum }_{jϵN}\left\{a_{ij}\mathrm{sgn}\left({\mathit{x}}_i-{\mathit{x}}_j\right)+g_i\mathrm{sgn}\left({\mathit{x}}_i-{\mathit{x}}_o\right)\right\}$$

(49)

$$\mathrm{sgn}\left(r\right)=\{\begin{array}{c}1,r>0\\ 0,r=0\\ -1,r<0\end{array}$$

(50)

 E. 

Output consensus.

The problem of output synchronization tracking is a unique variation, and is shown below in Equations (51) and (52). It is frequently the case that consensus of only the outputs is required. This particular problem has attracted significant research interest, particularly for nonlinear agent dynamics, and includes designs for distributed observers and other methods [62].

$${\dot{\mathit{x}}}_i=A_i{\mathit{x}}_i+B_i{u}_i,y_i=C_i{\mathit{x}}_i$$

(51)

$$u_i=K{\displaystyle \sum }_{j\in N_i^c}{a}_{ij}\left(y_j-y_i\right)$$

(52)

 F. 

Linear interconnected multi-agent systems (LIMAS).

As has been remarked on several occasions, power system agents are physically coupled through electrical interactions, as well as coupled over the communication graph. In this case, a more appropriate modeling framework is the linear interconnected multi-agent systems (LIMAS), assuming that a linearized model is available. For example, assuming physical coupling over graph $G_P$ and communication coupling over graph $G_C$, and using a proportional input protocol, then an appropriate model is given by Equations (53) and (54).

$${\dot{\mathit{x}}}_i=A{\mathit{x}}_i+A_p{\displaystyle \sum }_{j\in N_i^p}{a}_{ij}\left({\mathit{x}}_j-{\mathit{x}}_i\right)+B{u}_i$$

(53)

$$u_i=K{\displaystyle \sum }_{j\in N_i^c}{b}_{ij}\left({\mathit{x}}_j-{\mathit{x}}_i\right)$$

(54)

In general, we are interested in how the selection of parameters affects the ‘consensusability’ of this system [30]. With an appropriate transformation of state variables, as deviations for the equilibrium value—by definition, the consensus state—then we instead have a stabilizability problem amenable to, for example, eigenstructure decomposition. Specifically, the eigenstructure of the global system matrix is shown in Equation (55) below. The salient point is that this system matrix expresses the dual-coupling structure through the physical graph and communication graph Laplacian matrices ${ℒ}_P$ and ${ℒ}_C$, respectively. In fact, the problem is equivalent to that of designing a global controller feedback gain matrix $K$, but with structural constraints on $K$ as discussed in Section 4.1. It is commonly assumed that matrices ${ℒ}_P$ and ${ℒ}_C$ commute [31].

$$\dot{\mathit{x}}=\left(I_N\otimes A-{ℒ}_P\otimes A_P+{ℒ}_C\otimes BK\right)\mathit{x}$$

(55)

$$\mathit{x}={\left[{\mathit{x}}_1^T\cdots {\mathit{x}}_i^T\cdots {\mathit{x}}_j^T\cdots {\mathit{x}}_N^T\right]}^T$$

(56)

 G. 

LIMAS with proportional and integral control action.

A generalization of the above model is to include both proportional and integral action in the control protocol. Further, it is conceivable that proportional action and integral action may operate using different connectivity between agents, known as multiplex network control [63]. The agent dynamics are shown in Equations (57) and (58), with $\sigma ,{\sigma }_P$ and ${\sigma }_I$ representing global coupling gains. The previous model may be considered a particular case of this more general model.

$${\dot{\mathit{x}}}_i=A_i{\mathit{x}}_i+{\mathbf{b}}_i-\sigma {\displaystyle \sum }_{j=1}^N{ℒ}_{C,ij}{\mathit{x}}_j\left(t\right)+{\mathit{u}}_i\left(t\right)$$

(57)

$${\mathit{u}}_i\left(t\right)={\sigma }_P{\displaystyle \sum }_{j=1}^N{a}_{ij}\left({\mathit{x}}_j\left(t\right)-{\mathit{x}}_i\left(t\right)\right)+{\sigma }_I{\displaystyle \sum }_{j=1}^N{\beta }_{ij}{{\displaystyle \int }}^{}\left({\mathit{x}}_j\left(\tau \right)-{\mathit{x}}_i\left(\tau \right)\right)d\tau$$

(58)

As previously done, we can form the global state vector around the consensus equilibrium, and introduce the new state variable $\mathit{z}\left(t\right)$ as Equation (59). Using an augmented state vector, the global system state-space equation is Equation (60), or using $\mathit{y}\left(t\right)=\mathit{z}\left(t\right)+B$ as the more concise expression in Equation (61), with $A$ and $B$ defined in Equation (62).

$$\mathit{z}\left(t\right)={\left[{\mathit{z}}_1^T\left(t\right),\cdots ,{\mathit{z}}_N^T\left(t\right)\right]}^T=-{\sigma }_{I}X{\int }^{}\mathit{x}\left(\tau \right)d\tau$$

(59)

$$\left[\begin{array}{c}\dot{\mathit{x}}\left(t\right)\\ \dot{\mathit{z}}\left(t\right)\end{array}\right]=\left[\begin{array}{cc}A-\sigma \left({ℒ}_C\otimes I_n\right)-{\sigma }_P\left({ℒ}_P\otimes I_n\right)& I_{nN}\\ -{\sigma }_I\left({ℒ}_I\otimes I_n\right)& {\mathbf{0}}_{nN\times nN}\end{array}\right]\left[\begin{array}{c}\mathit{x}\left(t\right)\\ \mathit{z}\left(t\right)\end{array}\right]+\left[\begin{array}{c}B\\ \mathbf{0}\end{array}\right]$$

(60)

$$\left[\begin{array}{c}\dot{\mathit{x}}\left(t\right)\\ \dot{\mathit{y}}\left(t\right)\end{array}\right]=\left[\begin{array}{cc}A-\sigma \left({ℒ}_C\otimes I_n\right)-{\sigma }_P\left({ℒ}_P\otimes I_n\right)& I_{nN}\\ -{\sigma }_I\left({ℒ}_I\otimes I_n\right)& {\mathbf{0}}_{nN\times nN}\end{array}\right]\left[\begin{array}{c}\mathit{x}\left(t\right)\\ \mathit{y}\left(t\right)\end{array}\right]$$

(61)

$$A=\mathrm{diag}\left\{A_1,\cdots ,A_N\right\},B={\left[b_1^T,\cdots ,b_N^T\right]}^T$$

(62)

The stability properties of this model are discussed in [63]. The consensus value is shown in Equation (63), and the ‘consensusability’ of this model depends on the structure of the inverted term in parentheses. Specifically, this term must be non-singular with a Hurwitz symmetric part. Where this is not the case, it is suggested to add an auxiliary control $v_i=K_i{\mathit{x}}_i$ to a subset of nodes, which can be shown to stabilize the system.

$$\underset{t\to \infty }{\lim }\mathit{x}=-\frac{1}{N}{\left(\frac{1}{N}{\displaystyle \sum }_{k=1}^N{A}_k\right)}^{-1}{\displaystyle \sum }_{k=1}^N{b}_k$$

(63)

4.2.3. Dynamic Decoupling Assumptions

Since agents in a microgrid are coupled through electrical interactions, the use of agent-based control protocols will be accompanied by a disturbance input from the natural state coupling, as shown in Figure 4. Cooperative techniques for frequency and voltage regulation are also complicated by the fact that active and reactive power flows are also coupled, which is discussed in Section 6. Feedforward disturbance decoupling is usually employed in the primary compensators of voltage-source converters. Therefore, it is standard to introduce some decoupling assumptions to simplify the cooperative problem. In particular, we can identify three normal methods used for the decoupling of agent-based control in microgrids.

• Use standard MAS control methods, and either add a disturbance-decoupling control design for measurable disturbance inputs (i.e., bus voltages or output currents), or ignore the disturbance under reasonable boundedness assumptions (e.g., using robust techniques).
• Include both agent physical and communication coupling in the model, utilizing LIMAS methods as described above. The problem can be simplified using small-signal linearization and, for example, active and reactive power decoupling.
• Do not use the linearized model; instead, study the nonlinear dynamic model with cyber–physical interconnections. In this case, other techniques are required, which normally appeal to nonlinear Lyapunov methods, passivity-based control, and research into the synchronization of complex coupled networks, as described in the following paragraphs.

4.2.4. Feedback Linearization and Neuro-Adaptive Agents

Since the full-order MAS dynamics for interconnected power electronic systems will be highly nonlinear, well-known techniques are available to cancel or partially reduce the effects of nonlinearities, such as feedback linearization [64]. Neuro-adaptive controller designs are described in [65], though these are beyond the scope of this paper. Consensus for second-order dynamics was described above, while consensus of higher-order dynamic agents can be seen in [59,65].

4.3. Game-Theoretic Approaches to Distributed Control

The concept of a microgrid architecture often involves information exchange, and decision-making, between intelligent agents that may not have access to global system information regarding constraints or costs, but only local information of nearest neighbors, some of which may be private or unavailable. Under these conditions, game-theoretic strategies naturally arise, and the control problem can be decomposed into coupled sub-problems, where the performance costs of sub-systems are coupled, either in the control inputs or states, or both, and where each “player” chooses their control actions to minimize their own performance cost. Game theory models these strategic considerations using a mathematical formalism that has successfully been applied in control theory—in particular, $H_{\infty }$ control—and increasingly in other areas. Multiplayer non-zero-sum games include both cooperative and non-cooperative elements, of which the zero-sum game may be considered a particular case.

Conceptually, we have either resource allocation problems or dynamic trajectory problems, with the former having static constraints while the latter have dynamic constraints. Regarding the latter, and recalling the optimal control problem in Equations (15)–(17), if, instead of an aggregate system cost, we allocate a cost to each sub-system or player, which is a function of its own state and its coupled neighbors, then we can define a dynamic multiplayer non-zero-sum game. The game objective is for each player to find an admissible control strategy that minimizes its (coupled) performance function, which is, in effect, an infinite horizon control problem for each player. The set of optimal control policies $u_1^{*},u_2^{*},\dots ,u_N^{*}$ is found by solving offline the coupled Algebraic Riccati Equations (ARE) or Game-theoretic ARE (GARE), which constitutes a Nash Equilibrium (NE) for the solution space as defined in Equation (64) [66].

Definition 2.

Nash Equilibrium Strategies [67].

An N-tuple of strategies $\left\{u_1^{*},u_2^{*},\dots ,u_N^{*}\right\}$, $u_i\in {\mathsf{\Omega }}_i,i\in N$ and associated performance costs is $J_i^{*}$ said to constitute a Nash equilibrium for an N-player finite game in extensive form if:

$$J_i^{*}=J_i\left(u_i^{*},u_{-i}^{*}\right)\le J_i\left(u_i,u_{-i}^{*}\right),i\in N$$

(64)

4.4. Synchronization of Large-Scale Complex Systems, Passivity-Based Methods, Energy and Stability Functions

For a more general class of nonlinear interconnected systems, particularly large-scale complex networks such as an interconnected electrical power system, several approaches have been adopted. These problems relate either to synchronization of state vectors, or synchronization of outputs, otherwise known as output consensus [32]. Early results focused on diagonally dominant matrices, e.g., M-matrices to derive stability conditions for interconnected systems, and other approaches include Master Stability Functions [68], while other studies focus on the topology of large-scale networks—for example scale-free and small-word networks [69]. Interestingly, it can be shown that strategies such as “pinning” a small subset of nodes to the required reference will bring a large-scale system to an equilibrium or synchronized to a dynamic reference (synchronization by pinning control). Alternatively, making dynamical adjustments through switching the topology can also synchronize the system, i.e., synchronization by topology switching control [70]. Indeed, it can be shown that a cooperative system requires strong connectivity “most of the time”.

Passivity-based control is an application of energy conservation principles for physical systems to the case of dynamic systems with a defined input and output, and an internal structure that includes energy storage elements such as capacitors and inductors. For the application of passivity to group coordination problems, see [71]. Normally, however, there is not an obvious equivalence between power flows through the system, and the selected input and output signals, nor an obvious choice for a “total energy” function for the system; see [72] for a microgrid application using a quasi-gradient model. In certain cases, a special feedback control structure can be designed to create an input–output relation that is passive—for example, using Interconnection Damping Assignment-Passivity Based Control (IDA-PBC)—and this procedure has been applied to power electronic converters in a DC microgrid [73].

A large-scale interconnected system such as an electrical power system has a state-space representation of the dynamics that can be summarized by Equations (65) and (66), where $i$ represents a sub-system interconnected with other sub-systems designated $jϵN_i$, and with an underlying graph structure, which, here, is assumed to be connected and balanced. Designing a control as Equations (67) and (68) to achieve output consensus as defined above is challenging in several respects, and the reader is referred to [74] for a comprehensive overview; suffice it to note that if the system structure satisfies certain passivity properties, then synchronization is guaranteed. Methods for designing a feedback controller, and stability theorems, can be found in [73,74], including power system examples.

$${\dot{\mathit{x}}}_i=f_i\left({\mathit{x}}_1,\dots ,{\mathit{x}}_i,\dots ,{\mathit{x}}_N,u_1,\dots ,u_i,\dots ,u_N\right)$$

(65)

$$y_i=s_i\left({\mathit{x}}_i\right)$$

(66)

$$u_i={\displaystyle \sum }_{j=1,j\ne i}^N{a}_{ij}\mathsf{\Gamma }\left(y_j-y_i\right)$$

(67)

$$u=-\left(L\otimes \mathsf{\Gamma }\right)y$$

(68)

5. Information Exchange Using Wireless Networks

5.1. Design Considerations for Packet-Switched Cooperative Networks

The development of packet radio has its origins in the experimental DARPA packet radio network [75], which was designed for resilience and adaptability. Ad-hoc wireless networks (WANET) are distributed wireless networks without fixed infrastructure such as access points, hubs, or routers; instead, they use the participating nodes themselves to route messages and manage network connectivity dynamically. The characteristics of these networks make them particularly well-suited to multi-agent control applications. Important technical issues regarding the practical application of ad-hoc wireless in multi-agent control problems is found in, for example, [49,76,77].

Unlike wired networks using fixed infrastructure, the possibility of radio interference from nearby nodes is a problem unique to wireless networks, and channel access and routing are inherently different problems. In terms of the ISO layered model, medium access control (MAC)-level protocols are particularly relevant [78], and can be probabilistic and contention-based, or deterministic scheduling methods for channel access [49]. A useful comparison in the context of this research study is found in [79]. Enhancements to support deterministic and real-time data transmission have been proposed [80,81], and novel protocols are being actively researched [82]. For microgrid examples, refer to [83,84], where efficient use of radio spectrum and low power consumption are important. Since agent-based control occurs over a communication graph $G$ topology, with unique connectivity properties, interference may limit the set of usable simultaneous links between any two nodes; refer to Figure 6. Several mathematical models have been developed to analyze the convergence speed and throughput of agent-based wireless networks, including [76,78,85]. These are stochastic models that utilize Markov chains and graph theory to assess the impact of physical geometry between nodes, channel access regimes and routing methods on throughput, and the convergence speed of the consensus algorithm.

Incorporating real channel characteristics into an agent-based dynamic model is not practical for most purposes and would be a mixed-modeling exercise of vast complexity, so that the only feasible option is using co-simulation or network emulation. Several authors have focused on particular aspects of packet switched networks and MAS—for example, [86,87,88,89]. Often, it is sufficient to introduce time delays to the dynamic model, which can be fixed or variable, homogenous across the network, or heterogenous and link-specific [90].

A review of cooperative control of microgrids with time delays is beyond the scope of the article; examples of microgrid control with time delays include [91,92,93]. However, some brief remarks are required. Time delays will impact the stability and performance of the microgrid. Both frequency-domain and Lyapunov techniques focus on determining the maximum delay bound ${\tau }_{MAX}$ (delay-dependent stability), for uniform or multiple delays over heterogenous links. However, performance degradation is frequently the designer’s main concern and can be improved through, for example, rate allocation or compensated for elsewhere; in this regard, the reader is referred to [94]. In this regard, recall that, for the single integrator cooperative regulator problem Equation (42), the eigenvalues of the graph Laplacian matrix determine the convergence speed and stability. The speed to consensus is driven by the second-largest eigenvalue ${\lambda }_2$ of $ℒ$. The maximum delay bound ${\tau }_{MAX}$ is $\pi /2{\lambda }_{max}$, and with $\mathsf{\Lambda }=\max \sum _{i\ne j}{a}_{ij}$ then ${\lambda }_{max}<2\mathsf{\Lambda }$ and it follows ${\tau }_{MAX}<\pi /4\mathsf{\Lambda }$ [57]. It can be concluded from this that networks with a few large hubs, implying large values of $\mathsf{\Lambda }$, are not as robust to time delays as, for example, random graphs. Further, for certain network topologies, increasing connectivity between nodes will degrade rather than enhance performance [44]. An area of significant recent interest is event-triggered communication, which has the potential to improve the efficiency and utilization of both processor and communication link bandwidth; see, for example, [95]. A microgrid secondary control strategy utilizing event-triggered communication is described in [96].

5.2. Microgrid Automation Frameworks and Agent-Based Development Platforms

Progress in the development of protocols and standardized automation frameworks covers peer-to-peer networking, ad-hoc networks, multi-agent systems, object-oriented languages, and client/server models, and utilizes protocols such as CIM, UDP, TCP, MQTT, MMS, and GOOSE. Existing international standards include IEC 61850, IEC 802.11, IEC 61968, IEC 61970, IEC 61499, and IEEE P2030. IEC 61850 has been utilized as a microgrid automation framework, both for the development and prototyping of microgrid control and protection systems in a laboratory setting using real-time emulation or simulation of microgrid devices [97,98], and for demonstration of its application to the control of real microgrid systems [99,100]. Other agent-based platforms include LEAP [101] and JADE, which is described in the next section. Communication network modeling for agent-based design often utilizes co-simulation or emulation platforms, e.g., OMNeT++, ns2/3, OPNET, and CORE.

Since MAS and cooperative control are similar concepts, this section provides a brief overview of application development platforms for implementing agent-based algorithms and strategies in power systems. Although there is no universally accepted definition, it is generally understood that an “agent” is defined by the following characteristics. Given more recent interest in deep learning and AI techniques, reflected in the last point, the terminology of “agent” and “intelligent agent” is assumed interchangeable.

• Autonomy: internal control over actions and behavior
• Social ability: agent communication language
• Reactivity: interact with environment, including stimulus and response
• Pro-activeness: goal-directed behavior
• Reinforcement learning: self-assessment of decisions, performance improvement

In the context of industrial automation, it is natural to consider “agents” as software objects running code on an application platform, either local or remote to the hardware environment or plant with which they interact—control, monitor, etc.—and an object-oriented approach is common for agent-based software development platforms. However, aspects of distributed coordination may occur at different levels through an interaction hierarchy; for example, negotiation of channel access or bandwidth may occur at the medium-access control layer of the ISO hierarchy.

Importantly, the concept of interoperability is implicit in the above definition. Agents communicating across a common platform using the same language is simple enough. In industrial applications, agents reside on different equipment from multiple vendors and, therefore, the central issue is the agent communication language—in particular, the agreed protocol and ontology for the specific use case. Ontologies allow agents to agree on the meaning of the message syntax for a given context. In the energy sector, this is a key issue for agent-based automation frameworks and the associated standards. Refer to [102,103] for a summary of research into MAS in power engineering applications, and to [104] for a discussion on the use of ontologies in the energy area.

JADE is an example of a widely used, open-source development platform for MAS applications, designed to be compliant with the Foundation for Intelligent Physical Agents (FIPA) specification. Developed using JAVA, the JADE architecture is shown in Figure 7 below. In the diagram, each running instance of JADE on the computer is called a “container”, which provides a suite of specialized runtime libraries and methods that constitute the software environment in which the agents operate.

Message queuing telemetry transport (MQTT) is an application layer publisher–subscriber protocol that runs normally over TCP/IP networks and is suitable for low-bandwidth M2M applications. Utilizing MQTT, a wireless microgrid energy management system (EMS) is described in [105], which highlights the need for cross-layer optimization. A comprehensive review of microgrid communication technology and trends is provided in [106]. A summary of current wireless technologies is provided in

Table 2. Summary of wireless technologies for cooperative control [107].

Technology	Standards	Data Rate	Distance	Latency	Fixed Cost
ZigBee	IEEE 802.15.4	<1 Mbps	100 m	50 ms	Low
	IEEE 802.11ax	>1 Gbps	70 m	3 ms	Medium
WLAN	IEEE 802.11ac	100 Mbps–1 Gbps)	70 m	10 ms	Low
	IEEE 802.11n	1–100 Mbps	50 m	15 ms	Low
	IEEE 802.11g	1–100 Mbps	50 m	15 ms	Low
Cellular	3G	100 Mbps–1 Gbps		100 ms	Low
	4G	100 Mbps–1 Gbps	35 km	10 ms	Low
	5G	>1 Gbps		<1 ms	Medium
WiMAX	IEEE 802.16	1–100 Mbps	30 km	50 ms	High

.

6. Application Areas for Cooperative Control of Microgrids

6.1. Secondary Voltage and Frequency Control

Secondary controls regulate the system voltage and frequency to within prescribed limits according to quality of supply standards, as described in Section 3.1, and may also include such tasks as synchronization of the microgrid following an islanding event. Secondary controls normally use longer time constants than primary controls, which facilitates the decoupling of primary and secondary control design [64]. Normally, distributed designs include an auxiliary input to Equation (5) for secondary frequency control or Equation (7) for voltage control. However, the actual inverter model is nonlinear—for example, see the large-signal microgrid equations developed in [108]—and there are multiple steps involved.

Cooperative secondary voltage control is modeled in [64] as a linearized second-order tracking synchronization problem, based on a sparse nearest neighbor communication network. If each inverter output voltage is given as $y_i$, then the secondary voltage control problem for a cooperative controller is to select a distributed input $u_i$ for each inverter such that $y_i\to v_{ref}$, i.e., all inverter output voltages track the reference value. To linearize the dynamic relationship between input $u_i$ and output $y_i$, input–output feedback linearization is used to derive a linear expression for the output in terms of the input [109]. The distributed secondary voltage control problem then reduces to finding an auxiliary distributed input $v_i$ so that all node output voltages converge to the reference value, i.e., ${\mathit{y}}_i\to {\mathit{y}}_o,\forall i$, where ${\mathit{y}}_o={\left[{\mathrm{V}}_{\mathrm{REF}},0\right]}^T$. The problem is equivalent to the synchronization tracking problem given by Equations (29) and (30); in this case, the exogenous reference is constant. A block diagram of the controller design is shown in Figure 8. The linearizer function blocks require calculation of the Lie derivatives of the state-space model, which requires data from the nonlinear model specification if they are known. An alternative approach is to use neural networks (NN) [110] or reinforcement learning [111] to approximate the dynamic model. The transient response is dependent on the ARE parameters in Equation (38) and may be adjusted through modification of matrices $Q$ and $R$.

Similar to above for voltage, [112] describes a secondary strategy for cooperating control of frequency. In this case, the agent dynamics are derived from the power flow Equation (69), where $v_{oi}$ and ${\delta }_i$ are the inverter output voltage and power angle, respectively, while $v_{bi}$ is the connection bus voltage and $X_{Ci}$ is the connection inductive impedance. For small ${\delta }_i$, $\sin \left({\delta }_i\right)\approx {\delta }_i$ and $h_i$ is assumed constant. By differentiating both sides of Equation (69) and noting ${\delta }_i={\omega }_i$, a simple single-order agent model may be utilized. As above, the problem is then equivalent to the synchronization tracking problem given by Equations (29) and (30). The design for a distributed cooperative control $u_i$ is illustrated in Figure 9, with an auxiliary term added to correct for output power sharing. Using the above assumptions, it can be shown that each inverter output frequency ${\omega }_i$ will converge to and then track an external frequency reference ${\omega }_{ref}$. An inverter is selected as the pinned node, which receives the reference value ${\omega }_{ref}$. For practical implementation purposes, it may be more advantageous to utilize multiple pinned nodes; otherwise, the distributed benefits are negated.

$$P_i=\frac{\left|v_{oi}\right|\left|v_{bi}\right|}{{\mathit{X}}_{ci}}\sin \left({\delta }_i\right)\equiv h_i\sin \left({\delta }_i\right)\approx h_i{\delta }_i$$

(69)

An equivalent protocol that achieves finite time secondary frequency control is shown in Equation (70), with $\mathrm{sgn}(.)$ defined as Equation (50). This results in both faster and more robust synchronizing behavior, as described in D part of Section 4.2.2. Finite-time distributed secondary frequency control for microgrids is described in [61,113].

$$u_i=-c_i\left\{{\displaystyle \sum }_{j\in N_i}{a}_{ij}\mathrm{sgn}\left({\omega }_i-{\omega }_j\right)+g_i\mathrm{sgn}\left({\omega }_i-{\omega }_{ref}\right)+{\displaystyle \sum }_{j\in N_i}{a}_{ij}\mathrm{sgn}\left(m_{Pi}{P}_i-m_{Pj}{P}_j\right)\right\}$$

(70)

Since the above protocols require knowledge of the graph topology in order to calculate the coupling gain $c_i$, as discussed in Section 4.2.2, [56] proposes an adaptive gain that does not require knowledge of ${\lambda }_2$, which is applied to secondary control in [114].

A consensus-based distributed voltage control scheme is described in [15], which does not rely on droops for reactive power sharing, but rather a direct consensus-based voltage reference adjustment. This is an alternative scheme to droops for reactive power sharing. Closed-loop stability can be shown under the assumption that ${\delta }_{ij}\approx 0$ so that $\cos {\delta }_{ij}\approx 1$, and referring to Equation (71), reactive power flows are controlled by only voltage manipulation of $V_i$ and $V_j$. Therefore, Equation (10) is simplified to give Equation (71) below.

$$Q_{iAC}=\left|B_{ii}\right|V_i^2-{\displaystyle \sum }_{j=1}^N{V}_i{V}_j\left|B_{ij}\right|$$

(71)

Using this decoupling assumption, the inverter dynamics are as Equation (73), and the proposed controller is given by Equations (74)–(76), where $V_i^d$ is the desired or nominal voltage reference and $k_i$ the controller gain. The system dynamics can be formulated as a second-order consensus problem with agent dynamics given by Equations (73) and (76). The communication graph $G$ is assumed to be undirected and connected, and exponential stability proven using the spectral properties of the composite system Laplacian.

$$V_i=u_i$$

(72)

$$D_q{\dot{Q}}_i=-Q_i+Q_{AC}$$

(73)

$$u_i=V_i^d-k_i{\int }_0^t{e}_i\left(\tau \right)d\tau$$

(74)

$$e_i\left(\tau \right)={\displaystyle \sum }_{j\in N_i}\left(\frac{Q_i}{Q_{i0}}-\frac{Q_j}{Q_{j0}}\right)$$

(75)

$${\dot{V}}_i=-k_i{\displaystyle \sum }_{j\in N_i}\left(\frac{Q_i}{Q_{i0}}-\frac{Q_j}{Q_{j0}}\right)$$

(76)

This results in a microgrid model equivalent to the LIMAS model described in Section 4.2.2, and consensus and stability can be shown using the approach described for Equations (46)–(48). As an ancillary result from this paper, the conservation law in Equation (77) is proven, which, if $k_i$ are interpreted as weighting coefficients, states that the weighted average trajectory of voltages in the microgrid is constant, with the constant value $\xi$ given by Equation (78). For $k_i=1$, the average trajectory of voltages in Equation (79) for the microgrid is the nominal voltage.

$$\Vert K^{-1}V\left(t\right)\Vert ={\displaystyle \sum }_{i=1}^N\frac{V_i\left(i\right)}{k_i}=\xi \left(V\left(0\right)\right)$$

(77)

$$\xi \left(V\left(0\right)\right)={\displaystyle \sum }_{i=1}^N\frac{V_i^d}{k_i}$$

(78)

$$\overline{V}\left(t\right)=\frac{1}{N}{\displaystyle \sum }_{i=1}^N{V}_i\left(t\right)=V_N$$

(79)

Given that droops are advantageous as a primary strategy for power sharing, however inexact, [115] proposes a scheme for cooperative adjustment of the droop gain. A practical implementation of the consensus-based secondary layer consistent with the Internet of Energy (IoE) framework is described in [116], utilizing TCP/IP and MQTT protocols for information exchange.

6.2. Load Sharing and Network Utilization

As discussed in Section 3.1, output power sharing using conventional frequency droops can result in steady-state errors and poor load sharing accuracy. Several schemes are proposed based on consensus-based control in [22,25]. The objective for the system is to attain group agreement of output power sharing ratios as shown in Equation (80), where $D_i$ is the rated or base output power of each inverter.

$$\underset{t\to \infty }{\lim }\left(\frac{P_i}{D_i}-\frac{P_j}{D_j}\right)=0,\forall i,j\in N$$

(80)

A secondary input $\xi$ is introduced in Equation (81), with the consensus controller as Equation (82), and, for a multi-inverter microgrid, consensus can be demonstrated using the techniques discussed in Section 4.2.2.

$$\frac{d{\delta }_i}{dt}=k_p\left(P_i^0-P_i+\xi \right)$$

(81)

$$\frac{d{\xi }_i}{dt}=-k_i{\displaystyle \sum }_{j\in N_i}{c}_{ij}\left(\frac{P_i}{D_i}-\frac{P_j}{D_j}\right)$$

(82)

A cooperative control scheme to improve voltage profiles and network utilization for distribution networks having many embedded solar photovoltaic generators is addressed using a cooperative control strategy in [117]. This paper proposes a ‘self-organizing’ power flow control strategy such that all PV generators track a specific utilization factor operating on the network. Similarly, secondary power balancing in an islanded microgrid using a sparse nearest neighbor communication network and the consensus problem is addressed in [118], and a ‘pinning-based control’ scheme is used. This allows for uncertain communications and plug-and-play operation, where the sufficient conditions are based on a predefined consensus among pinned agents. The theory is based on cooperation and information flow between autonomous networked MAS [44], and pinning strategies for multi-agent systems [69,119].

An enhancement to the use of consensus-based droop control is described in [54], utilizing ADMM. An overview of ADMM was provided in C part of Section 4.2.2. It has been demonstrated that consensus using ADMM converges faster and is more robust to communication packet errors [55], particularly for large-scale networks. The inverter equations are given as Equations (83) and (84), where $D_{p,i}$ and $D_{q,i}$ are the active and reactive power droop constants.

$$D_{p,i}{\dot{\theta }}_i=P_i^{*}-P_i-p_i$$

(83)

$$D_{q,i}{\dot{V}}_i=Q_i^{*}-Q_i-q_i$$

(84)

The proposed update rules for adjustment of active and reactive power are Equations (85)–(86) and (87)–(88), respectively, where $p_i$ is the active power droop control compensation term and $q_i$ the reactive power droop control compensation term. The auxiliary variable ${\lambda }_i$ is the Lagrangian multipler, while constants $b_{i,j}$, $a_{k,j}$, and $d_i$ depend on the graph structure (Laplacian). The variables $u_{p,i}$ and $u_{q,i}$ are the current values of the Lagrange multipliers for the active and reactive power update steps shown in Equations (85) and (87), respectively. This is based on the dual decomposition theory described in Section 4.1 and C part of Section 4.2.1, and the reader is referred to [120] for the original derivation and convergence proofs.

$${\dot{p}}_i=\left(1-d_i\right)\left(P_i^{*}-P_i-p_i\right)+D_{p,i}{\lambda }_{p,i}+2D_{p,i}{u}_{p,i}$$

(85)

$${\dot{\lambda }}_{p,i}=u_{p,i}={\displaystyle \sum }_{j\in N_i}{b}_{i,j}{\displaystyle \sum }_{k\in N_j}{a}_{k,j}\frac{p_k}{D_{p,k}}-d_i\frac{p_i}{D_{p,i}}$$

(86)

$${\dot{q}}_i=\left(1-d_i\right)\left(Q_i^{*}-Q_i-q_i\right)+D_{q,i}{\lambda }_{q,i}+2D_{q,i}{u}_{q,i}$$

(87)

$${\dot{\lambda }}_{q,i}=u_{q,i}={\displaystyle \sum }_{j\in N_i}{b}_{i,j}{\displaystyle \sum }_{k\in N_j}{a}_{k,j}\frac{q_k}{D_{q,k}}-d_i\frac{q_i}{D_{q,i}}$$

(88)

Additionally, using ADMM, in [53], the topic is optimal flow of reactive power from PV inverters on a distribution network. The problem is cast as a convex optimization of reactive power generation from each inverter, under the constraints of upper and lower voltage limits, and rated capacity. A distributed algorithm is proposed using local cost functions and only nearest neighbor information exchange.

Rather than assuming an ideal DC source, as is often the case for inverter models, a cooperative scheme to synchronize the storage levels in distributed energy storage systems (ESS) is found in [121], where the group objective is for a consensus of energy storage levels at each system. In cases where several microgrids exist in clusters, or hybrid architectures consisting of multiple AC or DC sub-nets, a further issue regarding load sharing between sub-systems arises [122,123].

6.3. Remedial Action Schemes

This topic includes the use of agent-based information discovery for fault location, isolation, and supply restoration (FLISR). An agent-based protection scheme based around IEC 61850 GOOSE is described in [124]. This particularly robust approach uses agent-based distributed communication for adjustment of critical overcurrent and frequency selectivity criteria and is demonstrated on an experimental setup that includes pulsed loads. IEDs communicate using agent-based communication to determine which circuit breakers should clear the fault. GOOSE peer-to-peer message traffic includes interlocking signals and mode and setting control. On transition to islanded operation after opening the grid-CB, operation of pulsed loads could result in voltage and frequency deviations that could potentially disrupt traditional frequency protection methods. This system uses agent-based communication and inertia estimation to implement adaptive blocking during pulsed load operating periods.

The multi-agent cooperative load restoration algorithm described in [125] is a frequently cited example of a novel consensus-based remedial action scheme, and, therefore, it is worth including a more extensive summary. The problem addressed concerns the selection of a feasible set of load buses for restoration, immediately following a fault and isolation event. Each bus in the electrical system represents a node, with an associated agent (node agent or NA). Connected to each node is either a load or generator, and each load is pre-allocated an associated load restoration priority value. The example demonstrates that it is possible to perform this action using only data from ad-hoc neighbors of each node. The algorithm is based on the application of the average consensus algorithm described in Section 4.2.1, for information discovery between nodes on the network for the purpose of ascertaining which loads should be restored.

Each node agent (NA) is assigned a $n\times 3$ matrix $M$, where $n$ is the number of agents, and for each agent $i,$ the column entries are initialized as follows: $M_{i,1}$ is either $i$ or $0,$ representing a normal functioning or faulty load or generator; $M_{i,2}$ is either $i$ or $0,$ representing whether an offline load is ready for restoration or not, and $M_{i,3}$ is the available capacity of the load or generator. All other rows are initialized to $0$.

$$M_i=\left[\begin{array}{ccc}{M}_{i,1}& M_{i,2}& M_{i,3}\\ ⋮& ⋮& ⋮\\ M_{n,1}& M_{n,2}& M_{n,3}\end{array}\right]$$

(89)

After a fault on the network, the discrete-time consensus algorithm in Equation (21) is applied to each element of the matrices $M_i$ of each agent. Each entry will converge to the average of all corresponding entries in the initial matrices, as shown in Equation (90). Therefore, all information matrices will converge to the same identical matrix $M$. The paper demonstrates that the converged or equilibrium values of each entry allow for sufficient information for load restoration and load shedding.

$${\mathit{X}}_{eq}=\frac{{\sum }_{i=1}^n{x}_i^0}{n}{\left(1,\dots ,1\dots ,1\right)}^T$$

(90)

The positions (indexes) of zero/non-zero entries in the matrix allow agents to determine which loads or generators are out-of-service and which are ready for restoration. The sum of the third column gives the average total net available power. The decision on which loads to restore is then based a dynamic linear programming (LP) algorithm by each node using identical information. Since all nodes share the same information and use the same rules, coordinated load restoration can be initiated and the microgrid recovered. The case study presented uses a 16-bus microgrid, with generation and load values as shown in

Table 3. Generation, load, and load restoration priority for the example in [125].

No.	${\mathit{P}}_{\mathit{G}\mathit{i}}$	${\mathit{P}}_{\mathit{L}\mathit{i}}$	${\mathit{X}}_{\mathit{i}}^0$	Priority
1	40	0	40	-
2	40	0	40	-
3	50	0	50	-
4	0	20	−25	P2
5	0	35	−35	P2
6	0	20	−20	P1
7	0	15	−15	P2
8	0	25	−25	P1
9	0	5	−5	P1
10	0	30	−30	P2
11	40	0	40	-
12	0	45	−45	P2
13	0	10	−10	P2
14	0	40	−40	P2
15	30	0	30	-
16	50	0	50	-

, and with the communication graph shown in Figure 10. In Table 3, ${\mathit{X}}_i^0$ is the net power exported at the bus, i.e., ${\mathit{P}}_{\mathit{G}\mathit{i}}-{\mathit{P}}_{\mathit{L}\mathit{i}}$, while P1 is high priority and P2 is low priority for restoration. Establishing appropriate values for the edge weights of the communication graph impacts the algorithm’s speed of convergence, as outlined in Section 4.2.1. For this example, the coefficients are adjusted automatically to achieve optimal performance, also known as adaptive coefficient setting. The method for updating the coefficients is the mean metropolis method.

The sequence of events for the test case is as follows: (i) a short-circuit fault occurs at node 11, (ii) protection operates immediately, switches isolate and sectionalize a subset of nodes, and (iii) the load restoration algorithm described above is initiated, and nodes in the solution set restore their connected loads. The value of the final converged matrix $M$ is shown in Equation (93). The load restoration decision at each load uses the LP optimization in Equations (91) and (92), where $P_{Li}$ and $P_{Gi}$ are the load and generator powers, respectively, and ${\rho }_i$ is a priority weighting factor, in this case set to 1 for low priority (P2) and 10 for high priority (P1). The constraint ensures that there is always sufficient generation to meet the aggregate load. A functional block diagram is shown in Figure 11.

$$\max {\displaystyle \sum }_{i=1}^N{\rho }_i{P}_{Li}$$

(91)

$${\displaystyle \sum }_{i=1}^N{P}_{Gi}-{\displaystyle \sum }_{i=1}^N{P}_{Li}\ge 0$$

(92)

The solution convergence time is not provided; however, the authors estimate that, for a 162-bus system with a network speed of 10 Mb/s, a realistic convergence time is 0.1283 s [94]. From a practical perspective, the convergence time to a consensus result is a critical factor. Similar restoration schemes have been applied to shipboard power systems [97].

$$M_{FINAL}=\left[\begin{array}{ccc}1/16& 0& 40/16\\ 2/16& 0& 40/16\\ 3/16& 0& 50/16\\ 0& 4/16& -25/16\\ 5/16& 0& -35/16\\ 0& 6/16& -20/16\\ 0& 7/16& -15/16\\ 0& 8/16& -25/16\\ 0& 9/16& -5/16\\ 0& 10/16& -30/16\\ 0& 0& 0\\ 0& 12/16& -45/16\\ 13/16& 0& -10/16\\ 14/16& 0& -40/16\\ 15/16& 0& 30/16\\ 16/16& 0& 50/16\end{array}\right]$$

(93)

6.4. Economic Dispatch and Scheduling

Distributed algorithms for agent-based economic dispatch and scheduling problems in microgrids are divided into the following general categories: (i) conventional least-cost dispatch based on aggregate generation and demand, and utilizing a dual decomposition method of Lagrange multipliers, and solution by primal-dual gradient descent using distributed exchange of multipliers between nodes; (ii) game-theoretic dispatch using local coupling of costs, local or global coupling of constraints, and a distributed solution to the Karush–Kuhn–Tucker (KKT) conditions, or (iii) consensus-based economic dispatch using distributed averaging. The convex optimization solver utilized in each case for the hard computation work includes various new techniques, as described below.

For economic dispatch of a community microgrid using heterogenous wireless networks, specifically both Wi-Fi and cellular, [50] has developed a distributed algorithm based on distributed averaging, as described in Section 4.2.1. Nodes are categorized as generation or load. Surprisingly, the authors prove that agents need only knowledge of averaged values across the network in order to solve the constrained least-cost optimization problem. The dispatch problem then reduces to an information discovery problem, where, over several iterations—the information exchange period—all nodes reach a consensus on the average values of the cumulative capacity and aggregate demand, and, based on the author’s theorem, can each dispatch the least-cost level of generation during the dispatch period. The scheme utilizes a combination of short-range ad-hoc wireless connections (Wi-Fi), with the cellular network available to a sub-set of “dual-mode” nodes, to increase the solution convergence speed, based on deterministic rather than ad-hoc connections. Dual mode nodes can be utilized that use both ad-hoc short-range and cellular backbone networks during the information update process. The use of network topology information by a few nodes for deterministic link scheduling can increase the solution convergence speed, as described in [7,49].

Since economic dispatch of a microgrid is a constrained resource optimization problem, it is amenable to solution using decomposition–coordination techniques that utilize distributed message passing, such a distributed gradient or sub-gradient descent, and dual decomposition. Several other distributed algorithms have been proposed for the economic dispatch problem in microgrids [126,127], otherwise known as optimal load sharing control. Often, there are subtle but important differences in the formulation of the dispatch problem, relating to the nature of constraints, whether constraints are local or global, and whether knowledge of cost functions is local (private) or global. Based on the algorithm developed in [128] for the optimal power control of wireless power transfer networks, [129] addresses the particular case of optimal power dispatch in a hybrid microgrid consisting of DC and AC sub-grids, and with a DC/AC converter-based controllable interconnection between sub-grids. Each sub-grid includes an energy storage unit and multiple generation units and loads. Storage units are equipped with wireless charge/discharge sensors. Using Lagrangian decomposition, a constrained least-cost dispatch problem is formulated. The authors of [129] show that the optimal power flow problem can be solved with only information from the two sensors. A hybrid game-theoretic method for optimizing storage capacity using multi-agent guiding particle swarm optimization (MAG-PSO) is described in [130], while in [131], a peer-to-peer MAS-based EMS uses a T-cell-inspired algorithm to optimize the operation of a microgrid consisting of RE sources, storage, and controllable loads. Further applications of MAS and game theory to scheduling and dispatch are found in [132,133,134].

Regarding other applications, voltage regulation in distribution network areas having high-photovoltaic penetration is increasingly an issue for utilities. Voltage regulation in distribution networks is normally solved using optimal power flow (OPF) and voltage constraints. Recently, several authors have proposed distributed techniques using agent-based coordination, as described in [135,136,137]. The algorithm in [135] is particularly suited to time-varying communication networks, while the scheme in [136] proposes a three-level coordination of (i) voltage ramp-rate, (ii) voltage deviations from nominal, (iii) and voltage exceedance of network limits. A more recent problem associated with distributed open-access energy markets, and the concept of an Internet of Energy (EI), is that of energy routing. Examples of energy routing protocols based on graph theory, agent-based consensus, and game theory can be found in [138].

7. Recent Developments and Open Problems

This section includes a few remarks regarding future directions for this research. While several applications for cooperative microgrid control have been described in the paper, it is certainly the case that the full range of operating issues experienced by microgrids is far wider than this, and is still to be explored. Power quality problems regarding the management of harmonics and unbalance problems are two examples [139,140]. It may be concluded from a review of the applications that certain aspects of microgrid operation still require centralized supervisory control.

Since islanded microgrids without sufficient energy storage or inertia require near instantaneous power balancing, future research into cooperative control must consider the critical time constraints in the context of the microgrid technical characteristics. Automation frameworks, interoperability, and security are also important areas that are related. Performance-oriented cross-layer optimization will require that end-to-end time delays are dynamically controlled with respect to both utilization of channel bandwidth and system performance indices. Autonomous operation requires seamless integration of the physical and cyber layers, and distributed intelligent agents embedded with all the multitude of functions required to manage a modern power system. Reinforcement learning for cooperative controller synthesis is a more recent innovation, with application to microgrids described in [111,141,142].

Several proposed smartgrid architectures are conceptually a vertical hierarchy [143]. However, one of the benefits of cooperative control is that using direct peer-to-peer transactions avoids the overhead of having to transfer messages up and down the layers for processing. It is efficient and fast because of its horizonal structure, and this enables autonomous operation. In this regard, whether proposed smartgrid architectures realize this practical advantage is an open question.

The next generation of 5G and IEEE 802.11 wireless will encourage the adoption of cooperative control methods for microgrids [5]. In particular, the evolution of M2M ultra-low latency wireless is enabled by innovations such as mmWave (24 GHz to 40 GHz) for small-cell networks, Network Functions Virtualization (NFV) using Software-Defined Networking (SDN), cognitive radio, and the Industrial Internet of Things (IIOT) [144]. The ultimate quality of end-to-end service depends on advances at all levels of the hierarchy, including medium-access control, routing, and application layers [82]. In conjunction with deep learning and AI techniques, this research will support the evolution of new cooperative agent-based methods for microgrid control.

8. Conclusions

The paper has reviewed the current research on cooperative control of microgrids, which is an area of increasing interest in the context of recent developments in wireless communication, which is enabling agent-based automation in a practical sense. The paper identifies and explains the theoretical frameworks for cooperation control, including a range of example applications for microgrids, which serve to illustrate the limitations and scope of the theory. In this way, the research study is a step towards a generalized framework for cooperative control in microgrids, which can inform the discussion of future architectures for microgrids. Several novel applications are described, while recent developments are discussed, and research directions for cooperative microgrid control are suggested. The paper also emphasizes the notion of the microgrid as a cooperative cyber–physical system that requires a reinterpretation of the conventional microgrid control model and performance-oriented design.