A Survey of Multi-Agent Systems for Smartgrids

Abstract

This paper provides a survey of the literature on the application of Multi-agent Systems (MAS) technology for Smartgrids. Smartgrids represent the next generation electric network, as communities are developing self-sufficient and environmentally friendly energy production. As a cyber-physical system, the development of the vision of Smartgrids requires the resolution of major technical problems; this has fed over a decade of research. Due to the stochastic, intermittent nature of renewable energy resources and the heterogeneity of the agents involved in a Smartgrid, demand and supply management, energy trade and control of grid elements constitute great challenges for stable operation. In addition, in order to offer resilience against faults and attacks, Smartgrids should also have restoration, self-recovery and security capabilities. Multi-agent systems (MAS) technology has been a popular approach to deal with these challenges in Smartgrids, due to their ability to support reasoning in a distributed context. This survey reviews the literature concerning the use of MAS models in each of the relevant research areas related to Smartgrids. The survey explores how researchers have utilized agent-based tools and methods to solve the main problems of Smartgrids. The survey also discusses the challenges in the advancement of Smartgrid technology and identifies the open problems for research from the view of multi-agent systems.

1. Introduction

One of the major paradigms in the energy sector in the 21st century will be the utilization of renewable, carbon-free energy resources for an efficient and sustainable society [1]. Climate change and the significant use of fossil resources have become significant concerns of contemporary societies. Population growth and industrialization have increased the demand for energy from households, businesses and factories. These factors bring the need for greater production of electricity in power plants which causes pollution, toxic waste and over consumption of natural resources.

Governments have started enacting laws to limit greenhouse gas (GHG) emissions from factories and thermic electric plants, in order to protect the environment for sustainable development. Furthermore, many countries are planning to close nuclear power plants and urge entrepreneurs to invest in renewable energy resources. Climate change, pollution and the scarcity of fossil fuels have increased the demand for renewable energy sources (RESs). These alternative energy resources include wind, solar, hydro, biomass and agricultural and industrial waste.

Several international protocols and agreements have been announced for the preservation of nature and reducing pollution. Key events and milestones are the enforcement of the United Nations Framework Convention on Climate Change in 1994 [2], the Kyoto Protocol in 2005 [3], the enforcement of the Paris Agreement in 2016 [4] and Directive 2009/28/EC [5]. These commitments have resulted in policies for decarbonizing the electricity sector and providing incentives towards the use of renewable resources and clean technologies to produce energy. Examples of such policies are the Energy Policy Act of 2005 [6,7] and the American Recovery and Reinvestment Act of 2009 [8,9] in the U.S., the Renewable Energy Law of 2005 [10] and the Golden Sun program [11] in China and the Renewable Energy Directive [12] and Emissions Trading Scheme in the EU [13].

Conventional centralized electricity systems consist of a small number of large power plants, high voltage transmission lines, transformers and local distribution networks. In this architecture, power flow is unidirectional (from plants to consumers) and there is no communication or feedback mechanism between producers and consumers [14]. The lack of predictability of energy demand requires plants to overproduce electricity for safety margins. In addition, considering the foregone heat during electric production and the losses over long transmission lines, the centralized electric production system has been recognized as inefficient and wasteful.

Aside from environmental concerns, another main factor that drives change in conventional electric grids is efficiency and scalability. As populations and cities grow, the demand for electric energy rises and consequently the load on central power plants, network and transmission lines increases. In transportation services, electric and hybrid vehicles are becoming more prevalent due to their efficiency. Existing plants, infrastructure and transformers have bounded capacity; thus, growing the grid constitutes a great challenge in the sense that this would incur a major revision to the infrastructure and significant cost. Moreover, as mentioned above, transmitting electricity over long distances requires costly infrastructure and causes losses; thus, a logical solution is to install local, decentralized energy generators close to the end users.

Along with these motivations, the electricity industry is experiencing a radical transformation from a few, centralized, large-scale thermic and nuclear plants to many, decentralized, small-scale clean producers. Thanks to the development of alternative and affordable energy sources, such as photovoltaic solar panels, wind turbines and natural gas microturbines, some households or businesses are currently installing new equipment to produce and store electricity to satisfy in part or in full their needs and even sell excess electricity to other consumers in the grid. These small and renewable generators have lower emissions and lower operational and maintenance costs. In this respect, a Smartgrid seems to be a promising direction to fulfill the increasing energy demand of the society and at the same time preserve the environment and decrease pollution. The term “Smartgrid” refers to a distributed system of electric energy generation with many intelligent producers and consumers who can communicate, reason and decide.

This radical change in the electricity industry also brings some new challenges for management, control, scheduling and security. Power obtained from solar panels and wind turbines depends on weather conditions; thus, the amount of generated energy varies both during the day and across days. In order to deal with the uncertainty, energy consumption of electric devices can be shifted or rescheduled dynamically over time. Hence, in a distributed grid with many producers and consumers, coordination, allocation and exchange of energy becomes an important challenge. Aside from these engineering aspects, there are also economic issues such as energy trade and allocation in the Smartgrid.

In order to manage the operation of a cyber-physical system such as a Smartgrid, we need software tools and computational infrastructure to handle communication and various functions of the grid elements. Multi-agent systems (MAS) seem to be a suitable framework for management and control of a highly distributed, dynamic grid of hard and soft components. A multi-agent system is a group of autonomous, independent entities (agents) acting in an environment to achieve their own objectives or group objectives. These agents can be software or a cyber-physical object as in a Smartgrid. MAS is a paradigm for modeling autonomous and intelligent agents who can perceive and perform actions in a dynamic environment. Agents are intelligent in the sense that they reason about the state, optimize their benefit, make rational decisions and learn from past experiences. In this respect, the MAS framework can be used to model a network of independent agents in the Smartgrid. The contemporary developments toward the future Smartgrid may require integration of millions of devices such as Distributed Energy Resources, loads, storage elements and sensors. The control systems will have to operate efficiently on a large scale; they should also be robust to faults and attacks on the system. The complexity of the control system can be highly reduced by distributing tasks among cooperative and communicating agents. Agents can operate autonomously and they cooperate or compete with each other depending on the context. This feature of MAS may play an important role in the management of a sophisticated Smartgrid with a divide-and-conquer method: the grid can be partitioned into several microgrids and intermediate layers (station, building, home) can also be added. Compared to the control methodologies of conventional power systems, this would be a bottom-up approach instead, where decisions are taken locally in a decentralized manner.

Alternatives to MAS exist, such as SCADA (supervisory control and data acquisition), expert systems and neural networks. However, these methods are either not suited or not scalable for a large electric grid. SCADA has a top to bottom structure and it is used for central management of the grid. MAS technology has been used by researchers in electrical engineering, computer science and economics to address various issues in Smartgrids.

In this paper, we survey the literature about applications of MASs for Smartgrids. We first give some definitions and explain the concept of MASs. We discuss the relevance and benefits of MASs for Smartgrids and distributed energy management systems. Then, we go through the literature on the following topics:

• MAS platforms and tools for energy management;
• Standards and protocols;
• Ontology for energy domains;
• Energy markets and trade;
• Control and management;
• Demand and supply management;
• Restoration and self-recovery;
• Protection and security;
• Simulation and implementation.

In each section, we report the main problems, challenges in that topic and how researchers have developed tools to solve these problems. At the end of the section, we discuss the remaining open problems. We have also created a separate section where we provide a meta-level analysis and explain the general challenges.

There are existing survey papers [15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31] for the application of MAS techniques to Smartgrid and energy systems. However, these surveys focus on one or several topics in the list above.

In

Table 1. Previous survey papers in the literature.

Survey Paper	Reviewed Topics
Bayram et al. (2014) [15]	Energy trade, EV (dis)charging, Market simulation
Coelho et al. (2017) [16]	Microgrid control, Energy storage units and EV charging, Demand management, Restoration, Security, Implementation
Gómez-Sanz et al. (2014) [17]	Energy trade, Control, Simulation
Kantamneni et al. (2015) [18]	MAS platforms, Energy trade, Control, Restoration
Kiran et al. (2017) [19]	MAS platforms, Energy trade, Energy market simulation
Kulasekera et al. (2011) [20]	Energy trade, Control, Restoration
Mahela et al. (2020) [21]	Smartgrid standards, Control, Building energy management, EV charging
McArthur et al. (2007) [22]	Protection, Simulation, Implementation, Technical Challenges
McArthur et al. (2007) [23]	MAS design methodologies, Standards, Ontologies
Halhoul Merabet et al. (2014) [24]	MAS platforms, Control, Implementation
Sukumaran Nair et al. (2018) [25]	Supply management (economic dispatch, unit commitment), Consensus algorithms
Vithanage et al. (2019) [26]	Control
Roche et al. (2010) [27]	MAS platforms, MAS design methodologies, Energy trade, Demand management, Simulation, Implementation, Future scope
Roche et al. (2013) [28]	MAS organization and design methodologies, Standards, Ontologies, Energy trade, Voltage control, Restoration, Future scope
Rohbogner et al. (2013) [29]	Energy trade, Microgrid control, Voltage and frequency stabilization
Hasanuzzaman Shawon et al. (2019) [30]	Energy trade, Control, Restoration, Security
Sujil et al. (2016) [31]	Energy trade, Control, Supply management, EV charging, Restoration

, we illustrate the differences in these surveys and the present survey in terms of the topics that they review. In particular, the previous surveys do not mention the role of knowledge reasoning and planning for future research. Furthermore these survey papers do not state specific challenges, alternative methods and problems in each field which are not yet studied or investigated in sufficient detail. One of the distinguishing features of this survey is that we review the relevant work in each topic in detail, state challenges and open problems and provide an upper picture of the state of the art, all from the perspective of MASs.

For the literature search, we first identified the main problems and research topics about Smartgrids on the list. In this process, we also benefited from policy papers and other survey papers cited above. Then, for each of these topics, we searched for the papers on Google Scholar. We included journal articles, conference articles, official policy papers and standardization documents about MAS applications on Smartgrids, smart buildings, smart homes and electric vehicles. Among them, we selected those papers which have seminal contributions or propose original solutions to the problem related to the topic. We excluded papers that have non-MAS solutions, study a subproblem or a specific problem, or study conventional power systems.

2. Smartgrid: Basic Concepts

Smartgrid is regarded as the new generation electricity system, which integrates Information Technology, distributed computation and Artificial Intelligence tools for efficient energy generation and delivery with renewable resources [21]. In addition to the use of renewable energy resources, other advantages of Smartgrid are accommodation of energy storage (batteries), monitoring and estimation of users’ consumption and adaptive configuration of the grid, e.g., in response to unexpected events [15].

A Smartgrid is a cyber-physical system, meaning that it has physical infrastructure (electrical and mechanical elements like lines, transformers, relays, etc.) and computational components (communication, software, applications, protocols, etc.) [32]. It has the capabilities of communication and real-time data collection and processing. A Smartgrid enables a two-way flow of electricity and information to create an automated, widely distributed energy network [33]. These capabilities of Smartgrids provide the benefits of monitoring grid elements, dynamic energy pricing, analysis and scheduling of energy usage, efficient allocation of energy in a wide network, resistance to cyber-attacks and auto-recovery [26].

A Smartgrid is responsible for managing electricity resources and loads. The Distributed Energy Resources (DER) in the Smartgrid are photo-voltaic panels, wind turbines and diesel generators and the storage devices are energy capacitors, batteries [34]. Critical loads are those higher-priority devices such as heaters, refrigerators, lights, security and fire alarm systems that require delivery of electricity immediately for the welfare of the household or community [35,36]. On the other hand, the energy demand of non-critical loads can be postponed when necessary, e.g., as is the case of laundry, dishwasher machines and electric vehicles (EVs).

A Smartgrid (see Figure 1) is an interconnected network of power plants, Virtual Power Plants, microgrids, sensors, meters, transmission elements and auxiliary components (stations, transformers, cables) [21]. It typically includes a main grid (also called utility/upstream/AC grid) and a multitude of smaller local microgrids and Virtual Power Plants (VPPs), that are connected to the main grid [37] (Figure 2). The main grid is the primary source of energy; it is powered by large-scale plants (thermic, hydroelectric, nuclear) and supplies electricity to the microgrids and VPPs in case their local electricity generation is insufficient. It is also possible that microgrids and VPPs transfer (e.g., by selling it) their excess energy to the main grid. The microgrids and Virtual Power Plants are connected to the main grid through the Point of Common Coupling (PCC) where the energy transfer occurs.

A microgrid consists of distributed small-scale renewable and non-renewable energy resources, storage devices and critical and non-critical loads [38,39]. A typical example of a microgrid is a geographically isolated area such as a campus or holiday resort which produces its own energy. The microgrid operates either in the connected mode or in the islanded mode. In the connected mode, the microgrid can transfer electricity from the main grid whereas in the islanded mode, the microgrid can only use its own resources. The islanded mode can occur as a result of a fault or outage at the main grid; yet there are also self-sufficient microgrids that always operate intentionally in the island mode.

Households in the Smartgrid are often called prosumers, which means they are both producers and consumers of electricity [40]. Prosumers exchange information and trade energy with each other for optimal utilization and sharing of available energy. A Virtual Power Plant (VPP) is a cluster of distributed energy generators which sell electricity in the market and compete with large-scale power plants [41]. The intuition behind VPPs is that a single household or distributed generator can only offer a negligible amount of electricity in the market and thus cannot be a major player or supplier. A VPP is a solution to this problem: it is a virtual coalition of small distributed resources and is seen as a single entity (power plant) in the energy market. VPP aggregates energy and information of its generators and sets its own production schedule, quota and price. In addition to energy generators, a Virtual Power Plant involves consumers, loads, storage elements, smart homes, smart buildings, electric vehicles and auxiliary components. The concept of VPP is similar to a microgrid, except that a microgrid is defined based on the spatial proximity of its grid elements and it is usually located in a well-defined geographical area. On the other hand, a VPP can be instituted at a wide scale (or at the desired scale) without a geographical limit. A VPP also has a more advanced management of energy generation, load profile and operation cost in order to obtain revenue in the market.

3. Multi-Agent System Concepts and Definitions

This section provides some preliminary definitions concerning the concepts of agent, MAS technology, multi-agent organizational paradigms, MAS platforms and software, standards and ontologies for the energy domain.

3.1. Definition of an Agent

In the literature, there are multiple definitions for the concept of an agent resulting from diverse applications and domain-specific features of MASs. One of these early definitions was proposed by Wooldridge [42], “an agent is a computer system that is situated in some environment and that is capable of autonomous action in this environment in order to meet its design objectives”. According to Russel and Norvig [43], an agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through effectors. Later, in the 2003 edition of their book [44], Russel and Norvig define an intelligent agent as an autonomous entity which has the following properties:

• It has the ability to communicate and interact with its environment;
• It is able to perceive the (local) environment;
• It is guided by basic objectives;
• It has feedback behavior.

Ref. [45] defines an agent as an entity which is placed in an environment and senses different parameters that are used to make a decision based on the goal of the entity. Ferber [46] proposes another definition: “An agent can be a physical or virtual entity that can act, perceive its environment (in a partial way) and communicate with others, is autonomous and has skills to achieve its goals and tendencies”.

Power engineering and the Smartgrid community have mostly adopted the definition of agents in Wooldridge and Jennings [47] as it fits the role of the agent in this context. An agent denotes a hardware- or software-based system which has the following properties:

• Autonomous: Agents exert partial control of their actions and internal state, seeking to influence outcomes without the intervention of humans or external devices.
• Social: Agents can communicate and negotiate with humans, external devices or other agents to coordinate actions and satisfy their objectives.
• Reactive: Agents react in a timely fashion to changes in their environment.
• Proactive: Agents exhibit goal-oriented behaviors and take initiative to satisfy objectives.

Ref. [48] lists eight properties of agents: location, mobility, autonomy, perception, communication, adaptation, reactivity, pro-activeness, rationality, socialness. A common methodology to implement agents is the Belief–Desire–Intention (BDI) model: Beliefs reflect the information of the agent about the environment. Desire represents the needs and objectives of the agent. Intention represents the actions and strategy of the agent to achieve its goals [49,50]. An overview of agent definitions and discussion can be found in [51,52].

3.2. Anatomy of Agents

The internal architecture (or anatomy) of an agent defines its behavior, i.e., the actions of the agent as a function of its perceptions and the changes in the environment [53]. This function can be simple, as in reactive agents, or very complex, as in cognitive agents. There are different types of agents, classified according to their level of autonomy, the way perception is achieved, and how they can be modeled [28]:

• Reflexive agents perform simple actions based on their perceptions; their behavior is based on an action-selection module that receives percepts from the environment and consults a database of condition–action rules, similar to if–then rules, to make a decision. Such agents can be useful when fast response times are needed, e.g., for protection. Their representation of the world (the environment) is minimal, but they may support emergent behaviors. Emergence occurs when new characteristics appear at a certain level of complexity.
• Goal-based agents are directed by goals set a priori by the user. They have an internal representation of their environment and can memorize previous percepts to make more elaborate decisions. More precisely, once a percept is received by the agent, the memory manager stores the information in the percept memory. A sequence of percepts is then built in order to be used subsequently by the action-selection module to select suitable actions in order to reach given goals.
• Utility-based agents use a performance-measurement index, referred to as the utility function, in order to evaluate their behavior. A utility-based agent chooses an action that optimizes its utility or achieves a certain satisfactory level of utility. Such an agent is rational and behaves efficiently, given its prior knowledge of the environment.
• Learning agents belong to one of the previous classes and in addition can learn to perform a given task more effectively. They are able to modify the function that codes their behavior while interacting with their environment, to be more precise in performing a given task. An agent running a load-forecasting algorithm would be a typical example. The learning can be supervised, unsupervised or reinforcement learning.

3.3. Multi-Agent Systems and Smartgrid

A multi-agent system (MAS) consists of a collection of agents that interact with each other and the environment and perform actions to satisfy their objectives [45,54]. An MAS supports a distributed computing paradigm: the idea of MAS is to decompose a complex problem into several simpler sub-problems and use a separate agent to deal with each sub-problem.

An MAS can have a different organizational structure (see Figure 3). An organizational structure specifies the connection and logical position of agents, their roles and privileges, information flow and coordination patterns [28]. There exist a variety of MAS organizational paradigms, such as hierarchy, holarchy, matrix, coalition, team, holon, congregation, society, federation and their compounds. Each organization has its own topology among agents. For instance, a hierarchy has a tree-like top to bottom structure; a coalition, team, holon maintain a cluster of agents; a matrix has multiple managers or peers. A description of these organizational paradigms can be found in [28,54,55].

As we will explain in subsequent sections, an MAS-based management system for a Smartgrid may involve a variety of agents operating within an organizational architecture. Some of these agents are associated with a device or component (e.g., wind turbine agent, battery agent, EV agent, bus agent) and some agents are associated with a specific task (e.g., auctioneer agent, control agent, energy-forecasting agent, validation agent). The MAS organizational structure describes the region and grouping of agents in Smartgrids and the communication topology between them. The MAS structures that are more commonly employed in Smartgrid applications are centralized, decentralized, hierarchical and hybrid [34]. In a centralized architecture, a collection of simple, uniform agents is managed by a single master controller agent, operating in a master–slave relationship. Namely, all primary tasks and functions of the Smartgrid are handled by the controller agent. In this sense, a centralized MAS structure resembles a traditional SCADA-type supervisory system. In contrast, in a decentralized structure, there is no hierarchy or master–slave relationship between any agent in the MAS system. Agents operate and decide in a fully independent manner. Typically, agents communicate and negotiate in a bilateral (peer-to-peer) fashion.

The centralized management framework for Smartgrid causes high data traffic load at the level of the controller, and hence requires a powerful data center and wide bandwidth for processing a large volume of data in a short period of time [56,57]. Another disadvantage is that it is susceptible to “Single Point of Failure”; in other words, a fault or an attack to the controller agent will impair the operation of the entire system [58,59,60]. Furthermore, the grid will be at risk of cascading failures [61,62,63]. In addition, a centralized model is not suitable for a Smartgrid which has a time-variant structure.

The distributed approach to MAS Smartgrid management is more reliable compared to the centralized approach [64]. It is more robust in terms of operation, control and monitoring. In addition, it is simpler and more cost-effective in terms of implementation [65]. An MAS is an effective way of distributed control of microgrids as an alternative to traditional hardware-based centralized control. The key advantages of MASs for Smartgrid applications are highlighted below [16,22,25,27].

• Distributed Nature: The entire Smartgrid can be divided into microgrids and VPPs. Moreover, intermediary layers and lower-level grid elements can be added in between. An MAS includes many autonomous agents computing and operating in a parallel and asynchronous manner. The decentralized structure of MAS and autonomous agents makes the control of Smartgrids easier.
• Flexibility: MAS supports the plug-and-play capability of microgrids, Distributed Energy Resources, storage elements and other equipment and can adjust the control of the grid accordingly. Agents have self-adaptive behavior to the Smartgrid environment and act to accomplish their goals.
• Fault tolerance: If one agent fails, the rest of the system can remain active and can adapt to the new state by its rules and behaviors. Thus, MAS-based Smartgrids can be more resilient to disturbances and faults.
• Responsiveness: As agents sense changes in real time, collect the relevant information and communicate with each other, MASs can quickly respond to the events in the environment.
• Scalability: The complexity of the energy-generation and -distribution system can be highly reduced by dividing it into layers, units and agents. Each autonomous agent is responsible for a component of the Smartgrid and agents are modular. Thus, the overall system can be expandable by simply adding new agents.
• Local knowledge: Each agent only needs information from its local environment and communicates with its neighbors for its own decision-making. Thus, the required data and communication in MAS are more controlled and limited compared to a centralized control system. This feature of MAS is especially beneficial for management of a large system like a Smartgrid.

Some scholars have a critical view of whether the agent-based system is the right technology for a Smartgrid. Ref. [17] argues that MASs may not be the right or optimal choice for every problem or domain and discusses the incorrect use of agents. In particular, the authors state that in some MAS models for Smartgrids, there are either too few (just one agent) or too many agents, agents do not have decision capabilities, there are very few choices for the agents, agent-to-agent relationships are client–server rather than peer-to-peer and decentralization is achieved through distribution which are two different concepts.

Between the two extremes, fully centralized and fully decentralized, researchers have also adopted hierarchical and hybrid MAS architectures for Smartgrid applications. In a hierarchical MAS, there are multiple levels (also called layers) from top to bottom. An agent at a given layer has control over agents at the lower layer. In a hierarchical architecture, agents communicate with those agents in the upper layer and the lower layer. This type of organizational structure emerges in Smartgrids as consecutive layers, e.g., device/appliance, house, building, station and microgrid in an ascending fashion. There is often a dedicated agent responsible for a layer such as a microgrid control agent, zone agent, transformer agent and home energy management agent. Note that the decision making across adjacent levels in a hierarchy does not always have to be in a master–slave relation. Each agent can have some autonomy and fulfill the relevant assigned task. As an example, each layer aggregates information from the lower layer and performs some optimizations before communicating to the higher layer.

Some agent-based designs for Smartgrids are hybrid (or mixed). For instance, the grid might be partitioned into different zones and each zone can have a different structure (centralized, decentralized or hierarchical). Another possible situation might be that the MAS has multiple layers and the relationship inside a layer might be different from others. Namely, sibling agents at a layer can be organized in a centralized manner, whereas at another layer they can be decentralized or they can form holons. Examples of such organizational paradigms will be provided in the subsequent sections.

To achieve proper operation of Smartgrids, agents in the system perform their assigned tasks, functions and optimization processes, e.g., opening/closing circuit, determining and evaluating bids, fault and anomaly detection, charge scheduling and restoration [66]. Note that the choice of decision rule or optimization method depends on the MAS structure and the type of application. For example, a consensus type algorithm (which uses information exchange between neighbors) is more suitable for a decentralized system whereas combined heat and energy optimization of a home is more suitable in a centralized system.

In MAS-based Smartgrid models, researchers have used the following control methods and decision functions: rule-based, fuzzy or probabilistic logic, expert systems, decision models, data-driven control, (non)linear control, model predictive control, consensus algorithms, convergence algorithms, game theory, historical data analysis, statistical methods (logit, probit), statistical filtering, neural networks, machine learning, human immune system and heuristic algorithms. The optimization methods used by the researchers are reinforcement learning, dynamic programming, mathematical programming, Lagrangian methods, gradient algorithms, swarm intelligence, evolutionary and genetic algorithms, graph and tree search, exhaustive search, local search and (meta)heuristic optimization algorithms. Refs. [34,67,68] give a description of these decision functions and optimization methods. The choice of the relevant method depends on the nature of the problem and the context.

3.4. MAS Design Methodologies

In the literature, several MAS design methodologies have been presented [69,70,71], yet the basic steps in these design methodologies are the same. These methodologies essentially employ software-engineering and knowledge-engineering approaches for the specification and design of agent-based systems. There are three fundamental stages in the design of an MAS system: conceptualization, analysis and design [31]. In the design process, the output of each stage is fed into the subsequent stage. The conceptualization stage defines the problem to be solved and specifies the system requirements. During the analysis stage, the problem and requirements are analyzed by the appropriate software- and knowledge-engineering techniques and the main task is decomposed into a hierarchy of subtasks. In the last stage (design stage), agents are created and assigned to these tasks in the hierarchy. The design stage also specifies agent anatomy, functionality, MAS architecture, agent behaviour, communication topology and an ontology for communication.

In Smartgrids, multi-agent system design steps and decomposition depend on the type of task, grid elements and architecture. For instance, if a Smartgrid security system requires each agent to monitor and assess neighbor agents, then the corresponding MAS structure should be decentralized and interaction is peer-to-peer. On the other hand, for a control system of a network of multiple microgrids, a hierarchical MAS would be better suited because there exist different layers like main grid, microgrid, zone and building in the network. In this case, energy transfer and communication take place across adjacent layers or inside the same layer.

Namely, the main problem is divided into subproblems in a suitable hierarchy and communication topology and agents are assigned to perform individual subtasks or functions in the respective position. If necessary, aggregator or coordinator agent(s) can be assigned at the respective layers or at the center of the MAS.

3.5. MAS Platforms and Software

There are many programming languages and software platforms available for designing and implementing multi-agent systems. Some of these MAS languages and platforms are JADE, Zeus, JADEx, JACK, EMERALD, JAS, Jason, AGLOBE, Agent Factory, SeSAm, GAMA, Cougaar, Swarm, MASON, INGENIAS, Kit, Cormas, Repast, MaDKit, CybelePro, JIAC, AgentScape, AnyLogic, Net-Logo, JAMES, OOA, Pangea, Pade, Gaml, Spade, JaCaMo, Goal, Plasa and Sarl. An overview and comparison of these platforms can be found in [72,73].

MAS development platforms are quite heterogeneous and have a number of plugins, frameworks and libraries for both commercial and academic audiences. Among these platforms, the ones that have been primarily used in the context of electric infrastructures and Smartgrid projects are JADE, ZEUS, PADE, JIAC and Volttron [74].

JADE is an open source multi-agent (multi-host) platform [75] developed and distributed by Turci by Telecom Italia Lab in 1999. JADE has the full support of FIPA standards, it has third-party plugins and a wide documentation. JADE includes Agent Communication Channel (ACC), Agent Management System (AMS) and Director Facilitator (DF). The Director Facilitator service, similar to yellow pages, is useful for listing agents and their abilities. JADE also partially supports Semantic Web technologies. In terms of implementation, agents in JADE can be seen as three-layered architectures: a message-handling layer for processing messages, a behavioral level for defining when tasks are to be carried out and a functional level for defining the actions the agent will perform [23].

Some of the applications of JADE in the power domain include:

• Ref. [76] develop a PowerSmartgrid Prototype at Illinois Institute of Technology. They use the JADE platform to model autonomous agents such as Distributed Energy Resource-generation agents and energy-storage agents. JADE also serves as a medium to communicate and post messages between agents.
• Refs. [77,78] use JADE to design a microgrid control infrastructure for an island in Greece. The objective of the application is to control the operation of non-critical loads in the microgrid.
• Ref. [79] use JADE to develop a microgrid management system to control generation and storage devices. The management system consists of a central controller, source controller and load controller. Agents can trade energy by submitting bids to the central market manager. The researchers tested this proposed architecture in laboratory facilities under different microgrid configurations.
• Ref. [80] propose a model of a microgrid simulation where agents are designed with JADE. Agents interact and negotiate with each other for demand management. In an experiment, they illustrate how agents react by adjusting their demand and how prices dynamically evolve upon a change in power supply.
• Ref. [81] propose a distributed energy- and resource-management system for multiple microgrids using JADE. Allocation of energy is achieved by an auction mechanism and JADE agents bid for energy in the market in real time.
• Another example is [82] which presents an MAS-based energy management system with JADE. In their setting, agents use the contract network protocol to allocate energy among them.

Other examples of Smartgrid control with JADE are [83,84,85,86]. Some other papers that use JADE to design agents for energy trade, scheduling and demand response are [38,87,88,89].

ZEUS [90,91] is a multi-agent platform developed by the British Telecom intelligent system research laboratory. ZEUS supports the design of agents in terms of their goals, tasks and factual knowledge of the environment. ZEUS is open source, FIPA compliant and enables agents’ communication using either the ACL or the KQML language. The user can specify agent properties using a graphical user interface, and the platform automatically generates the corresponding Java code. ZEUS provides a run-time environment, debugging tools and planning- and process-scheduling tools. However, it supports only one agent model, which limits the range of possible MAS designs.

A number of Smartgrid applications have been developed using ZEUS [74,92,93,94]. Refs. [74,92] discuss the design of an intelligent, distributed, autonomous power system which can operate in “normal mode” as well as “outage mode”. There are four types of agents: a control agent, a DER agent, a user agent and a database agent. The ZEUS platform is used to model the agents and to support the communication among them. The agents work in collaboration to detect upstream outages and react accordingly to allow the microgrid to operate autonomously in islanded mode. During normal operating conditions, the microgrid runs as a part of the local utility and coordinates its internal loads and Distributed Energy Resources for optimal operation. In the event of an upstream outage, the microgrid performs load controls based on a predefined prioritized list and activates its internal generators to secure critical loads.

Refs. [93,94] adopt a hierarchical MAS system using ZEUS and Matlab for coordination and control of a microgrid. Their system includes three levels: master level, substation level and terminal level. DER and load agents are located at the terminal level. Simulations reveal that MAS-based microgrid systems can perform stable operations: the microgrid can switch from connected mode to islanded mode in cases of substation failure and then back to connected mode when the failure is eliminated.

PADE [95] is a Python-based platform, specifically designed for power-engineering applications. PADE is an open source platform for MAS development that makes use of standard communication tools, enables the execution of predefined behaviors, and provides a tool for tracking agents’ actions. PADE provides agents with communication capabilities over standard protocols such as Generic Object Orientated Substation Event protocol (GOOSE) and Manufacturing Message Specification protocol (MMS) of IEC 61850, along with data standards such as the Common Information Model (CIM). PADE is thus appropriate for power engineering applications such as monitoring and diagnostic, automation, self-healing, adaptive protection and microgrid control.

JIAC [96] is an open source multi-agent platform which supports a distributed agent framework, comprising agent nodes and enables the run-time environment for the agents. The run-time environment can be monitored and controlled by Java Management Extension Standard (JMX). JIAC involves debugging capabilities and has been utilized in a series of industrial applications. In particular, Ref. [97] describes an MAS-based decentralized Smartgrid management system, which monitors and controls the grid. They also developed an application that optimizes the charging schedule of electric vehicles and stations where users, vehicles and stations are represented by JIAC agents. Ref. [98] provides an MAS-simulation framework, NeSSi, using JIAC for interconnected power and telecommunication networks.

Volttron [99] is another MAS platform specialized for energy system applications. It was developed by the Pacific Northwest National Laboratory (PNNL). Volttron is open source and developed in Python, but capable of also supporting agents written in other languages. Communication is established through a central “MessageBus” in the form of topics and subtopics. Its control architecture is modeled as a three-level hierarchy of agent classes: cloud agent (publishing data to/from a remote platform), control agent (interacting with the devices), and passive agent (interacting with the sensors and recording data). Volttron has been applied to the integration of electric vehicles and distributed energy generators in a Smartgrid [100]. Volttron has been used to develop the TNP system, in which agents perform grid operational services such as demand response, fault detection and energy transactions [101]. Volttron is also employed as the base of other energy management platforms, like the BEMOSS system [102,103].

3.6. MAS Toolkits for Energy Management

In addition to the general purpose MAS programming languages, a number of out-of-the-box, ready to use MAS toolkits for Smartgrid and energy management domains have been created. Notable examples of such toolkits are GridAgent, HomeBots, IDAPS, Ideas and PowerMatcher. These toolkits are based on the principles of MAS and implement different types of agents. An overview and evaluation of these toolkits is provided in [41].

The majority of these toolkits include components related to the challenges of energy trading based on dynamic pricing schemes. Agents revise their energy demand based on prices and market transactions. The majority of these toolkits are also associated with the direct management of physical entities for grid control (the only exception is Homebots). Homebots [104,105] is a framework to manage distributed equipment in a home environment. It involves a load-management system: each load (e.g., lights, appliances) is represented by an autonomous agent with a utility function.

GridAgent [88,106] is designed to manage a network of Distributed Energy Resources, transformers and secondary networks. Loads represent buildings, such as private homes and factories. Agents are plug and play and they can schedule the charging of electric vehicles.

IDAPS [35,74,107] has been developed by the Advanced Research Institute of Virginia Polytechnic Institute and State University for the purpose of controlling a microgrid. Its typical coverage is a residential distribution circuit with houses and transformers. Aside from energy management and trade, another feature of IDAPS is that it can detect power outages in the main grid and operate the microgrid autonomously in an islanded mode.

The IDEAS project [108,109] was developed for coordinating a large collection of houses with a focus on demand side management. Market prices are predicted daily and the adaptive mechanism reschedules deferrable loads (e.g., laundry machine, dishwasher) based on predicted market prices. The demand-management model also optimizes the non-deferrable loads, such as thermal heating of the house, to minimize the overall energy cost. IDEAS supports the concept of Virtual Power Plants for a cost-efficient way to incorporate Distributed Energy Resources among houses. VPPs enable electricity producers to participate in the energy market and trade energy with consumers on a daily basis.

PowerMatcher [110,111] was developed as part of the Smartgrid European FP7 project. PowerMatcher supports a coordination mechanism to balance demand and supply in a multi-microgrid environment with Distributed Energy Resources. The grid is considered as a collection of smaller Virtual Power Plants. In PowerMatcher cluster, the agents are organized as a logical tree. The auctioneer agent is at the root node and local device agents are at the leaf nodes. This system has been installed in three pilot regions in the Netherlands, Germany and Greece.

3.7. MAS Standards and Protocols

Standards are required for the development, communication, interoperability and integration of MAS systems. In addition, common languages and protocols are required for agents to communicate. The IEEE standards committee has identified the challenge of interoperable protocols and data formats and stated that open communication between smart devices using common protocols is crucial to interoperability [23,112].

An IEEE Computer Society organization FIPA (Foundation for Intelligent Physical Agents) has established standards for the abstract architecture of MAS (the entities and the environment), agent management specifications, agent communication language and message transport protocol [113,114]. FIPA standards have become the de facto standards used by MAS developers in the engineering and computer science communities [115]. The FIPA standard identifies two core services: the AMS (agent management service), which maintains a directory of registered agents, and the DF (directory facilitator), which provides a searchable directory of services that agents offer to other agents. Thus, agents do not need to be hard coded with the contact details of other agents whose services they need; instead, agents can search the DF for the currently available providers of services. Detailed information about FIPA can be found in [23].

FIPA provides a standard communication language called ACL (Agent Communication Language) [116], defining the communication protocols among agents and the meaning of messages. The standard described by ACL only defines the structure of messages and interactions, not the actual content of messages and vocabulary used by agents. ACL has origins in another communication protocol, the Knowledge Query and Manipulation Language (KQML) [117]. KQML is a language and protocol for communication between software agents and knowledge-based systems, proposed in the early 1990s as part of a DARPA effort.

Special to energy generation and distribution systems, some standards have been developed to promote coordination between devices, communication architecture and integration of Distributed Energy Resources [112]. IEEE 2030-2011 is one of the most widely adopted standards to support Smartgrid architectures [118]. IEEE 2030 classifies Smartgrid communication architectures into three subgroups:

• Home area network (HAN);
• Neighborhood area network (NAN);
• Wide area network (WAN). [119].

Typically, a WAN covers a distance greater than 10 Km, a NAN covers a distance between 100 m and 10 Km and a HAN covers a distance smaller than 100 m. In the context of an electric grid, a WAN involves the components associated with bulk generation and transmission (power stations, PMUs, protection and control units, several NANs), while NANs address the needs of electric distribution (DERs, protection and control unit, multiple HANs) and HANs support the individual consumers/prosumers (smart meters, EVs, sensors). All the communication technologies are facilitated by both wired and wireless media.

Another institution, the IEC (International Electro-technical Committee), has developed standards for electric substations, process automation and information transfer in power systems. IEC 60870 [120,121] and DNP3 [122] describe the standards for communication protocols and process automation in SCADA systems. The IEC 61850 standard [123,124] is about the design of the automation system for an electric substation and it includes communication protocols between power plants based on Ethernet. IEC 61850 is object-oriented and splits a physical device into logical devices, which can be further divided into logical nodes, data objects and data attributes. The IEC 61850 communication architecture consists of three levels: station level, bay level and process level. IEC 61850 specifies communication protocols for the client–server-based system (SCADA) and also for the publisher–subscriber system [125]. For the client–server architecture, the ACSI protocol is used whereas for the publisher-subscriber, GOOSE and GSSE protocols are used for communication [126]. An application of multi-agent-based Smartgrid automation architecture employing IEC 61850/61499 intelligent logical nodes is described in [127].

Other standards proposed by IEC for power systems are IEC 61970, for interfaces with energy management systems and IEC 61968 for interfacing the main applications for electrical distribution in a utility [128]. IEC 61970 describes a Common Information Model (CIM) [129] which defines how application software can exchange information about the configuration and status of an electrical network. CIM provides a structured class hierarchy in the description of power system plants and topology. CIM is a three-layer domain model; it defines a common vocabulary to describe the basic components used in electricity transportation and distribution. CIM aims to facilitate power-management processes, e.g., outage management, asset management and customer information management [130,131].

MultiSpeak [132] is another standard designed for interfacing software applications of utilities and devices in the grid. It defines common data semantics (in XML format), the message structure for data exchange and the messages required to support specific business process steps. Although CIM and MultiSpeak are similar, the latter is more oriented towards the distribution of energy.

As for the integration of distributed energy-generation sources into a Smartgrid, the technical requirements are provided by the IEEE 1547.1, 1547.2, 1547.13 standards [133,134]. These standards involve monitoring and controlling interconnected distributed resources, information transmission and test procedures for equipment.

Given the multiplicity and diversity of standards, operating a Smartgrid using an MAS-based approach requires a system compatible with these standards. Achieving a coherent system respecting these specifications and harmonization has been the subject of several proposals [135,136].

3.8. Ontology for Energy Domains

For the successful operation of an MAS, a knowledge base or ontology may be necessary. An ontology is a formal representation of knowledge, under the form of a set of concepts and of relationships between such concepts. In the Artificial Intelligence community, ontologies describe entities and their properties, relationships, constraints and behavior that are not only machine-readable but also machine-understandable [137,138]. According to [139], the functions of ontology are communication, interoperation and acquisition, reuse and sharing of knowledge.

Some ontologies have been specifically developed for Smartgrids and the energy domain. FIPA-SL is an ontology standard [140] adopted by FIPA and includes three types of elements: concepts (components, data types), agent actions and predicates. FIPA-SL includes predicates, relations, actions suitable for the energy domain such as microgrid, substation, DER, load, storage and their status [112,141]. This ontology can be used by agents for exchanging information, asking questions and requesting execution of an action [142]. Ref. [143] constructed an ontology for the multi-agent-based electricity market and [144] constructed an ontology for the customer portfolio. Kofler et al. [145] develop an ontology for the energy efficiency in a smart home. The ontology involves representation of home facilities, their energy demand and supply and use cases. Information about energy type, cost, tariff and provider are also included.

The literature has also provided ontologies more specifically focused on the management of electricity markets. For example, Ref. [146] describes an ontology named Electricity Market Ontology (ELMO). This ontology was developed primarily for the electricity market of Greece. ELMO uses a multi-layered architecture divided into extendible and reusable modules. These modules can be used by organizations or transmission system operators. Santos et al. [147] also present an Electricity Market Ontology (EMO). The EMO is an upper ontology for the electricity market, from which other low-level ontologies can be extended. In particular, ontologies for the EPEX [148] and Nord Pool spot market [149] are developed as extensions of EMO.

4. Energy Markets and Trade

Energy markets serve the function of allocating electricity between agents in the context of an energy economy (

Table 2. MAS-based approaches to energy markets and trade.

Problem	Proposed Methods
Design market and trade models for microgrid	Auction [81,87,150,151,152,153,154,155,156,157,158,159,160], contract networks [89,161,162,163], negotiation (bargaining) [164,165], non-market methods [35,166,167,168]
Multi-microgrids and large scale	Extended models [81,157,159,160,168,169]
Demand and supply forecasting	Neural networks [170,171], historical data [172], support vector machine [151]

). For this, we need to design a medium and exchange mechanisms for prosumer agents to exchange energy with each other. Intuitively, producers, consumers and prosumers in the Smartgrid trade with each other, in order to balance energy demand and supply. Researchers have employed different methods and MAS structures for energy trade: auction, contract network, negotiation (bargaining) and non-market methods.

4.1. Microgrid Level

One method of allocating energy among agents in a microgrid is through auction. In auctions, agents determine the quantity of energy to buy/sell and then submit their bids. Auction models for electricity markets are typically double-sided with multiple buyers and multiple sellers. The type of auction can be an ascending auction, sealed bid, Vickrey, English, Dutch or others [150,151]. Auction models mostly have a centralized structure [151,152,153,154] with an auctioneer agent, consumer agents and producer agents. The auctioneer agent collects bids and announces the auction result. Note that an auction may involve multiple rounds or iterations until the market clears [81,153,154]. Excess or less energy can be sold or bought from the main grid, respectively.

As an alternative to the centralized auction models, Ref. [87] describes a hierarchical MAS structure with market clearing agents, utility grid agents, coordination agents and auxiliary agents at different levels to realize the auction. Ref. [173] explores a two-layered hierarchical MAS for auctions, where the first layer is responsible for the bid evaluation and the swap operation across agents while the second layer performs the consensus procedure. Among decentralized auction models, Ref. [151] explores a peer-to-peer bidding model between the generator and the load agents. Another example of decentralized auction models is provided in [150], where the generator agents can act as auctioneers and sell auctioned energy to the participating load agents; alternatively, load agents can act as auctioneers and conduct auctioning to secure the right of deriving energy from the participating generator agents. Auctions can also take place in hybrid MAS architecture, such as holons [155]. Each holon (a group of generator and load agents) participates in the auction as a whole and holons can change dynamically over time. In the prosumer setting, an agent may switch its role from seller to buyer in auctions and vice versa. Consumer agents can also decide whether some part of their demand can be dispatchable (shiftable to later periods) [87,152]. Auctions can be performed to trade energy on a daily basis [156,157], hourly basis [151], 15 min intervals [81] or continuously [158]. Some auction schemes have a day-ahead market and a real-time market [159].

There exist variations of auction and bidding mechanisms in the literature. Ref. [153] explores risk-based auctions, where a buyer or seller agent updates its target bid price at every round using its own risk model. Ref. [156] offers a complex bid model for non-critical (dispatchable) loads. The bid involves parameters like the earliest start time point, the latest finish time, the (minimum) duration that the device will be switched on and the power rating of the device. In [160], the modified auction scheme involves incentive compatibility and rationality conditions for agents.

MASs have also been used in contract networks for energy trading [89,161,162,163]. Contract networks are for next-hour or day-ahead trading and the price is fixed. Seller agents offer alternative contracts where the energy is sold in batches and consumers select among contracts. Contracts are offered and discussed in a decentralized structure. Namely, the buyer and seller agents communicate and agree on the contract bilaterally. Agents can make forward contracts as well as real-time contracts [163].

In negotiation models, producers and consumers bargain on the price of electricity, with a negotiation deadline [164,165]. Ref. [164] develops a negotiation strategy between a smart building and the main grid. Negotiation may span multiple rounds, the buyer and the seller have adaptive ask price and bid price. The MAS structure for the building is hierarchical with a negotiation agent, central coordinator agent, multiple local coordinator agents and the load agent. The central coordinator agent is responsible for comfort optimization; total energy demand of the building is determined by the local controller agents and the load agent. Ref. [165] studies peer-to-peer negotiation between agents inside a microgrid. At the beginning, the main grid operator agent announces the price for selling and buying one unit of energy. Local generator and load agents set their initial buy/sell price based on their operational costs and then the negotiation starts.

There are also non-market mechanisms for energy trade with multi-agent systems. Consensus algorithms [166] have decentralized structure and agents communicate with their neighbors to set up their energy demand. Ref. [167] develops a quantum-inspired energy-allocation algorithm with multi-objective optimization. Their framework has three levels of hierarchy, namely the coordination agent, the local controller agents and the device controller agents. In [168], there is a public bulletin for bilateral energy exchange and a mediator agent. The trading agents search each other via the posts in the bulletin. A manual approach to energy management is provided by [35]. In their IDAPS system, the human user agent enters the electricity price of different utility companies to a bulletin board where the resource, load and storage agents can access.

4.2. Multi-Microgrids and Large Scale

Energy-trade models have been built for large and small scale grids. Refs. [159,169] have a wholesale and retail sale market together. In these works, the MAS has two layers; the upper layer includes aggregate agents such as generator companies, retailer agents and deals with the wholesale market. The lower layer includes end-users and serves as the retail electricity market. While [159] employs an auction to allocate energy, Ref. [169] uses negotiation and bilateral contracts between the generator companies and retailers and between retailers and end users. Refs. [81,157] propose auction mechanisms for multiple microgrids and the main grid. Their MAS architecture is hierarchical with multiple layers, in order to manage energy inside and across the microgrids. Each microgrid optimizes its benefit and bids as a single unit in the market. Microgrids can also buy/sell the less/excess energy from the main grid. Ref. [168] studies self organization of individual agents to form a coalition. Agents (producer, consumer, storage) coordinate among each other to form coalitions (associations) and enter the electricity spot market as a single unit. This allows them to overcome capacity-related entry barriers and compete to increase their benefit. Agents negotiate the potential value and surplus distribution during the formation stage of the coalition. Ref. [160] proposes a modified auction with special payment for energy-storage (battery) sharing among residential homes inside a community. A residential unit determines the fraction of its storage to sell and the reservation price. The community is partitioned into blocks and each block has a shared facility controller agent in the MAS hierarchy.

4.3. Demand and Supply Forecasting

We should note that the trade and demand response models of Smartgrid involve energy demand and supply forecasting to estimate the prices and the energy consumption. For example, Ref. [170] utilizes neural networks to predict the next-day load demand. The output of the neural network forecaster is fed to a hierarchical MAS which performs peak-load reduction. MAS consists of Load Management Agent (LMA), Feeder Agent (FA), Distributed Generator Agent (DGA) and Demand Response Agent (DRA). A feeder agent collects data from DGA and DRA, combines the DGA and DRA capacities and sends the available load reduction and its price to LMA. LMA decides to dispatch the load reduction based on the forecasted demand, FA data and the main grid price. Ref. [171] uses neural network to forecast the load demand of a Virtual Power Plant. Ref. [172] implements day-ahead and next-hour forecasting based on historical data. Based on prediction, prosumer agents in MAS cooperate with each other to optimize their profit or cost. Ref. [151] tests various forecasting algorithms such as weighted average of previous periods or support vector machine for individual generator and load agents.

Challenges and Open Problems: Authors have designed energy-trade mechanisms for sharing energy between prosumers in the stochastic environment of Smartgrids. Namely, agent-based energy market and non-market models for both microgrid level and network of microgrids have been constructed. In this manner, the produced energy can be allocated among the prosumer agents in the Smartgrid.

The main issue in the above energy-trade models is that they do not consider network hierarchy or geographical proximity. In particular, a Smartgrid has a hierarchical structure: houses, buildings, microgrids and multi-microgrids. Normally, agents in the same house share energy; thus, they should enter an auction as a single entity, rather than individual agents. The same idea can be applied to the microgrid level, namely each microgrid enters the market as a single unit and then distributes the net energy inside it. As such, the energy trade and allocation becomes a hierarchical problem: trade across entities (house, microgrid) and then trade inside the entity. A related issue which has not been addressed in the trade models is the geographical proximity. It is better to trade energy with a neighbor agent rather than another agent further away, in order to mitigate transmission loss and network traffic. This might be achieved by introducing a cost proportional to the distance or block in the auction mechanism.

Another critical factor missing in the existing market models is communication. Agents can communicate and exchange information with each other about operation time, load amount before the auction, negotiation or contract, in order to improve the outcome and convergence speed. In particular, the neighbor agents should communicate with each other to trade. The information-exchange procedure will require the design of a communication topology and order.

Energy markets operate based on predicted renewable energy generation and predicted demand. Then, the question is what would happen if actual energy supply is less than demand. To deal with this case, alternative ready-to-use strategies should be developed such as secondary market to switch energy among consumers, rescheduling loads inside the day, pricing of energy from storage elements (batteries) and diesel generators. These strategies should be investigated in more detail to enhance operation of energy markets.

The existing energy-trade models assume that agents know or predict their energy supply/demand and enter the auction. It would be interesting to study a problem where a human agent can enter the auction and submit bids and quantity of energy manually.

5. Smartgrid Control and Management

The aim of Smartgrid control is to manage microgrid resources and elements in a smooth and efficient manner. The control and management system must ensure stable delivery of electrical power to consumers while optimizing energy usage towards additional objectives such as maximum renewable energy utilization, minimum energy cost, weather conditions, etc. This is a major problem of the Smartgrid due to the intermittent and stochastic nature of renewable energy resources and dynamic energy demand from loads. The main issues in Smartgrid control are securing critical loads, regulating frequency and voltage during an outage, reactive power compensation and minimization of power loss.

Researchers have addressed Smartgrid control at different scales, which we review below (see also

Table 3. An overview of MAS-based methods for control and management.

Problem	Proposed Methods
Home/building energy management	Combined heat and power optimization [36,174,175,176,177,178,179,180]
Microgrid control in connected mode	Centralized [181,182], decentralized [38,183,184,185], hierarchical [186,187,188,189,190,191], reinforcement learning [192,193]
Microgrid control in islanded mode	Centralized [194,195,196], decentralized [185,197], hierarchical [190,191]
Transition between modes	Securing critical loads [35,74,83,85,93,198], voltage and frequency regulation [199,200,201,202,203,204,205,206,207,208]
Network of microgrids and buildings	Decentralized network of buildings [209,210], hierarchical MAS of microgrids [157,211,212], decentralized (consensus or peer-to-peer) [213,214]

).

5.1. Smart Home and Building

Some studies deal with energy management on a small scale, i.e., homes, buildings or factories [36,174,175,176]. These home or building energy management systems (Figure 4) are oriented towards Combined Heat and Power (CHP) optimization, i.e., manage electricity consumption, lighting, heating and cooling together. These systems generally exhibit a centralized control or a layered MAS structure consisting of zone, floor, home and building. The objective function typically involves multiple elements such as minimizing the cost of energy, reducing emissions and maximizing the occupants’ comfort.

In [175], the central controller agent receives sensor data (temperature, humidity, occupancy), decides on the amount of heating/cooling and sends commands to the actuators. The controller agent is hybrid and employs reinforcement learning, Bayesian learning, dynamic programming and a fuzzy logic kernel. Ref. [36] investigates energy management of a self-sufficient smart home in islanded mode. Solar panel, wind turbines and storage devices supply energy to the building. MAS is hierarchical and consists of a central coordinator agent and multiple local controller agents. Users can specify their preferences and thermal comfort, then the central coordinator agent uses particle swarm optimization (PSO) to determine the best allocation of resources. The system sheds non-critical loads in case of insufficient power supply. Ref. [177] creates an energy management algorithm for a smart home in islanded mode. MAS handles the flow of energy between the energy resources and the storage units. In particular, the supervisory agent charges or discharges the battery depending on the surplus or shortage of power. In the Building Energy Management System (BEMS) of [178], fuzzy logic rules are used to monitor and control the indoor energy flow. The controller increases or decreases the power based on the current comfort level and the desired comfort level. Ref. [176] constructs a CHP system where the electricity agent, heating agent, cooling agent measure and keep track of the electricity and heat flow. The agents request electricity, hot water, heat based on their needs and the central system controls the flow. The electricity agent continuously receives electricity prices from the main grid and attempts to reduce the local demand.

Machine learning algorithms have been applied to solve the home energy management. In [179], the photovoltaic panel output and the main grid prices are predicted by neural networks. Reinforcement learning is used for hour-ahead energy-consumption decisions. Ref. [174] proposes multi-agent reinforcement learning for an industrial site to manage on-site energy generation, energy storage devices and transfer from the main grid. Ref. [180] considers the problem of management of energy-storage devices at homes. Each household is an agent, obtains its electricity from the main grid and wants to minimize its electricity bill. Households have fixed energy demand profiles within a day but energy prices change depending on demand and supply. Agents use a weighted moving average algorithm to predict day-ahead market prices and make their best-response strategy in a game-theoretic framework. Authors develop a novel learning mechanism for optimum storage charging.

5.2. Microgrid Level

At the scale of a microgrid, researchers have utilized MAS to control microgrid resources and energy management in the connected mode and the islanded mode. Transition from the connected mode to the islanded mode (and the reverse direction) is also part of the microgrid control problem. The control process should be rapid, adaptable and reliable: microgrid assets must be managed in real-time and protected during transitions.

5.2.1. Operation of Microgrid in Normal Mode (Connected/Islanded)

Refs. [38,186,187,188,189] study the control problem of the microgrid in the connected mode. Ref. [38] constructs a decentralized MAS model with three types of agents: producer agent, consumer agent, observer agent. Their setting involves renewable and non-renewable resources (diesel generator), loads and batteries. A producer agent is assigned to each energy resource; a consumer agent to a load; a producer and a consumer agent to a battery; and observer agents to the AC Bus and PCC breaker nodes. MAS agents act independently and they decide to buy/sell from local resources or from the main grid, depending on the spot market price and local generation price. Refs. [186,187,188] consider a microgrid consisting of two systems (department and hostel), each containing a Photo Voltaic (PV) panel, wind turbine, local consumer, battery and a diesel generator. MAS is hierarchical and contains a grid agent, a control agent and an individual agent for each device mentioned above. They propose an intuitive algorithm for the control agent as follows. If the renewable resource supply is insufficient, the controller obtains the required energy from the battery. If the battery is depleted, the controller sheds the non-critical loads and obtains the energy from the diesel generator or the main grid (the one with lower price). In [181], MAS is centralized and consists of station agent, control agent, load agent, generation agent and storage agent. The objective is cost minimization; the station agent performs optimization and supervision of the microgrid. Control agents function as a circuit breaker or a switching agent. Excess/less energy is transferred to/from the battery or the main grid, depending on the price. Ref. [182] also adopts a centralized microgrid control. The Smartgrid controller agent (SGC) is located at the point of common coupling to the main grid; it handles the energy transfer inside the microgrid as well as energy transfer from the main grid. Based on their prediction, the load, generator, battery agents submit their power to buy/sell and its price to the SGC, for the next period. The SGC decides the amount of power for each agent and informs them. Ref. [183] handles electricity and heat management of a microgrid using a decentralized MAS with generator agents, load agents, storage agents and thermal agents. Microturbine agent generates electricity and heat together. Every agent targets maximizing social welfare and they perform a convergence algorithm where the price and energy demand are updated iteratively. In another decentralized model [184], agents do not directly communicate between one another to prevent heavy traffic, instead they post messages on a bulletin board (stigspace). There are resource agents, load agents and a broker agent. The broker agent examines the summary data, market, grid information to set a cap on the total power drawn from the main grid. The resource agents revise their generation plans to satisfy the cap while adhering to their local constraints. Ref. [189] proposes a hybrid, hierarchical MAS structure with three layers for microgrid control. The upper layer works in a centralized manner for energy management, the upper layer agent solves a multi-objective optimization (including cost of electricity, emission, line losses) by particle swarm optimization. Several coordinated middle layer agents switch the operation mode of individual generators depending on the surplus or shortage of energy. The lower layer contains generator, load and storage agents. Generator agents also perform local control by sharing active and reactive power among themselves.

Researchers have also adopted reinforcement learning methods to deal with the energy management of the microgrid. In [192], there is only photovoltaic panel (with unpredictable output) and battery as internal energy resources. They propose a supply control mechanism, namely to learn charging and discharging of storage devices as a function of the system load and available resources. The goal is to minimize the purchase of the additional energy from the main grid. Ref. [193] adopts a fuzzy-logic controller to manage renewable, non-renewable resources, storage devices and loads in the microgrid. Reinforcement learning is used to train the fuzzy logic controller.

The control problem of a microgrid operating in islanded mode has been addressed by [194,195,196]. These works explore a centralized MAS structure for microgrid control. In [194], there are seven types of agents: Single Smartgrid Controller (SGC), Load Agents (LAGs), a Wind Turbine Agent (WTAG), Photo-Voltaic Agents (PVAGs), a Micro-Hydro Turbine Agent (MHTAG), Diesel Generator Agents (DGAGs) and a Battery Agent (BAG). The SGC agent coordinates the negotiation process between the buyers and the sellers. Namely, it sorts the producers with respect to the energy price and fulfills the demand based on the sorted price. If the electricity generated from the renewable resources in the microgrid is insufficient, the shortage is compensated by the diesel generators. Refs. [195,196] have no critical loads and the central controller agent uses fuzzy decision making. In [195], the user agent, the control agent (energy control center), the database agent, and the Distributed Energy Resources (DER) agent works collaboratively to perform the assigned tasks. The ECC agent uses fuzzy logic in such a way that, depending on the output of solar and wind generators, the circuit breakers switch to the load or to the battery. Ref. [196] also includes a diesel generator to compensate the energy demand. The fuzzy logic controller aims to fulfill the demand using solar and wind generators and minimizes the use of diesel generators. Ref. [197] proposes a distributed MAS for controlling a self-sufficient microgrid. Energy management is achieved by an incremental cost-based consensus algorithm. Communication between agents is specified by a graph with weighted edges. Agents communicate with each other according to the graph structure to implement the consensus algorithm.

There are also studies which deal with the control problem of microgrids in both connected and islanded modes. Ref. [185] proposes a decentralized MAS system with four types of agents: source agent, load agent, breaker agent and switch agent. Breaker and switch agents connect or disconnect loads and batteries from the energy resources. They study different operation modes where either the main utility grid or the renewable energy serves as the primary power source or they are integrated. They also provide algorithms for choosing the energy source and delivering energy to the loads in these different modes. In a more sophisticated framework [190,191], the fuzzy decision maker agent and an evolutionary particle swarm optimization agent work together to minimize the energy cost. The System Operator agent performs negotiations and handles the energy exchange with the main grid. MAS has a physical layer, a communication layer and a control layer and the communication structure is a graph (social network of agents).

5.2.2. Operation during Transition Mode

Another critical problem in microgrid control is how to provide sound transition of microgrids from the connected mode to the islanded mode when a fault or outage occurs. Typically, there is an agent such as PCC agent [83] or central controller agent [85] or main grid agent [93], who is responsible for monitoring the voltage, phase, frequency and disconnecting the microgrid upon a fault in the main grid. This problem has several aspects: managing the critical and non-critical loads, restoring frequency and voltage to their normal values and reducing power loss. Researchers have addressed these problems with different methods and MAS architectures. Refs. [85,93,198] have proposed a simple solution with a centralized MAS structure. In this approach, the central control agent disconnects all non-critical loads upon an outage. When the fault or outage at the main grid is fixed, the microgrid makes a transition to the connected mode again and the non-critical loads are reconnected. However, this method may cause under-utilization of energy resources, since the available energy may still feed a portion of non-critical loads in the islanded mode. To remedy this pitfall, the IDAPS system of [35,74] has a priority list of loads and disconnects non-critical loads according to this priority, during transition to the islanded mode. In [83], non-critical loads are disconnected in islanded mode only if the total energy demand is greater than the renewable energy supply. The diesel operator is started if the renewable energy supply is not sufficient to feed even the critical loads. Namely, the control algorithm maximizes the renewable energy usage and minimizes the operation of the diesel generator.

Another issue in transition from the connected to the islanded mode is stabilizing the frequency and voltage and restoring them to their normal values. As for frequency and voltage control, authors have mostly used decentralized MAS structure with peer-to-peer communication [199,200,201,202]. Load agents communicate with neighbor agents to restore them back to the nominal values. Refs. [202,203] have used consensus algorithms, while [199,200] have used an input–output feedback linearization algorithm. Ref. [201] attempts distributed voltage control by splitting the feeder into a series of overlapping segments. An agent in each segment senses the voltage level in its own segment and exchanges this information between the agents in the adjacent segments. Then, the agent formulates the reactive power compensation sufficient to restore the voltage in that segment. Ref. [204] achieves voltage regulation with multi-agent systems and expert systems. The expert system decreases or increases voltage based on a set of conditional rules and electric circuit equations. In the model of [205], two types of agents (the estimator agent and the control agent) collaborate for frequency regulation. The first agent estimates the frequency-bias coefficients and provides the area-error signal; the second agent compensates the power imbalance between generations based on the error signal. Ref. [206] constructed a centralized MAS structure for voltage control of a microgrid. The control agent (on-load tap changer (LTAC)) uses fuzzy logic rules like predefined if–then–else rules, to change the load or transformer tap and at the same time to prevent excessive tap operation. This allows LTAC to keep the voltage of feeders in the standard range. Ref. [207] uses the graph-partitioning method to regulate the voltage by the master and local control agents in the hierarchy, after a fault occurrence. Ref. [208] focuses on restoring the frequency of storage devices in the microgrid to the reference range. The cooperative control system modifies the output power of storage devices so that they reach a balanced energy state and maintain load sharing.

To minimize power loss in electrical systems, reactive power compensation is necessary. Ref. [56] proposes a hierarchical MAS consisting of holons and a decomposition algorithm for reactive power control. Ref. [215] tackles minimization of reactive power needed by the distributed generators. This challenge is addressed by solving a linear programming problem iteratively, which converges to the optimal solution. The contract net protocol is used for distributed coordination and assigning moderator, monitor and dispatch roles to the agents. Decentralized leaderless or leader–follower consensus algorithms have also been applied to the reactive power compensation [216,217].

5.3. Network of Microgrids and Smart Buildings

The overall electricity-distribution system of a Smartgrid consists of the main grid, multiple microgrids and the smart buildings which interact with each other and the main grid. In this context, the objective of microgrids is to share energy and information with each other to optimize their cost and energy need.

Ref. [209] presents a method for optimal power flow and energy sharing among smart buildings. Each building is an MAS agent which contains a smart meter and other agents. However, in their framework, houses only have batteries and no renewable energy resources. Hence, buildings share energy from their batteries. The objective function is the summation of cost curves of battery-storage systems subject to the active and reactive power flow constraints. Ref. [210] constructs a decentralized MAS for multiple smart homes in a neighborhood. Smart homes are equipped with a photovoltaic panel and battery and can trade energy with each other. In this non-cooperative game, the objective of home agents is to minimize their electric bill. Each home agent uses a genetic algorithm to optimize its energy consumption and battery (dis)charge policy.

Multi-agent systems for management of multiple microgrids and the main grid generally have a hierarchical structure due to the nature of the grid. Ref. [211] deals with the control of energy transfer between multiple microgrid systems. Each microgrid consists of photovoltaic arrays (PVs), batteries as a storage units, a diesel generator and critical and non-critical loads. MAS is decentralized and there is a grid agent which monitors the main grid condition. Each microgrid includes a microgrid agent which can negotiate energy exchange across other microgrids. The objective is to share energy against dynamic load and generation conditions. Ref. [212] also considers energy sharing with multiple microgrids and multiple substations. Energy sharing and payoff distribution is based on coalition game theory with transferable utilities. Power flow is managed by a hierarchical MAS which comprises four types of agents, namely the Point of Common Coupling (PCC) agents, the Grid Facilitator (GF) agents, the Grid Management (GM) agent and the MicroGrid (MG) agents. The PCC agent coordinates the overall power operations between the main grid and each linked microgrid. The MG agent is responsible for the power balance of its own microgrid at the distribution level. Ref. [157] studies optimal energy exchange between multiple microgrids and the main grid. There is a day-ahead energy market for the integrated microgrids. Each microgrid bids for the energy to trade with other microgrids or buys from the main grid. Ref. [213] has a different approach for management of microgrids: a consensus algorithm for distributed coordinated control. Agents discover global information by communicating with their neighbors. Each microgrid is operated at an optimum economic point with respect to its incremental cost. Ref. [214] proposes a multi-layer MAS architecture with peer-to-peer communication to control a network of microgrids. Primary, secondary and tertiary control are realized at separate layers.

Challenges and Open Problems: Smartgrid control and energy management models are designed to handle energy usage problems of home, microgrid and multi-microgrid levels under given conditions, i.e., fixed price, network architecture. These models can be used to manage Smartgrid operation during normal mode (connected or island) and transition mode (during outage or reconnection). The system objectives in these models are maximum renewable resource utilization, minimizing energy cost, reducing non-renewable energy usage and energy purchase from the main grid. Additional objectives are reactive power compensation and voltage, and frequency stabilization.

One main open problem in Smartgrid management is how to handle control and energy trade together in a unified framework. Energy market models determine the electricity price assuming that the demand and supply are fixed. On the other hand, control models assume the price is exogenous and optimize the consumption of agents. Ideally, both price and quantity should be determined together in the market. There are some preliminary attempts [157] to integrate Smartgrid control and energy trade, but this topic has not been addressed in sufficient detail.

The advantage of MAS for Smartgrids is its decentralized nature, which is more reliable and efficient. However, the existing control models for Smartgrids often have a centralized structure. Therefore, future research should strive towards developing decentralized control and management models.

In some MAS-based control models, the agents are not really autonomous or intelligent. For example, there are circuit breaker or PCC agents whose task is just to execute the commands sent to them by the central controller agent. A better alternative is a decentralized MAS where the circuit breaker agents decide to turn on/off autonomously, based on the outage or connection status of the microgrid. Likewise, for energy cost optimization during the normal operation, agents should communicate and find the optimal schedule of energy usage on their own. In this manner, the need for a central controller agent would be alleviated.

It would be interesting to study Smartgrid control problems in an adaptive, dynamic setting. Recall that a Smartgrid has a stochastic structure due to the weather-dependent nature of resources and changing operation time of loads. Namely, agents’ quantity and temporal schedule of production and consumption may change over time. Existing control models often use simple methods like shedding non-critical loads when (renewable) energy supply is insufficient. To enhance energy usage and prevent sudden interruption of load devices, more intelligent algorithms can be developed, e.g., scheduling of loads over time to reduce instantenous peak energy usage. For example, oven, laundry machine operation and EV charging can be made sequential rather than simultaneous. This necessitates agents to be more proactive in making plans and performing actions. Agents should inform each other about the relevant variables and make new schedules accordingly. Thus, researchers should deal with the dynamic control problem with more adaptive, intelligent agents.

In Smartgrid management, the presence and actions of the human user(s) have not been considered in the control mechanisms. During normal operation, the user may switch on/off some critical or non-critical loads depending on his need or other conditions. Likewise, during an outage, the user may intentionally switch off appliances based on his own priority at that moment, which may differ from the built-in priority list of the Smartgrid control. Hence, Smartgrid management should integrate users’ preferences into control and power optimization. If necessary, the control software should take the relevant input from the user about the operation of the appliances.

6. Demand and Supply Management

A critical paradigm in Smartgrids is demand and supply management. These are tools to assist energy management of Smartgrids and prevent energy shortage. Demand and supply management assume the other side is fixed and optimize their own side by proper scheduling, cost minimization and developing other necessary mechanisms. The goal is to attain the utmost utilization of scarce energy resources and shape the load profile to prevent overloading. A variety of approaches have been investigated (

Table 4. MAS-based solutions to demand and supply management.

Problem	Proposed Methods
Supply side management	Economic dispatch [173,191,218,219,220,221,222,223,224], unit commitment [141,225,226,227,228]
Demand response (residential)	Direct control [229,230,231], indirect control [232,233,234], unified [235,236]
Demand response (smartgrid)	Direct control [237,238,239,240], indirect control [170,241,242,243,244,245,246], consensus algorithm [247]
Electric vehicle charge scheduling	Centralized [248,249], hierarchical cooperative [248,250,251,252,253,254,255], hierarchical non-cooperative [256,257]
Electric vehicle to building	Building consumption optimization [258], unidirectional [259], bidirectional [260], VPP [261], global objective [262]

) and are discussed next.

6.1. Supply Side Management

Supply side management mainly deals with allocating the energy production among generators to minimize the total cost of production. The aggregate demand of loads in the microgrid is assumed to be fixed. This problem is also known as economic dispatch. MAS-based solutions to the economic dispatch problem have a mainly decentralized structure: generator agents need to communicate only with their neighbors. These models mostly utilize a consensus algorithm in which agents propagate their incremental cost information [218,219,220,221,222]. Ref. [223] studies energy cost minimization with generators and storage devices. The solution method is again a decentralized consensus protocol between agents. Alternatives to the consensus algorithm also exist. Ref. [224] uses a distributed gradient algorithm to allocate output between generators. Each agent computes the gradient of its local cost function and acquires the gradients from its neighbors. Then, the agent updates its local generation according to a weighted sum of the gradient values. In [173], generator agents engage in auction and exchange output power to minimize the cost of production. In the approach of [191], a dedicated optimization agent uses particle swarm optimization to solve the economic dispatch problem.

Another problem in the supply side management is the unit commitment: how to schedule generation time period of distributed energy sources over a time horizon to fulfill the demand. An example of MAS-based solutions to the unit commitment problem is presented in [141]. In this paper, generator agents submit their bids to a pool to sell their energy. Ref. [225] designs a rule-based algorithm for day-ahead and hour-ahead scheduling of Distributed Energy Resources, considering dynamic market prices. The Microgrid Management, Monitoring, Control (MMC) agent receives the forecasted price data from the Distribution System Operator (DSO) agent and runs the day-ahead and hour-ahead scheduling algorithms. Ref. [226] proposes a heuristic unit commitment algorithm for a central controller. Storage units are also considered as an energy resource, in addition to renewable and non-renewable generators. Each energy resource agent has a priority and the controller assigns output to an agent from the list of available agents according to their priority. If total local production is less than demand, energy is purchased from the main grid at an external market. Reinforcement learning has also been applied to the unit commitment problem in a decentralized cooperative setting [227,228]. Distributed generator agents learn to satisfy the demand profile with minimum cost subject to the constraints. Ref. [228] formulates the unit commitment problem as a Markov decision process and employs multi-step deep reinforcement learning to solve it. A survey of economic dispatch and unit commitment models using multi-agent systems can be found at [25].

6.2. Demand Side Management

Demand side management (DSM) or demand response refers to the act of postponing or rescheduling loads and devices in order to balance the aggregate energy supply and the aggregate energy demand in the Smartgrid. Recall that the outputs of renewable energy sources are also time-dependent due to weather conditions. During the day, there are periods like morning and evening when residential energy consumption tends to be high, so-called peak demand periods. In the case of peak demand, local renewable energy generation is often insufficient; hence, transfer from the main grid or operation of expensive resources, like diesel generators, might be necessary. Consequently, peak demand may also impact power plant capacity and cause transmission line congestion and infrastructure damage. For optimal utilization of renewable resources, an efficient demand response mechanism should be designed in order to shift the peak energy usage to periods of low demand or to the periods of high availability of renewable energy. Thus, consumers may need to reschedule their non-critical loads accordingly.

Approaches to demand response are broadly divided into incentive-based programs and time-based programs [263]. The incentive-based programs are usually more suitable for industrial facilities and time-based programs are more suitable for residential users. In both types, the day is divided into a number of periods and the goal is to attain relatively smooth energy usage.

The focal problem in demand response programs is how to coordinate energy consumption of independent agents in order to alleviate the peak demand problem. For this purpose, authors have mainly used hierarchical or decentralized MAS structures. A simple electric price tariff is typically not sufficient to prevent peak demand because the tariff scheme just shifts peak demand to the beginning of the low price period. Several demand response strategies have been proposed in the literature: peak shaving, valley filling, load shifting, strategic conservation, strategic load growth and flexible load shape [264,265]. These strategies only deal with how to allocate energy usage across various time periods inside the day, but not how to coordinate the individual agents or give incentive to them. In order to achieve coordination among users, researchers have developed direct and indirect load control methods. Direct load control attempts to schedule loads of each consumer to smooth energy usage. Centralized direct load control achieves optimal demand scheduling; however, it requires perfect and complete information. Indirect control methods try to adjust demand using price signalling or market mechanisms.

MAS-based demand response methods have been applied to Smartgrids, including residential load scheduling and electric vehicle charging.

6.2.1. Residential Demand Response

Residential Demand Response deals with management and scheduling of home appliances; some models may also involve heating/cooling equipment or (hybrid) electric vehicles. In these models, shedding of loads is based on their priority and they use machine learning or other algorithms for scheduling.

Ref. [266] introduces the Smart Home Device Scheduling (SHDS) problem, where each device at home is an individual agent. The authors model SHDS as a distributed constraint-optimization problem under real-time pricing schemes of the electric company. A distributed local search algorithm is developed to find the locally optimal DCOP solutions.

Among direct load control approaches, Ref. [229] studies demand side management for Home Energy Management System (HEMS). The MAS behind such a model is hierarchical and consists of an HEMS agent, a Demand Side Management agent and an agent for each home appliance, at consecutive layers. Each appliance has a priority level and the DSM agent uses an hourly based algorithm which turns off home appliances based on their priority when the energy supply is less than the demand. In [230], the Energy Management System (EMS) agent at each house employs reinforcement learning to schedule the operating time of the devices. At the beginning of each period, the EMS agent receives the current price level from the utility grid and the energy request from devices. Ref. [231] tackles the problem of combined optimization of distributed energy-generation management and demand response for residential agents and electric vehicles (EVs) in a microgrid. Grid agents, control agents and residential agents form the hierarchical MAS from top to bottom layer. The grid agent compares the total on-site energy generation, battery storage and aggregate residential demand and informs the control agent to shift the interruptable and deferrable loads. The control agent receives the current load requirement from each residential agent and the charge status of the EV agents. Depending on the electric price of the main utility grid, the control agent runs a demand response algorithm every hour which shifts loads or EV charging to off-peak periods, based on their priority.

As for indirect control approaches, imposing time-dependent price is a common approach to shape residential demand. Ref. [232] considers the demand management of N houses each having M appliances. The MAS is decentralized; there is a grid operator agent and several household agents. The grid agent implements a real-time pricing scheme which depends on prices in the previous periods. The house agents utilize reinforcement learning methods to schedule their appliances and minimize their energy cost under such dynamic pricing schemes. Ref. [233] studies scheduling of residential loads and electric vehicle charging together with a decentralized MAS. Every appliance is an agent and fed from the main grid via the transformer. The transformer agent predicts the total load and the price for the next 24 h and sends predicted and current load and price information to each load and EV agent. Each load agent employs reinforcement learning to schedule their loads and minimize their costs, subject to the constraint of not overloading the transformer. Ref. [267] has a layered MAS with a market layer, an energy management layer and a household layer. Generator agents offer their prices; load agents evaluate and accept them. In order to minimize the total energy cost, device agents at houses utilize distributed, cooperative ant colony optimization to schedule their energy consumption.

Market mechanisms have also been used for residential demand response. Ref. [234] employs a market mechanism for demand management of a home. Load, storage and generator agents send their bids to an aggregator agent at an upper layer in the hierarchy. The aggregator compiles those bids and forwards to the central market. In the central market, an equilibrium price is calculated and returned to the agents. Shiftable load agents compute their dynamic threshold value, which depends on historical price information and decide whether to start the device or not, by comparing the market price to the threshold value.

The literature has also addressed the challenge of exploring supply side and demand side management of smart homes together in a unified framework [235,236]. In [236], home, building and the grid constitute different levels of the architecture. Both the home agent and the building agent have their demand side and supply side management algorithms. In [235], the HEMS agent is at the upper level of the MAS hierarchy and manages the SSM agent and the DSM agent. The SSM agent manages the power flow from the electrical supplies i.e., the main power grid (MG), electric vehicle (EV), solar panel and energy storage. The DSM agent controls home appliances according to their priorities.

6.2.2. Demand Management of Smartgrids

Both direct and indirect load-control approaches have been proposed for the demand management of microgrids and Smartgrids. As for direct load-control methods, Ref. [237] studies optimal scheduling and shedding of loads and storage devices in the Smartgrid. MAS consists of generation agents, load agents, energy market agents (EMAs), auxiliary energy resource management agents (AMAs) and energy storage agents (ESAs). Generator and load agents publish their day-ahead forecast data to the EMA which computes the power imbalance and estimates the market clearing price. Upon receiving the forecasted mismatch and prices, AMA performs direct load control; it computes the optimal operating schedule of storage devices and non-critical loads by a genetic algorithm. Ref. [238] develops a load-shedding method for Smartgrids which is composed of subareas. The mediator agent, which is at the highest layer, computes the total power imbalance and distributes the imbalance among the areas. The generation and load agents are at the lowest layer and communicate with their area agents. The amount of demand to shed is transmitted to the area agent who distributes it among its load agents. Ref. [239] considers demand response of prosumers in the Smartgrid, which includes renewable and non-renewable energy resources and storage devices. Decentralized MAS contains load-forecasting agents, load agents, generator agents, market agents, distribution-management system agents and demand side management (DSM) agents. The DSM agent organizes an electronic auction platform to allocate the generated renewable energy between the agents. In cases of energy shortage, the DSM agent can utilize two possible algorithms:

• The shifting algorithm schedules the loads in order to bring the total load-consumption curve close to the objective load-consumption curve, which is inversely proportional to the market price at the main grid. Based on the demand forecast and auction result, the DSM agent decides whether to defer non-critical loads to other time slots.
• The load-curtailment algorithm charges the batteries or starts the diesel generators depending on surplus or shortage of energy.

In [240], the Master agent coordinates the individual load/EV agents for shifting demand to the most convenient hours. At each iteration, the load and EV agents solve an optimization problem which involves own energy cost and a global objective to reduce the peak demand. The Master agent and the individual agents iteratively exchange information until the schedule converges.

A variety of indirect control approaches have been studied, such as price signaling, auctions, market and contracts. Ref. [241] adopts price signal to reduce the peak demand. Day-ahead demand is forecasted by a neural network using historical data. Load management agents (LMAs), zone agents (ZAs), load agents and generator agents form a hierarchical MAS. Based on the forecasted load, the LMA identifies the peak demand level and its time. The LMA solves an optimization problem to allocate the load and energy generation over time, in order to reduce the peak demand at the minimum cost. This allows the LMA agent to send the operation time to the load and generation agents via the zone agent. In [170], a network feeder agent aims to reduce its load and sends the price of available load reduction to the load-management agent. The network feeder agent receives the generation capacity and the price from the distributed generator agent and receives the demand-reduction capacity from the demand response agent. Then, the load-management agent dispatches the load reduction between different feeders based on the forecasted load, the price of each FA and the main grid. Ref. [242] proposes a signaling scheme for demand management at the substation level. There is a physical layer and a logical (communication) layer. The MAS hierarchy includes an agent for physical layer, an agent for logical layer and an agent for each house. The control unit at the substation calibrates itself and sends an on-peak message to the house agents. A house agent reacts to this signal by calculating its own power flow factor and brings its power consumption to within allowable limits.

Auction and markets are also possible methods to allocate or exchange demand. In [243], there is a decentralized market for trading demand response exchange. The market participants (aggregator agents, retailer agents, distributor agents) engage in an auction with multiple rounds. DR buyers are willing to pay for demand while DR sellers have the capacity to curtail their loads to supply demand on request. The market operator agent adjusts DR prices iteratively in a prescribed way until convergence. Ref. [244] adopts a decentralized MAS where each house is an agent. In their setting, there are multiple houses and critical and non-critical loads. They assume that outputs of photovoltaic panels and wind generators are constant over days, so forecasting is not necessary. Households buy electricity from the main grid at the market price for critical loads. Finally, they engage in auctions among each other to allocate their renewable energy production to feed the non-critical loads. The house agents use reinforcement learning to determine their bidding function.

Another approach for demand response management is offering contracts to consumer agents [245,246]. In [246], the Curtailment Service Provider (CSP) agent offers load-curtailment contracts or direct load-control contracts. In the first type of contract, consumers receive a payment for the amount of load curtailed. In the second option, the CSP can cut or reduce directly the consumer loads only with a prior notification. Consumer agents’ participation in the demand response program is voluntary and depends on the contract. Ref. [245] studies contract design for the demand response. Every contract proposal includes time periods and rewards to the consumer for up/down capacity. In this model, the aggregator agent offers contracts, then the agents engage in a cooperative game to form coalitions.

Decentralized consensus algorithms have also been applied to the demand side management [247]. Agents only communicate with their immediate neighbors. Through information exchange, they discover the total net active power for the decision making about load shedding. When an agent decides to shed its load, it signs the corresponding element of a vector and shares with other agents to achieve synchronization.

6.2.3. Electric Vehicles

Electric vehicles (EVs) are becoming more and more popular since their prices are decreasing and their fuel cost is lower compared to the conventional vehicles. In addition, electric vehicles do not emit greenhouse gases and are environmentally friendly. However, the issue with electric vehicles is how to schedule their charging times. Electric cars have large battery capacity and, during charging, they draw an electric power which is almost an average house. Due to their high power requirement, uncoordinated charging may cause peak demand which overloads the transformer and damages the electricity infrastructure [31]. Hence, coordinating electric vehicle charging is a critical problem in the near future.

Multi-agent systems have also been utilized by researchers to address the EV charging problem. These models have centralized or hierarchical MAS structure, the majority being the latter one. In the centralized models [248,249], a central coordinator agent obtains information from all EVs in each period and computes the optimum charging schedule to minimize the total cost of charging. As for optimization method, Ref. [249] uses quadratic programming while [248] uses evolutionary algorithm and linear programming. However, centralized scheduling is not practical since it needs perfect departure and arrival information from every vehicle inside the day. Furthermore, the centralized model works efficiently for a limited number of EVs and is not useful in a large fleet.

Hierarchical models have layers and agents defined by the geographical zones and the electrical distribution system. Namely, the multi-agent system includes EV agent, area/region agent, transformer agent, line/grid agent and distribution system operator agent. Most hierarchical MAS models are cooperative [250,251,252,253,254], though non-cooperative models also exist [256,257]. In cooperative models, EV agents share their information and cooperate to schedule their charging periods whereas in non-cooperative models agents decide their charging schedule on their own. Both models require tentative charging plans of EV agents across time periods to estimate the demand and take measure against overloading. Non-cooperative models utilize dynamic price signals to prevent overloading and indirectly adjust agents’ charging schedules. EV agents decide their charging time according to their state of charge and also the market price to minimize their cost. For example, Ref. [257] assumes that each EV agent knows the mass charging behavior of other EVs and they examine the Nash equilibrium of this non-cooperative game where agents determine their charging periods based on the price control. In [256], MASs have two layers of hierarchy; the upper layer is managed by the DSO agent and the lower layer is managed by the fleet operator agent. In both layers, the market mechanism is used to allocate energy across fleets and EVs. According to the market price at the lower layer, each EV computes its optimal charging schedule.

In cooperative MASs, agents coordinate and arrange their schedule according to the hierarchy. Typically, EV agents send their preliminary charging periods or preferred charging periods during the day to the aggregator agent at the upper level. Depending on the hierarchy, the aggregator agent can be the fleet operator agent, the area agent or the transformer agent. EV agents may also send their criticality or risk factors to the aggregator agent which is calculated by their current state of charge, desired level of charge, connection duration and departure time [250,252]. Then, the aggregator agent sums the charging requests and calculates the total load in each period of the day. In cases of overloading or imbalance of energy demand and supply, the aggregator agent revises the charging periods of its associated EV agents. Namely, the aggregator computes a new schedule by optimizing the total cost subject to the balance and capacity constraints. Various algorithms have been proposed for computing optimum schedules such as exhaustive search [252], reactive/proactive scheduling [254], mixed integer programming [251] and optimization toolkits [253]. Some models [252,253] introduce more than two layers and aggregation occurs at multiple levels. That is, EV loads are aggregated at the fleet/area level, fleet loads are aggregated at MV/LV substation (transformer) level, MV/LV substation loads are aggregated at LV/HV substation level and so on. In [248], EV charge demands are aggregated at a single (transformer) level, but there are multiple transformer agents at the same level. The transformer agents execute a negotiation algorithm to allocate energy among each other and then distribute among their associated EV agents. In [255], the EV demands are aggregated at the charging station level. Communication and control signals are transferred between multiple layers (Distributed System Operator, Local Cluster, EV aggregator).

An interesting application of electric vehicles is that they can be used as storage devices which feed residential loads or transfer their energy to the grid. Ref. [258] investigates combined heat and electricity consumption optimization of a building where EVs can discharge to provide energy needs of home appliances. In other words, EVs are not solely a load to charge, but they can also discharge to fulfill the energy demand of buildings. In their vehicle-to-building (V2B) model, the building energy management system (BEMS) charges EVs with priority based on their state of charge and connection duration. Ref. [260] considers a microgrid with bidirectional grid-to-vehicle and vehicle-to-grid services. The number of EVs is fixed and they are connected to the grid for a certain time interval. A decentralized consensus algorithm is proposed to schedule charging and discharging of a population of EV agents in the grid. Ref. [259] studies unidirectional vehicle-to-grid energy transfer. The authors argue that individual EVs have little power to contribute to the grid, hence in their setting EV agents form a coalition or a Virtual Power Plant to sell their energy in the electricity market. In [261], electric vehicles also form a VPP to coordinate, but they have bidirectional energy transfer. They propose EV (dis)charging algorithms for two different cases: Vehicles are managed by a central controller or they are personally managed by their owners. In the first case, the objective of the central controller is to minimize the purchased energy from the main grid. If the amount of renewable energy generation is less than non-EV loads, then the shortage of energy is obtained from electric vehicles. In the second case, the EV owner employs a mixed logit algorithm for (dis)charging decision which accounts for individual cost and benefit of charging, time spent, existing battery level and travel distance. In [262], EV agents decide their charging or feeding strategy to optimize their own benefit and cost, but they also consider the global objective. Each EV agent has a symmetric objective function which involves a penalty for overloading the transmission lines based on the charging power and the predicted non-EV demand.

Challenges and Open Problems: For supply side management, researchers have studied economic dispatch and unit commitment and developed algorithms for these problems. As for demand management, researchers have utilized demand response strategies (allocating demand across time periods), direct control (direct scheduling of loads), indirect control (price signaling, market, auction, contract) for home and microgrid environment. In addition, previous research have addressed charge scheduling of a group of electric vehicles (cooperative, noncooperative methods) and constructed models for vehicle-to-building energy transfer.

In the literature, the demand response problem of the house and the microgrid have been studied separately. The reason is that the household, as an entity, pays its own electric bill. However, from an engineering perspective, demand management of the house and the microgrid should be handled together. This can be achieved in a layered MAS structure where agents cooperate to alleviate the peak demand. Thus, one direction for future research is constructing agent-based systems and demand response programs for the house and the microgrid.

We also note that the two tasks of energy trade and demand response are closely related to each other. Therefore, we need MAS-based models which integrate energy market and demand and supply management. On their own, demand and supply management tools assume the other side is fixed and try to schedule only loads or generators. In a unified framework, generators and loads should interact with each other and they should be both scheduled dynamically. Another aspect of the problem is how to design the MAS structure to coordinate various agents in the grid and at the same time to respect geographical proximity.

Based on the fact that energy generation and energy usage of consumers are highly volatile over time periods, the demand response problem is a continuous, dynamic problem. One potential method to tackle this difficult problem is communication between agents and adaptive scheduling. Namely, agents can post messages about the change in demand and supply conditions.

As for Electric Vehicle battery charging, a decentralized MAS structure should be designed due to the nature of the problem. Existing models assume that Electric Vehicles can only be charged at home or office at a single duration. However, charging can be interleaved in space and time. To deal with this discontinuous problem, an EV agent should exchange information with other EV agents and also adaptively schedule charging on its own.

7. Restoration and Self-Recovery

During normal operation of the electric grid, faults may happen, such as a component failure, phase-to-earth contact and short circuit which affects stability and energy supply to the rest of the system. Service restoration involves fault detection, location, isolation of the faulted section from the network and restoring power to the de-energized areas. A fault in energy-distribution systems might create other faults and outages in the network; therefore, rapid restoration process is necessary for system resilience.

Automatic fault detection and prevention are not possible in the conventional grid due to the lack of communication or information transfer [268]. This leads to unanticipated power outages, thus a reliable power supply cannot be ensured to the end-users. Furthermore, the conventional large and centralized electric grid is prone to massive cascading failures in the network when a single fault occurs in the transmission and/or distribution lines. System restoration in the present electric grid is manual, the process is slow and depends on the operator’s experience. Multi-agent systems provide an effective alternative to this complex problem, thanks to their capability to communicate information and distribute tasks among agents. Thus, a major benefit of MAS for the future Smartgrid is the automated self-healing capability.

The restoration problem in Smartgrid is a constrained optimization problem whose objective is to provide electricity to as many loads in the rest of the network with available sources until the fault is amended. The objective function may include priority of loads (i.e., critical or deferrable) and the optimization is subject to the network topology, quantity of energy resources, transmission line capacity, voltage and frequency limits [269,270,271,272,273,274]. A new configuration of energy supply to the loads is found by adjusting the switches and circuit breakers. Hence, the restoration problem is essentially a combinatorial search problem to find the optimum arrangement of switches and circuit breakers. Some restoration systems aim to optimize a multi-objective function which also incorporates minimization of power loss and minimization of switching operations [269,271,275,276,277].

In the literature, various MAS and non-MAS-based automated restoration systems have been constructed, a survey of these systems is provided in [270]. Non-MAS-based restoration systems utilize centralized optimization with different techniques such as knowledge-based systems, expert systems, heuristic search, tabu search, fuzzy logic, integer programming, neural network, evolutionary algorithms and hybrid models. However, centralized optimization is computationally expensive; in addition, a central restoration scheme is susceptible to a single point of failure and not adaptive enough for a time-varying, dynamic grid [278].

Multi-agent-based restoration systems exhibit a variety of MAS architectures (

Table 5. MAS-based solutions to restoration and recovery problems.

Problem	Proposed Methods
Fault identification and restoration	Centralized [279,280], rule-based [275,278,281,282], proposal & negotiation [269,272,276,283,284], graph search [271,285], consensus algorithms [286], reinforcement learning [274,287], stochastic nogotiation [288], others [273,277,289,290,291]

), optimization methods and assumptions. The majority of these systems have a decentralized or hierarchical MAS architecture and utilize communication, information sharing and collaboration among agents for fault analysis and recovery.

In [281], fault identification and reconfiguration of the grid is rule-based. The generator and load agents collect real-time voltage and current data; the line agent detects the fault based on these data and commands the circuit breakers. Ref. [282] uses fuzzy logic for self-healing. Prevention control agents collect system data (like voltage, current, power level, erroneous component) and weather conditions and predict states that might lead to failures. Response control agents analyze the state data and determine the malfunction in the network. Then, the supervisor agent uses the state data and the evidence of faults to populate the fuzzy logic rules. Examples of other rule-based systems are [275,278].

Some MAS models involve proposal and negotiation processes for service restoration. That is, agent(s) in the faulty zone compute a reconfiguration plan and send the restoration proposal to other agents. In [283], the bus agent of the de-energized zone communicates with its adjacent bus agent to find the path to re-energize itself. The bus agent informs the facilitator agent that its associated bus can be energized by the proposal of its adjacent bus agent. Then, the facilitator agent updates the system topology accordingly. In [284], every bus agent keeps track about the power flow situation in its neighborhood. In cases of a line fault or load/supply change, the bus agents negotiate with each other and shed some loads to prevent damage. Ref. [272] creates a MARS system which also employs negotiation. Their framework includes four classes of agents: substation, feeder, branch and equipment. Feeder agents propose restoration configurations of the available power and negotiate among each other. The initiator feeder agent evaluates the proposals and chooses the configuration with the highest power. In a similar setting [276], the initiator feeder agent negotiates with other feeder agents using a contract net protocol. The initiator feeder agent sends call-for-proposal messages and the respondent feeder agents reply with their available remaining power capacity to the initiator agent. Afterwards, the initiator decides on the zone combinations making use of expert-based rules extracted from the past scenarios. In [269], the affected zone agents inform their feeder agent, who then initiates restoration and starts negotiations with its neighboring feeder agents by sending a call for proposal messages. After collecting data, this feeder agent finds a solution (local optimum) and then requests neighbor feeder and zone agents to take control actions to change the switches.

Graph-based search models have also been applied to restoration [271,285]. Ref. [271] encodes the restoration problem as a maximum feasible network flow problem for a microgrid in the connected mode or islanded mode. The electric grid is viewed as an undirected weighted graph and an improved version of the shortest argument-path (SPA) algorithm is used to solve the network flow problem. Another graph-based modeling of restoration is in [285]. The lower layer of the MAS includes DER and load agents, the upper layer includes the switch and team agents. The team agents communicate with their peers and implement a subgraph tree search to find the new network configuration.

As for other alternative approaches, Ref. [289] constructs a multi-agent system which mimics the human immune system for self-healing. The agents in MAS and their functions have a one-to-one analogy with the cells in the immunity system. The zone agent (antibody B-cell) uses the Clonal selection block algorithm to find the fault and isolate it. In the self-healing algorithm of [290], the fault-identification layer collects system data and notifies the fault-diagnosis layer. The diagnosis layer searches for the fault identification and fetches a solution from a list of fault-handling routines. Then, the corrective action layer is notified about the solution to perform.

Reinforcement learning, especially Q-learning, has also been applied to solve the restoration problem. The switch agent in [274] and the feeder agent in [287] learn the optimal policy to reconfigure the network. Heuristic algorithms [277,291], consensus algorithms [286] and expert systems [273] with decentralized MAS structure have also been utilized for fault detection, isolation and network reconfiguration.

Note that decentralized restoration approaches may yield local optima rather than global optimum. In order to escape from suboptimal configurations, as a partial remedy, Ref. [288] offers stochastic negotiation which involve probabilistic expected utility. In their framework, feeder agents represent a distributed energy source which has the ability to restore a blackout region and zone agents represent a load zone which demands power be restored. Feeder agents negotiate between each other and a feeder agent proposes candidate target zone to deliver the power. Agents evaluate whether to accept the proposal of another agent by using two alternative stochastic decision functions. This creates the possibility of avoiding the trap of a local optima. Another attempt to reach a global optimum is by dynamic team forming or reforming algorithm in [292]. Bus agents in the proactive and reactive layer try to restore the physical bus by some predetermined countermeasure action plans. If the countermeasure plans are not successful for restoration, i.e., the fault is serious, then the bus agents delegate restoration to the social layer. The coordination agents in the social layer form a team to restore the bus locally. The team size can be modified according to the complexity of the fault.

Centralized MAS systems for restoration also exist. Ref. [279] offers a centralized MAS structure which actuates an auto-reclosurer algorithm in order to restore power to the unaffected parts of the grid. This algorithm distinguishes a temporary fault from a permanent fault by performing multiple checks for the presence of the fault at several periods. In [280], the microgrid operator agent (MOPA) collects data from equipment and measurement agents such as current, voltage and impedance to perform fault analysis. Based on these data, MOPA computes the final network configuration and the protection configuration and then sends the configurations to the switcher and protection relay agents.

Some studies restrict attention to correcting one type of fault like phase-to-earth-contact [293]. An identification method dedicated to this special type of fault is developed. Node agents collect transient reactive power data and judge whether the feeder associated with this node is faulty or not. The control agent receives the fault information from the node agent, sends control commands to the node agents and starts the restoration process.

Until now, all restoration systems assume that there is only single fault in the network. Refs. [294,295] deals with correcting multiple faults in the grid. In these approaches, each fault is considered as a single fault and the solutions of individual faults are combined to have the final reconfiguration. In [295], MAS is comprised of a management and execution layer; the management layer is responsible for the recovery process. In [294], fault detection is performed in a decentralized manner by the local agents (load, switch agents) whereas the fault reconfiguration is performed by the global agent in a centralized fashion. The reconfiguration algorithm of the global agent searches for the optimum switching combination.

Challenges and Open Problems: Previous works have studied fault detection, isolation and reconfiguration problems and developed methods explained above: Rule-based, fuzzy logic, proposal and negotiation, graph search, human immune system, reinforcement learning, stochastic negotiation, team formation and other algorithms.

Nevertheless more robust and effective restoration methods are required to deal with faults in Smartgrid. Accidents and failures can occur anytime and anywhere in the grid; hence, fault detection requires continuous monitoring of the transmission line and electrical elements. Ideally, a recovery plan should be developed before the faults occur. One benefit of multi-agent systems for fault detection and recovery is local knowledge. Intelligent and autonomous agents know the state of the neighbor agents and the local grid. Hence, nearby agents can identify the fault and its location rapidly by sensing electrical values and checking the statuses of each other. Moreover, small faults can be handled easily at the local level without the need for large scale optimization. This feature of MAS systems has not been utilized to a great extent for restoration purposes in the literature.

The existing restoration methods detect the fault and compute the optimum network configuration in real time (online), which depends on the location and type of the fault. Observe that the restoration problem does not have to be solved in real time; consequences of each possible fault with a specific location and type are known beforehand; hence, we can construct predefined logical rules and lookup tables to detect the fault, based on abnormal electrical and spatial data. In a similar fashion, the optimal network configuration can be precomputed for each possible fault location and type. The precomputation can be performed by the system operator in the meta level; hence, a global optimum can be achieved. Thus, local agents can pinpoint the location of the fault and then realize the predefined restoration plans accordingly. How to implement these logical rules with an agent-based system in a centralized or decentralized manner is part of future research.

Interestingly, previous research has not considered demand response or deferring loads during fault and restoration. In our opinion, this is a critical issue because outage of energy (at any time of the day) should also shape agents’ consumption profile and load scheduling. In particular, non-critical load agents can shift their energy consumption to later periods after recovery. Other agents can also reschedule their demand over time to match available power by negotiating with each other. In cases of shortage of power or urgency, non-renewable resources like diesel generators can be operated to feed the critical loads or part of the affected zones. Note that for such a demand-management program, all agents in the microgrid must be informed about the existence and location of the fault. This can be accomplished by posting the fault event to a public bulletin which every agent checks regularly. Thus, another open problem is how to notify agents and how to determine and reschedule their energy-consumption profile upon a fault or outage in the system.

The case of multiple faults inside the network has not been studied in sufficient detail by the existing research. In particular, determining whether there is a single fault or multiple faults may require more sophisticated algorithms which include testing various electrical conditions and checking signals and values. Note that those multiple faults are likely to be at nearby locations; thus, neighbor agents should collaborate on detecting and distinguishing the faults. Then, the problem is developing algorithms to coordinate the agents for fault detection.

8. Protection and Security

A Smartgrid is a cyber-physical system which involves energy and information transmission, and communication and software applications; moreover, it performs computationally intensive operations. Hence, a Smartgrid is highly susceptible to both physical and cyber attacks. Physical threats include terrorism, natural disasters and industrial catastrophes. Cyber threats include viruses, malicious software, false data injection, manipulation of components, denial of service, theft of private and financial data and attacks against an agent or an application. Most agent-based systems designed for Smartgrids use TCP/IP and the Internet for communication which increases the risk and vulnerability to cyber-attacks. Due to the multi-layered structure, the scope of cyber attacks can be at the user, component, operating system, protocol or network level. The main responsibilities of a security system are to detect the location and type of the attack, isolate the threat, deploy preventive measures and coordinate among units and agents for stronger resilience, relying on records of attacks and learning to adapt and improve the security system for future threats. More detailed information about types of attacks and security issues in energy-distribution networks can be found in [296,297].

Several security guidelines and requirements for Smartgrids have been developed; an overview is provided by [298,299]. Some key guidelines are the European Network and Information Security Agency (ENISA) Smartgrid Security report [300], the ENISA guidelines for security measures [301] and the U.S. National Institute of Standards and Technology report NISTIR-7628 [302]. The International Organization for Standardization ISO/IEC TR 27019 report [303] provides security guidelines based on ISO/IEC 27002 for process-control systems specific to the energy industry.

Note that there is always a trade-off between the extent of the security measures and the computational overhead associated with it [304]. Therefore, installing security components should be a wise decision which does not hamper the real-time operation.

In the security literature, it is well recognized that central management systems are more vulnerable to cyber threats [56,58,59,60]. The primary concern is that the central controller is an ideal target for a malicious attack; compromising just this agent will make the whole system totally ineffective. On the other hand, a decentralized system is more resilient to attacks thanks to its distributed intelligence, labor division and coordination. For this reason, many Smartgrid-protection and security systems adopt decentralized MAS architectures.

In the literature, multi-agent systems have been used to provide solutions to data encryption and authentication, user privacy, threat detection, intrusion prevention and system protection in the Smartgrid (

Table 6. MAS-based solutions to security and protection of Smartgrids.

Problem	Proposed Methods
Data encryption and authentication	Public key infrastructure and digital certificate [304]
System protection and monitoring	Structural measures [305], human immune system [306]
Data privacy	Conditions on communication order and content [307]
Intrusion attacks	Statistical anomaly detection [308,309,310], machine learning [311,312], consensus algorithms [312,313], trust-based filtering [314,315,316]
False and malicious data	Monitor state variables and control signals [317,318]

). Researchers have utilized different measures and policies to deal with the above security problems. For data encryption and authentication, Ref. [304] proposes a public key infrastructure (PKI) for the microgrid. The protocol has a certificate-issuing process and a certificate-authentication process. The certificate authority (CA) agent issues digital certificates to the user/device agents in the microgrid. The validation authority (VA) agent verifies the validity of the certificate. By issuing and validating digital certificates, an agent can authenticate the originality of the messages sent by another agent.

Ref. [305] investigates upper-level system protection and monitoring of Smartgrids. Ref. [305] takes structural measures to prevent cyber attacks. The input and output format of the data is specified for each agent and the tasks of agents are sorted in a queue. Their approach uses continuous observation of the overall system. MAS is hierarchical and involves a functional layer and a logical layer. The functional layer is responsible for protection and task execution. Ref. [306] constructs an artificial immune system which mimics the human immune system. There are monitoring and forecasting agents which can learn, predict and evaluate risks and detect dangerous deviations from the model state.

Ref. [307] studies protection of user data privacy. In their setting, prosumer agents communicate the quantity of energy demand, supply and price information with the energy management system (EMS) agent. To ensure privacy of prosumers, the authors impose two conditions: no prosumer can enter into communication channels of another prosumer and no prosumer can infer the consumption plan of any other prosumer from his received information. To satisfy the first requirement, they take structural measures such that the channels of prosumer agents are separated and agents cannot directly communicate with one another. For the second requirement, the authors impose some constraints on the content of the data sent by each prosumer.

There is also a great deal of work that tackles the intrusion attacks and false data injection [308,309,310,311,312,313,314,315,316]. In these models, agents collect local data and exchange them with their peers; the reported data from agents are inspected to detect attacks. Consequently, most of these intrusion-prevention models have a decentralized MAS structure, but some hierarchical MASs also exist [309]. The methods of attack detection are statistical data analysis (anomaly or irregularity) [308,309,310], machine learning [311,312], consensus theory [312,313] and trust-based filtering [314,315,316]. Among statistical data-analysis models, in [309], the master agent (overseer) collects measurement of state variables from each agent and identifies an anomaly, i.e., a compromised agent, using an algorithm which accounts for the majority vote. Ref. [308] utilizes descriptive statistics to detect anomalies. First, energy flow data are gathered from agents in the grid and the key characteristics of the data are determined. Then, the profiling agent monitors the real-time incoming data in the grid and detects an irregularity. Ref. [310] combines anomaly-based data analysis with model-based probabilistic techniques for intrusion detection. MAS architecture is composed of three layers: power, control and security. The security agents identify an intrusive event by collecting data about system variables, CPU, memory usage and communication traffic.

Trust-based filtering methods assign a trust metric to each agent and data coming from an agent are respected according to their trust value. Ref. [315] uses trust-based filtering for robust estimation of state variables such as voltage magnitudes, angles, transformer ratios and power flows. In their setting, every agent has a local estimate of the state variable and updates the trust value of a neighbor agent based on the accuracy of the estimate reported by that agent. The value of the state variable is calculated using the weighted estimates of agents, where the weights correspond to the trust values. Ref. [316] also employs accuracy of the reported state variable to update the trust value of an agent. The authors also propose an additional data-retransmission scheme to combat cyber attacks: every load and generator agent regularly transmits its set of previous observations (a digest update). Control agents examine the history contained in the digest updates to reconstruct past system states in order to maintain the situational awareness. Trust and data-retransmission schemes can be combined. In [314], households, as prosumers, trade energy with each other via blockchain technology and each household agent has a trust value. Agents evaluate the trust of other agents and declare these trust values to their respective aggregator agents. Both the evaluator and the evaluated agent are judged based on their honesty. They have a repeated game theory setting and a defaulter agent may regain its trust by reporting correct evaluations in the consecutive periods. Another trust-based security mechanism for Smartgrids that uses blockchain technology is developed by [319].

Consensus theory can be also used for analyzing suspicious data flows and infection as in [313]. In their ELVIRA framework, an agent iterates the state variable using the data coming from its neighbors and the expression converges to the truth asymptotically. Hence, the agent can compare this value to the collected data in order to identify a compromised agent.

Machine learning approaches have been devised to detect data anomaly and intrusion attacks. Ref. [312] considers anomaly detection as a multi-class classification problem and implements a supervised learning algorithm, named Support Vector Machine embedded Layered Decision Tree. The aim is to prevent Denial of Service (DoS) attacks which target the load-shedding scheme. The status of the power system is classified into five different modes. Every agent broadcasts its load profile and an adaptive load-rejection strategy which considers historical records is realized. Ref. [311] has rule-based learning for threat detection. An agent can learn and memorize cyber-attack alert rules from historical observations. The agent becomes skeptical when the incoming data violate these rules. Whenever an alert rule is triggered, the agent sends out a data inquiry to find out the real system status.

False data injection is a type of intrusion where the adversary tries to compromise a component or alters a state/control variable. In particular, the attacker may inject false values or modify a control signal, which causes undesired opening of a circuit breaker. Ref. [317] provides algorithms for the identification of false data injection attacks and the identification of alert manipulation attacks. The network is divided into multiple subsystems which include a Phasor Measurement Unit (PMU) agent, Detection and Identification (DI) agent and Dynamic Control (DC) agent. To mitigate an attack on PMU agents, DI agents perform a sensitivity analysis which checks the deviation of line current and voltage magnitudes and monitors the circuit breaker status. Ref. [318] designs a Breaker Supervision Agent (BSA) to prevent malicious false tripping of circuit breakers which cause power outages. In a decentralized MAS structure, agents measure the line current values and breaker states to identify an out-of-service line and subsequently inform the BSA agent about the outage. Then, the BSA agent checks the node voltages, relay currents and control signals to identify a malicious trip command.

Challenges and Open Problems: Researchers have addressed protection and security problems in the Smartgrid and developed various MAS-based solutions for system monitoring, data privacy and encryption, intrusion, attack detection and prevention, malicious data flow, denial of service, false data injection and anomaly detection. Formal guidelines and requirement documents for Smartgrid protection and security have also been created.

As explained in the text, decentralized security management is more robust compared to a centralized security system. But then the imminent question is how to maintain overall security and protection in a decentralized security system. In particular, when an agent faces an attack, how does it notify the whole grid about the existence of the attack and how do agents make a joint effort to remove the threat? To deal with these challenges, emergency situation action plans and communication mechanisms which are suitable for the MAS structures should be prepared beforehand. In a hierarchical MAS, the information should propagate both upward and downward, while in a decentralized MAS information should propagate towards each neighbor. To cope with false alarms, authentication of the source and verification from multiple agents can be implemented.

To this end, we should note that the existing security and protection systems are not truly decentralized: there is still a central coordinator or overseer agent who is susceptible to attack. One method to handle this issue in a complex system is to partition the Smartgrid into zones and assign the security duty to the zone agents. If the MAS system is layered, an agent at each layer can be assigned to monitor the security in its own region. A Smartgrid will become more resistant to attacks when multiple agents carry out the protection task.

As for information safety, additional measures other than encryption and authentication can be implemented. For example, agents should agree on certain time points to perform periodic communication. In addition, an agent should be allowed to send only a set of variables depending on its status and privileges in the grid. Information received outside of agreed time points or privileges are classified as unauthorized/alien and ignored.

Encryption, authentication and security protocols for Smartgrid require further research and investigation. A Smartgrid is a large-scale network; thus, whether all agents and all zones will use the same encryption, decryption and authentication algorithm is an issue. One potential solution might be agents inside a microgrid use the same encryption algorithm; communication across microgrids employs a different one. Another issue to resolve is the scope of data privacy inside the hierarchy. In particular, should the agents in an apartment share their data with other apartments in the building?

Previous research has concentrated on statistical analysis or trust score to detect anomalies. In an autonomous, distributed system like a Smartgrid, individual agents should be endowed with intelligence and reasoning to protect them against various malicious attacks, in order to strengthen the whole system. An agent should have the capability to monitor the state, possess predefined safety rules, reason with these rules and learn new rules from experiences. For this, each agent must have a knowledge base which involves information about his own state, nearby agents and the environment. Furthermore, neighbor agents should periodically exchange data with each other to check whether any of them have been compromised or not. Developing a robust, agent-based, intelligent protection system is a promising direction for further research.

9. Simulation and Implementation

Simulation studies in the literature have tested their Smartgrid model on a virtual environment in computer with MAS platform and simulation software (Table 7). Implementation projects have physically deployed their MAS and Smartgrid model on a real site to perform as an energy management system. Various attempts at simulation and implemention of Smartgrids have been made and these simulated and implemented systems vary with respect to scale, platform and functionality. We go over the simulation and implemention projects of Smartgrids in this section.

9.1. Simulation

It is necessary to simulate a control system to assess its robustness and effectiveness before physically deploying it to the Smartgrid. Simulation is also beneficial to prevent physical damage and costs in cases of unintended consequences. Each of the papers reviewed in the previous sections has performed simulation or test of their proposed model, typically in Matlab or Simulink environment. Other simulation platforms specific to multi-agent systems such as Symphony [320], Mosaik [321], OPAL-RT [80] have also been used.

Table 7. MAS-based simulation and implementation of Smartgrids.

Problem	Proposed Methods
Generic simulation platform	MACSim [322], MACSimJX [323], Mac-Sim [324], MECSYCO [325], MASGriP [326,327], co-simulation [285,328]
Simulating special function of Smartgrid	Energy markets (MASCEM [329], EMCAS [330], AMES [331], GAPEX [332], Power TAC [333,334]), security [335], demand response [80], restoration [285]
Implementation	Physical installation [76,77,78,88,154,336,337,338,339,340], laboratory facilities [79,154,221,341,342], hardware-in-the-loop [343,344]

Table 7. MAS-based simulation and implementation of Smartgrids.

Problem	Proposed Methods
Generic simulation platform	MACSim [322], MACSimJX [323], Mac-Sim [324], MECSYCO [325], MASGriP [326,327], co-simulation [285,328]
Simulating special function of Smartgrid	Energy markets (MASCEM [329], EMCAS [330], AMES [331], GAPEX [332], Power TAC [333,334]), security [335], demand response [80], restoration [285]
Implementation	Physical installation [76,77,78,88,154,336,337,338,339,340], laboratory facilities [79,154,221,341,342], hardware-in-the-loop [343,344]

9.1.1. MAS-Based Generic Simulation Platforms

Researchers have also constructed generic as well as special purpose MAS-based simulation platforms for Smartgrids. Among the generic platforms, MACSim (Multi-Agent Control for Simulink program) [322] is a medium where a Java or C/C++ program which has agent-based design can transfer data with Simulink. MACSim has a client–server architecture, where the client part is embedded in Simulink and the server is in the main Java or C/C++ program which works in a multi-threaded fashion. MACSim has been extended to MACSimJX [323] by incorporating JADE to the server. Hence, MACSimJX allows the use of a generic MAS platform and MAS designs in this platform to be embedded into simulation. In the MACSimJX framework, JADE can exchange data and messages with Simulink through the MACSim interface. MACSimJX has been applied to control of microgrids [85,87,345].

Some of the simulation platforms are actually co-simulation platforms; namely, they are composed of several simulators or software tools. For example, Mac-Sim (Multi-Agent and Communication Simulator) [324] includes three components: JADE for MAS framework, Opnet for network simulation framework and a run-time infrastructure which handles messaging, object management, time synchronization and acts a mediator between JADE and Opnet. Note that though the names are similar, MacSim and Mac-Sim are different simulation platforms with different structure. Ref. [285] created a co-simulation platform which has one more component, PSCAD/EMTDC, to integrate power system simulations. Ref. [328] also created another co-simulation platform which involves MAS, power and network domains, but this platform uses Powerfactory for power simulator and OmNET++ for network simulator.

Ref. [325] addresses a more generic and broader view of co-simulation: multi-model simulation. Their MECSYCO platform can integrate arbitrary tools and simulators by using the MAS paradigm. Each tool or simulator is an individual agent of a decentralized MAS. These tools/simulators interact and exchange messages via the communication network OMNet++.

MASGriP [326,327] is an agent-based Smartgrid simulation platform which is, by design, integrated into another simulator MASCEM for Competitive Electricity Markets. In this manner, agents can negotiate in the electricity market. Its MAS architecture includes base agents as well as two types of aggregator agents, the Virtual Power Plants (VPPs) and the Curtailment Service Providers (CSPs). A laboratory experiment of MASGriP has been performed to test energy management and demand response of buildings [346]. MARTINE [347] is MAS-based simulation infrastructure for energy management of smart buildings and microgrids. MARTINE has decision support, optimization, negotiation and reinforcement learning capabilities. It has four layers: real-time simulation, building, MAS and decision making.

9.1.2. Specialized Simulation Platforms

There are multi-agent simulators specially designed for energy trade markets such as MASCEM [329], EMCAS [330], AMES [331] and GAPEX [332]. MASCEM (Multi-Agent System for Competitive Electricity Markets) is FIPA-compliant and can cooperate with other MAS societies via ontologies. MASCEM can simulate popular market models like day-ahead pool, bilateral contracts, balancing market, forward market and hybrid markets. Its MAS architecture contains five types of agents: main agent, management information base agent, market operator agent, system operator agent and player agent. EMCAS (Electricity Market Complex Adaptive System) allows dynamic and adaptive agent strategies in the market. The framework has three components: agents, interaction layers and planning periods. The types of agents in EMCAS are generation companies, demand companies, transmission companies, distribution companies, independent system operators, consumers and regulators. In EMCAS, agents are free to establish their own objective functions and apply their decision rules. Using exploration-based learning, agents explore entirely new market strategies and observe the results of their actions. Hence, agents can learn from their previous experiences and change their future behavior.

AMES (Agent-based Modeling of Electricity Systems) is oriented towards the study of the U.S. wholesale electric market in accordance with the Federal Energy Regulatory Commission (FERC) market design. Originally, AMES was designed for research and teaching purposes rather than commercial-grade application. Its multi-agent system includes an independent system operator agent (ISO), load-serving agents and generation companies (GenCos), which are distributed across the buses of the transmission grid. AMES supports stochastic reinforcement learning algorithms for generation companies; it facilitates the augmentation of the empirical input data with the simulated input data. GAPEX (Genoa Artificial Power Exchange) can simulate market-clearing procedures of most European power-exchange markets. GAPEX is developed in Matlab, it supports several market mechanisms and machine learning for agents and it has a statistical analysis module. Its agent concept is abstract and can range from a simple reactive agent to a more sophisticated cognitive agent.

In the scope of energy market simulation, we should also mention a relevant initiative called Power TAC (Trading Agents Competition) which is organized annually since 2012. Power TAC provides a simulation platform with many energy producer, consumer agents and the broker agents (competitors). The energy market, suppliers, customers and the distribution utility is modeled by the Power TAC simulation platform [333,334]. Research groups prepare their own software and enter the competition as broker agents. The task of the broker agent is to design and offer tariff contracts to consumers and producers in order to allocate the energy among the agents in the Smartgrid. The consumers of electricity are the households, enterprises and owners of electric vehicles. A tariff specifies flat prices, time-dependent prices, peak prices, load caps for certain periods of the day, contract duration, signup bonuses, early withdrawal penalties, etc. Thus, the role of the broker agent is to mediate the flow of electricity between energy producers and consumers. The broker who achieves the most profit over a range of scenarios is the winner of the competition. Note that Power TAC organizers can also set some social welfare objectives such as fairness, utilization of renewable resources and emissions. Hence, the market designers can create incentive mechanisms in addition to profit maximization, in order to attain socially desirable outcomes [348].

In addition to energy markets, simulation tools for other functions of the Smartgrid have also been developed. Ref. [335] has created a simulation system for multi-agent protection and security. The simulation framework consists of Simulink and JADE; the interfacing agents mediate the data exchange between the two. Their multi-agent protection system (MAPS) includes coordination agent, fault detection agent and communication agent. Ref. [80] creates a microgrid simulation environment for demand response of agents and the evolution of price. These agents interact with the real physical installations, OPAL-RT is used to simulate resources that are not physically available. The co-simulation platform of [285] tests a restoration and network-reconfiguration problem by a fault scenario.

9.2. Implementation

The ultimate objective is the implementation of the Smartgrid infrastructure: deploying the necessary and well-functioning physical components and software to operate the network. Aside from simulation, another major concern is how the above proposed control mechanisms behave in a real, physical context. Some models have been physically implemented on a real Smartgrid in a campus or a pilot location; others are tested in a laboratory environment or hardware-in-the-loop.

9.2.1. On-Site Implementation

Refs. [77,78] designed and installed a microgrid controller for an isolated Greek island Kythnos as well as a test site in Athens. Household agents have renewable energy resources, batteries, water pump and other loads. An Intelligent Load Controller (ILC) agent monitors the power line and measures the voltage and current values. ILC has an algorithm whose main objective is to control the operation of non-critical loads, in particular the water pump depending on the available energy in the batteries. Load shedding is equally divided among all houses in the settlement.

An open-source multi-agent system for residential homes and businesses is deployed at trial test locations in Australia [88]. MAS agents are implemented on personal computers and personal digital assistants. Prosumers actively participate in the market and trade energy. DER agents communicate through the bulletin board, they can post and query topics in the bulletin. Ref. [154] installed a restoration system on an urban electric grid in Jiangning County, China. The grid at the pilot site is comprised of three substations, five branches and six transformers. The system has five operating states and four subcontrols: emergency control, restorative control, corrective control and preventive control. The entire self-healing functionality is controlled by a multi-agent system consisting of response layer, coordination layer, organization layer.

Some implemented Smartgrid management systems include a microcontroller device like Arduino, which receives input and gives output from/to grid elements, sensors and actuators. Ref. [336] presents a deployment of an MAS-based controller on Arduino for a rural Indian microgrid. The control system has a simple, compact design in order to be low cost and affordable. Agents are hierarchically grouped into three classes: distributed grid agents, micro agents and D agent (cloud or computer). Ref. [337] creates a controller on Arduino for a Smartgrid test bed where three microgrids are interconnected to the main grid. Each microgrid has its own photovoltaic panel, wind turbine, diesel generator, battery, critical and non-critical load units. The controller agent of a microgrid handles the energy transfer between its own elements as well as the energy transfer from/to other microgrids and the main grid. Another Arduino-based Smartgrid management system is implemented by [349] for two interconnected microgrids, one in a university department and the other in a hostel. The grid agent and the controller agent manage the energy transfer between the (non-)renewable resources, battery units, loads. In the experiments, the agents on the controller receive input from Simulink (through MACSimJX interface), instead of physical hardware. Ref. [338] deploys a microgrid management system to a commercial building with 16 offices and 40 photovoltaic panels. Each zone is managed by a unique mGIM agent and agents run on Raspberry Pi boards. Each agent is equipped with an hour-ahead forecasting algorithm and agents can trade energy by peer-to-peer local auctions. The aim is to implement an architecture for end-user representation and a light-weight solution that can be deployed on a single board computer. In [339], a self-healing microgrid system is implemented. A test bed is assembled which consists of a physical DC electric grid, Arduino microcontrollers and Raspberry Pi computers. The agent software is hosted and executed on the Raspberry Pi and its physical interconnection to the electric grid is achieved by the Arduino microcontroller. The load control agent and the restoration agent together execute the Prim’s minimum spanning tree algorithm to solve the service-restoration problem.

Some universities and research groups have installed Smartgrid management systems on their campuses. Illinois Institute of Technology has initiated the Perfect Power Smartgrid project [76] upon many power outages at its campus between 2004 and 2006. The goal of the project is peak load reduction on distribution feeders by on-site distributed energy resources and energy management systems at costs competitive with the system/capacity upgrades. Distributed resources can reduce peak demand and decrease total energy costs. The “Advanced Distribution Automation and Recovery System” was implemented in 2013 and it achieved %50 peak load reduction, uninterruptible power to critical facilities and resilience to a single point of failure. Its MAS architecture is composed of a group of teams. The agents within a team communicate with each other, while a team negotiates with other teams by its mates. Ref. [340] presents a Multi-Agent Management System (MAMS) for the microgrid deployed at UADY Engineering Faculty. MAMS regulates the energy consumption of the microgrid and the energy transfer from the main grid in order to minimize the total cost and at the same time maintain the microgrid energy balance. MAMS is composed of seven types of agents, solar- and wind-generation agent, battery bank agent, electric vehicle agent, public grid agent and critical and ordinary load agents. MAMS performs power scheduling of the loads based on the solar and wind output prediction from a neural network.

9.2.2. Implementation in Laboratory

Some authors implement and test their system under laboratory facilities [79,154,221,341,342]. Ref. [79] deploys a microgrid management system in a laboratory with a diesel generator, battery banks and controllable loads. They used real, physical equipment in their experiments. The effectiveness of their load shifting and curtailment algorithms was tested under different microgrid configurations. Ref. [221] realizes the decentralized consensus protocol for the economic dispatch problem (scheduling of energy generators) in a microgrid testbed. The controller runs on a dSpace platform with control desk software. The setting includes three physical inverters and two resistive load units. A multi-agent system for microgrid control is tested at the laboratory microgrid of National Technical University, Athens [154]. The system handles energy trade by auction and transition between the connected mode and the islanded mode. The setup contains a photovoltaic panel, battery bank and loads. The control system is implemented as software on a personal computer. Another multi-agent system for control and monitoring is implemented on a laboratory Smartgrid test bed at Florida International University [341]. The framework consists of renewable and non-renewable generation units, loads, transmission line models, field sensors and actuators. The agent platform is implemented on a personal computer; an OPC UA server handles the measurements and the information transfer. The area power system operator agent and the DER agent are responsible for reducing excessive energy demand and overloading during peak hours. In another project at the same university, an MAS-based communication-assisted fault localization, isolation and restoration method has been tested on their test bed [342].

9.2.3. Hardware-in-the-Loop

An alternative to the physical experimentation is hardware-in-the-loop simulation where the controller or management system is implemented on physical hardware in the laboratory, but some peripheral devices (generator, storage, load, sensor) run on a simulator like Opal-RT and their input/output are obtained from the simulator. For example, Ref. [343] performed a laboratory test of a fault-isolation and restoration system. The MAS agents are designed according to the Human Immune System cells for feeder fault location, isolation and self-healing. Agents run on electronic boards with microprocessors and a three-phase fault experiment is conducted in the Analogue Power Simulator hardware. Ref. [344] designs an MAS-based microgrid control system for an emergency demand response (EDR) program and conducts its hardware-in-the-loop simulation in the laboratory. Agents are implemented on microcontrollers and the real-time data from generators and loads are obtained from the Opal-RT system. The microgrid contains two DERs, a micro-gas turbine (MGT), a battery energy-storage system (BESS) and a load. In cases of a power shortage, the main grid operator requests emergency demand response for the microgrids. The microgrid central coordinator agent (MGCC) decides to join the EDR request depending on the energy status of its microgrid. The MGCC agent uses Contract Net Protocol for participation of the generator and load agents in the demand response.

Challenges and Open Problems: Most researchers actually created their own model in Matlab, Simulink and JADE to simulate their system. In addition, researchers have also constructed a number of simulation platforms for testing Smartgrid operation. Some of the existing simulation platforms for Smartgrids are designed as a generic agent-based system while others are oriented towards simulating a specific function such as energy markets, security, demand response and restoration. In this scope, energy market simulations have received special attention.

For some aspects of Smartgrids such as security or restoration, more simulation studies and experiments are required to understand the behavior and the issues in the grid. In addition to security and restoration, some other functions like supply side management and EV charging have not been tested with the generic MAS-based simulation platforms. Thus, the effectiveness of these generic simulation platforms for all functions of Smartgrids must be verified.

As for implementation, physical Smartgrid systems have been installed at pilot locations or at laboratory facilities. Nonetheless, these physically implemented systems so far are small scale, e.g., building, business, campus. Moreover, energy trade between houses or microgrids has not been implemented. An open problem is whether these models would work on a large scale such as a district or town. Researchers should investigate whether an MAS-based framework is efficient for managing Smartgrids and examine the problems encountered in practice.

Physical MAS systems deployed at trial locations focus on a single functionality of the grid e.g., control, demand response or restoration. Furthermore, some of these implementations handle the main tasks of the Smartgrid by a microcontroller or software, rather than autonomous agents. To construct a self-operating Smartgrid, ideally we require an MAS-based setting that can handle all functions including control, trade, demand/supply management, restoration and security. For this purpose, future studies should utilize independent, intelligent agents working and deciding on their own. Hence, one of the focuses of implementation should be designing agent-based systems and testing various aspects of the grid for robustness.

10. Discussion and Meta-Level Analysis

At the end of this survey, what are the final conclusions? Each of the papers reviewed in the preceding sections tackles a problem in a particular field of Smartgrids. The MAS architectures and methods utilized in these papers are also different: they use different types of agents, MAS layers and organization. As the reader will have noticed, there is no universal model that can handle all areas of Smartgrids. Even the papers that address the same problem use different configurations of grid elements, energy resources, load profiles and operation modes. Thus, an important challenge is how to construct a single MAS structure or integrate the above models in order to obtain a robust and well-functioning Smartgrid. One solution might be combining different functionalities into a single layer or agent. For example, control, demand scheduling, local recovery and security monitoring duties can be assigned to the zone agent. A similar argument can also be made for various optimization tasks: how to optimize market price, energy consumption, load/generator scheduling, restoration together and which agents will be responsible?

We also note that some models utilize sophisticated methods and/or MAS structures, especially the auction and demand response programs. This stems from the difficulty in energy management of the Smartgrid and also from the usual tradeoff between efficiency and simplicity. A complex framework can utilize and allocate resources close to optimum, but tends to be less flexible and hard to implement. Thus, designing MAS systems that are both efficient and practical for Smartgrids is a promising direction for research.

Some of the proposed systems in the literature are not truly agent-based. In these systems, most agents are simple electronic components which perform the instructions sent by the main software or the central controller. That is, the relationship is basically master–slave and these agents are not really autonomous or intelligent. In our opinion, the core ingredient of MASs is independent and self-operating agents.

As discussed in the introduction, the main paradigm of Smartgrids is energy management, with consideration for the weather-dependent nature of renewable energy resources. The problem of Smartgrids is essentially balancing the demand and supply, utilizing available resources to the utmost extent and scheduling/reshaping the demand. Numerous models of energy trade, control and demand response have been proposed in the literature. These models tend to be heuristic or ad hoc, tailored for a particular setting. Authors have tested their models on their own in a simulation environment. However, it is unknown how these different models proposed by authors compare to each other in terms of efficiency and reliability. In addition, MAS-based solutions should also be compared to non-MAS-based solutions to figure out whether MAS is a proper framework for Smartgrids. In this respect, part of the effort in future research should focus on preparing benchmark problem instances for Smartgrids to evaluate the proposed models.

Another challenge of Smartgrids is their geographical and spatial aspects. It is relatively easy to assign a microgrid and design an MAS structure for an isolated, small region such as a campus, village or military camp. However, for large, continuous residential settlements in urban areas, it is difficult to implement Smartgrids in the whole territory and partition them into virtual microgrids or Virtual Power Plants. Moreover, each VPP may have a different size and configuration; therefore, how to construct an MAS design generic enough for VPPs is a problem to deal with.

The existing frameworks in the literature consider a microgrid with fixed size and arrangement of resources and loads. Their MAS structure, hierarchy and types of agents are designed for this certain microgrid size and element. However, as cities grow, we expect new districts and regions to join the electric grid. Therefore, how to adapt the microgrid and the corresponding MAS structure for an enlarging network is another issue. To cope with this, the multi-agent system for the microgrid should have alternative designs and flexible topology, layer and zone architecture. The number and type of agents can be adaptive depending on the scale of the Smartgrid.

Due to digitization and advancement of information technologies, energy demand of data centers, cloud computing and website hostings have been growing every year [350,351,352,353,354]. These data and computing centers constitute concentrated, heavy energy-consumption points, which is a challenge for conventional electric production and distribution networks. Though there are preliminary works [355,356,357,358,359] in the literature, whether (or how) Smartgrid, renewable energy and multi-agent system technology can be used to fulfill the high energy need of these information-processing centers is an open direction for future research.

10.1. Review of Challenges and Open Research Problems

This section presents a sum-up and upper-level review of challenges and open problems mentioned in previous sections to create an entire agenda of key issues for Smartgrid research, as listed below.

• Since centralized control is computationally heavy and prone to single-point-of-failure, we need truly decentralized models for Smartgrid control, energy management and security. Local knowledge of distributed agents and local solutions would be both a simple and an effective way to deal with the above problems.
• Then, a related problem is how to achieve overall coordination of agents in a decentralized system against power outages, faults, overloading and security attacks. For this purpose, collaborative ready-to-use strategies, emergency action plans and communication protocols must be developed.
• Agents in Smartgrid system should possess intelligence and reasoning capabilities to detect abnormal events, perform action planning and collaboration.
• There is also need for communication and information exchange mechanisms between agents in order to enhance energy trade, security, restoration, demand and supply management.
• Since Smartgrid control, energy trade, demand and supply management are closely related to each other, we need a unified framework to handle these functions using effective yet practical algorithms.
• A prominent problem is embedding network hierarchy and geographical proximity into energy trade and allocation, as these are critical factors for convenient energy distribution and reducing transmission loss. In particular, energy sharing inside the same building or layer should be studied.
• Another issue is how to perform energy management and trade when actual supply and demand differ from forecasted values, namely actual renewable energy production is less than demand. In this situation, alternative strategies, secondary markets, rescheduling of loads and non-renewable energy resources can be utilized.
• As demand management involves all loads in the grid, a major challenge is how to integrate demand response programs of house, building and microgrid. From home appliances to vehicles and plants, alleviating peak demand constitutes a complex, hierarchical problem to deal with.
• Spatial and temporal reasoning should be utilized in electric vehicle charging, in order to consider alternative time periods and locations. Vehicle-to-building models and energy management of buildings need more detailed analyses.
• In restoration and self-recovery, it is necessary to incorporate load shedding and demand response. In addition, case-based predetermined rules and strategies are required for rapid and efficient fault identification and recovery.
• Another challenge in Smartgrid control is how to incorporate manual actions and preferences of human agents into the energy management system. Human actors tend to intervene into the process (especially in control, trade and restoration) and set their own bids, load priorities and device on/off actions.
• As for protection and security, researchers should develop encryption, decryption and authentication algorithms specific to Smartgrid domains which respect network hierarchy and agent privacy. More advanced methods for information safety and communication protocols are also promising directions for research.
• A primary problem is implementation of Smartgrid and microgrids/VPPs on a large scale, such as city or region. Moreover, whether MAS technology is a proper choice for Smartgrid operation and its efficiency should be investigated.
• Designing MAS architecture, hierarchy and agents to perform all functions of Smartgrids (energy trade, control, security, restoration and demand supply management) constitutes a great challenge and problem for future research. Previous simulation and implementation projects have not covered all the above functions and thus more studies are required to explore these aspects.

10.2. Knowledge Reasoning and Planning for the Smartgrid

Aside from fixed network topology and grid configuration, the models for Smartgrid reviewed above lack an important feature. Each paper has addressed a specific problem which has a certain objective function, e.g., bidding, control, demand/supply response, restoration, anomaly detection. And their solutions mostly employ a hard-coded algorithm because there are no unknown elements or external constraints. Namely, agents execute a predefined sequence of actions to achieve their objectives stated in the problem. Consequently, these algorithms do not work for any other task or in a setting which has additional constraints or a slightly different goal. Therefore, agents should be able to construct an action plan for an arbitrary objective function in any setting with special configuration and constraints. Doing so, the agent should take into account both the individual/local goals as well as the global (system-wide) goals. As an example, consider a scenario where the Photovoltaic Panel agent has detected a fault in its inverter component and needs to inform the utility agent. The PV agent can communicate with only the local controller agent, EM agent and neighbor agents, but it does not have direct access to the utility agent. The local controller agent is under maintenance that day; thus, the PV agent has to find sequence of communication actions based on the network topology to accomplish its goal. There is no solution in the literature for this particular problem and we cannot write a hard-coded algorithm for every special situation. Then, the agent needs to develop its own action plan. As another example, suppose that the electric car has a low battery level, but its owner decided to drive in the evening. In this situation, the EV agent needs to inform the other agents in the building to replan their energy usage. In order to charge its batteries, the EV agent should find a new schedule; for example, it proposes that the laundry agent and the home battery agent postpone their energy consumption, as a modification to the existing schedule. Therefore, we need to devise more generic methods that can achieve arbitrary objectives and allow for unexpected situations and additional, external constraints.

In order to perform automated action planning, the agent should be endowed with a knowledge base which includes possible actions, domain information, the agent’s belief about the world and his beliefs about beliefs of other agents. According to the BDI concept, the agent has Beliefs (about the physical world and beliefs of other agents), Desires (goals) and Intentions (actions). The domain description in the knowledge base also includes fluents (variables) that describe the state of the world, their possible values and the action description (precondition, effect and observability of each action). Recall that agents are autonomous and independent; thus, they have private beliefs, their own domain description and local knowledge. In particular, the domain description of an agent involves only the fluents and the actions which are relevant for him. For example, the domain of the AC agent consists of the fluents on, heat_mode, connected and temperature and the actions turn_on, turn_off, heat, cool and inform_temperature. Then, one part of the problem is how to represent the action description and the private beliefs of an agent. There are action languages such as $m{A}_0$ [360] and $m{A}^{\ast }$ [361] designed for the multi-agent setting; however, whether they are suitable for the Smartgrid context and suitable for BDI and the private view of agents should be investigated.

In a dynamic environment, agents communicate and execute actions; thus, another important problem is how to update beliefs of agents upon an action occurrence. Belief update is necessary for both the agent having a correct view of the world and also to determine whether a sequence of actions attains a goal condition. For instance, consider a home environment with energy-storage agents. At one moment, storage 1 is supplying electricity to the other appliances while storage 2 is charging. Later, storage 1 becomes depleted, but it knows that there is another storage which has been charging for 1.5 h. The storage 1 agent informs the home energy management (HEM) agent about its storage level, and the HEM agent communicates with the storage agent 2 to learn its status. Then, the HEM agent sends instructions for storage 1 to switch to charge mode and storage 2 to switch to supply mode. After these ontic and communication actions, each of the three agents should update their belief about the mode of each agent. Moreover, the home energy management agent must verify that this action plan maintains the goal of proper energy delivery.

The challenges in this belief-update and state-transition problem are the agents might have initially incomplete or incorrect beliefs and they might have different levels of observability of an action. In a multi-agent context, not all agents might observe the execution of an action and its effects. In particular, in a communication action, an agent sends message to a certain set of agents; other agents outside of this audience are unaware of this action. In some cases, an agent might partially observe the effect of the action. As an example, the central controller agent knows that the heater agent periodically senses the temperature level, but he cannot observe the sensed value of the temperature. Following this idea, agents can be classified into three categories: Full observers who observe the action and its effects, partial observers who only observe that the action takes place but not its effects and oblivious agents who are completely unaware of the action occurrence.

In the uncertain and time-varying setting of the Smartgrid, pre-specified algorithms for energy distribution and load scheduling may not work. For instance, EV agents may arrive and leave, and operation time of the appliances depends on the human users and the temperature. In such a dynamic medium of the Smartgrid, neighbor agents should communicate with each other and plan/replan scheduling of their energy consumption and charging. Developing (re)planning, decentralized scheduling for agents will be useful for better utilization of the available energy.

The knowledge base model also allows for representation of different kinds of knowledge such as commonsense rules, defaults and causal laws. Examples are “By default, the wind turbine is on and the diesel generator is off”, “Electric vehicle is charging implies it is connected to the grid” and “If the temperature is below 20 °C, the heater must be on”. The rules of commonsense are beneficial for circumventing incomplete information and enhance reasoning. Creating an ontology for commonsense knowledge, default rules, causal laws and integrating into reasoning, planning is a promising work. As for other tasks in the Smartgrid, the knowledge base of an agent should also include the necessary variables and elements for auction, energy trade, load profile, security.

A Smartgrid has a hierarchical and hybrid organization. It consists of a house, a building, a microgrid/VPP and main grid layers and each layer can have a different structure. In addition, there may also be coalitions, holons and zones in some layer(s). An agent should only consider local and relevant variables and actions because it cannot consider the fluents of all agents in the system or update them in a reasonable manner. Then, the problem is how to design the knowledge base of an agent appropriate for the hybrid structure of the Smartgrid.

Henceforth, knowledge reasoning and planning are important features of intelligent agents, yet they have not been studied in the context of Smartgrids. Future research should proceed in this direction in order to endow agents with tools to operate efficiently in an uncertain environment.

11. Conclusions

Smartgrids represent the next generation in energy production and sharing. Humanity is shifting from high-pollution nuclear, thermic plants towards carbon-free, clean and renewable resources. Environmental regulations and tightening government policies about carbon emissions are putting pressure on the energy production and transportation sectors and this trend seems likely to continue in the coming decades. Moreover, households are evolving into economically motivated prosumers who can also produce, store and sell electricity in addition to consumption.

This paradigmatic shift necessitates a Smartgrid which can deal with the challenges of decentralized production such as control, demand management, restoration, energy trade and security. Volatility of weather-dependent generators and time-dependent electricity consumption require accurate control, scheduling and storage of energy. In addition, prosumers need to exchange energy among each other to balance demand and supply. Thus, market and auction mechanisms should be designed and implemented. Microgrids can become efficient energy ecosystems if prosumers can communicate, access the market and have the necessary background to achieve their objectives.

Advancement of Smartgrids and related technologies need integration of Artificial Intelligence, distributed systems, information and communication technologies. In this survey, we focused on the use of multi-agent system technology for Smartgrids. Multi-agent system theories have been developed by computer scientists to represent autonomous agents interacting with each other and with the environment. Agents have a knowledge base; they can decide and act on their own without human intervention. Moreover, agents are goal-driven; they perform actions to satisfy or optimize their objectives. As a system of autonomous and interacting entities, MASs seem to be a potential framework for modeling distributed producers and consumers in the Smartgrid.

At the beginning of the survey, we provided some definitions and background information about Smartgrids, multi-agent system concepts, MAS platforms, ontologies and standards. Then, we reviewed the literature about how researchers have used MAS technology for energy trade, control, demand and supply management, restoration, security, simulation and implementation of the Smartgrid. In each of these fields, we highlighted the challenges and the remaining open problems for future research. In addition, we conducted an upper-level analysis of existing MAS models and research. Observe that in each field, a great deal of work has already been done; however, it is not clear whether the main problems of the Smartgrid have been solved. In particular, integrating these different models for full functionality is an issue waiting to be dealt with. Researchers should assess whether the proposed solutions are practical and flexible enough, and for this purpose, physical implementation may be necessary. Moreover, agents should possess intelligence to perform automated planning and knowledge reasoning in order to accomplish their objectives in a generic setting and maintain a consistent, up-to-date knowledge base and infer new information.

In addition to the hardware, a Smartgrid also involves software and cyber components which need to achieve a number of complex and important tasks mentioned above. Multi-agent systems have the advantage of dividing a complicated problem into smaller subproblems and using agents to solve the smaller problems. However, whether an agent-based approach is efficient and yields the optimal or desired outcome in the Smartgrid domain should be investigated in more detail and comparison with non-MAS methods should be made. In particular, coordination of many autonomous and self-interested small agents to accomplish various functions of the grid is a major concern. In this sense, possible pitfalls and disadvantages of MAS technology, if any, should be identified and whether MAS is the right technology for Smartgrid management (or whichever aspect) should be determined.

In the future, whether (or how) Smartgrids can be deployed on a large scale or an economy-wide scale for electricity allocation is unclear. Doing so would require, in addition to the massive changes in the physical infrastructure, control and coordination of millions of independent prosumers and distributed generators/loads. Therefore, developing a robust and scalable MAS framework for the energy systems is a promising direction for medium- and long-term research. We hope this survey on the applications of multi-agent systems for Smartgrids will be beneficial for researchers, engineers, designers, prosumers and other stakeholders in terms of exploring further topics and solutions.