Distributed Reinforcement Learning for the Management of a Smart Grid Interconnecting Independent Prosumers

Abstract

In the context of an eco-responsible production and distribution of electrical energy at the local scale of an urban territory, we consider a smart grid as a system interconnecting different prosumers, which all retain their decision-making autonomy and defend their own interests in a comprehensive system where the rules, accepted by all, encourage virtuous behavior. In this paper, we present and analyze a model and a management method for smart grids that is shared between different kinds of independent actors, who respect their own interests, and that encourages each actor to behavior that allows, as much as possible, an energy independence of the smart grid from external energy suppliers. We consider here a game theory model, in which each actor of the smart grid is a player, and we investigate distributed machine-learning algorithms to allow decision-making, thus, leading the game to converge to stable situations, in particular to a Nash equilibrium. We propose a Linear Reward Inaction algorithm that achieves Nash equilibria most of the time, both for a single time slot and across time, allowing the smart grid to maximize its energy independence from external energy suppliers.

1. Introduction

The eco-responsible production and distribution of electrical energy at the local scale of an urban territory (a district—an activity zone for example) is today an environmental and economic key objective in the development or the planning of such territories [1,2]. Indeed, the trend of governance of urban territories, which often wish to keep or gradually regain control of their infrastructure networks (water, energy, mobility, and waste), is to deploy, in the energy sector, smart grids interconnecting private and public actors (companies, administrations, and residential buildings, for example) [1,3], each potentially having a green electricity production and storage capacity; they act as prosumer actors [4,5].

In this context of a smart grid interconnecting different prosumers, two operating paradigms can be considered. On the one hand, a totally collaborative operating mode in which each actor leaves its autonomy of decision to a centralized system, which acts towards global optimization and distributes the final costs to each actor (for example through a mechanism using Shapley values [6]). On the other hand, a more selfish mode of operation in which each actor retains its decision-making autonomy and defends its own interests in a comprehensive system with rules, which are accepted by all, that encourage virtuous behavior. It is this second paradigm that we consider here. Indeed, as it is unlikely that an actor, in particular an industrial actor, would agree to leave, to an independent operator, the control of the use of its own means of production and storage of energy, this second paradigm seems more realistic.

To be accepted by all, a strategy must be able to guarantee to each actor that it will be more advantageous than a situation of pure selfishness, which is the objective of this article. Thus, our objective is to propose and analyze a model and a management method for a smart grid that is shared between different independent actors, who respect their own interests, and that encourages each to a behavior allowing as much as possible an energy independence of the smart grid from external energy suppliers. This respect for the interests and independence of each actor naturally leads us to consider not the search for a global optimum between the actors, but more of a virtuous balance of their politics that is beneficial to all.

This is why we consider here a game theory model, in which each actor of the smart grid is a player, and we investigate distributed machine-learning algorithms to allow decision-making, thus, leading the game to converge to a stable situation, in particular Nash equilibria if they exist (see [7] for a definition). From a control systems engineering perspective, reinforcement learning (RL) can be considered a closed-loop process [8,9], which means that, at each learning step, the control system is feedback-regulated (as opposed to open-loop controllers, which do not consider process output and only calculate commands using internal or external parameters).

In our context of using RL, the feedback resides in the reward values of each step that are used to calculate new probabilities of actions vis-à-vis of the smart grid, actions to be selected by each prosumer at the next step. The closed-loop process is therefore considered here at the level of granularity of the set of prosumers and their decisions across time, and not at that of the management of electrical devices for which closed-loop processes are also proposed [10,11]. At the level of granularity we consider here, we make no fixed assumptions about the type of control used for real-time management of electrical devices, closed-loop or open-loop.

Some early works (as of the 2010s) focused mainly on the use of game theory for energy supply to end consumers [12,13,14]. As renewable local energy production capabilities became increasingly available, some work focused on considering those energy production capabilities in smart grid models, leading to including prosumers (i.e., producers/consumers) in those models. In [15], the authors proposed a game theory approach to model the interactions among prosumers and distribution system operators for the control of electricity flows in real-time.

The authors in [16] proposed a game theory energy management based on the Stackelberg leadership model system, which seeks for social optimality when each energy consumer focuses only on its own energy efficiency; the existence of a Nash equilibrium for any instance of the game model that is proposed in the paper was proven (we will see that this is not the case for the game model that we consider, see Lemma 1), and an optimal distributed deterministic algorithm is proposed that reaches this equilibrium.

Some other works seek to propose pricing policies for energy suppliers, considering various optimization objectives. In [17], the authors considered a smart power infrastructure where several subscribers share a common energy source, and proposed a distributed algorithm that automatically manages the interactions among subscribers and the energy provider. The authors in [18] presented an equilibrium Selection Multi-Agent Reinforcement Learning for consumer energy scheduling of a residential microgrid, based on private negotiation between each consumer and the energy provider, and the work of [19] include reliability parameters and energy fluctuations and propose a Deep Reinforcement Learning algorithm based on Q-Learning and Deep Neural Networks to determine a pricing policy for a power supplier that balances those two parameters.

The authors in [20] also proposed a Vickrey–Clarke–Groves (VCG)-based auction mechanism aiming to maximize the aggregate utility functions of all users minus the total energy cost. In [21], the authors proposed a game theory approach for selecting the best power subscription when several power suppliers are available, based on a two-stage process.

Taking into account the availability of several power suppliers leads to various Demand Response Management (DRM) problems. The results of [22] focused on the real-time interactions among multiple utility companies and multiple users, and they proposed a distributed real-time demand response algorithm to determine each user’s demand and each utility company’s supply simultaneously.

The contributions of [23] also focused on DRM but solved electricity pricing questions using a reinforcement learning-based decision system. The goal in this approach is to minimize the electricity payment and consumption dissatisfaction for end users. Some works focused on more distributed models with autonomy goals. The authors in [24] proposed a coalition game between a number of micro-grids (energy producers) that are each servicing a group of consumers (or an area), and [25] also considered autonomy goals for a distributed set of micro-grids without support from a traditional centralized grid and proposed a distributed algorithm based on a cooperative game theory approach. In [26], a Q-learning method was proposed in a multi-agent model where each consumer (and more particularly each smart meter) adapts its consumption and the control of its electrical equipment to the state of the grid and to the prices of electricity.

This synthetic review on related works leads to one observation: very few works focused on a general smart grid model, in which consumers may or may not have power production capabilities and power storage facilities, and considered learning approaches. To our knowledge, only [27] proposed a multi-agent reinforcement-learning approach to controlling a smart grid composed of production resources, battery storage, electricity self-supply, and short-term market trading. Event though the proposed algorithm offers a significantly high computation speed, it has local optima issues that cause it to be out-performed by the use of a simulated annealing in terms of energy costs.

We propose here a new game theory model for a smart grid interconnecting different prosumers. This game theory model is based on the proposal of a virtual economic model between the players, which has an impact on the amount of energy that the smart grid has to buy from an external supplier. In a discrete time context, in order to allow the smart grids to obtain an operation that is as little dependent as possible on such an external energy supplier, while respecting the constraints and interests of each actor, we develop and experiment on different scenarios a distributed reinforcement-learning approach aimed at achieving each time period of use of the smart grid. Indeed, the objective here is to propose a decision algorithm that, on the one hand, respects the choices and the interests of each actor and, on the other hand, does not require a large data history. This is why we decided to use a reinforcement-learning approach distributed between the actors and not a centralized supervised or unsupervised learning approach (see for example [28,29]).

The remainder of the article is structured as follows: first, in Section 2—Materials and Methods, we define the architecture of the considered system, i.e., the various actors, their interactions, and their actions. This section then specifies the game model considered, as well as the internal economic model that this game implements. In Section 2.2—Distributed Reinforcement Learning for the Game in Each Period, the distributed reinforcement-learning model that will be used for the simulations is defined. In Section 3—Results, the performance evaluation of the proposed reinforcement learning executed at each period is analyzed. The behavior of this approach over a series of periods is then analyzed in Section 3.2—Multi-Period Simulation.

2. Materials and Methods

2.1. Smart Grid Model

In this section, we define the actors of the smart grid, their interactions, the associated multi-agent game model, and the virtual economic model that governs it. The main parameters and notations of the proposed model are given in

Table 1. List of main notations defining the components of the smart grid model and their interactions.

${\mathit{a}}_{\mathit{i}}$	$1\le \mathit{i}\le \mathit{N}$, ith actor in the smart grid.
$S_i^{max}$	maximum quantity of electricity stocked by $a_i$ at any period.
$P_i^t$	quantity of electricity produced by $a_i$ at period t.
$C_i^t$	quantity of electricity consumed by $a_i$ at period t.
$S_i^t$	quantity of electricity stored by $a_i$ at beginning of period t.
$R_i^t$	residual capacity of the battery storage.
$pro{d}_i^t$	quantity of electricity inserted by $a_i$ in SG at period t.
$con{s}_i^t$	quantity of electricity consumed from SG by $a_i$ at period t.
$Ou{t}_{SG}^t$	the sum of the agents’ consumption.
$I{n}_{SG}^t$	the sum of the agents’ production.
$Stat{e}_i^t$	state of $a_i$ in $Deficit,Self,Surplus$ at period t.
$mod{e}_i^t$	mode of $a_i$ in $\{CON{S}^{-},CON{S}^{+},DIS,PROD\}$ at period t.
${\gamma }_i^t$	value of the incentive to store or preserve electricity of $a_i$ at period t.
$r_i^t$	the amount of energy preserved by $a_i$ for the next period $t+1$.
${\pi }_0^{+,t}$	unitary benefit of electricity sold by an actor to SG (independently from EPO) at period t.
${\pi }_0^{-,t}$	unitary cost of electricity bought by an actor from SG (independently from EPO) at period t.
$b_0^t$	unitary benefit of an actor selling electricity to SG (possibly partially to EPO) at period t.
$c_0^t$	unitary cost of an actor buying electricity (possibly partially from EPO) to SG at period t.
$be{n}_i^t$	virtual benefit of $a_i$ at period t.
$cs{t}_i^t$	virtual cost of $a_i$ at period t.
$V_i^t$	learning utility function of player $a_i$ at period t.
${\pi }_{EPO,t}^{+}$	unitary price of electricity purchased by EPO at period t.
${\pi }_{EPO,t}^{-}$	unitary price of electricity sold by EPO at period t.
${\varphi }_{EPO}^{+}$	benefit function of electricity sold by SG to EPO.
${\varphi }_{EPO}^{-}$	cost function of electricity bought by SG to EPO.
$B{B}_i$	whole benefit of $a_i$ during the T periods.
$C{C}_i$	whole cost for $a_i$ during the T periods.
$I{B}_i$	the internal benefit of $a_i$ during the T periods.
$I{C}_i$	the internal cost of $a_i$ during the T periods.

—a list of the main notations defining the components of the smart grid model and their interactions.

2.1.1. Smart Grid Actors Model

Our model assumes that each day is divided into homogeneous consecutive periods of a few hours. We make two assumptions here. First, that the consumption and production values of each actor in each period can be considered as a fixed value. Secondly, that it is possible at each period and for each actor to sufficiently predict precisely these consumption and production values for the following period. In [30], it is shown that such a model is realistic considering half-day periods, but it is important to note that our model is independent of the choice of periods—the execution of the method requires a short time in each one.

In this discrete time context, each energy indicator of each actor can be considered as a fixed value in each period. For each period, the system representing the smart grid is considered here as a multi-agent system. Let $AG=\{a_1,\dots ,a_N\}$ be the set of N actors connected by the smart grid (SG), each actor $a_{i:1\le i\le N}$ being potentially able to produce electricity, in particular with renewable energies, and to store it. Each actor’s production and storage capacities are limited. Production varies over periods.

Each actor $a_i$, with $1\le i\le N$, at each period t is characterized by

• $P_i^t$ is the production of electricity by the actor during the period.
• $C_i^t$ is the electricity consumption of the actor during the period.
• $S_i^t$ is the quantity of electricity stored by the actor at the beginning of the period. We denote by $S_i^{max}$ the maximal storage capacity, and we define the residual capacity of battery storage $R_i^t=S_i^{max}-S_i^t$, i.e., the maximum quantity of energy that the actor can store during the period. Note that, during a period, an actor cannot both use and provide storage.

As has been done in different game models applied to Smart Grids [31,32,33], at the level of granularity corresponding to the model we use here, where a single prosumer can be a micro-grid, we consider at a first approach that the values of consumption and production are inelastic in each period, which thus makes it possible to assess the relevance of algorithmic game theory to arbitrate the choices of the various independent actors. This relevance established, taking into account inelastic data at a finer granularity could then be considered [34,35,36], in game models that are likely to be more complex to handle.

The smart grid is also connected to an external energy production operator (EPO), who can supply it with electricity if necessary and to which the smart grid can sell unused electricity produced by the actors. Note that the actors do not directly interact with this EPO.

Thus, the smart grid itself can be considered as an energy container connecting all the actors $a_{i:1\le i\le N}$; each actor $a_i$ at each period t provides a quantity of electricity $pro{d}_i^t$ to SG and uses a quantity of electricity $con{s}_i^t$ from SG (note that $pro{d}_i^t$ and $con{s}_i^t$ cannot be both greater than 0). We denote $I{n}_{SG}^t={\displaystyle \sum _{i=1}^N}pro{d}_i^t$ and $Ou{t}_{SG}^t={\displaystyle \sum _{i=1}^N}con{s}_i^t$. Then, if $I{n}_{SG}^t<Ou{t}_{SG}^t$, the energy production operator EPO provides a quantity of energy $q=I{n}_{SG}^t-Ou{t}_{SG}^t$ to SG from a given linear increasing cost function ${\varphi }_{EPO}^{-}\left(q\right)$ (see Section 3.1—Learning Performance Analysis); else if $Ou{t}_{SG}^t\le I{n}_{SG}^t$, SG sells a quantity of energy $q^{\prime }=Ou{t}_{SG}^t-I{n}_{SG}^t$ from a given linear increasing benefit function ${\varphi }_{EPO}^{+}\left(q^{\prime }\right)$. Definitions of ${\varphi }_{EPO}^{-}$ and ${\varphi }_{EPO}^{+}$ are inputs of the model.

2.1.2. Action Modes of Each Actor

At each period t, each actor $a_{i:1\le i\le N}$ chooses one of four possible modes of action $mod{e}_i^t\in \mathcal{M}=\{CON{S}^{+},CON{S}^{-},DIS,PROD\}$. Modes $CON{S}^{-}$ and $CON{S}^{+}$ mean that the actor requires energy from SG for its consumption, i.e., $con{s}_i^t>0$ and its production is not sufficient to cover its consumption (the two modes differ on whether the actor uses its storage or not, as detailed below). Mode $DIS$ means that the actor decides to be independent of SG (i.e., $pro{d}_i^t=con{s}_i^t=0$). Finally, mode $PROD$ means that the actor chooses to provide energy to SG (i.e., $pro{d}_i^t>0$, and it is autonomous for its own consumption). The conditions to choose one mode in $\mathcal{M}$ respect the following rules considering three disjoint states for an actor depending only on $P_i^t,S_i^t,C_i^t$.

• State Deficit: $P_i^t+S_i^t\le C_i^t$. In this state, actor $a_i$ needs energy from SG for its consumption, then $pro{d}_i^t=0$; it can also choose to use its storage or not. Thus, in this state, the chosen mode can be $mod{e}_i^t=CON{S}^{+}$, in which case $con{s}_i^t=C_i^t-(P_i^t+S_i^t)$ and thus $S_i^{t+1}=0$, or $mod{e}_i^t=CON{S}^{-}$, in which case $con{s}_i^t=C_i^t-P_i^t$ and thus $S_i^{t+1}=S_i^t$.
• State Self: $P_i^t+S_i^t>C_i^t$ and $P_i^t\le C_i$. In this state, two modes can be chosen by $a_i$. First, $mod{e}_i^t=CON{S}^{-}$, where $con{s}_i^t=C_i^t-P_i^t$: the actor does not use its storage, $S_i^{t+1}=S_i^t$. Secondly $mod{e}_i^t=DIS$, where $con{s}_i^t=0$ and $S_i^{t+1}=S_i^t-(C_i^t-P_i^t)$. In both cases, $pro{d}_i^t=0$.
• State Surplus: $P_i^t>C_i^t$. Two modes can be chosen by $a_i$. First, $mod{e}_i^t=PROD$, where $pro{d}_i^t=P_i^t-C_i^t$: the actor provides all its overall produced energy to SG and $S_i^t$ remains the same. Secondly, $mod{e}_i^t=DIS$ in which $S_i^{t+1}=Min(S_i^{Max},S_i^t+(P_i^t-C_i^t))$ and $pro{d}_i^t={|P_i^t-C_i^t-R_i^t|}^{+}$: the actor favors the storage of its remaining produced energy. In both cases, $con{s}_i^t=0$.

The logical architecture of the system thus proposed is synthesized in Figure 1.

Thus, in State Deficit, each actor $a_i$ can choose at each period t its mode $mod{e}_i^t$ in $\{CON{S}^{+},CON{S}^{-}\}$. In State Self, an actor $a_i$ can choose its mode in $\{CON{S}^{-},DIS\}$. Finally, in State Surplus, an actor $a_i$ can choose its mode in $\{PROD,DIS\}$. Thus, in any case, an actor has always two possible strategies.

These modes ensure that the production of the actor $a_i$ is intended primarily for its consumption, and the stored energy of $a_i$ is never imported from SG but is only filled by its own production. In fact, each actor $a_i$ consumes the total amount of production (and eventually its current storage) before importing electricity from SG. The choice of actors therefore only concerns the policy of supply or use of the stock.

Let us now define two parameters $q^{t,-}$ and $q^{t,+}$ such as $q^{t,-}=$${\left|{\displaystyle \sum _i}|C_i^t-P_i^t{|}^{+}-{|P_i^t-(C_i^t+S_i^{max}-S_i^t)|}^{+}\right|}^{+}$ and $q^{t,+}={\left|{\displaystyle \sum _i}|P_i^t-C_i^t{|}^{+}-{|C_i^t-(P_i^t+S_i^t)|}^{+}\right|}^{+}$. Note that $q^{t,-}$ is an upper bound of the energy quantity bought by SG to EPO at period t and that $q^{t,+}$ is an upper bound of the energy quantity sold by SG to EPO at period t. We then also define ${\pi }_{EPO,t}^{-}=\frac{{\varphi }_{EPO}^{-}\left(q^{t,-}\right)}{q^{t,-}}$ and ${\pi }_{EPO,t}^{+}=\frac{{\varphi }_{EPO}^{+}\left(q^{t,+}\right)}{q^{t,+}}$ (in fact, we consider that way a kind of linearization of functions ${\varphi }_{EPO}^{-}$ and ${\varphi }_{EPO}^{+}$).

We define two other parameters, ${\pi }_0^{t,-}\le {\pi }_{EPO}^{-}$ and ${\pi }_0^{t,+}\le {\pi }_{EPO}^{+}$. First, ${\pi }_0^{t,-}$ is a cost each actor has to pay for one unit of energy provided to it by SG. Secondly, ${\pi }_0^{t,+}$ is a benefit each actor makes for one unit of energy sold to SG (see Figure 2 Unitary prices in the model used for learning process). This guaranties that, for any $0\le x\le q^{t,-}$, we have ${\varphi }_{EPO}^{-}\left(x\right)\ge x\times {\pi }_0^{t,-}$. These two parameters are not input data but rather parameters of the virtual economic internal model of SG, needed for its management; the way to choose values for ${\pi }_0^{t,-}$ and ${\pi }_0^{t,+}$ will be discussed in Section 2.1.2—Action Modes of Each Actor.

Then, the internal cost and benefits of each actor $a_i$ defined in the model depend on the value of $pro{d}_i^t$, $con{s}_i^t$, $I{n}_{SG}^t$ and $Ou{t}_{SG}^t$. We first define a unitary cost $c_0^t$ Equation (1) and a unitary benefit $b_0^t$ Equation (2) of a unit of energy in SG as follows:

$$c_0^t=\left\{\begin{array}{cc}{\pi }_0^{t,-}\hfill & ifI{n}_{SG}^t\ge Ou{t}_{SG}^t\hfill \\ \frac{{\varphi }_{EPO}^{-}(Ou{t}_{SG}^t-I{n}_{SG}^t)+I{n}_{SG}^t\times {\pi }_0^{t,-}}{Ou{t}_{SG}^t}\hfill & else\hfill \end{array}\right.$$

(1)

$$b_0^t=\left\{\begin{array}{cc}{\pi }_0^{t,+}\hfill & ifOu{t}_{SG}^t>I{n}_{SG}^t\hfill \\ \frac{Ou{t}_{SG}^t*{\pi }_0^{t,+}+{\varphi }_{EPO}^{+}(I{n}_{SG}^t-Ou{t}_{SG}^t)}{I{n}_{SG}^t}\hfill & else\hfill \end{array}\right.$$

(2)

These definitions are based on choices relating to the proposed model. If $I{n}_{SG}^t>Ou{t}_{SG}^t$, then we consider that each actor in state Deficit, or in state Self with mode $CON{S}^{-}$, receives the same percentage $(I{n}_{SG}^t-Ou{t}_{SG}^t)/I{n}_{SG}^t$ of energy supplied globally by EPO. This percentage is then reflected on the unit cost $c_0^t$. On the contrary, if $Ou{t}_{SG}^t>I{n}_{SG}^t$, then we consider that each actor in state Surplus sells the same percentage $(Ou{t}_{SG}^t-I{n}_{SG}^t)/Ou{t}_{SG}^t$ of energy to EPO, which is reflected in the unit benefit $b_0^t$.

To finally define the cost and benefit functions of each actor at each period t, we consider a given parameter ${\gamma }_i^t$ which indicates the importance given by each actor $a_i$ of storing energy for the next period, with ${\gamma }_i^t\in \{0,{\pi }_0^{t,-}+1,{\pi }_0^{t,+}+1\}$. Each actor at each period t knows $C_i^{t+1}$ and $P_i^{t+1}$. The way to determine the value of each ${\gamma }_i^t$ will be considered in Section 2.2.1—Learning strategies.

The (virtual) whole cost of $a_i$ at period t is $cs{t}_i^t=c_0^t\times con{s}_i^t$ and its whole benefit is $be{n}_i^t=(b_0^t\times pro{d}_i^t)+({\gamma }_i^t\times r_i^t)$, with the additional or preserved stock $r_i^t$ indicating the amount of energy stored or preserved by the actor for the next period defined as follows:

• If $mod{e}_i^t=CON{S}^{+}$ then $r_i^t=0$.
• If $mod{e}_i^t=CON{S}^{-}$ then $r_i^t=S_i^t$.
• If $mod{e}_i^t=DIS$ then $r_i^t=0$ in State Self and $r_i^t=Min(S_i^{max}-S_i^t,P_i^t-C_i^t)$ in State Surplus.
• If $mod{e}_i^t=PROD$ then $r_i^t=0$.

Thus, we denote by $V_i^t=be{n}_i^t-cs{t}_i^t$ the value of actor $a_i$ at period t. We also consider $I{B}_i={\displaystyle \sum _{t=1}^T}{b}_0^t\times pro{d}_i^t$ and $I{C}_i={\displaystyle \sum _{t=1}^T}{c}_0^t\times con{s}_i^t$ being respectively the internal benefit and cost of actor $a_i$ for all the periods in the virtual economic model we propose inside SG.

2.1.3. Game Model

From the definitions of the previous section, we consider, in each period, a simultaneous game model [7], called SGGame, in which players are the actors $\{a_1,\dots ,a_N\}$ and where the set of strategies of player $a_i$ is $S{t}_i=\{CON{S}^{-},CON{S}^{+}\}$ if $a_i$ is in State Deficit, $S{t}_i=\{CON{S}^{-},DIS\}$ if $a_i$ is in State Self and $S{t}_i=\{DIS,PROD\}$ if $a_i$ is in State Surplus.

Given a strategy profile $\mu =s_1\dots s_N$, with $s_i\in S{t}_i$ a strategy for each player $a_i$, we define the utility of player $a_i$ by $V_i=(be{n}_i-cs{t}_i)$. Note that, since ${\pi }_{EPO}^{-}$ and ${\pi }_{EPO}^{+}$ can be different from the other and different from ${\pi }_0^{+}$ and ${\pi }_0^{-}$, this game called SGGame is not a zero-sum game [7].

Let us recall that a Nash equilibrium is a strategic profile in which the unilateral modification of the strategy of any actor degrades its utility [7] (thus, no actor has any interest in changing its strategy alone). As shown in Section 3.1—Learning performance analysis, non-trivial instances of the game admit pure Nash equilibria. The following result proves that this is not always the case.

Lemma 1.

SGGame does not admit a pure Nash equilibrium for all instances.

Proof. 

We consider an instance of SGGame with two players $a_1$ in State Deficit and $a_2$ in State Surplus, with

• $C_1-P_1=2$ and $S_1=1$.
• $P_2-C_2=2$ and $S_2^{max}=S_2+1$.
• ${\gamma }_1={\pi }_0^{-}+1<{\pi }_{EPO}^{-}={\varphi }_{EPO}^{-}\left(1\right)$ and ${\gamma }_2={\pi }_0^{+}+1<{\pi }_{EPO}^{+}={\varphi }_{EPO}^{+}\left(1\right)$.

Note that these values of ${\gamma }_1$ and ${\gamma }_2$ are compatible with what will be defined in Section 3.1. □

Then, we can have four possible strategy profiles depending on strategies $\{CON{S}^{+},CON{S}^{-}\}$ for $a_1$ and $\{DIS,PROD\}$ for $a_2$, and the utilities are given in

Table 2. Utilities for each possible strategy profile.

	$\mathit{m}\mathit{o}\mathit{d}{\mathit{e}}_2=\mathit{P}\mathit{R}\mathit{O}\mathit{D}$	$\mathit{m}\mathit{o}\mathit{d}{\mathit{e}}_2=\mathit{D}\mathit{I}\mathit{S}$
	(then $r_2^t=0$)	(then $r_2^t=1$)
$mod{e}_1=CON{S}^{-}$	$I{n}_{SG}^t=2$, $Ou{t}_{SG}^t=2$	$I{n}_{SG}^t=1$, $Ou{t}_{SG}^t=2$
(then $r_1^t=1$)	$be{n}_1^t={\gamma }_1$, $cs{t}_1^t=2\times {\pi }_0^{-}$	$be{n}_1^t={\gamma }_1$, $cs{t}_1^t={\pi }_{EPO}^{-}+{\pi }_0^{-}$
	$V_1={\gamma }_1-2\times {\pi }_0^{-}$	$V_1={\gamma }_1-{\pi }_{EPO}^{-}-{\pi }_0^{-}$
	$be{n}_2^t=2\times {\pi }_0^{+}$, $cs{t}_2^t=-0$	$be{n}_2^t={\pi }_0^{+}+{\gamma }_2$, $cs{t}_2^t=0$
	$V_2=2\times {\pi }_0^{+}$	$V_2={\pi }_0^{+}+{\gamma }_2$
$mod{e}_1=CON{S}^{+}$	$I{n}_{SG}^t=2$, $Ou{t}_{SG}^t=1$	$I{n}_{SG}^t=1$, $Ou{t}_{SG}^t=1$
(then $r_1^t=0$)	$be{n}_1^t=0$, $cs{t}_1^t={\pi }_0^{-}$	$be{n}_1^t=0$, $cs{t}_1^t={\pi }_0^{-}$
	$V_1=-{\pi }_0^{-}$	$V_1=-{\pi }_0^{-}$
	$be{n}_2^t={\pi }_0^{+}+{\pi }_{EPO}^{+}$, $cs{t}_2^t=0$	$be{n}_2^t={\pi }_0^{+}+{\gamma }_2$, $cs{t}_2^t=0$
	$V_2={\pi }_0^{+}+{\pi }_{EPO}^{+}$	$V_2={\pi }_0^{+}+{\gamma }_2$

The definition of $r_1^t$ and $r_2^t$ is given at the end of Section 2.1.2—Action modes of each actor. Concerning $a_1$, line 1 dominates line 2 in column 1, and line 2 dominates line 1 in column 2. Concerning $a_2$, column 2 dominates column 1 in line 1, and column 1 dominates column 2 in line 2. Thus, there is no PNE.

.

Note that this lemma is a consequence of the differences between the values of ${\pi }_{EPO}^{-}$, ${\pi }_{EPO}^{+}$, ${\pi }_0^{+}$ and ${\pi }_0^{-}$.

As we have seen, we consider the time of the smart grid as a sequence of T homogeneous periods of the same duration. Each actor $a_i$ is in a chosen mode $mod{e}_i^t$ in $\{CONS-,CONS+,PROD,DIS\}$ for each period (modes determined by running the game considering the state of each actor). Thus, the system can be seen as a non-stationary repeated game. It is non-stationary because the values of $C_i$ and of $P_i$ at each step do not depend (only) on the strategies chosen during the previous steps. Moreover, since the choice of strategies for each period, apart from the value of the stock of each actor, does not depend on the strategies chosen during previous periods, we cannot here directly consider concepts of the theory of repeated games [37].

2.1.4. A Pricing Model for a Smart Grid

As we have seen, the model proposed here is an internal and virtual economic model between SG actors to be used as a cost allocation key for the real payment of electricity purchased from EPO by all the actors.

The objective is thus here to propose a mechanism to compute a unit price of purchase and sale of electricity to each actor for the whole periods, prices whose modifications compared to the real price of EPO will depend on the choice of all the players during all the periods. Thus, there is no real financial exchange here between the actors, or between the actors and SG (again, the internal economic model is virtual). This model will be the basis of how to fix ${\pi }_0^{t,-}$ and ${\pi }_0^{t,+}$ for each period t.

For each actor $a_i$, let us denote $CON{S}_i={\displaystyle \sum _{t=1}^t}con{s}_i^t$ and $PRO{D}_i={\displaystyle \sum _{t=1}^T}pro{d}_i^t$. It is clear that $PRO{D}_i$, i.e., all of what is produced in SG by $a_i$, is not sold to EPO, and that $CON{S}_i$, ie all of what is consumed from SG by $a_i$, is not purchased from EPO. Thus, we propose here that the real cost $a_i$ has to pay for the T periods is $C{C}_i=({\beta }_{SG,T}^{-}\times CON{S}_i)$ and the real benefit of $a_i$ is $B{B}_i=({\beta }_{SG,T}^{+}\times PRO{D}_i)$ with

$${\beta }_{SG,T}^{-}=\frac{{\varphi }_{EPO}^{-}\left({\displaystyle \sum _{k=1}^T}\left(|{\displaystyle \sum _{i=1}^N}con{s}_i^k-{\displaystyle \sum _{i=1}^N}pro{d}_i^k{|}^{+}\right)\right)}{{\displaystyle \sum _{k=1}^T}{\displaystyle \sum _{i=1}^N}con{s}_i^k}$$

and

$${\beta }_{SG,T}^{+}=\frac{{\varphi }_{EPO}^{+}\left({\displaystyle \sum _{k=1}^T}\left(|{\displaystyle \sum _{i=1}^N}pro{d}_i^k-{\displaystyle \sum _{i=1}^N}con{s}_i^k{|}^{+}\right)\right)}{{\displaystyle \sum _{k=1}^T}{\displaystyle \sum _{i=1}^N}pro{d}_i^k}$$

Then, the real economic balance of actor $a_i$ for all the periods is $E{B}_i=B{B}_i-C{C}_i$. Note that the sum of the actors profits is equal to the benefit of SG on EPO, and that the sum of the actors costs is equal to the overall price paid by SG to EPO.

It will therefore be a question of determining at each period t the values of ${\pi }_0^{t,-}$ and ${\pi }_0^{t,+}$, which offers the best compromise between the value ${\beta }_{SG,t}^{-}$ and ${\beta }_{SG,t}^{+}$ that are implied by ${\pi }_0^{-,t}$ and ${\pi }_0^{+,t}$. Since the purpose of each actor of the game in each period t is to make the best decision regarding the whole proposed economic model, we use definitions of ${\beta }_{SG,T}^{-}$ and ${\beta }_{SG,T}^{+}$ to set values of ${\pi }_0^{+,t}$ and ${\pi }_0^{-,t}$ at each period t as follows: ${\pi }_0^{+,t}=\frac{{\beta }_{SG,t-1}^{-}}{{\pi }_{EPO,t}^{-}}\times {\pi }_{EPO,t}^{+}$ and ${\pi }_0^{-,t}={\beta }_{SG,t-1}^{-}$. Note that more ${\beta }_{SG,t-1}^{-}$ is close to $\frac{{\varphi }_{EPO}^{-}\left(q\right)}{q}$, with $q={\displaystyle \sum _{k=1}^t}{\displaystyle \sum _{i=1}^N}con{s}_i^k$, more the fact that an actor in state Surplus provides electricity to the smart grid is required, which is why ${\pi }_0^t$ has to be close from ${\beta }_{SG,t-1}^{+}$ in such cases.

We consider that initially ${\beta }_{SG,0}^{-}={\pi }_{EPO,1}^{-}-1$ and ${\beta }_{SG,0}^{+}={\pi }_{EPO,1}^{+}-1$. As said in Section 2.1.3—Game model, due to the energy market rules, since ${\pi }_0^{-,t}$ and ${\pi }_0^{+,t}$, ${\pi }_{EPO,t}^{-}$ and ${\pi }_{EPO,t}^{+}$ are not necessarily equal, each period t does not consist in a zero-sum game and thus some situations could occur in which all actors utilities are positive (or negative).

2.2. Distributed Reinforcement Learning for the Game in Each Period

Based on the models given above, we propose here a distributed reinforcement-learning approach which objective is to make it possible to converge towards situations that are stable, i.e., in which a choice of strategy for each actor having a significant impact on the state of the smart grid is on one hand clearly established with a high probability and on the other hand optimized from the point of view of the cost for all the actors. In particular, in the game model we are introducing, we observe experimentally on the instances we generate that Nash equilibria (see [7] for a definition) are, when they exist, such stable situations.

2.2.1. Learning Strategies

We consider at each period t the game defined in Section 2.1.3—Game model. Note that in a distributed reinforcement learning context, if some actors $a_i$ in state Deficit are such that $S_i^t=0$, then any strategy in $S{t}_i=\{CON{S}^{-},CONS+\}$ lets them in a same situation. Thus, in the following, we do not consider such an actor as a player, even if we consider its impact in the computation of the utilities for the other (real) players.

The objective of the learning strategy is to attempt to reach a good stable situation, or even a Nash equilibrium, at each period, considering the benefit function $V_i^t$ defined in Section 2.1.3—Game model. The utility function we use in this learning strategy is based on a benefit/cost parameter $b{g}^k\left(a_i\right)$ for each player $a_i$ at each learning step k, depending on the storage parameter ${\gamma }_i^t$ computed as follows:

Given a period $t\le T$, whatever is the state of the actor $a_i$, the two modes this actor can choose to have different impact on the storage quantity at the end of the period. Let us denote, by $S_i^{t,+}$, the maximum of these two values and $S_i^{t,-}$ the minimum one.

• State Deficit: $S_i^{t,-}=0$, considering $mod{e}_i^t=CON{S}^{+}$ and $S_i^{t,+}=S_i^t$, considering $mod{e}_i^t=CON{S}^{-}$.
• State Self: $S_i^{t,-}=S_i^t-(C_i^t-P_i^t)$, considering $mod{e}_i=DIS$, and $S_i^{t,+}=S_i^t$, considering $mod{e}_i^t=CON{S}^{-}$.
• State Surplus: $S_i^{t,-}=S_i^t$, considering $mod{e}_i^t=PROD$, and $S_i^{t,+}=Max(S_i^{Max},S_i^t+(P_i^t-C_i^t))$, considering $mod{e}_i=DIS$.

One can check that $S_i^{t,-}\le S_i^{t,+}$. Consider now the probability

$$p{p}_i^t=\sqrt{Min\left(\frac{\left|\right|C_i^{t+1}-P_i^{t+1}{{|}^{+}-S_i^{t,-}|}^{+}}{S_i^{t,+}-S_i^{t,-}},1\right)}$$

$|C_i^{t+1}-P_i^{t+1}{|}^{+}$ is the ideal quantity of energy that actor $a_i$ should have in stock for period $t+1$. Let ${\rho }_i^t$ be a random variable where the value is 1 with probability $p{p}_i^t$, else ${\rho }_i^t=0$. Then, we set ${\gamma }_i^t={\rho }_i^t\times (X+1)$ with $X={\pi }_0^{-,t}$ if $a_i$ is in State Deficit or Self, else $X={\pi }_0^{+,t}$ if $a_i$ is in state Surplus.

We can now define the utility function we propose to use.

First, considering Section 2.1.3—Game model, we compute the following bounds of $I{n}_{SG}$ and $Ou{t}_{SG}$ (see end of Section 2.1.2—Action modes of each actor for definitions):

$$I_m=\sum _{a_i:State{\left(a_i\right)}^t=Surplus}{|P_i^t-(C_i^t+(S_i^{max}-S_i^t)|}^{+}\le I{n}_{SG}\le I_M=\sum _{a_i:State{\left(a_i\right)}^t=Surplus}{P}_i^t-C_i^t$$

$$O_m=\sum _{a_i:State{\left(a_i\right)}^t=Deficit}{C}_i^t-(P_i^t+S_i^t)\le Ou{t}_{SG}\le O_M=\sum _{a_i:State{\left(a_i\right)}^t\in \{Deficit,Self\}}{C}_i^t-P_i^t$$

Then, we can conclude that

$$c_0\le c_0^M=Min\left(\frac{(O_M-I_m)\times {\Pi }_{EPO}^{-}+I_M\times {\pi }_0^{-,t}}{O_m},{\pi }_0^{-,t}\right)$$

Note that $I_m$ (resp. $I_M$) is computed by considering that each actor $a_i$ in State Surplus is in mode $DIS$ (resp. $PROD$). The value of $O_m$ (resp $O_M$) is computed by considering that each actor $a_i$ in State Deficit is in mode $CON{S}^{+}$ and that each actor $a_i$ in State Self is in mode $DIS$; finally, $O_M$ is computed by considering that each actor $a_i$ in State Deficit or Self is in mode $CON{S}^{-}$. The game benefit of a player $a_i$ at step k is defined by $b{g}^k\left(a_i\right)=be{n}_i+\left(c_0^M\times \left(\right|C_i^t-P_i^t{|}^{+})\right)-cs{t}_i$. $cs{t}_i^t=c_0^t\times con{s}_i^t$ and $be{n}_i^t=(b_0^t\times pro{d}_i^t)+({\gamma }_i^t\times r_i^t)$. Note also that $c_0^M\times \left(\right|C_i^t-P_i^t{|}^{+})$ is an upper bound of the cost actor $a_i$ could have to pay.

We denote by $b{g}_{min}\left(a_i\right)$ and $b{g}_{max}\left(a_i\right)$ the minimum and maximum benefits obtained by the player $a_i$ during learning step from 1 to k. Then, the utility of $a_i$ during step k of the learning process of period t is equal to $u_i^k=1-\frac{b{g}_{max}\left(a_i\right)-b{g}^k\left(a_i\right)}{b{g}_{max}\left(a_i\right)-b{g}_{min}\left(a_i\right)}$.

2.2.2. Learning Process

Based on the strategy given above, we consider here a distributed Linear Reward Inaction (LRI) reinforcement learning process (see [38]).

Each actor $a_i$ is characterized by two possible strategies $s_1^i$ and $s_2^i$. Let us denote by $p_{1,k}^i$ the probability to choose strategy $s_1^i$ at step k and by $p_{2,k}^i=1-p_{1,k}^i$ the probability to choose strategy $s_2^i$ at step k. Let $s_j^{i,k}\in \{s_1^i,s_2^i\}$ the chosen strategy, with $j\in \{1,2\}$. Then,

$$p_{j,k+1}^i=p_{j,k}^i+b\times u_i^k\times (1-p_{j,k}^i),$$

with $b<1$ a learning parameter called slowdown factor. The probability of the other strategy at step $k+1$ is of course $1-p_{j,k+1}^i$.

3. Results

We present in this section several experiments and performance evaluations of the LRI reinforcement learning process presented above.

The performance evaluation of the approach proposed here is based on a set of data generated as indicated in Section 3.1 and Section 3.2, for about twenty actors. The purpose of this generation is on the one hand to represent realistic situations in terms of heterogeneity of actors’ behavior (production and consumption) and on the other hand to be sufficiently complex in terms of learning process to evaluate the performances of the method (in particular on the existence or not of pure Nash equilibria).

We start by studying the performances of the learning process during only one period. We then extend our simulations to cases in which the learning process runs on several successive periods.

3.1. Learning Performance Analysis

We first study here the performance of the learning strategies on only one period t. To do this, considered data are generated as follows: Experiments are done on 50 independent period instances. The values of $C_i$ and $P_i$ of actor $a_i$ is computed by:

• If $State\left(a_i\right)=Deficit$, then $C_i=15$ and $P_i$ is uniformly chosen in $\{5,\dots ,10\}$; we consider $S_i=0$.
• If $Stat{e}^t\left(a_i\right)=Self$, then with probability $1/2$, we set $C_i=10$ and $P_i$ uniformly chosen in $\{5,\dots ,8\}$ and we consider $S_i=6$. Else, we set $C_i=31$ and $P_i$ uniformly chosen in $\{21,\dots ,30\}$, and we consider $S_i=8$
• If $State\left(a_i\right)=Surplus$, then $C_i=20$ and $P_i$ uniformly chosen in $\{21,\dots ,30\}$, and we consider $S_i=8$.

We consider 15 actors in state Deficit, 10 actors in State Surplus and 10 actors in State Self. For each actor $a_i$, we set $S_I^{max}=20$. Finally, we consider ${\varphi }_{EPO}^{+}\left(x\right)=10\times x$, ${\varphi }_{EPO}^{-}\left(x\right)=30\times x$, ${\pi }_0^{+}=4$ and ${\pi }_0^{-}=3$. The value of each value ${\gamma }_i$ is computed as defined in Section 2.2.1—Learning strategies, considering that $p{p}_i^t=0.8$.

The 50 instances generated correspond each to a difficult context for SG, i.e., in which SG is highly dependent on EPO. For each instance, we consider the average value of $Per{f}_t={\displaystyle \sum _{1\le i\le N}}{V}_i^t$ over the 50 tested instances (see the end of Section 2.1.2—Action modes of each actor for definition of $V_i^t$).

We compare the performances of the reinforcement learning process (RL) described above, with parameter $b=0.01$, with a total exploration method (TEM) to find a strategy profile maximizing $Per{f}_t$ (note that such method seems to be impracticable in real situations because of computation time when N is not small), considering also the maximum value of $Per{f}_t$ over all Nash equilibria, if at least one exists. The learning algorithm RL is stopped when, for each actor and before 50,000 learning steps (maximum number of learning steps), the highest probability of the strategic of each actor is greater than or equal to $0.9$; we refer to this situation as stabilization, and it is the strategy with such maximum probability that is chosen for each actor. Otherwise, the learning process is stopped after 50,000 steps. The chosen strategic profile is made up for each player in the strategy with the highest probability.

Experiments show first that the set of generated instances each admits a non-empty set of Nash equilibria, which are all equivalent, i.e., where only variations concern the strategies of certain actors in deficit mode, variations that do not change the utility of any player. These Nash equilibria therefore all have the same $Per{f}_t$ value. This value is minimum on all possible strategic profiles, and only these balances reach this value, as indicated by the execution of TEM. The ratio between the best and the worst value of $Perf$ for a profile is on average $0.25$ among all the instances, which shows the variability of the set of strategic profiles for each instance in terms of performance.

The learning method stabilizes for 12 instances over 36, with an average of 18,000 learning steps for those 12 instances. Whether the method has stabilized or not, the strategic profile provided is in all cases a Nash equilibrium. We can conclude that on this set of instances, RL requires a relatively small number of steps to determine the Nash equilibrium, whether it stabilizes or not. Moreover, the evolution of the strategy probabilities during the learning process reveals two types of well-discriminated instances. By definition, actors and instances for which learning stabilizers have a maximum probability greater than $0.9$. For each of the other instances, the mean value of the maximum probability is globally comprised between $0.78$ and $0.82$.

Figure 3 Examples of stochastic vectors evolution for two instances compares the evolution of the probabilities during RL execution on two instances from the set we generated, instances 46 and 35. For instance 46, the RL process stabilizes after 9500 steps, while for instance 35 it doesn’t achieve stabilization. The figure compares the evolution of the probabilities during RL execution for three actors (one for each state in $\{Deficit,Self,Surplus\}$) chosen randomly, from those two instances. As for the majority of Deficit actors in stabilizing instances, the evolution of the probabilities of such a player in instance 46 shows that the number of learning steps is only due to time needed by such players to choose a strategy, whereas the players in state Self or Surplus choose the right strategy very quickly.

The learning time is therefore mainly used by the players in Deficit to arbitrate between the use of their stock and the use of SG. As illustrated by curves concerning Instance 35 in the figure, the instances that do not stabilize see the probabilities of the actors in state Self remaining at $0.5$ during the training, which has a negative impact on the average of the best probabilities of each actor (even if this situation does not prevent the instance from finally choosing a Nash equilibrium). Indeed, in these instances, some actors $a_i$ in State Self are such that $C_i^t=P_i^t>0$ and $S_i^t=0$; for these actors, the two strategies are equivalent, with a utility equal to zero. These are the worst cases for the effectiveness of learning, but RL still succeeds in determining the Nash equilibrium.

We consider also another type of instances with 10 actors for which there does not systematically exist pure Nash equilibria.

• 4 actor $a_i$ in State Deficit with $S_i^t=0$, $C_i^t=15$, $P_i^t=0$.
• 3 actor $a_i$ in State Self with two possible situations with probability $0.5$: one in which $S_i^t=0$, $C_i^t=10$ and $P_i^t$ uniformly chosen in $\{8,\dots ,12\}$; the other in which $S_i^t=0$, $C_i^t=31$ and $P_i^t$ uniformly chosen in $\{21,\dots ,30\}$.
• 3 actor $a_i$ in State Surplus with $S_i^t=0$, $C_i^t=20$, $P_i^t=26$.

We also consider $S_{max}=50$ and $p{p}_i^t=0,3$ for each actor $a_i$. Note that ${\varphi }_{EPO}^{+}$, ${\varphi }_{EPO}^{-}$, ${\pi }_0^{+}$ and ${\pi }_0^{-}$ are the same as in the previous instance generation scenario.

In the simulation, we run at most 30,000 learning steps in each of the 50 period instances. In this instance set, only 4 instances admit a Nash equilibrium, with the same properties as the ones of stabilizing instances of the previous scenario. As regards the 6 other instances, there is no stabilization. The ratio of values of $Per{f}^t$ given by RL and the one given by TEM varies between $0.85$ and $0.92$, knowing that the mean value of $Pe{r}^t$ by RL is equal to $-1466$ for these six instances. The difference between the actor profiles given by RL and TEM is that the values of $Per{f}^t$ is $3\%$ less in TEM profiles than in RL ones, ie that these actors in State Surplus are solicited by SG upon to contribute beyond their own interest, which contradicts the objectives of the proposed model.

Finally, we focus on the evolution of the strategy probabilities during the execution of RL for a two-player instance resulting from the proof of Lemma 1. This instance is defined by

• $S_1=1$, $C_1=3$, $P_1=1$ and ${\gamma }_1=2$,
• $S_2=1$, $C_2=1$, $P_2=3$ and ${\gamma }_2=3$,
• $S_{max}=2$, ${\pi }_0^{-}=1$, ${\pi }_0^{+}=2$, ${\Pi }_{EPO}^{-}=6$ and ${\Pi }_{EPO}^{+}=3$.

In this case where there is no Nash equilibrium, as shown in Figure 4 Evolution of the probabilities of the strategies of the two actors during RL steps, at the beginning of learning, actor 2 quickly confirms its choice of the DIS strategy, that is to say to favor its stock. It is the consequence of actor 1 first favoring the $CON{S}^{-}$, which leads it to use all the energy that actor 2 overproduces. Therefore, actor 2 does not derive any benefit from a sale of electricity to EPO.

When actor 1 eventually learns to prioritize its stock (strategy $CON{S}^{+}$), actor 2’s decision to choose DIS has reached a probability too high to be challenged by RL, even if it begins to decline again slightly. In this example, it is therefore the initial decision of the actor in the Surplus state, which dominates the learning process.

Thus, to summarize all this RL performance evaluation, through all these experiments, it appears that when a Nash equilibrium exists and corresponds to an optimal value of $Perf$, RL succeeds in determining it (note that it appears experimentally difficult to generate instances having a Nash equilibrium but which is not an optimal). Among these instances, those that do not stabilize are those in which the two strategies are equivalent for a certain number of actors in state Self, without this preventing RL from determining the Nash equilibrium. When a Nash equilibrium does not exist, RL determines a profile, which remains efficient in terms of $Per{f}^t$ compare to the one provided by TEM, without sacrificing the situation of actors in Surplus state in particular. This experimentation shows the interest to use RL in this context, and it is now a question of evaluating the performance of RL for a series of periods, which is studied in the next section.

3.2. Multi-Periods Simulation

3.2.1. Instances Generation

We consider the following scenario of instances generation over consecutive periods, where the situation of each of these actors at any period $t>1$ is determined from the one at period $t-1$ by considering one of the transition automata given in Figure 5 Transition states automata for instances generation.

Data for each actor $a_i$ at each period t are generated as follows. Consider first that each actor $a_i$ is in one of four situations defined as follows:

• Situation A: $C_i^t=10$ and $P_i^t$ is uniformly chosen in $\{2,\dots ,4\}$; $S_i^{max}=10$.
• Situation B1: $C_i^t=10$ and $P_i^t$ is uniformly chosen in $\{8,\dots ,12\}$; $S_i^{max}=6$.
• Situation B2: $C_i^t=22$ and $P_i^t$ is uniformly chosen in $\{18,\dots ,22\}$; $S_i^{max}=15$.
• Situation C: $C_i^t=20$, $P_i^t=26$; $S_i^{max}=20$.

Considering all the periods, the average electricity quantity that is consumed per period by actors is equal to 400 and the average produced one is 380.

This instance generation tends to represent a situation in which a smart grid connects two types of actors. On the one hand, players that are not very productive relatively to their needs (situations A and B1), typically old collective housing or administrative buildings, on the other hand players with high consumption but with significant production capacities and energy storage (situations B2 and C), typically eco-responsible buildings or recent industrial buildings.

For $t=1$, we consider initially 8 actors in Situation A (with initial stock $S_i^1=3$), 5 actors in Situation $B1$ (with initial stock $S_i^1=4$), 5 actors in Situation $B2$ (with initial stock $S_i^1=10$) and 8 actors in Situation C (with initial stock $S_i^1=10$).

Finally, as in Section 3.1—Learning performance analysis, we consider ${\varphi }_{EPO}^{+}\left(x\right)=10\times x$, ${\varphi }_{EPO}^{-}\left(x\right)=20\times x$, ${\pi }_0^{+,1}=4$ and ${\pi }_0^{-,1}=3$.

3.2.2. Performances Evaluation

Considering each of the three scenario above, we generate, for each actor, values $C_i^t$ and $P_i^t$ for 50 consecutive periods. On each of the such obtained three sequences of values, we compare at each period the performances of RL, a Systematic algorithm (SyA) and a Selfish-deterministic algorithm (SDA). It should be noted that since the methods we propose in this article do not consist in constructing precise predictions of consummations and productions, but rather in calculating actions that tend to optimize a controlled distributed system, based on feedback from the environment of the system, we think that the performance of such a system cannot be summed up in a single global indicator (as for example the accuracy index of a predictive algorithm using statistical learning methods such as Deep Learning). Several parameters must be considered together in order to decide which solution is the best fit. We therefore propose here to take into account three indicators evaluating learning performance (Aperf), effective revenue (ER) and virtual revenue (VR), as defined below.

Regarding RL, we consider a maximum number of 10,000 learning steps; indeed Section 3.1—Learning performance analysis shows that even if there is no stabilization before 10,000 steps, the obtained profile is the same as the one after 50,000 steps. The purpose of these algorithms is to determine a strategy profile for actors at each period.

The Systematic algorithm fixes mode $CONS+$ for any actor in State Deficit and mode $DIS$ for any actor in States Self or Surplus; this algorithm consists of having each actor systematically feeding or using its stock.

The Selfish-deterministic algorithm (SDA, Algorithm 1) is defined as follows:

Algorithm 1: SDA

In this algorithm, at each period, each actor unilaterally manages its stock considering its needs in the next period.

For each algorithm, we focus on the average value $APerf=\frac{{\displaystyle \sum _{1\le t\le T}}\left(Per{f}_t={\displaystyle \sum _{1\le i\le N}}{V}_i^t\right)}{T}$ of $Per{f}_t$ over all periods. $V_i^t$ is the utility of actor $a_i$ at period t, considering its benefit $be{n}_i^t$, cost $cs{t}_i^t$ and stock incentive ${\gamma }_i^t$ (see Section 2.2.1 and Section 3.1). Moreover, for each of the two methods we focus on the values of the Effective Revenue $ER={\displaystyle \sum _{i=1}^N}E{B}_i$, and the Virtual Revenue $VR={\displaystyle \sum _{i=1}^N}I{B}_i-I{C}_i$, i.e., the one obtain by considering virtual prices ${\pi }_0^{-,t}$ and ${\pi }_0^{+,t}$ at each period $t\le T$; remind also that $E{B}_i$ is the real economic balance (see Section 2.1.4—A pricing model for a smart grid) and that $I{B}_i$ and $I{C}_i$ are internal benefit and cost of actor $a_i$ respectively (see Section 2.1.2—Action modes of each actor).

Finally, to evaluate if the situations obtained in each period by RL do not disadvantage the actors in state Surplus, compared to situations obtained by SDA, we define two partial metrics concerning $ER$. We thus also define experimentation parameters ${\tau }_I=E{R}_{RL}^I-E{R}_{SDA}^I$ and ${\tau }_{II}=E{R}_{RL}^{II}-E{R}_{SDA}^{II}$, where $E{R}_{RL}^I$ and $E{R}_{RL}^{II}$ are the values of $ER$ restricted to players initially in Situations A and B1 and the ones in Situations B2 and C, respectively, and where similarly $E{R}_{SDA}^I$ and $E{R}_{SDA}^{II}$ are the values when using SDA.

The experimentation provides the metric values presented in the following

Table 3. Performance comparison of learning algorithms.

Metrics	RL	SDA	SyA
$Aperf$	$-592$	$-782$	$-404$
$ER$	$-9802$	−12,790	−12,054
$VR$	$-1236$	−15,652	−14,240

.

These results show first of all that the SyA method has slightly better performances than those of SDA concerning ER, which seems to indicate that a selfish approach systematically favoring the current stock of each actor is more effective than a deterministic approach aiming to keep into account needs of future stock and sharing within SG. This is no longer the case when such a context is handled by RL, which gives the best performances concerning ER. It should also be noted that the differences in values of V between on the one hand RL and on the other hand SyA and SDA show that these two last methods provide strategic profiles of very poor performances in the sense of the game defined within SG.

Finally, we note that concerning $Aperf$, which calculates the profit in the game virtual economic model of SG, SyA is more efficient than RL because it converges towards profiles that are not Nash equilibria and which, therefore, appear less efficient in terms of real profits (ER). In all the periods generated on the basis of the instance generation scenario described above, it therefore clearly appears that the use of a reinforcement learning method is necessary to allow all the actors of the smart grid to converge towards a less costly situation for all, while preserving the independence of decision of each.

Moreover, the couples of values $({\tau }_I,{\tau }_{II})$ are equal to $(4509,-1521)$. This shows that when considering the objective of keeping account for future stock needs and sharing within the smart grid, with SDA, the overproduction of actors in Situations B2 and C is totally solicited by SG to provide as much energy as possible to the actors in states Situations A and Situations B1, which then have a very low cost to pay. With RL, we see that the interest of each over-productive actor (states Situations B2 and C) is preserved, which increases the cost of other actors. But overall the cost paid by SG to EPO is falling with RL as seen previously. As indicated in Section 2.1.4—A pricing model for a smart grid, a cost allocation solution between actors to globally distribute the cost of the smart grid on each actor could also be considered from the result provided by RL.

3.3. Technical Implementation and Code Performance

Simulations and learning algorithms were implemented on a computer composed of i7-10870H CPU @ 2.20GHz, 8 cores, 16 threads, 64Go RAM. Both are implemented using Python 3.8, with libraries = NumPy (1.20.1) and Pandas (1.2.4).

Even though the system is distributed by nature, simulations for all agents are centralized, in order to compute values of the utility functions. On one period with 50 agents, and 50,000 learning steps for each agent, one simulation takes between 30 min and 1 h. It should be noted that what takes most computing time is the computation of all function utility values.

4. Conclusions

In this article, we proposed a distributed and autonomous decision model based on game theory of a smart grid interconnecting prosumers, associated with an internal economic virtual model that must converge during the execution time towards a real economic model imposed by an external energy supplier (EPO). We show that, by integrating a forecast of the necessary energy stock, the use of learning techniques by reinforcement allows convergence on economically relevant situations for the smart grid while respecting the interests of each prosumer.

When considering a single period of time, these learning approaches proved to be optimal in terms of the energy consumption for all the instances that we considered and allowed for reducing energy costs for all actors of the smart grid. When learning is done across several time periods, it was more efficient than both a simple deterministic algorithm that feeds or uses its stock and a deterministic selfish algorithm that uses a consumption forecast. Moreover, convergence time, which is often an issue with reinforcement learning, remained low here.

As a first approach, the physical operation of smart grids was considered as being able to respond to any set of actions by prosumers without problems in the distribution capacity. Thus, we assumed that the smart grid is able to meet any demand for power distribution. An extension of our approach will have to take into account the dynamic configuration of the network, meaning that the capacity of its components must be able to adapt to each set of prosumer demands. The proposed RL approach will then have to be coupled with a dynamic configuration algorithm of the smart grid topologies, inspired, for example, by [39].

We also assume that the smart grid does not experience any failures or malfunctions. Such malfunctions would impact the actions chosen by each actor and would therefore imply a change of strategic profile "on the fly" imposed by the control of the smart grid by real-time processing when the energy distribution faults are perceived by prosumers. From the point of view of the model to be considered, such failures could follow a stochastic model (for example based on Bayesian networks).

Moreover, since, in this article, we wanted to demonstrate the relevance of an approach using reinforcement learning based on a game model for the distributed management of a smart grid interconnecting prosumers, production and consumption data were considered certain. It would be interesting in future work to perform a sensitivity analysis in order to study the robustness of this approach in the face of the uncertainty of the energy production and consumption data. As already studied in other fields of application, it will be a question in particular of developing RL methods capable of adapting to elastic data considering production and consumption and modeled by stochastic processes.

Finally, in typical smart grids, some prosumers may include electric car charging stations. These charging stations have a significant impact on the management of the configuration of the smart grid, since the energy stored at one point of the network can be relocated (minus a certain consumption made by the electric vehicle during its movement) to another point of gate entrance. In addition, all the energy behaviors of these prosumers depend on the road topology on the territory covered by the smart grid (or even wider), which must therefore be taken into account. Further work could be carried out in order to learn about these types of profiles.