Model-Free Approach to DC Microgrid Optimal Operation under System Uncertainty Based on Reinforcement Learning

Abstract

There has been tremendous interest in the development of DC microgrid systems which consist of interconnected DC renewable energy sources. However, operating a DC microgrid system optimally by minimizing operational cost and ensures stability remains a problem when the system’s model is not available. In this paper, a novel model-free approach to perform operation control of DC microgrids based on reinforcement learning algorithms, specifically Q-learning and Q-network, has been proposed. This approach circumvents the need to know the accurate model of a DC grid by exploiting an interaction with the DC microgrids to learn the best policy, which leads to more optimal operation. The proposed approach has been compared with with mixed-integer quadratic programming (MIQP) as the baseline deterministic model that requires an accurate system model. The result shows that, in a system of three nodes, both Q-learning (74.2707) and Q-network (74.4254) are able to learn to make a control decision that is close to the MIQP (75.0489) solution. With the introduction of both model uncertainty and noisy sensor measurements, the Q-network performs better (72.3714) compared to MIQP (72.1596), whereas Q-learn fails to learn.

1. Introduction

There has been tremendous interest in generating electric energy from renewable energy resources [1,2] owing to environmental concerns and current advances in power electronics. Unlike conventional power plants, which often come in large sizes, the capacity of renewable power plants strongly depends on local potential and ranges from a few kilowatts to several megawatts.

Currently, small-sized renewable energy power plants such as micro hydro, photo-voltaic (PV), or wind turbines, are often connected to local loads at the distribution level to configure autonomous alternating current (AC) microgrids [3]. These systems are expected to be able to work with and without the grid connections. However, the problems of reactive power compensation and frequency control in AC microgrids (ACMG) can be more challenging than conventional AC systems due to the intermittent nature of wind and solar power sources [4].

As most renewable energies, especially wind turbine type-4 and PV, inherently produce direct current (DC) outputs, it is natural to interconnect them using DC voltage to configure direct current microgrids (DCMG). Different from AC microgrids, DC microgrids have no problems with reactive and frequency controls [4]. Moreover, in a DC microgrid system, the use of an energy storage system (ESS), notably a battery energy storage system (BESS), becomes indispensable to balance the intermittency. This has been demonstrated in [5] where a method to coordinate several ESSs has been proposed.

DC voltage in a DC grid can be considered as the power balance indicator, i.e., similar to frequency in an AC grid [4,6,7,8,9]. By controlling the DC voltage level within the grid, a certain power flow can be achieved. The DC source connected to a DC grid can be operated in three different modes, i.e.,: droop-controlled voltage, constant power, or constant voltage mode. The response of these modes when subjected to a disturbance is illustrated in Figure 1.

Wind or PV power plants are usually operated at their maximum power point, hence this DC source is often operated in active power control, i.e., injecting active power to the DC microgrid system regardless of the DC voltage condition on the DC microgrid. This also means that the intermittent operation of these power plants is reflected in the DC voltage fluctuations. In order to counter the changes in the DC voltage, BESS can be operated in the DC voltage mode to maintain a constant DC voltage. This means that BESS can be absorbing or injecting power to the grid whenever there is a surplus or deficiency of power in the DC microgrid. However, the operation of BESS is a complex problem as it has special constraints such as Depth of Discharge (DOD) requirements, State of Charge (SOC) limitations, Charge and Discharge rates, among others [10]. Therefore, it might happen that BESS switches its mode to DC voltage droop mode to limit the contribution to maintain the DC voltage within the DC microgrid or even operate in the constant power mode when reaching its charge or discharge limit.

1.1. Related Work

In order to operate DC microgrids in an optimal way, a method to minimize operating costs has been proposed in [11] using a particle swarm optimization (PSO) algorithm. A similar idea of minimizing operating cost while improving stability has been proposed in [12] using a switch system approach called supervisory control (SC), which appropriately selects the operating mode to optimize the control objectives. In another study using adaptive dynamic programming (ADP) [13] and tabu search (TS) [14], researchers found that using a central control system that coordinates distributed control devices across multiple nodes in the microgrid can help optimize a common objective. The switching problem of voltage sources across multiple nodes has been further studied in [15] to guarantee the analytic performance of the proposed controller. The author in [15] formulated the switching problem of multi-node voltage sources as mixed-integer linear programming (MILP) problem. The stability study used in [15] is based on the previous study of a small signal model for low-voltage DCMG (LVDCMG) in [16].

Although the proposed methods in the literature have demonstrated satisfactory performances in the scenarios presented in the corresponding papers, they necessitate the exact model of DC grids. This assumption is very restrictive for several reasons. Firstly, from the practical aspect, DC microgrids consist of several components coming from different vendors. Each vendor is normally unwilling to share the detail of components, e.g., the transfer functions of the components are their controllers, to safeguard their intellectual properties. Secondly, even if accurate models are available, some components experience wear and tear after a certain time, which leads to model uncertainty. A study exploring the incomplete state model is explored in [17] by considering communication delay. The author overcame the lack of an exact system model by implementing a heuristic-based approach, that is, a genetic algorithm (GA). That is because GA did not require an exact model to obtain a sufficient solution approximation. Hence, the natural remedy for this is to use a method that does not rely on the exact model of the DC grids.

In the current power system hierarchy, there have been large amounts of data generated by, for example, smart meters and phasor measurement units (PMUs). These data contain very rich information regarding operating status of the systems at a time instance. On the other hand, recent progress in applied mathematics and data science leads to the possibility to extract useful information for power system operation without prior supervisory knowledge on how to extract meaningful information from data. An example of this is the reinforcement learning (RL) [18] approach, which is a method in machine learning allowing an agent to take a sequence of appropriate actions to maximize a certain objective. This method has recently been adopted in power systems [19], for instance to solve the Volt/Var control (VVC) problem in medium voltage AC (MVAC) distribution systems [20], real-time wide-area stabilizing control in a high-voltage AC (HVAC) system [21], and a data-driven load frequency control in a multi-area HVAC system [22].

The purpose of this study is to propose a novel method to solve the DC microgrid switching problem optimally given limited information about the exact model of DC grids based on a reinforcement learning algorithm. The limitation is modeled as noise and error in the system matrix model, as a form of simulating noise in sensor measurements and an imprecise power system model. The table highlighting the difference between our study compared with other works can be seen in the comparison of related works in

Table 1. Comparison of Related Works.

Paper	Systems	Methods
		Algorithm	Objective	Unknown
[11]	DCMG	PSO	Economic, Environment	Droop
[12]	DCMG	SC	Stability	Mode
[15]	DCMG	MILP	Economic, Stability	Mode
[13]	DCMG	ADP	Stability	Policy
[14]	DCMG	RL, TS	Topology, Stability	Edge, Policy
[17]	ACMG	GA	Stability, Frequency	Generation
[20]	AC	RL	Economic, Stability	Tap Changer
[21]	AC	RL	Stability	Control Signal
Ours	DCMG	RL	Economic, Stability	Mode
Paper	Voltage Level	Data Challenge
		Delay	Noise	Error
[11]	LV	-	-	-
[12]	LV	-	-	-
[15]	LV	-	-	-
[13]	LV	-	-	-
[14]	LV	-	-	-
[17]	LV	✓	-	-
[20]	MV	-	-	-
[21]	HV	-	-	-
[22]	HV	-	-	-
Ours	LV	-	✓	✓

. The data challenges shown in the table refer to communication delay, measurement noise, and model error.

1.2. Contributions

Due to the fact that accurate models of DC grids are not always readily available, this paper proposes a novel model-free approach to performing operation control of DC microgrids based on reinforcement learning. The variances of reinforcement learning algorithm is proposed in the paper because of its ability to learn efficiently from past data of the state and action, such as the current and operation cost data which correspond to any mode of operation in the context of a DC microgrid system. In addition, we show that our proposed approach can cope with the model uncertainty due to noise in sensor measurement and imprecise power system model, while producing near-optimal policy which takes both stability and operational cost into consideration.

The contributions of this paper can be summarized as follows:

• Propose a novel model-free approach for solving the LVDCMG optimal switching problem
• Demonstrate the ability of a reinforcement learning algorithm to solve LVDCMG optimal switching problem under measurement noise and imprecise power system mode
• Provide a minimal working example for applying reinforcement learning parameters in the LVDCMG optimal switching problem

2. Operation Control of DC Microgrids

2.1. Models of DC Microgrids

Each power source in distributed droop-controlled LVDCMG can be operated in various different modes of operation, that is droop-controlled voltage source (DVS), a constant power source (CPS), or a constant power load (CPL) [15,16] as can be seen in Figure 2. For instance, a battery can be operated as either a CPS on discharging, CPL on charging, or a DVS both at charging and discharging.

For each droop-controlled voltage source $j\in {\mathcal{N}}_d$,

$$V_j=V_j^0-d_j{P}_{Dj},$$

(1)

where $V_j$ is the droop controller output voltage, $V_j^0$ is the droop controller nominal voltage value, $d_j$ is the droop controller gain, and $P_{Dj}$ is the droop controller output power. The small-signal representation of the droop-controlled voltage source can be modeled as

$$v_j=-(\frac{d_j{\overline{V}}_j}{1+d_j{\overline{I}}_j})i_{Dj}$$

(2)

where $v_j$ is the small-signal perturbation of the voltage, ${\overline{V}}_j$ and ${\overline{I}}_j$ are the DC voltage and current steady-state operating point, respectively, and $i_{Dj}$ is the corresponding small-signal perturbation of the current. The individual small-signal models for each node are aggregated which leads to

$$\mathbf{v}=-\tilde{\mathbf{d}}{\mathbf{i}}_D,$$

(3)

where $\mathbf{v}:=[v_1,v_2,\cdots ,v_{N_d}]$, ${\mathbf{i}}_D:=[i_1,i_2,\cdots ,i_{N_d}]$, and $\tilde{\mathbf{d}}$ are a diagonal matrix with diagonal elements $\frac{d_j{\overline{V}}_j}{1+d_j{\overline{I}}_j}$, and $j=1,2,\cdots ,N_d$.

Suppose that the microgrids consist of $N_{CPS}$ number of CPsS out of $N_s$ sources where the set of CPSs is given by ${\mathcal{N}}_{CPS}:=\left\{1,2,\cdots ,N_{CPS}\right\}$. By modeling a CPS as a current source in parallel with a conductance for each CPS $j\in {\mathcal{N}}_{CPS}$, we have the following models

$$i_{CP{S}_j}=g_{CP{S}_j}{v}_j+{\tilde{i}}_{CP{S}_j},$$

(4)

where $i_{CP{S}_j}$ and $v_j$ are the corresponding CPS small-signal output current and voltage. The values of the corresponding conductance $g_{CP{S}_j}$ and current source ${\tilde{i}}_{CP{S}_j}$ are determined by the power and voltage at an operating point as follows

$$g_{CP{S}_j}=\frac{P_{CP{S}_j}}{V_{D{C}_j}^2},$$

(5)

$$I_{CP{S}_j}=\frac{2P_{CP{S}_j}}{V_{D{C}_j}},$$

(6)

where $P_{CP{S}_j}$ and $V_{D{C}_j}$ are output power and nominal voltage at operating point of the CPS. It needs to be noted that $I_{CP{S}_j}$ is the nominal current source value, while ${\tilde{i}}_{CP{S}_j}$ is the small-signal perturbation of the current source. All the small-signal model can be aggregated into the following equation

$${\mathbf{i}}_{CPS}={\mathbf{g}}_{CPS}{\mathbf{v}}_{CPS}+{\tilde{\mathbf{i}}}_{CP{S}_j},$$

(7)

where

$${\mathbf{i}}_{CPS}:=[i_{CP{S}_1},i_{CP{S}_2},\cdots ,i_{CP{S}_{N_{CPS}}}],$$

$${\mathbf{v}}_{CPS}=[v_1,v_2,\cdots ,v_{N_{CPS}}],$$

$${\tilde{\mathbf{i}}}_{CPS}=[{\tilde{i}}_{CP{S}_1},{\tilde{i}}_{CP{S}_2},\cdots ,{\tilde{i}}_{CP{S}_{N_{CPS}}}],$$

$${\mathbf{g}}_{CPS}=\left[\begin{array}{ccc}{g}_{CP{S}_1}& 0& 0\\ 0& \cdots & 0\\ 0& 0& g_{CP{S}_{N_{CPS}}}\end{array}\right],$$

Similar to CPS, a CPL is modeled as a negative conductance in parallel to a current sink. Suppose that the set of CPLs is given by ${\mathcal{N}}_{CPL}:=\left\{1,2,\cdots ,N_{CPL}\right\}$ where $N_{CPL}$ refers to the number of CPLs in the microgrids. Each CPL $j\in {\mathcal{N}}_{CPL}$ can be modeled as follows

$$i_{CP{L}_j}=-g_{CP{L}_j}{v}_j+{\tilde{i}}_{CP{L}_j},$$

(8)

where $i_{CP{L}_j}$ and $v_j$ are the corresponding small-signal output current and voltage of the CPL, respectively. The values of the corresponding conductance $g_{CP{L}_j}$ and current sink ${\tilde{i}}_{CP{L}_j}$ are determined by the power and voltage at an operating point as follows:

$$g_{CP{L}_j}=\frac{P_{CP{L}_j}}{V_{D{C}_j}^2},$$

(9)

$$I_{CP{L}_j}=\frac{2P_{CP{L}_j}}{V_{D{C}_j}},$$

(10)

where $P_{CP{L}_j}$ and $V_{D{C}_j}$ are the output power and DC nominal voltage at the operating point of the CPL. The small-signal model can be aggregated into

$${\mathbf{i}}_{CPL}=-{\mathbf{g}}_{CPL}{\mathbf{v}}_{CPL}+{\tilde{\mathbf{i}}}_{CP{L}_j},$$

(11)

where

$${\mathbf{i}}_{CPL}:=[i_{CP{L}_1},i_{CP{L}_2},\cdots ,i_{CP{L}_{N_{CPL}}}],$$

$${\mathbf{v}}_{CPL}=[v_1,v_2,\cdots ,v_{N_{CPL}}],$$

$${\tilde{\mathbf{i}}}_{CPL}=[{\tilde{i}}_{CP{L}_1},{\tilde{i}}_{CP{L}_2},\cdots ,{\tilde{i}}_{CP{L}_{N_{CPL}}}],$$

$${\mathbf{g}}_{CPL}=\left[\begin{array}{ccc}{g}_{CP{L}_1}& 0& 0\\ 0& \cdots & 0\\ 0& 0& g_{CP{L}_{N_{CPL}}}\end{array}\right].$$

Suppose that a set of $N_b$ power lines connecting the adjacent vertices (sources and loads) in the microgrid is given by ${\mathcal{N}}_b:=\left\{1,2,\cdots ,N_b\right\}$. Each power line $j\in {\mathcal{N}}_b$ can be modeled as follows

$$v_{b_j}=l_{b_j}\frac{d{i}_{b_j}}{dt}+r_{b_j}{i}_{b_j},$$

(12)

where $v_{b_j}$ and $i_{b_j}$ are the power line voltage and current, respectively, and $l_{b_j}$ and $r_{b_j}$ are the power line inductance and resistance with length ${\ell }_j$, respectivley. The aggregate of all individual small-signal models can be expressed as

$${\mathbf{v}}_b={\mathbf{l}}_b\frac{d{\mathbf{i}}_b}{dt}+{\mathbf{r}}_b{\mathbf{i}}_b,$$

(13)

where

$${\mathbf{v}}_b:=[v_{b_1},v_{b_2},\cdots ,v_{b_{N_b}}],$$

$${\mathbf{i}}_b=[i_{b_1},i_{b_2},\cdots ,i_{b_{N_b}}],$$

$${\mathbf{l}}_b=\left[\begin{array}{ccc}{l}_{b_1}& 0& 0\\ 0& \cdots & 0\\ 0& 0& l_{b_{N_b}}\end{array}\right],$$

$${\mathbf{r}}_b=\left[\begin{array}{ccc}{r}_{b_1}& 0& 0\\ 0& \cdots & 0\\ 0& 0& r_{b_{N_b}}\end{array}\right].$$

Finally, the DC microgrids follow Kirchhoff’s voltage law (KVL) and Kirchhoff’s current law (KCL) as follows

$${\mathbf{v}}_b=\mathbf{Mv},$$

(14)

$${\mathbf{i}}_D-{\mathbf{i}}_{CP}={\mathbf{M}}^T{\mathbf{i}}_b.$$

(15)

By combining (3), (7), (11) and (13)–(15), the dynamics of the DC microgrids can be summarized in the following state equations:

$$\dot{\mathbf{x}}=\mathbf{A}\mathbf{x}+\mathbf{B},$$

(16)

where $\mathbf{x}={\mathbf{i}}_b\in {\mathbb{R}}^{N_x}$ and

$$\mathbf{A}=-{\mathbf{l}}_b^{-1}(\mathbf{M}{({\tilde{\mathbf{d}}}^{-1}-{\mathbf{g}}_{CP})}^{-1}{\mathbf{M}}^T+{\mathbf{r}}_b),$$

(17)

$$\mathbf{B}=-{\mathbf{l}}_b^{-1}\mathbf{M}{({\tilde{\mathbf{d}}}^{-1}-{\mathbf{g}}_{CP})}^{-1}.$$

(18)

Any operational modes (i.e., DVS, CPS, or CPL) will follow linear state equations described in (16), albeit with different values of $\mathbf{A}$ and $\mathbf{B}$. Therefore, the dynamics can be slightly modified as follows

$$\dot{\mathbf{x}}={\mathbf{A}}^{\sigma }\mathbf{x}+{\mathbf{B}}^{\sigma },$$

(19)

where ${\mathbf{A}}^{\sigma }$ and ${\mathbf{B}}^{\sigma }$ stand for matrix $\mathbf{A}$ and $\mathbf{B}$, respectively, for mode $\sigma \in (1,N)$, where N refers to the total number of modes under consideration.

The DC microgrid’s dynamics can be reformulated in discrete forms as follows

$$\mathbf{x}(k+1)=\mathbf{x}(k)+\dot{\mathbf{x}}(k)\Delta t,$$

(20)

where $\Delta t$ is a time-integration constant. Substituting (19) to (20), the complete dynamics in discrete forms can be rewritten as

$$\mathbf{x}(k+1)={\mathbf{A}}_{\mathbf{k}}^{\sigma }\mathbf{x}(k)+{\mathbf{B}}_{\mathbf{k}}^{\sigma },$$

(21)

where ${\mathbf{A}}_{\mathbf{k}}^{\sigma }={\mathbf{A}}^{\sigma }\Delta t+1$ and ${\mathbf{B}}_{\mathbf{k}}^{\sigma }={\mathbf{B}}^{\sigma }\Delta t$. Therefore, the decision variable that we can give to the system is $\mathbf{u}(k)={\left[\begin{array}{cccc}{u}_1(k)& u_2(k)& \cdots & u_N(k)\end{array}\right]}^T$, where $u_{\sigma }\in (0,1)$ refers to the status of mode $\sigma$ of the DC microgrid and is given by

$$u_{\sigma }=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\mathrm{mode}\sigma \mathrm{is}\mathrm{active}\hfill \\ 0\hfill & \mathrm{if}\mathrm{mode}\sigma \mathrm{is}\mathrm{inactive}\hfill \end{array}\right..$$

(22)

Note that only one mode can be active at a time.

2.2. Problem Statement

In this paper, we try to find the most optimal operational control for the DC microgrids modeled in Section 2.1. The problem can be formulated as follows:

Problem 1

(Optimal operation control). Given a microgrid system following a certain dynamics $\mathbf{x}(k+1)=\mathbf{f}(\mathbf{x}(k),\mathbf{u}(k))$ which produces an operational cost $c(\mathbf{x}(k),\mathbf{u}(k))$, what is the input signal ${\mathbf{u}}^{*}(k)={\left[\begin{array}{cccc}{u}_1^{*}(k)& u_2^{*}(k)& \cdots & u_N^{*}(k)\end{array}\right]}^T$ which will minimize the total cost C defined as

$$C=\sum _{k=0}^{M}c(\mathbf{x}(k),\mathbf{u}(k)).$$

(23)

Here, N denotes the number of available modes of operation and $u_i\in (0,1)$ for every $i\in (1,N)$ and M denotes the number of steps.

Suppose that the cost function at iteration-k is given by

$$c(k)=\frac{1}{2}\mathbf{x}{(k)}^T\mathbf{Q}\mathbf{x}(k)+{\gamma }_{\sigma }{u}_{\sigma }(k).$$

(24)

where $\mathbf{Q}\in {\mathbb{R}}^{N_x\times N_x}$ stands for a positive definite matrix while ${\gamma }_{\sigma }$ stands for the cost of operating the current active mode $\sigma$. Here, the first term corresponds to the stability of the system (i.e., an unstable system causing large values of state $\mathbf{x}$ will be penalized) while the second term corresponds to the operation cost of the operational mode being employed. If the system dynamics $\mathbf{f}(\mathbf{x}(k),\mathbf{u}(k))$ is fully known and linear while the cost function $c(\mathbf{x}(k),\mathbf{u}(k))$ is linear or quadratic, one way to solve the presented optimization problem is to employ linear/quadratic programming. In this case, the system dynamics in (19) can be written in the form of a constraint to the optimization problem. One can then employ algorithm such as the mixed-integer quadratic programming (MIQP) to solve the optimization problem as reported in [15].

However, the requirement to fully know the system dynamics accurately might be difficult to realize in many practical situations. Some parameters in a DC microgrid system might not be known beforehand due to communication delay [17]. Meanwhile, system identification to acquire the model parameters could be complicated, especially if the system contains a large number of components.

3. Reinforcement-Learning-Based Operation Control

3.1. DC Microgrids as a Markov Decision Process

To solve the optimization problem in Problem 1, we can consider the DC Microgrids with unknown system dynamics as a Markov Decision Process (MDP) [23]. By doing so, we assume that the DC Microgrids system fulfills the following criteria:

• the future state $\mathbf{s}(k+1)$ only depends on the current state $\mathbf{s}(k)$, not the previous state history,
• the system accepts a finite set of actions $\mathbf{a}(k)$ at every step,
• the system will provide state information $\mathbf{s}(k)$ and reward $r(k)$ at every step.

The reward function $r(k)$ denotes how good the performance of the state $\mathbf{s}(k)$ at iteration-k is. Therefore, the optimization problem described in Problem 1 where the cost function C needs to be minimized can be transformed into an optimization problem to find a policy $\pi ={\mathbf{u}}^{*}(k)$ which will maximize the total reward $R={\sum }_{k=0}^{M}r(k)$ over M steps.

This discrete dynamics equation in (21) can be transformed into an MDP where the state $\mathbf{s}$ is given by $\mathbf{s}={\left[\begin{array}{cc}\mathbf{x}& c\end{array}\right]}^T$ and the action $\mathbf{a}=\mathbf{u}$. We assume that the cost $c(k)$ follows Equation (24). The reward function $r(k)$ can be derived from the cost function $c(k)$ as follows:

$$r(k)=\left\{\begin{array}{cc}{c}_s-c(k)\hfill & \mathrm{if}{c}_s>c(k)\hfill \\ 0\hfill & \mathrm{if}{c}_s\le c(k)\hfill \end{array}\right.,$$

(25)

where $c_s$ denotes a positive constant. Thus, the objective function used to evaluate each model is

$$\underset{\mathbf{u}}{max}\sum _{k=0}^{M}r(k).$$

(26)

3.2. Q-Learning for Near-Optimal Operation Control

To solve the optimization problem for MDP modeling dynamics of DC microgrids, a reinforcement learning approach is employed. This approach allows an agent to interact with an environment (i.e., system) without knowledge of the system’s dynamics by applying an action $\mathbf{a}$ and receiving a state information $\mathbf{s}$ as well as a reward r as shown in Figure 3. The agent then learns from this information to decide the most optimal policy $\pi =\mathbf{a}(k)$, where $k\in (0,M)$, which will maximize the cumulative rewards $R={\sum }_{k=0}^{M}r(k)$ for M number of steps.

The reinforcement learning algorithm employed in this paper falls under the class of algorithm called Q-learning [18] which has been successfully employed for various engineering problems, such as gaming [24] and robotics [25]. The algorithm works by predicting an action-value function $Q(\mathbf{s},\mathbf{a})$ for every state $\mathbf{s}$ and action $\mathbf{a}$ which reflects the expected final reward when choosing an action $\mathbf{a}$ in the state $\mathbf{s}$. Once $Q(\mathbf{s},\mathbf{a})$ is established, the optimum policy $\pi (\mathbf{s})$ for every state $\mathbf{s}$ will select an action which has the most action-value (i.e., will yield the most rewards in the long-term) as given by [18],

$$\pi (\mathbf{s})=arg\underset{\mathbf{a}}{max}Q(\mathbf{s},\mathbf{a}).$$

(27)

For a discrete number of states, the way to build an estimate of action-value function $Q(\mathbf{s},\mathbf{a})$ is by constructing a table which maps a state $\mathbf{s}$ and action $\mathbf{a}$ into a value $Q(\mathbf{s},\mathbf{a})$ as shown in Figure 4. This algorithm, called tabular Q-learning, starts with an empty table. The table will be updated upon interaction with the environment by taking into account information regarding current state $\mathbf{s}$, the given action $\mathbf{a}$, the current reward $r(\mathbf{s})$, and the next state ${\mathbf{s}}^{\prime }=\mathbf{s}(k+1)$. The update equation is based on Bellman’s optimality equation and is given by [18],

$$Q(\mathbf{s},\mathbf{a})\stackrel{}{\leftarrow }Q(\mathbf{s},\mathbf{a})+\alpha (r(\mathbf{s})+\gamma \underset{{\mathbf{a}}^{\prime }}{max}Q({\mathbf{s}}^{\prime },{\mathbf{a}}^{\prime })-Q(\mathbf{s},\mathbf{a})).$$

(28)

Here, $\alpha$ stands for a learning rate constant which determines how much we allow the value of $Q(\mathbf{s},\mathbf{a})$ to change at every iteration and $\gamma$ stands for the reward’s discount factor which determines how much we value the contribution of future rewards to the value of the current state. The iteration process stops once the cumulative reward R over M number of steps reach a certain threshold $R_s$.

One of the main reason in choosing Q-learning to solve the optimization problem in Problem 1 is because it is an off-policy algorithm, i.e., it is able to learn from and reuse any past experience regardless of the policy being used to gather this experience. In the context of DC microgrids, Q-learning is able to learn from past data of the state and action pairs, such as the current and operation cost data which corresponds to any mode of operation. Due to this fact, Q-learning is better in terms of data sampling efficiency compared to employing online reinforcement learning problem such as policy gradient method [26]. Another reason is because it is designed specifically for problems with discrete action space which is the characteristics of the operation control problem, i.e., to determine which mode of operation is the most optimal among a discrete number of possible operations.

3.3. Q-Network for Operation Control under Uncertainty

The tabular Q-learning approach found to be extremely effective in various applications when the number of state $\mathbf{s}$ is discrete. However, this is mostly not the case for a DC microgrid systems which can have a large number of possible current values and operational cost as a state. The presence of uncertainty in the measurement of the state value adds further complications. To handle a system with a large number of states or even a continuous state, the Q-table can be replaced with a function approximator which directly maps state $\mathbf{s}$ and action $\mathbf{a}$ into $Q(\mathbf{s},\mathbf{a})$. One such function approximator which has gained prominence in recent years is a neural network, due to its ability to approximate highly non-linear functions solely from raw data [27]. Recent study in employing neural networks into Q-learning as a function approximator of action-value function $Q(\mathbf{s},\mathbf{a})$ has been carried out in [28]. This approach, called Q-network, works essentially the same as Q-learning, except that the Q-table is replaced with a neural network and the update equation in (28) is replaced with training steps of the neural network.

The neural network $\varphi (\mathbf{s})$, as depicted in Figure 5, accepts a state $\mathbf{s}$ as an input and returns values $Q(\mathbf{s},\mathbf{a})$ of each possible discrete actions $\mathbf{a}$. The target output of the network is derived from Bellman’s equation, similar to the term in Equation (28), as follows:

$$Q_t(\mathbf{s},\mathbf{a})=r(\mathbf{s})+\gamma \underset{{\mathbf{a}}^{\prime }}{max}Q({\mathbf{s}}^{\prime },{\mathbf{a}}^{\prime }).$$

(29)

The losses, given by $\mathcal{L}=Q(\mathbf{s},\mathbf{a})-Q_t(\mathbf{s},\mathbf{a})$ are used to update the network’s parameter $\varphi (\mathbf{s})$ using the gradient descent algorithm.

To facilitate a better training of the neural network (which ideally requires uncorrelated training data), a strategy called experience buffer is employed. Upon interacting with the environment, the agent does not update the neural network’s parameter solely based on the current information, but saves a tuple $\tau$ defined as $\tau =(\mathbf{s},\mathbf{a},{\mathbf{s}}^{\prime },r)$ into a buffer of length $L_b$. In every iteration, a total of $N_t<L_b$ of tuples are randomly sampled from the experience buffer and used as data training. The training process stops once the average cumulative reward over M number of steps $R_{avg}$ in the last $N_E$ number of completed episode reaches a certain threshold $R_s$. To obtain diverse data at the initial stage of training, the agent initially starts with purely random action, i.e., the probability $ϵ$ of choosing random action is set to 1. This parameter $ϵ$ is then linearly decreased at every iteration until it reaches a certain minimum value ${ϵ}_{min}$ after $N_{ϵ}$ number of iterations. Afterwards, the agent will have a probability of ${ϵ}_{min}$ of choosing a random action and a probability of $1-{ϵ}_{min}$ of choosing the most optimal policy in (27) from the output $Q(\mathbf{s},\mathbf{a})$ provided by the neural network.

4. Methodologies

To evaluate the performance of the algorithm, simulations consisting of 2 environments are used as follows:

• A simple environment: a simplified version of DC microgrid system consisting of a single source and a single load connected via a transmission line ($N_x=1$). The system can be operated in 2 different modalities ($N=2$). The number of update steps in a single episode of operation is set to be $M=28$. For simplicity, the state equation matrices are simplified and assumed to be $\mathbf{A}=\mathbf{B}=1$ for mode $\sigma =1$ and $\mathbf{A}=-10,\mathbf{B}=1$ for mode $\sigma =2$.
• A complex environment: a more realistic DC microgrid system consisting of 3 nodes. Each node consists of a source and/or a load. Two transmission lines ($N_x=2$) connect Node 1 and Node 2 as well as Node 2 and Node 3 as shown in Figure 6. The system can be operated in 2 different modalities ($N=2$). The number of update steps in a single episode of operation is set to be $M=40$. In this environment, it is assumed that the system’s mode of operation can only be updated once every 1 s. The state equation matrices used in this environment are derived from (16)–(22).

The two environments are chosen because of their simplicity, yet are representative of simulating a DCMG power system model in the form of an MIQP problem. The first environment is the most possible simple representation of the system model, while the second environment is a simple building block for any DCMG system topology. For future research, this second environment can be used to build a bigger and more complex DCMG system topology.

For evaluating the performance of the algorithm, we design 3 different scenarios. The first one is when the model is assumed to be perfectly known to the MIQP algorithm and the current measurement is perfectly accurate, i.e., no modeling and measurement uncertainty. The second one is when there is a modeling error, i.e., the model information used by the MIQP algorithm does not perfectly match the real model of the system. In this scenario, the real transition matrix ${\mathbf{A}}^{\sigma }$ and input vector ${\mathbf{B}}^{\sigma }$ for each mode $\sigma$ have an offset of $\Delta \mathbf{A}$ and $\Delta \mathbf{B}$ respectively with respect to the model information known to the MIQP algorithm. Finally, the third and last scenario is when there is a Gaussian noise with mean $\mu$ and standard deviation $\upsilon$ added to the current measurement $\mathbf{x}$. Thus, the third scenario contains both the Gaussian noise and modeling error at the same time.

Apart from implementing Q-learning and Q-network, we also implement an optimization technique based on mixed-integer quadratic programming (MIQP) employed in [15]. The MIQP approach is used as the baseline approach and simulated under the assumption that the dynamics model of the DC microgrid is known. We tested the Q-learning, the Q-network, and the MIQP algorithm to find the most optimal action for each scenario in the given environment. Note that the Q-learning and Q-network do not know anything about the dynamics of the system while the MIQP requires information regarding the model’s dynamics. The neural network employed in the Q-network algorithm contains 1 hidden layer with 4 nodes. The parameters used in the algorithm are listed in detail in

Table 2. List of Parameter Values.

Param	Values		Units
	Simple	Complex
${\gamma }_{Q-learning}$	0.9	0.9	-
${\alpha }_{Q-learning}$	0.2	0.2	-
${\gamma }_{Q-network}$	0.99	0.99	-
${\alpha }_{Q-network}$	0.01	0.001	-
$c_s$	0.2	2	-
$\mu$	0	0	ampere
$\upsilon$	0.01	0.1	ampere
$\Delta t$	0.1	0.0001	second
$\Delta \mathbf{A},\Delta \mathbf{B}$	30	10	%
$\mathbf{Q}$	1	0.00001	-
${\gamma }_{\sigma }$	0.1	1	-
$N_t$	28	40	-
$L_b$	30,000	50,000	-
$N_{ϵ}$	10,000	250,000	-
${ϵ}_{min}$	0.02	0.02	-

.

5. Results

5.1. Simple Environment

First, we present the results of the simulation in the first scenario when there is no modeling or measurement error. For this problem, the Q-learning and Q-network algorithm produce an exactly identical solution shown in Figure 7a,b. The best policy produced by both algorithms is shown in Figure 7a and the resulting state dynamics is shown in Figure 7b.

To evaluate the performance of both algorithms, we compare the cumulative reward per episode as function of training iteration number in Figure 8a. The red line, blue line, and green line indicate the cumulative reward produced by Q-learning, Q-network, and MIQP algorithm, respectively. Note that the cumulative reward of the Q-network is presented as an average cumulative reward over the last 100 episodes to take into account the stochastic nature of the Q-network algorithm. We can observe that the Q-learning algorithm (red line) converges very quickly towards its most optimal solution. The produced cumulative reward is also very close to the optimal solution produced by MIQP (green line). The Q-network algorithm (blue line) produces a solution with less average cumulative reward compared to Q-learning and MIQP. However, as can be observed in Figure 7a, the best solution produced by Q-network matches the one produced by Q-learning. Consequently, as shown in Figure 8b, the reward accumulated by Q-network over 28 steps match the reward accumulated by Q-learning. Overall, the total reward accumulated by both of these algorithm (red line for Q-learning and blue line for Q-network) over 28 steps is only slightly smaller than the optimal solution predicted by MIQP (green line for MIQP) as shown in Figure 8b. This can also be concluded from

Table 3. Summary of Results.

System	Model Error	Noise	Best Reward/Cost per Episode
			Q-Learning	Q-Network	MIQP
Simple	x	x	3.5867	3.5867	3.5871
	✓	x	3.3092	3.6554	3.6293
	✓	✓	x	3.6685	3.6667
Complex	x	x	74.2707	74.4254	75.0489
	✓	x	72.4335	72.3714	72.1596
	✓	✓	x	72.3714	72.1596

where the final reward of Q-learning and Q-network (3.5867) is shown to be really close to the reward of MIQP (3.5871). Note that this performance is achieved by the Q-learning and Q-network without prior information of the system’s dynamics contrary to the MIQP.

The simulation results of the second scenario where there is a modeling error in the system is shown in Figure 9. From the plot of cumulative reward per episode as function of training iteration number in Figure 9a, we can observe that the Q-learning algorithm (red line) once again converges rapidly to the final solution. However, we can observe that the solution reached by this algorithm is no longer close enough to the solution provided by MIQP (green line) even in the presence of model error. The average cumulative reward produced by Q-network (blue line) is also less than the case where there is no model error. However, if we observe the cumulative reward gathered over the curse of 1 episode (28 steps) in Figure 9b, the best solution of Q-network (blue line) is slightly better than the one produced by MIQP (green line). This is the case since MIQP no longer provides the global optimum solution for the system due to the presence of modeling error. The Q-network algorithm, on the other hand, learns directly from the system without prior information regarding the model, and thus, its performance is not affected by the modeling error. We can observe this more clearly from Table 3 where the best reward produced by Q-network (3.6554) is better than the reward produced by Q-learning (3.3092) and MIQP (3.6293).

The simulation results of the third scenario in the presence of both the modeling error and the measurement noise is shown in Figure 10. In this scenario, the Q-learning algorithm fails to converge towards a solution. This is mainly caused by the fact that the measurement noise causes a lot of variation in the state values which consequently causes the Q-Table’s dimension to be too large. In contrast, the Q-network algorithm does not fall into this problem because it works using continuous approximation function in the form of a neural network. In Figure 10, we can observe that the Q-network algorithm in this case converges much faster (slightly after 10,000 iterations) compared to the previous cases. It demonstrates how the Q-network algorithm actually works better in a more complex scenario, such as the one with modeling and measurement uncertainty. The best solution produced by Q-network algorithm (blue line) also produces cumulative reward which is very close to the solution provided by MIQP (green line) in Figure 10. In fact, the best cumulative reward of Q-network (3.6685) is once again better than the one produced by MIQP (3.6667) as shown in Table 3. This demonstrates the power of Q-network algorithm to solve complex optimization problems in the presence of uncertainty without relying on prior information about the system’s model. These features make the Q-network algorithm a very strong candidate to be used in a real operational control problem of DC microgrid system.

5.2. Complex Environment

In this Section 5.2, we present the results of the complex simulation consisting of three nodes. The configuration of the network is shown in Figure 6. This Section 5.2 is almost identical with Section 5.1, only differing in system parameters. The system parameters are derived from state equation matrices presented in (16)–(22). The resulting sample of the transient current transfer between nodes as an impact of the switching can be seen in Figure 11. First, in a system without modeling and measurement error, the cumulative reward per episode, consisting of 40 steps, as a function of training iteration number can be observed in Figure 12. We can observe that Q-learning algorithm (red line) behaves as in Section 5.1, converging very quickly towards its most optimal solution. However, as we can see from Figure 12, Q-learning achieves the lowest score, followed by Q-network, while MIQP achieves the highest score. Furthermore, an increase in systems’ complexity can be seen to have an impact on the model-free approach as the difference of both Q-learning (74.27071075) and Q-network (74.42539101) to the global optimum solution by MIQP (75.04887579) widens compared to the simple system in Section 5.1. This happens because the information gap between the model-free approach and the fully known system’s dynamics parameters on MIQP solution is also widened.

The simulation results of the second scenario of a complex system with a modeling error in the system is shown in Figure 13. From the plot of cumulative reward per episode as a function of training iteration number in Figure 13, we can observe that the Q-learning algorithm (red line) keeps converging rapidly to the final solution, even achieving the best score followed by Q-network. From Figure 13, we can see that in the course of 1 episode (40 steps), both Q-learning (72.4335) and Q-network (72.3714) could perform relatively well even without prior knowledge of the model’s dynamics compared to MIQP (72.1596). Furthermore, the presence of modeling error greatly shifts the MIQP solution from the global optimum in the complex system scenario, showing the vulnerability of the MIQP approach under inaccurate system model. Nevertheless, Q-network requires many iterations to converge (around 500,000 iterations) compared to the first scenario where no model error exists.

The simulation results of the third scenario where there are both modeling error and measurement noise in the system is shown in Figure 14. In this scenario, the Q-learning algorithm fails to converge again towards a solution, just like in the simple model. In contrast, the Q-network algorithm is still capable of handling noise using its neural network model. Furthermore, we can see from Figure 14 that the solution of Q-network (72.3714) achieves a better score compared to the solution provided by MIQP (72.1596). This result shows that the Q-network algorithm is actually capable of handling both modeling and measurement uncertainty even in complex scenario. These features once again further solidify Q-network algorithm as very strong candidate to be used in a real operational control problem of DC microgrid system. Nevertheless, further research is needed to better understanding the limit of system complexity and measurement error that Q-network able to handle in feasible manner with acceptable results.

To investigate the impact of node quantity on the performance and training time of the Q-network, a series of experiments was conducted. The simulations involved placing clusters of nodes in a sequential arrangement, following the configuration of the complex network illustrated in Figure 6. The clusters consisted of 50, 100, and 1000 nodes, which can be equivalently represented as 150, 300, and 3000 nodes, respectively. The results of these experiments can be found in

Table 4. Impact of System Size.

System Multiplier	Number of Nodes	Training Time (s)
1	3	13.949822300113738
100	300	28.190732199931517
1000	3000	125.25556980003603

. The findings indicate that increasing the system size leads to a higher computational burden for Q-learning during training. However, the rate of increase in training time is lower than the rate of increase in the number of nodes. This suggests that the computational time increase is primarily caused by small-signal calculations, rather than the Q-network algorithm itself.

6. Conclusions and Future Works

In this paper, we propose a reinforcement learning algorithm to solve the optimal operational control problem of a DC microgrid system. The proposed algorithm works without relying on prior information regarding the system’s dynamics, but rather it learns the most optimal policy by interacting directly with the system. We tested two algorithms in this paper, namely Q-learning and Q-network, and compared the performance with the MIQP algorithm found in the literature as the baseline. The proposed algorithms are tested in two different environments, simple and complex environments; and three scenarios, namely perfect information, modeling error, and measurement noise. Both in the first and second environments, the Q-learning (3.5867 and 74.2707) and Q-network (3.5867 and 74.4254) algorithm are shown to produce solutions very close to the optimal solution of MIQP (3.5871 and 75.0489). These results are achieved by the RL methods without knowledge of the system’s dynamics, whereas the MIQP solution requires an accurate system model. It is shown that, when an accurate system model did not exist, meaning there is error in system’s model, both Q-learning (3.3092 and 72.4335) and Q-network (3.6554 and 72.3714) could perform better than MIQP (3.6293 and 72.1596) that requires an accurate system model. Moreover, with the introduction of measurement noise on top of the modeling error, the proposed algorithm of the Q-network is shown to produce a better solution to MIQP. These preliminary results demonstrate that the reinforcement learning algorithm is a strong candidate to solve optimal operation control of a DC microgrid system, especially when the system’s dynamics model is not available.

By showing the capabilities of reinforcement learning-based algorithms in solving model-free DC microgrids distributed droop switching problems, this paper suggests that other off-policy algorithms that did not require an exact system model might also be able to solve the same problem. This paper as a basic starting point in implementing a reinforcement learning algorithm in such environments also opens a challenge for a newer and more complex reinforcement learning algorithm that can be explored to achieve better results. Other kinds of machine-learning algorithms that did not require an exact system model that can learn from experience are also interesting topics to be explored.

In the future, more complicated environments and scenarios, such as the ones consisting of huge numbers of sources or modes, and more complicated constraints can be used, further exploring the limits of system complexity, mode choices, and uncertainties due to modeling and measurement error that Q-network can handle. We will also explore the use of more complex neural network architecture to improve the performance of the Q-network algorithm. Finally, employing the real data from DC microgrid system in the training process will also be investigated.