Active Exploration by Chance-Constrained Optimization for Voltage Regulation with Reinforcement Learning

Abstract

Voltage regulation in distribution networks encounters a challenge of handling uncertainties caused by the high penetration of photovoltaics (PV). This research proposes an active exploration (AE) method based on reinforcement learning (RL) to respond to the uncertainties by regulating the voltage of a distribution network with battery energy storage systems (BESS). The proposed method integrates engineering knowledge to accelerate the training process of RL. The engineering knowledge is the chance-constrained optimization. We formulate the problem in a chance-constrained optimization with a linear load flow approximation. The optimization results are used to guide the action selection of the exploration for improving training efficiency and reducing the conserveness characteristic. The comparison of methods focuses on how BESSs are used, training efficiency, and robustness under varying uncertainties and BESS sizes. We implement the proposed algorithm, a chance-constrained optimization, and a traditional Q-learning in the IEEE 13 Node Test Feeder. Our evaluation shows that the proposed AE method has a better response to the training efficiency compared to traditional Q-learning. Meanwhile, the proposed method has advantages in BESS usage in conserveness compared to the chance-constrained optimization.

1. Introduction

With increasing PV penetration, power system operation suffers from the uncertainty between a forecast and actual generation. A major research focus is the fact that the voltage of the distribution network becomes vulnerable under the PV impact [1]. BESS is a potential solution to mitigate the effect of the PV uncertainty.

Engineering has a long history of handling uncertainties in application [2,3,4]. From the mathematical point of view, solutions mitigating the effect of uncertainties can be classified into two major groups. The first group is probability-based optimization. Stochastic programming [5] is a standard algorithm in this group. These techniques are well developed and have been used in real power system applications [6,7]. Chance-constrained optimization, which is a subset of stochastic programming, is a natural framework because it can quantify the probability of the voltage violation [8,9]. While probability-based optimization methods are useful, it is difficult to maintain the balance between its reliability and conservativeness [10], thereby often leading to over-engineered solutions. Conservativeness is necessary for some applications, but sometimes it can increase the control cost of the system [5]. Systems that use probability-based optimization to operate BESS-supplemented grids often need larger than necessary batteries. Another limitation of the probability-based optimization is the model of the uncertainty. The probability-based optimization requires the knowledge of the uncertainty to estimate its distribution or the boundary. In practice, the model of the uncertainty could be inaccurate and hard to derive.

A recent popular group of solutions managing uncertainty is RL. RL was designed to solve the Markov decision process (MDP) which could regard uncertainties without the uncertainty model. The RL agent can grasp the uncertainty features of the historical training data. In voltage control, researchers have already proposed RL methods for different applications. The tap-changer transformer [11], the capacitor-based Volt/Var control [12,13], the load restoration [14], and the reinforcement learning for the autonomous voltage control [15] have been proved to be effective in power systems. RL for BESS is also studied for voltage control [16], but the algorithm proposed in that paper is using the RL to select which BESS should be used instead of directly controlling the charge/discharge power of BESSs. Even though the achievements are made by RL, the long training time or the low training efficiency is always a challenge in real applications [17].

One of the reasons for the low training efficiency of RL is the large exploration process. The exploration strategies of the RL can be categorized as directed and undirected. Exploration is undirected when the action selection is fully random. Typical exploration methods, such as $\mathsf{\epsilon }$-greedy, Boltzmann distribution, and SoftMax [18], are undirected exploration strategies. The lack of the utilization of specific information characterizes the undirected exploration strategy, in which RL agent are randomly selects action choices with uniform probability for exploration. The directed exploration methods take advantage of specific information or knowledge to guide the exploration. The active exploration (AE) method was first proposed in [19], whereby human knowledge is used to correct the improper actions during the training process of an inverted pendulum system. Such a knowledge-based exploration method is also developed for guiding the RL agent in a maze environment by modification of the activation function [20]. A dynamic parameter tuning for exploration of a meta-learning method with a human interaction robot system is verified in [21]. The parameter changes the probability distribution of the action selection, which avoids the ineffective training. In a recent study, the AE concept is applied to controlling an HVAC system by a deep Q-learning method to accelerate the training speed [22]. The authors use engineering knowledge to judge the action picked by RL agents to achieve faster convergence purpose. Traditional reinforcement learning agents adopt a simple strategy, by randomly selecting an action from a set with all the available actions, which wastes training time significantly. The above studies point out that if the human knowledge or engineering sense can be utilized to guide the action selection during the training process, then the overall training time reduces.

The existing literature shows the potential of probability-based optimization and RL methods in obtaining better performance of voltage regulation with renewable uncertainties. However, answers to the following three issues are still unsettled, which motivates this paper’s research:

(1)

The forecasting model of the renewable generation is hard to obtain accurately. The mismatch between the predicted and actual value diminishes the performance of the probability-based optimization. Therefore, how to overcome the effect of the forecasting mismatch is the challenge of the probability-based optimization methods.

(2)

The conserveness characteristic of the probability-based optimization methods requires a larger size of the BESS to compensate for the forecasting mismatch. Knowing how to reduce the conserveness of the traditional methods so that the BESS size can be managed properly is valuable for industries.

(3)

RL has advantages in handling uncertainties. However, the low training efficiency of RL limits its application. Utilizing the probability-based optimization to improve the training time is meaningful.

In this paper, we integrate the conventional engineering knowledge with a relatively novel RL to conquer the aforementioned issues. Specifically, the chance-constrained optimization is employed as the engineering knowledge of proposed AE methods to improve the training efficiency in voltage regulation problems, and the RL framework relieves the dependency on the forecasting accuracy and the conserveness. The contributions of the work are concluded as:

(1)

We propose an AE method for the voltage regulation problem by applying BESSs in an RL framework. The performance of the proposed method is verified in an IEEE standard test feeder. Simulation results reveal that the proposed method has a better performance.

(2)

The proposed method speeds up the training process by modifying the action selection distribution according to the engineering knowledge, which discourages unnecessary exploration. To validate the improvement in the training efficiency, the proposed method is compared with the conventional Q-learning method as a benchmark method.

(3)

The effectiveness of the BESSs’ usage is improved by the proposed method. By comparing the proposed method with a chance-constrained optimization, the proposed method can achieve the voltage regulation with a smaller BESS size while the chance-constrained optimization method returns infeasible solution.

The rest of the paper is organized as follows. Section 2 demonstrates the chance-constrained optimization, which is regarded as the engineering knowledge. A conventional Q-learning and our proposed method is given in Section 3. Section 4 illustrates the case studies. A conclusion is drawn in Section 5.

2. Engineering Knowledge of Voltage Regulation

One of the limits of the RL is the low exploration efficiency. In this section, an active exploration approach is introduced to enhance exploration efficiency. To elaborate, we engage engineering knowledge to guide the exploration process by the action selection probability distribution. A chance-constrained optimization is firstly applied to obtain an optimal operation profile for BESSs. Then, the obtained optimal BESSs’ profile will guide the action selection in the exploration phases.

2.1. Forecasting Mismatch Model for Uncertainty Distribution

The initial process of the chance-constrained optimization is to obtain a mismatch uncertainty distribution, which is determined by different weather conditions, as Figure 1 shows. The forecasting mismatch refers to the difference between the forecasted power and the actual PV generation. The PV forecasting is sensitive to the weather condition. According to our previous study [23], the accuracy of the forecasting depends on the weather condition. For example, the mismatch between the PV prediction and the actual generation in cloudy days is larger than the sunny and overcast days. Therefore, the model of the mismatch uncertainty depends on the weather conditions. Figure 1 gives the probability density function (PDF) of the forecasting mismatch between the actual PV generation and the forecasted generation under various weather conditions. The PDF is the regression results from the original forecasting mismatch data, which describes the average mismatch of a day. We record the average mismatch of 138 days with the classification of weather conditions and obtained different PDFs of the forecast mismatch. The forecasting algorithm has better performance in accuracy under the weather of the clear and overcast skies than cloudy condition. Therefore, voltage regulation algorithms would encounter a different uncertainty set in this study.

2.2. Voltage Regulation with Chance-Constrained Optimization

The goal we set for the voltage regulation algorithm is to regulate the feeder voltage and minimize the BESS operation cost. The cost is modelled as the throughput degradation cost of batteries [24].

$$\min C\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right),$$

(1)

Subject to

$$SO{C}_{min}\le SO{C}_{k,t-1}+\Delta t·{\mathsf{ɳ}}_k{P}_{BESS,k,t}\le SO{C}_{max},$$

(2)

$${\mathit{V}}_{\mathit{m}\mathit{i}\mathit{n}}\le Loadflow\left(\mathit{P},\mathit{Q},{\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)\le {\mathit{V}}_{\mathit{m}\mathit{a}\mathit{x}},$$

(3)

$${\mathit{P}}_{\mathit{m}\mathit{i}\mathit{n}}\le {\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\le {\mathit{P}}_{\mathit{m}\mathit{a}\mathit{x}},$$

(4)

where ${\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}$ denotes a vector of the BESS operation profile, which is the optimization variable. The element in the vector is $P_{BESS,k,t}$. $P_{BESS,k,t}$ denotes the $k$th BESS operating power at time slot t. $\mathit{P}$ and $\mathit{Q}$ are the forecasted real and reactive power. ${\mathsf{ɳ}}_k$ is the charge or discharge efficiency of the BESS. ${\mathsf{ɳ}}_k={\mathsf{ɳ}}_c$ if the BESS is charging and ${\mathsf{ɳ}}_k=1/{\mathsf{ɳ}}_d$ if the BESS is discharging. ${\mathsf{ɳ}}_c$ denotes the charging efficiency and ${\mathsf{ɳ}}_d$ refers to the discharging efficiency. The SOC of BESS updates according to the $P_{BESS,k,t}$. $\Delta t$ is a 15-min operation interval. $SO{C}_{min}$ and $SO{C}_{max}$ are the minimum and maximum SOC values.$Loadflow\left(\mathit{P},\mathit{Q},{\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)$ means the load flow calculation, which outputs voltages. The ${\mathit{V}}_{\mathit{m}\mathit{i}\mathit{n}}$ and ${\mathit{V}}_{\mathit{m}\mathit{a}\mathit{x}}$ are vectors containing the minimum voltage and maximum voltage values. The operation range of the BESS is from −200 kW to 200 kW, which determines the vectors of ${\mathit{P}}_{\mathit{m}\mathit{i}\mathit{n}}$ and ${\mathit{P}}_{\mathit{m}\mathit{a}\mathit{x}}$. $C\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)$ is the degradation of the BESS. It is linear with the ${\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}$.

$$C\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)={\displaystyle \sum }_{t=1}^T\frac{C_{bat}}{2·L_{bat}\left(DOD\right)·DOD}·\left|P_{BESS,k,t}\right|·\Delta t,$$

where $C_{bat}$ denotes the battery capital cost, $L_{bat}\left(DOD\right)$ determines the battery life coefficient by considering the depth of discharge (DOD). $C_{bat}$, $DOD$ and $L_{bat}\left(DOD\right)$ are predefined values.

The voltage in the distribution network changes with the load and PV. A load flow calculation is necessary to obtain the accurate voltage value of a specific node. Many researchers agree that the voltage can be linearly expressed by the injected power [9]

$$\mathit{V}=\mathit{R}\mathit{P}+\mathit{X}\mathit{Q}+{\mathit{V}}_0,$$

(5)

where $\mathit{V}={\left[V_1,\dots ,V_N\right]}^{\mathit{T}}$ denotes the voltage profile of the whole feeder. $\mathit{R},\mathit{X}\in {ℝ}^{N\times N}$ are matrices containing the elements $R_{ij}$ and $X_{ij}$ which are in the $i$th row and the $j$th column, respectively. Note that $\mathit{R}$ and $\mathit{X}$ is different from the reactance and inductance of the system impedance. $\mathit{R}=\frac{\partial \mathit{V}}{\partial \mathit{P}}$, and $\mathit{X}=\frac{\partial \mathit{V}}{\partial \mathit{Q}}$. N is the total number of phases. $\mathit{P}={\left[P_1,\dots ,P_{\mathit{N}}\right]}^{\mathit{T}}$ and $\mathit{Q}={\left[Q_1,\dots ,Q_{\mathit{N}}\right]}^{\mathit{T}}$ are the active power and reactive power injections. ${\mathit{V}}_0$ is the reference voltage that can be obtained from the default net power.

Because the active power has a major impact on the voltage magnitude under high $\raisebox{1ex}{$\mathit{R}$}\!\left/ \!\raisebox{-1ex}{$\mathit{X}$}\right.$ ratio distribution networks, only the active power injection is considered. Therefore, the voltage variation is:

$$\Delta \mathit{V}=\mathit{R}\Delta \mathit{P},$$

(6)

where $\Delta \mathit{V}$ is a vector containing the voltage variation caused by the fluctuation of the active power injections $\Delta \mathit{P}$.

When the forecasting mismatch and the linearization are introduced, the (1c) are modified as:

$${\mathit{V}}_{\mathit{m}\mathit{i}\mathit{n}}\le \mathit{R}\left(\mathit{P}+{\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}+\mathit{\xi }\right)+\mathit{X}\mathit{Q}+{\mathit{V}}_0\le {\mathit{V}}_{\mathit{m}\mathit{a}\mathit{x}},$$

(7)

where $\mathit{\xi }$ is the uncertainty of the net power, which can be derived from Section 2.1.

Equation (1) can be formulated in a probabilistic fashion.

$$\min C\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right),$$

(8)

Subject to

$$SO{C}_{min}\le SO{C}_{k,t-1}+\Delta t·{\mathsf{ɳ}}_k{P}_{bat,k,t}\le SO{C}_{max},$$

(9)

$$\mathbb{P}\left({\mathit{V}}_{\mathit{m}\mathit{i}\mathit{n}}\le \mathit{R}\left(\mathit{P}+{\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}+\mathit{\xi }\right)+\mathit{X}\mathit{Q}+V_0\right)\ge \alpha ,$$

(10)

$$\mathbb{P}\left(\mathit{R}\left(\mathit{P}+{\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}+\mathit{\xi }\right)+\mathit{X}\mathit{Q}+V_0\le {\mathit{V}}_{\mathit{m}\mathit{a}\mathit{x}}\right)\ge \alpha ,$$

(11)

$${\mathit{P}}_{\mathit{m}\mathit{i}\mathit{n}}\le {\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\le {\mathit{P}}_{\mathit{m}\mathit{a}\mathit{x}},$$

(12)

where $\mathbb{P}(·)$ is the probability of the event. Equation (10) and Equation (11) mean the probability that voltage is within the designed range in every time slot t is more than or equal to $\alpha$. We use the individual constrained format. In this work, the $\alpha$ is set to 0.95.

Define,

$$g_1\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}},\mathit{\xi }\right)=\mathit{R}\left(\mathit{P}+{\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}+\mathit{\xi }\right)+\mathit{X}\mathit{Q}+V_0-V_{min},$$

(13)

$$g_2\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}},\mathit{\xi }\right)={\mathit{V}}_{\mathit{m}\mathit{a}\mathit{x}}-\mathit{R}\left(\mathit{P}+{\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}+\mathit{\xi }\right)+\mathit{X}\mathit{Q}+V_0.$$

(14)

The Formulations (10) and (11) can be rewritten as:

$$\mathbb{P}\left(g_1\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}},\mathit{\xi }\right)\ge 0\right)\ge \alpha ,$$

(15)

$$\mathbb{P}\left(g_2\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}},\mathit{\xi }\right)\ge 0\right)\ge \alpha .$$

(16)

In our formulation, the voltage approximation is linear, so that Equations (15) and (16) have a decoupled format to ${\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}$ and $\mathit{\xi }$.

$$g_1\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}},\mathit{\xi }\right)=h_1\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)-k_1\left(\mathit{\xi }\right),$$

(17)

$$g_2\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}},\mathit{\xi }\right)=h_2\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)-k_2\left(\mathit{\xi }\right),$$

(18)

where,

$$\begin{gathered} h_1\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)=\mathit{R}\left(\mathit{P}+{\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)+\mathit{X}\mathit{Q}+V_0-{\mathit{V}}_{\mathit{m}\mathit{i}\mathit{n}} \\ {k}_1\left(\mathit{\xi }\right)=-\mathit{R}\mathit{\xi } \\ {h}_2\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)={\mathit{V}}_{\mathit{m}\mathit{a}\mathit{x}}-\mathit{R}\left(\mathit{P}+{\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)+\mathit{X}\mathit{Q}+V_0 \\ {k}_2\left(\mathit{\xi }\right)=\mathit{R}\mathit{\xi } \end{gathered}$$

Then the chance constraints (15) and (16) get numerically simplified to linear [23].

$$\mathbb{P}\left(g_1\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}},\mathit{\xi }\right)\ge 0\right)\ge \alpha \iff h_1\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)\ge q_{\alpha }\left(k_1\left(\mathit{\xi }\right)\right),$$

(19)

$$\mathbb{P}\left(g_2\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}},\mathit{\xi }\right)\ge 0\right)\ge \alpha \iff h_2\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)\ge q_{\alpha }\left(k_2\left(\mathit{\xi }\right)\right),$$

(20)

where $q_{\alpha }\left(k\left(\mathit{\xi }\right)\right)$ is the $\alpha$-quantile of $k\left(\mathit{\xi }\right)$. At this point, the uncertainties are formulated linearly.

Then the overall chance-constrained optimization formulation will be convex.

$$\min C\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right),$$

(21)

Subject to:

$$SO{C}_{min}\le SO{C}_{k,t-1}+\Delta t·{\mathsf{ɳ}}_k{P}_{bat,k,t}\le SO{C}_{max},$$

(22)

$$h_1\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)\ge q_{\alpha }\left(k\left(\mathit{\xi }\right)\right),$$

(23)

$$h_2\left({\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\right)\ge q_{\alpha }\left(k\left(\mathit{\xi }\right)\right),$$

(24)

$${\mathit{P}}_{\mathit{m}\mathit{i}\mathit{n}}\le {\mathit{P}}_{\mathit{B}\mathit{E}\mathit{S}\mathit{S}}\le {\mathit{P}}_{\mathit{m}\mathit{a}\mathit{x}}.$$

(25)

The optimization solver used in the work is the cvx_toolbox published by Stanford University [25].

3. Proposed Method: Battery Energy Storage Systems (BESS) Operation by an Active Exploration (AE) Reinforcement Learning

The Markov decision process is the base of the RL. In this section, we discuss the MDP formulation of the BESS operation. Then, a modified AE Q-learning framework is proposed to solve the problem.

3.1. Markov Decision Process (MDP) of BESS Operation

MDP is with a 4-tuple of $S, A, P{r}_a,$ and $R_a$. $S$ denotes the state space where $s\in S$. $A$ represents the action space where $a\in A$ The transition probability is $P{r}_a=P{r}_a\left(s_{t+1}|a, s_t\right)$. $R_a$ is the immediate reward obtained after transitioning from state $s_t$ to state $s_{t+1}$ with the action a.

State: In our problem, the first state is the voltage magnitude of the node per unit.

$$s_v=\{\begin{array}{c}\begin{array}{c}1, if v<0.945 \\ 2, if 0.945\le v\le 0.95\end{array}\\ 3, if 0.95\le v\le 1.05\\ \begin{array}{c} 4, if 1.05\le v\le 1.055\\ 5, if 1.055<v\end{array}\end{array},$$

(26)

The second state is the SOC of the BESS. The range of the SOC is from 0 to 1. The interval between each SOC state is determined by the value of the charging or discharging power of the action space. The total state will be the product of the voltage state and the SOC state, $\mathit{S}={\mathit{S}}_{\mathit{v}}\times {\mathit{S}}_{\mathit{S}\mathit{O}\mathit{C}}$.

Action: Given a state S, a BESS executes the action $a_{BESS}$ at the time slot t. The $a_{BESS}$ represents the charging or discharging power. To avoid violating the power constraints, the action is within the minimum and maximum power as follows:

$$a_{BESS}\in \left[P_{b,min},P_{b,min}+{\Delta }^b,P_{b,min}+2{\Delta }^b,\dots ,P_{b,max}\right],$$

(27)

where ${\Delta }^b=\frac{P_{b,max}-P_{b,min}}{N^b}$ is used to discretize the action set with $N^b$ which is the number of the actions for a single BESS. In this work, $N^b$ is set to 21 and $P_{b,min}$ is −200 kW and $P_{b,max}$ is 200 kW. ${\Delta }^b$ is 20 kW. This range is the same as the chance-constrained optimization.

Transition probability: the state space contains two terms, the voltage and SOC. For the voltage of the feeder, it is updated with the change of net power injected. For the state of SOC, it is determined by the previous state and actions.

$$S_{SOC,t+1}=S_{SOC,t}+\mathsf{ɳ}{a}_{BESS,t}\Delta t,$$

(28)

where $\mathsf{ɳ}$ is the charge/discharge efficiency. $\mathsf{ɳ}={\mathsf{ɳ}}_c$, if $a_{BESS,t}$ is positive, which means the BESS is charging. $\mathsf{ɳ}=1/{\mathsf{ɳ}}_d$, if $a_{BESS,t}$ is negative, which means the BESS is discharging. ${\mathsf{ɳ}}_c$ and ${\mathsf{ɳ}}_d$ refer to the charging and discharging efficiency. $\Delta t$ is the time interval between each action. Our problem sets 15 min as the time interval. Therefore, the total number of steps for one day is 96.

Reward: The principle of the reward design is to encourage the proper actions and discourage improper actions. The total reward consists of different categories: voltage, SOC and degradation. The following is a detailed explanation of the rewards.

For voltage reward, if the selected action makes the voltage magnitudes hold within the safety range of [0.95, 1.05] in p.u., then the reward for the voltage is designed to be 1. If the action causes any voltage violation, the punishment is set to −100. The punishment should have a greater absolute value than the reward because the total steps in one episode are 96. If the punishment is smaller than 96, the training process may converge to a local optimal.

$$R_{v,t}=\{\begin{array}{c}1, if v\in \left[V_{min},V_{max}\right]\\ -100, else\end{array}.$$

(29)

The reward of the SOC follows the similar logic: if the BESS’s SOC is in [$SO{C}_{min}, SO{C}_{max}$], the reward is 1. Otherwise, the punishment is −100.

$$R_{SOC,t}=\{\begin{array}{c}1, if SOC\in \left[SO{C}_{min},SO{C}_{max}\right]\\ -100, else\end{array}.$$

(30)

The reward of the degradation cost is the same as the chance-constrained optimization. The Q-learning looks for the maximum value of the sum of the reward. The degradation reward is negative value.

$$R_{D,t}=-k\left|a_{BESS}\right|,$$

(31)

where $k$ is a constant related to the degradation coefficient of BESS as Equation (1).

The overall immediate reward is a weighted combination of each individual reward.

$$r_t={\theta }_1{R}_{v,t}+{\theta }_2{R}_{SOC,t}+{\theta }_3{R}_{D,t},$$

(32)

where ${\theta }_1$, ${\theta }_2$ and ${\theta }_3$ are coefficients of the reward.

3.2. Active Exploration (AE) during the Training

The final product of Q-learning is an optimal policy ${\pi }^{*}$ which maximizes the $Q_{{\pi }^{*}}\left(s_t,a_t\right)$ such that:

$$Q_{{\pi }^{*}}\left(s_t,a_t\right)=\underset{\pi }{\max }{Q}_{\pi }\left(s_t,a_t\right),$$

(33)

where $Q_{\pi }\left(s_t,a_t\right)$ denotes the grade of selecting the action $a_t$ at state $s_t$. $\mathsf{\pi }$ is the policy determining the action of BESSs under a certain state. According to the observed the state, the traditional RL agent selects actions based on the Ɛ-greedy algorithms [26].

$$a_t=\{\begin{array}{c}argma{x}_{a_t}Q\left(s_t,a_t\right), with probability 1-\mathsf{Ɛ}\\ random a_t, with probability \mathsf{Ɛ}\end{array}.$$

(34)

By contrast with the traditional action selection approach, modifications are introduced to improve the training efficiency. The conventional RL processes the exploration by assuming the action selection follows a distributed uniform distribution. The RL agent has $N^b$ different action that can be taken, and each possible action has equal probability. Then, the exploration is with probability as follows,

$$\mathbb{P}\left(P_{b,min}\le a_t\le P_{b,max}\right)=\frac{1}{N^b},$$

(35)

where $\mathbb{P}(•)$ is the probability distribution function.

After the optimization problem (10) is solved, the optimal BESS operation result $P_{bat}^{*}\left(t\right)$ is known. The action $a_t$ can be selected based on that value. In this study, we regard the $P_{bat}^{*}\left(t\right)$ as the engineering knowledge. The engineering knowledge or experience can accelerate the training process of active exploration. The chance-constraint optimization in our study can only derive a relatively conservative solution. The RL method is supposed to be less conservative because RL converges to an average optimal instead of worst-case satisfactory.

In this study, we define the active exploration that the action selection is actively assumed to follow a Gaussian distribution during the exploration.

$$\mathbb{P}\left(P_{b,min}\le a_t\le P_{b,max}\right)=\frac{1}{\sigma \sqrt{2\mathsf{\pi }}}\underset{P_{b,min}}{\overset{P_{b,max}}{{\displaystyle \int }}}{e}^{-\frac{{\left(a_t-\mu \right)}^2}{2{\sigma }^2}}d{a}_t,$$

(36)

where ${\sigma }^2$ is the variance of the distribution and $\mu$ is the mean of distribution. In our active exploration, $\mu =P_{bat}^{*}\left(t\right)$, so that the RL agent has a higher probability to select a rational action during the training phase, which results in the improvement of the convergence rate and the accuracy. The variance should be designed according to the confidence level of the engineering knowledge. If the forecasting algorithm performs well in terms of accuracy, then the chance-constrained optimization could generate reliable results. The $\sigma$ could be given as a relatively large value. Otherwise, $\sigma$ can be a small value to extend the exploration area and corresponding probability. As a trade-off, the smaller $\sigma$ results in a longer training period.

The comparison between the active exploration and the conventional exploration is illustrated in Figure 2. Compared to the uniform distribution exploration used by the conventional reinforcement, the probability that an RL agent chooses the action that is close to the $P_{bat}^{*}\left(t\right)$ is higher under the Gaussian distribution exploration. The corresponding reward of the active exploration is supposed to be larger than the conventional uniform distribution because of the engineering knowledge. The larger reward can be regarded as positive feedback to accelerate the speed of the learning.

After the actions of BESSs are selected, the environment interacts with the actions and generates the immediate rewards. The environment is the IEEE 13 Node Test Feeder in the OpenDSS [27]. The implementation of the RL is demonstrated in Figure 3.

By contrast with the optimization method, the RL method does not require the linear approximation of the voltage variation. The function of voltage to the injected power is a non-linear and even non-convex mapping, which ensures the accuracy of the power flow.

$$\mathit{V}=loadflow\left({\mathit{P}}_{\mathit{t}},{\mathit{Q}}_{\mathit{t}},{\mathit{a}}_{\mathit{t}}\right),$$

(37)

The unbalanced load flow calculation algorithm is backward forward swept (BFS) algorithm in the OpenDSS software. The SOC update refers to (3).

When the action $a_t$ is executed, the reward $r_t\left(s_t,a_t\right)$ in (17) is calculated and the next state $s_{t+1}$ is evaluated. The rule for the update of the action-value function $Q\left(s_t,a_t\right)$ is [26].

$$Q\left(s_t,a_t\right)\leftarrow Q\left(s_t,a_t\right)+\beta \left[r\left(s_t,a_t\right)+\mathsf{\gamma }·\underset{a^{\prime }}{\max }\mathrm{Q}\left(s_{t+1},a^{\prime }\right)-Q\left(s_t,a_t\right)\right],$$

(38)

where $\beta$ and $\mathsf{\gamma }$ stand for the learning rate and discount rate. $a^{\prime }$ denotes all potential actions. The Q-learning update (23) takes place in the Matlab.

4. Case Study and Results

We evaluate the proposed algorithm in four aspects: (1) voltage regulation performance; (2) training efficiency improvement; (3) conserveness; (4) success rate of the voltage regulation under different uncertainty levels and different BESS sizes. Specifically, the feasibility of proposed method in voltage regulation is given in Section 4.1. The proposed AE method is compared with the conventional RL to demonstrate the training efficiency improvement in Section 4.2. The conserveness study between the proposed method and the chance-constrained optimization occurs in Section 4.3. In the last section, the success rates of the proposed method, the conventional Q-learning, and the chance constrained optimization are studied to reveal their performance under different weather conditions.

Two BESSs are placed in the IEEE 13 Node Test Feeder. BESSs are installed in node 611 and node 675. Node 611 is the location with the most vulnerable node to the undervoltage problem. Node 675 is the most possible location of the overvoltage problem. The rest of the feeder maintains the default values. In our experiment, the tap position of the transformer remains unchanged. The power capacity of each BESS is 200 kW. The individual robust parameter $\alpha$ is set to 0.99. For RL, the learning rate $\beta$ is 0.8 and the discount factor $\mathsf{\gamma }$ is set to be 0.5. The aforementioned parameters are tuned based on the training performance to the historical data. Different parameters will result in different performance in training speed, but the reinforcement learning agent will converge to the same optimal policy with different learning and discount rates.

4.1. Voltage Regulation Results of the Proposed Method

In this section, the feasibility of the proposed method is tested. We use the 300 days of the load and PV data as input, and examine the proposed method in the case of exterior day. The energy capacity is 400 kWh for each BESS.

Figure 4 shows the results of the voltage regulation, and it illustrates the voltage profile for algorithms. In the voltage regulation results, the blue lines denote the voltage profiles of node 611, and the red lines denote the result in node 675. The x-axis is the hour of the day, and the y-axis denotes the voltage magnitude in per unit. The minimal allowed voltage in the out formulation is 0.95 p.u. and the maximum voltage limit is 1.05 p.u. The thin curves are the original voltage profiles. The bold lines are the results of the proposed method. We can see that, after the RL agent executes, the voltage profiles are managed in the range of [0.95, 1.05] p.u.

Undervoltage violations happen in node 611 after sunset. Overvoltage violations take place in node 675 at noon. The bold curves represent the voltage results from the proposed method. The red curve shows the voltage results from our method. Overvoltage and undervoltage issues caused by a PV are resolved by the proposed algorithm.

In Figure 5, the operation power and SOC of two BESSs are given. The x-axis is the hour of the day. The curves with cycles represent the operation power in kW, and the rest of the lines denote the state of charge of the BESSs. From the simulation, we can conclude that the BESSs run within the designed SOC constraints. The feasibility of the proposed method is verified.

4.2. Compare with Conventional Q-Learning by the Training Efficiency

The feasibility and the training efficiency of the proposed method and the conventional Q-learning (QL) are tested in this section. We use 100 days of the load and PV data as inputs, and examine the proposed method in the case of exterior day for the feasibility test. We would like to show our proposed method has advantages in training efficiency compared to QL.

In Figure 6, the operation power and SOC of two BESSs are given. The maximum of the training episode is set to 3000 in this case. The x-axis is the hour of the day. The left y-axis is the operating power of BESS in kW. The right y-axis denotes the SOC. From the simulation with the 3000 episodes, we can conclude that the BESSs run within the designed constraints of the proposed AE method. However, the SOC of QL is out of the limits of [0, 1]. That means the proposed method converges to an optimal policy at 3000 episodes while the conventional QL is infeasible at the same training efforts, which indicates the improvement of the proposed AE method.

Figure 7 gives the accumulated rewards and the average values of QL and AE. The maximum of the training episode is still set to 3000. The x-axis is the number of the episode. Y-axis is the reward value. According to the reward formulation, if the final reward value is larger than zero, then no voltage violation and all operation constraints are satisfied. If the final reward value is smaller than zero, that means the constraints are violated. Both the reward and the average reward of the proposed method is higher than the corresponding QL. At the end of the training, the reward value of the proposed method is positive, but the QL is negative. From the numerical simulation, we can conclude that the proposed AE method accelerates the training process comparing to the conventional QL.

Figure 8 gives the success rates of QL and AE. The maximum of the training episode is set from 1000 to 6000 in this case. We use 300 days of the load and PV data as inputs, and exam the proposed method in the case of 100 different days.

The x-axis is the number of the episode. Y-axis is the success rate. In a specific test profile, the OpenDSS calculates the voltage magnitudes. MATLAB updates the SOC of all BESS according to (13). We set the voltage allowed range as [0.95, 1.05] and the SOC range as [0, 1]. If the test results tell us that all the voltages and SOCs are within the allowed range, we define the corresponding algorithm successes rate. We use 100 different profiles to test the trained policy of AE in the Q-learning algorithms. The total number of successful cases is recorded. The success rate is calculated as:

$$\mathrm{Success} \mathrm{rate}=\frac{Number of success profiles}{Total tested profiles}\times 100\%,$$

(39)

From the simulation, the success rate of QL is zero before 3000 episodes, and it is always lower than AE. When the maximum episode increases to 6000, the success rate of the QL is the same with AE, which indicates both algorithms converge to the same optimal policy if the training efforts are large enough. With the limited training episode, the proposed AE has a better training efficiency than the conventional QL.

4.3. Compare with Chance-Constrained Optimization for Conserveness

In this part, the conserveness of the proposed method and the conventional chance-constrained optimization is studied. The conserveness of the method can be regarded as the BESS capacity needed for a certain voltage regulation case. In other words, the difference between SOC usage can reflect the conserveness of the algorithms.

In Figure 9, the voltage regulation performances of chance-constrained optimization and AE are given. We use the 300 days of the load and PV data as input, and examine the proposed method in the case of a different day. The x-axis means the hour of the day, and the Y-axis is the voltage magnitude. From the simulation, both the proposed AE and chance-constrained optimization can achieve voltage regulation for the selected case.

The BESS usage is directly related to the SOC. High variation of the SOC profile represents the large utilization of BESS. In Figure 10, the SOC of chance-constrained optimization and AE are given. The x-axis is the hour of the day, and Y-axis is the SOC. Because the SOC is constrained in [0, 1] in this study, both the AE method and the optimization fulfill the SOC constraints. From the results, the proposed AE uses less SOC than the chance-constrained optimization in the voltage regulation, which indicates the proposed method has lower conserveness than the chance-constrained optimization. In a real application, this feature can reduce the investment of the BESSs because less capacity is required. The reason behind this conservativeness is that the chance-constrained optimization needs the BESS to operate like that for the whole training profiles. To handle all the uncertainties in the training data, the optimization solver generates a conservative operation strategy and give more stable margin on the voltage profile. This conservative characteristic improves the reliability but will cause an unnecessary BESS size.

4.4. Success Rate Comparison under Different Sizes and Weather Conditions

In this part, we examine the success rate under different BESS capacities and weather conditions. We use 300 days of the load and PV data as inputs, and exam the proposed method in the case of 150 exterior days. In the selected 150 days, the amounts of clear sky, cloudy, and overcast days are equal to 50.

Figure 11a shows the results of the success rate of the 3000 maximum training episodes, and Figure 11b illustrates the success rate of the 6000 maximum training episodes. In both figures, the proposed AE method, conventional QL and the chance-constrained optimization are compared. This comparison is to show the training efficiency.

The success rate is the robustness metric in this work. We investigated the effect of both weather conditions and BESS size. The BESS in Section 4.1 is a 4-h BESS (200 kW with 800 kWh). In this part, a 3 h BESS and a 2 h BESS are studied. We maintained the power rate and change the energy capacity. The 3 h BESS had 200 kW and 600 kWh and the 2 h BESS had 200 kW and 400 kWh.

In Figure 11, the x-axis is the BESS size in kWh, and we have the values of 400 kWh, 600 kWh and 800 kWh. The y-axis denotes the weather condition. The z-axis represents the success rate. With a fixed weather condition, the success rate of all algorithms decreases with the reducing BESS size. In general, the success rate of the RL is larger than the chance-constrained optimization if the training episode is large enough.

The success rate of the chance-constrained optimization is zero in some cases. The reason is that the chance-constrained optimization solver claimed these cases were infeasible because of the insufficient BESS size. The operation profiles in these cases are only based on the forecasted value without introducing uncertainties. Unfortunately, the mismatch between the forecasting data and actual net power creates voltage violations.

From Figure 11, the success rate of the chance-constrained optimization drops with weather conditions. The cloudy day has the highest forecasting mismatch, and the clear sky has the lowest mismatch. The success of the AE and Q-learning drops as well, but the speed is much lower than that of the chance-constrained optimization. That means the chance-constrained optimization is more vulnerable to uncertainties. One potential explanation could be that with the increasing uncertainty level, the difference between the training profile and the test profile grows. The chance-constrained optimization may handle cases in training profiles. However, the probability that it encounters a new worse profile during the test process increases due to the rising difference. The optimization did not meet this new worse profile before the test process, and naturally, the generated BESS operation cannot perform well. A possible solution for the chance-constrained optimization is to enlarge the number of training profiles so the optimizer can meet more uncertainties. For fairness in our comparison, the same training profiles were provided to the three algorithms.

During the optimization calculation, the chance-constrained method would like to address cases of the training profiles as much as possible. That results in over-engineered solutions (for example, a large BESS size in our application). If the size of the BESS fails to satisfy the requirement of the chance-constrained optimization, the solver would not generate any profile. However, the goal of the RL agent of Q-learning and AE is to find an average optimal policy for BESS. Although the BESS size is small, the agent can still find a policy for the operation.

When the maximum episode is set to 3000, the Q-learning cannot converge to the feasible policy, so that the corresponding success rate in Figure 11a is low. When the maximum episode becomes 6000, the success rate of Q-learning is similar to AE, which means both algorithms are converging to the optimal policy, which proves the convergence of the AE and Q-learning.

5. Conclusions

We proposed an active exploration algorithm to integrate engineering knowledge into the reinforcement learning process. The proposed algorithm can improve the training speed and accuracy compared to the traditional QL algorithm. The proposed method was used to control the BESS for voltage regulation in the IEEE 13 Node Test Feeder. The results of an empirical evaluation led to the following observations: (1) the proposed AE method can accelerate the training process and have a better success rate in voltage regulation than QL when the training time is limited. (2) The control generated by the chance-constrained optimization tends to use more battery energy in general, and it needs a larger BESS to handle uncertainties, but the proposed method uses less BESS energy and performs relatively well. (3) When the forecasting mismatch increases, the success rate of the chance-constrained optimization drops faster than the proposed method with the same BESS size. The future directions include optimal parameter tuning and involving advanced reinforcement algorithms.