Data-Driven Distributionally Robust Stochastic Control of Energy Storage for Wind Power Ramp Management Using the Wasserstein Metric

Abstract

The integration of wind energy into the power grid is challenging because of its variability, which causes high ramp events that may threaten the reliability and efficiency of power systems. In this paper, we propose a novel distributionally robust solution to wind power ramp management using energy storage. The proposed storage operation strategy minimizes the expected ramp penalty under the worst-case wind power ramp distribution in the Wasserstein ambiguity set, a statistical ball centered at an empirical distribution obtained from historical data. Thus, the resulting distributionally robust control policy presents a robust ramp management performance even when the future wind power ramp distribution deviates from the empirical distribution, unlike the standard stochastic optimal control method. For a tractable numerical solution, a duality-based dynamic programming algorithm is designed with a piecewise linear approximation of the optimal value function. The performance and utility of the proposed method are demonstrated and analyzed through case studies using the wind power data in the Bonneville Power Administration area for the year 2018.

1. Introduction

To decarbonize the electric power grid, there have been growing efforts to utilize clean, renewable energy sources. The utilization of wind and solar power generation is challenging because these energy sources are uncertain, intermittent, and nondispatchable. In particular, as the penetration of wind power increases, fast-ramping generators must be called upon more frequently to balance supply and demand, or wind power production must be curtailed [1,2]. Such ancillary services and wind power curtailments will offset the economic and environmental benefits of wind energy.

One possible way to alleviate the negative impact of a growing wind power ramp rate is to utilize the flexibility that energy storage can offer. Energy storage devices are capable of shifting wind generation to reduce the ramp rate of wind generation [3,4]. For an efficient charging/discharging operation of battery energy storage systems, a model predictive control approach was proposed in [5]. However, a certain amount of wind generation must be curtailed when using this method. In [6], the wind power ramp control problem using energy storage was formulated as a social welfare maximization problem. As the optimal solution to the problem requires information about the future wind generation and demand, a suboptimal online algorithm is presented; however, the suboptimal approach suffers a performance loss. In [7], a storage control approach using a two-stage stochastic optimization was proposed. This operational strategy utilizes the forecast of wind energy obtained by a Gaussian process. Another optimization-based method was developed by using ramp scenario forecasts [8]. The performance of both methods depends on the accuracy of forecasted information because the optimization problems in [7,8] directly use wind forecasts. Arguably, the most popular method for efficient energy storage operation is stochastic optimal control [9,10,11,12]. The associated stochastic optimal control problems are solved by dynamic programming or its approximate version, which often allows important structural properties of optimal strategies. Unfortunately, this method requires knowledge about the probability distribution of all the uncertainties such as future wind power generation. However, accurate distribution models are difficult to obtain in practice. Thus, the effectiveness of stochastic optimal control methods is limited as the wind power distribution at any given time deviates from the distribution estimated using historical data.

The methods mentioned above either require reliable information about future wind power generation or compromise the control performance. To account for these limitations, we seek an efficient storage operation strategy for wind power ramp management when only an inaccurate probability distribution of wind power generation is available. This method is based on distributionally robust stochastic control, which minimizes the expected value of a given cost function in the face of the worst-case distribution drawn from a known set, called the ambiguity set [13,14,15,16,17,18,19]. In this work, the ambiguity set is chosen as the set of all probability distribution whose Wasserstein distance from an empirical distribution constructed from data is no greater than a certain threshold [20,21,22]. The proposed storage control strategy is robust against wind ramp distribution errors characterized by the Wasserstein ambiguity set. It is worth mentioning that some storage control techniques do not require the exact distribution of uncertainties [23,24]. However, these approaches do not aim to design a controller that is robust against distribution errors, unlike our method.

The contributions of this work can be summarized as follows. First, a novel storage operation strategy is proposed to provide a robust ramp management performance even when future wind power ramp distribution deviates from the empirical distribution obtained by historical data. Second, we develop a computationally tractable dynamic programming (DP) algorithm by using a piecewise linear approximation of the optimal value function with a uniform convergence property and Kantorovich duality. Thus, in each DP iteration, it suffices to solve linear programs for all grid points that discretize the state space. Third, the performance of the distributionally robust method is evaluated using the wind power generation data in the Bonneville Power Administration (BPA) control area and is compared with that of the standard stochastic optimal control method. Our simulation studies indicate that the proposed method reduces the ramp penalty by 4.82% on average compared to the standard stochastic optimal control method. We also examine how the ambiguity set size and the storage size affect the ramp management performance of the distributionally robust control method. This paper is significantly expanded from a preliminary conference version in many aspects [25]. The problem studied in this paper is wind power ramp management, while [25] considers a wind energy balancing problem. In addition, we use the Wasserstein ambiguity set and examine the effect of the set size, unlike [25], which employs the moment-based ambiguity set (The performance of the moment-based approach in [25] depends on the reliability of moments estimated from wind power generation data. However, the proposed Wasserstein approach does not have such an issue because it does not use information about moments.) Furthermore, a tractable dynamic programming solution is carefully developed in this work, using a linear programming approximation with a uniform convergence property.

The remainder of this paper is organized as follows. In Section 2, the Wasserstein distributionally robust storage control problem is formulated for wind power ramp management using historical data. Its dynamic programming solution with a piecewise linear approximation is proposed in Section 3. In Section 4, the performance and utility of the proposed method are demonstrated and analyzed using the wind power generation data in BPA for the year 2018.

2. Problem Formulation

2.1. Energy Storage Model

Consider an energy storage device whose state of charge (SOC) $x_t^s$ evolves as

$$x_{t+1}^s=\eta [x_t^s+({\alpha }^c{u}_t^c-u_t^d)\Delta t],t=0,1,\dots ,T-1,$$

where $\Delta t$ is the length of each time interval. Here, the control inputs $u_t^c$ and $u_t^d$ determine the amount of power (MW) by which the storage device is charged and discharged, respectively, at time t. The coefficients $\eta \in (0,1]$ and ${\alpha }^c\in (0,1]$ account for the dissipation loss and the charging inefficiency, respectively. Given the capacity of the storage device, $x_t^s\in [x_{min}^s,x_{max}^s]$, the decision variable $u_t:=(u_t^c,u_t^d)$ must satisfy the following constraints:

$$\begin{array}{cc}\hfill & 0\le u_t^c\le min\{(x_{max}^s-x_t^s)/({\alpha }^c\Delta t),u_{max}^c\}\hfill \\ \hfill & 0\le u_t^d\le min\{x_t^s/\Delta t,u_{max}^d\},\hfill \end{array}$$

(1)

where $u_{max}^c$ and $u_{max}^d$ denote the charging and discharging limit, respectively. If the storage device is connected to a bus, the amount of power drawn from the bus at time t is given by

$$h\left(u_t\right):=u_t^c-{\alpha }_d{u}_t^d,$$

where ${\alpha }^d\in (0,1]$ represents the discharging efficiency. This energy storage model is illustrated in Figure 1.

2.2. Wind Power Ramp Management

Let $w_t$ denote the amount of power generated by wind power plants in an area of interest at time t. We model the evolution of wind power production by

$$w_{t+1}=w_t+{\xi }_t,$$

where ${\xi }_t$ is a random variable in $\Xi \subseteq \mathbb{R}$. Note that $(w_{t+1}-w_t)/\Delta t={\xi }_t/\Delta t$ represents the (average) ramp rate of the wind power production. We introduce a new state $x_t^p$, which evolves according to

$$x_{t+1}^p=h\left(u_t\right)+{\xi }_t.$$

Let $x_t:=(x_t^s,x_t^p)\in {\mathbb{R}}^2$ be the augmented state, and $\mathcal{X}:=[x_{min}^s,x_{max}^s]\times [x_{min}^p,x_{max}^p]$ be the augmented state space.

Suppose the wind power plants and the energy storage device are located in the same area (possibly connected in the same bus). Their net power production in period t is given by

$$y_t:=w_t-h\left(u_t\right),$$

as illustrated in Figure 1. Let

$$d_t:=y_t-y_{t-1}={\xi }_{t-1}-h\left(u_t\right)+h\left(u_{t-1}\right)=x_t^p-h\left(u_t\right).$$

(2)

Then, $d_t/\Delta t$ represents the ramp rate of net power production.

Let $R_u>0$ and $R_d>0$ denote the ramp-up limit and ramp-down limit (MW), respectively. To compute the ramp penalty when $d_t\ge 0$, consider the following cases:

• If $0\le d_t<R_u$, then the ramp penalty, denoted by $r_t(x_t,u_t)$, is linear in $d_t$, i.e.,

  $$r_t(x_t,u_t):=p{d}_t.$$
• If $d_t\ge R_u$, then the ramp penalty is given by

  $$r_t(x_t,u_t):=p_u(d_t-R_u)+p{R}_u,$$

  where $p_u>p$. In other words, for the amount exceeding the ramp-up limit, we are penalized with price $p_u$ greater than p.

When $d_t\le 0$, by symmetry, we have the following:

• If $-R_d<d_t\le 0$, then the ramp penalty is given by

  $$r_t(x_t,u_t):=-p{d}_t.$$
• If $d_t\le -R_d$, then the amount below the ramp-down limit is penalized with price $p_d>p$, i.e.,

  $$r_t(x_t,u_t):=-p_d(d_t+R_d)+p{R}_d.$$

Note that the ramp penalty depends on the storage control action $u_t$ as well as the new state $x_t^p$ through the ramp $d_t$ of net power production. Thus, the ramp penalty can be reduced by carefully controlling the storage device. The ramp management problem using energy storage can be formulated as (To focus on the wind power ramp management capability of our method, we use this stylized control problem, neglecting the energy cost and the aging of batteries (in the case of battery energy storage systems). However, these additional factors can be incorporated by modifying the cost function.)

$$\underset{\pi \in \Pi }{min}{\mathbb{E}}^{\pi }\left[\sum _{t=0}^{T-1}{r}_t(x_t,u_t)\mid x_0=\mathit{x}\right],$$

(3)

where $\Pi$ is the set of admissible control policies,

$$\begin{array}{cc}\hfill \Pi :=\{\pi =({\pi }_0,\dots ,{\pi }_{T-1})\mid & u_t={\pi }_t\left(x_t\right)\hfill \\ \hfill & \in \mathcal{U}\left(x_t\right):=[0,min\{(x_{max}^s-x_t^s)/({\alpha }^c\Delta t),u_{max}^c\}]\times [0,min\{x_t^s/\Delta t,u_{max}^d\}]\}.\hfill \end{array}$$

The stochastic optimal control problem (3) can be solved when the probability distribution ${\mu }_t$ of the wind ramp variable ${\xi }_t$ is available. Unfortunately, it is difficult to accurately estimate the probability distribution ${\mu }_t$. Suppose that $T=288$ and $\Delta t=5$ min. In other words, by solving this problem, we can design an optimal controller that operates for 24 h. To estimate the distribution of ${\xi }_t$, we may use its historical data for the past 15 days. However, this distribution may not be valid for the next 15 days. For example, as shown in Figure 2, there is a clear discrepancy between the empirical distribution of wind power ramp rate at 12PM in the BPA area obtained by the data for 1–15 April 2018 and that obtained by the data for 16–30 April 2018. If an optimal controller is constructed using distributional information that will be invalid at the test time, it may not perform well. Our distributionally robust method aims to resolve this issue, as we will see later.

2.3. Ambiguity of Wind Ramp Distribution

One of the simplest ways to estimate the probability distribution of the wind ramp variable ${\xi }_t$ is to use the empirical distribution constructed from the (historical) data $\{{\widehat{\xi }}_t^{\left(1\right)},\dots ,{\widehat{\xi }}_t^{\left(N\right)}\}$:

$${\nu }_t:=\frac{1}{N}\sum _{n=1}^N{\delta }_{{\widehat{\xi }}_t^{\left(n\right)}},$$

where ${\delta }_{{\widehat{\xi }}_t^{\left(n\right)}}$ denotes the Dirac delta measure concentrated at ${\widehat{\xi }}_t^{\left(n\right)}$. However, this empirical distribution may not reflect the actual behavior of ${\xi }_t$, particularly at the (future) test time or operation time of energy storage. To characterize errors in the empirical distribution, we introduce the following set called the ambiguity set, of probability distributions:

$${\mathcal{D}}_t:=\{{\mu }_t\in \mathcal{P}(\Xi )\mid W({\mu }_t,{\nu }_t)\le \theta \},$$

(4)

where $\mathcal{P}(\Xi )$ is the set of probability distributions on $\Xi$. The ambiguity set is a statistical ball centered at the empirical distribution with radius $\theta >0$. Here, the distance between two probability distributions is measured by the Wasserstein metric of order 1,

$$W(\mu ,\nu ):=\underset{\kappa \in \mathcal{P}\left({\Xi }^2\right)}{min}\left\{{\int }_{{\mathbb{R}}^2}|\xi -{\xi }^{\prime }|\kappa (\mathrm{d}\xi ,\mathrm{d}{\xi }^{\prime })\mid {\Pi }^1\kappa =\mu ,{\Pi }^2\kappa =\nu \right\},$$

where ${\Pi }^i\kappa$ denotes the ith marginal of $\kappa$. The Wasserstein distance between two probability distributions can be interpreted as the minimum cost of transporting mass from one to another using nonuniform perturbation. The true distribution of ${\xi }_t$ lies in the ambiguity set if a sufficiently large $\theta$ is chosen. We use the Wasserstein ambiguity set ${\mathcal{D}}_t$ to design a control policy that is robust against errors in the empirical distribution ${\nu }_t$ of the ramp variable ${\xi }_t$. The Wasserstein ambiguity sets have received much attention because the associated distributionally robust optimization (DRO) problems provide solutions with probabilistic out-of-sample performance guarantees and have equivalent tractable forms [20,21,22].

2.4. Wasserstein Distributionally Robust Stochastic Control

Our goal is to construct a control policy that performs well even when the true distribution ${\mu }_t$ of the wind ramp variable ${\xi }_t$ deviates from its empirical distribution ${\nu }_t$ obtained from (historical) data. We take a game-theoretic approach. Consider a two-player zero-sum game in which Player I selects a storage control action to minimize the total ramp penalty while Player II, an adversary, chooses the probability distribution of ${\xi }_t$ from the Wasserstein ambiguity set ${\mathcal{D}}_t$ to maximize the same penalty. The strategy space for Player I is given by $\Pi$. The policy space ${\Gamma }_{\mathcal{D}}$ for Player II is chosen as

$${\Gamma }_{\mathcal{D}}:=\{\gamma :=({\gamma }_0,\dots ,{\gamma }_{T-1})\mid {\mu }_t={\gamma }_t\left(x_t\right)\in {\mathcal{D}}_t\},$$

where the subscript $\mathcal{D}$ is used to emphasize the fact that the adversary’s strategy space depends on the Wasserstein ambiguity set. The Wasserstein distributionally robust stochastic control problem for ramp management is then formulated as

$$\underset{\pi \in \Pi }{min}\underset{\gamma \in {\Gamma }_{\mathcal{D}}}{max}{\mathbb{E}}^{\pi ,\gamma }\left[\sum _{t=0}^{T-1}{r}_t(x_t,u_t)\mid x_0=\mathit{x}\right].$$

(5)

An optimal distributionally robust policy ${\pi }^{🟉}$ minimizes the worst-case ramp penalty under the most adversarial wind ramp distributions in the ambiguity set. Thus, ${\pi }^{🟉}$ is robust against distribution errors characterized by the Wasserstein ambiguity set. The radius $\theta$ of the Wasserstein ball controls the robustness and conservativeness of control policy ${\pi }^{🟉}$. A detailed discussion of tuning $\theta$ and theoretical properties of ${\pi }^{🟉}$ can be found in [19].

3. Solution via Dynamic Programming

To solve the minimax stochastic control problem (5), we use dynamic programming in conjunction with modern distributionally robust optimization (DRO). To describe the evolution of the augmented state $x_t:=(x_t^s,x_t^p)\in \mathcal{X}$, we use the following notation:

$$x_{t+1}=f(x_t,u_t,{\xi }_t),$$

where $f(\mathit{x},\mathit{u},\mathit{\xi }):=\left(\eta [{\mathit{x}}^s+({\alpha }^c{\mathit{u}}^c-{\mathit{u}}^d)\Delta t],h\left(\mathit{u}\right)+\mathit{\xi }\right)$. Note that the ramp penalty is a convex piecewise linear function and can be expressed as

$$\begin{array}{c}{r}_t(x_t,u_t)=max\{p{d}_t,p_u(d_t-R_u)+p{R}_u,-p{d}_t,-p_d(d_t+R_d)+p{R}_d\}\hfill \\ =max\{p(x_t^p-h\left(u_t\right)),p_u(x_t^p-h\left(u_t\right)-R_u)+p{R}_u,-p(x_t^p-h\left(u_t\right)),-p_d(x_t^p-h\left(u_t\right)+R_d)+p{R}_d\}.\hfill \end{array}$$

(6)

3.1. Bellman Equation

We define the (optimal) value function $v_t:\mathcal{X}\to \mathbb{R}$ of (5) as

$$v_t\left(\mathit{x}\right):=\underset{\pi \in \Pi }{min}\underset{\gamma \in {\Gamma }_{\mathcal{D}}}{max}{\mathbb{E}}^{\pi ,\gamma }\left[\sum _{\tau =t}^{T-1}{r}_{\tau }(x_{\tau },u_{\tau })\mid x_t=\mathit{x}\right],$$

which represents the minimum cost-to-go under the most adversarial distributions chosen from $\mathcal{D}$. By the dynamic programming principle, the value function satisfies the following Bellman equation:

$$v_t\left(\mathit{x}\right)=\underset{\mathit{u}\in \mathcal{U}\left(\mathit{x}\right)}{inf}\underset{\mathit{\mu }\in {\mathcal{D}}_t}{sup}{\mathbb{E}}^{\mathit{\mu }}[r_t(\mathit{x},\mathit{u})+v_{t+1}\left(f(\mathit{x},\mathit{u},{\xi }_t)\right)],t=0,\dots ,T-1$$

with $v_T\left(\mathit{x}\right)\equiv 0$ (More precisely, the dynamic programming principle holds under the measurable selection condition, which we assume throughout this paper. More technical details can be found in [26].) This Bellman equation may be solved backward in time, i.e., from T to 0. However, it requires an optimal solution to the minimax optimization problem, which is computationally challenging to solve. This is because the inner minimization problem over probability distributions is infinite dimensional.

3.2. Tractable Reformulation

To alleviate the computational difficulty, we reformulate the Bellman equation into a tractable form by using Wasserstein DRO based on the Kantorovich duality principle [20,22]. Specifically, the right-hand side can be expressed in the following dual form:

$$\begin{array}{cc}\hfill v_t\left(\mathit{x}\right)=\underset{\mathit{u},\lambda ,z}{inf}& \theta \lambda +r_t(\mathit{x},\mathit{u})+\frac{1}{N}\sum _{n=1}^N{z}_n\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& v_{t+1}\left(f(\mathit{x},\mathit{u},\mathit{\xi })\right)-\lambda |{\widehat{\xi }}_t^{\left(n\right)}-\mathit{\xi }|\le z_n\forall \mathit{\xi }\in \Xi \forall n=1,\dots ,N\hfill \\ \hfill & 0\le {\mathit{u}}^c\le min\{(x_{max}^s-{\mathit{x}}^s)/({\alpha }^c\Delta t),u_{max}^c\}\hfill \\ \hfill & 0\le {\mathit{u}}^d\le min\{{\mathit{x}}^s/\Delta t,u_{max}^d\}\hfill \\ \hfill & \lambda \ge 0.\hfill \end{array}$$

(7)

It is shown that strong duality holds under a minor technical condition [22]. Note that the reformulated Bellman equation involves a single minimization problem instead of the minimax problem in the original one. This minimization problem is a convex optimization problem given any $\mathit{x}\in \mathcal{X}$ because $r_t$ and $v_t$ are convex and f is affine. Furthermore, all the optimization variables $u,\lambda$, and z are finite dimensional, unlike those of the original Bellman equation. However, the first inequality constraint must be satisfied for all $\mathit{\xi }$ in the support $\Xi$, which is a dense set. Thus, the reformulated minimization problem is a convex semi-infinite program. This can be numerically solved by several existing algorithms, such as discretization and sampling-based methods (see [27,28,29] and the references therein).

3.3. Controller Design Algorithm Using Linear Programming

By using the piecewise linear structure of the ramp penalty function, we can further rewrite the Bellman Equation (7), with a slack variable b, as

$$\begin{array}{cc}\hfill v_t\left(\mathit{x}\right)=\underset{\mathit{u},\lambda ,z,b}{inf}& \theta \lambda +b+\frac{1}{N}\sum _{n=1}^N{z}_n\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& v_{t+1}\left(f(\mathit{x},\mathit{u},\mathit{\xi })\right)-\lambda |{\widehat{\xi }}_t^{\left(n\right)}-\mathit{\xi }|\le z_i\forall \mathit{\xi }\in \Xi \forall n=1,\dots ,N\hfill \\ \hfill & p({\mathit{x}}^p-h\left(\mathit{u}\right))\le b\hfill \\ \hfill & p_u({\mathit{x}}^p-h\left(\mathit{u}\right)-R_u)+p{R}_u\le b\hfill \\ \hfill & -p({\mathit{x}}^p-h\left(\mathit{u}\right))\le b\hfill \\ \hfill & -p_d({\mathit{x}}^p-h\left(\mathit{u}\right)-R_d)+p{R}_d\le b\hfill \\ \hfill & 0\le {\mathit{u}}^c\le min\{(x_{max}^s-{\mathit{x}}^s)/({\alpha }^c\Delta t),u_{max}^c\}\hfill \\ \hfill & 0\le {\mathit{u}}^d\le min\{{\mathit{x}}^s/\Delta t,u_{max}^d\}\hfill \\ \hfill & \lambda \ge 0.\hfill \end{array}$$

(8)

This optimization problem has a linear objective function. Furthermore, all the constraint functions are affine except the first. Observing that $\mathit{u}\mapsto v_{t+1}\left(f(\mathit{x},\mathit{u},\mathit{\xi })\right)$ is a convex function, we approximate it as a convex piecewise linear function, i.e.,

$$v_{t+1}(f(\mathit{x},\mathit{u},{\xi }^{\left[k\right]}))\approx \sum _{i,j=1}^M{\gamma }_{i,j,k}{v}_{t+1}(x^{\left[i\right]},y^{\left[j\right]}),$$

where ${\{(x^{\left[i\right]},y^{\left[j\right]})\}}_{i,j=1}^M$ is the set of rectangular grid points that discretize the state space (Here, we assume that there are the same number of grid points on each axis. However, this assumption can easily be relaxed)and ${\left\{{\xi }^{\left[k\right]}\right\}}_{k=1}^K$ is the set of grid points that discretize the support $\Xi$. The uniform convergence property of this approximation scheme is shown in [30]. Here, the weight ${\gamma }_{i,j,k}$ represents the contribution of $(x^{\left[i\right]},y^{\left[j\right]})$ to approximating $f(\mathit{x},\mathit{u},{\xi }^{\left[k\right]})$, and thus satisfies

$$f(\mathit{x},\mathit{u},{\xi }^{\left[k\right]})=\sum _{i,j=1}^M{\gamma }_{i,j,k}\left[\begin{array}{c}{x}^{\left[i\right]}\\ y^{\left[j\right]}\end{array}\right],\sum _{i,j=1}^M{\gamma }_{i,j,k}=1,{\gamma }_{i,j,k}\ge 0.$$

With this piecewise linear approximation, the Bellman Equation (8) can be expressed as

$$\begin{array}{cc}\hfill v_t(\mathit{x})=\underset{\mathit{u},\lambda ,z,b,\gamma }{inf}& \theta \lambda +b+\frac{1}{N}\sum _{n=1}^N{z}_n\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \sum _{i,j=1}^M{\gamma }_{i,j,k}{v}_{t+1}(x^{\left[i\right]},y^{\left[j\right]})-\lambda |{\widehat{\xi }}_t^{(n)}-{\xi }^{\left[k\right]}|\le z_n\forall k=1,\dots ,K,\forall n=1,\dots ,N\hfill \\ \hfill & f(\mathit{x},\mathit{u},{\xi }^{\left[k\right]})=\sum _{i,j=1}^M{\gamma }_{i,j,k}·(x^{\left[i\right]},y^{\left[j\right]})\forall k=1,\dots ,K\hfill \\ \hfill & \sum _{i,j=1}^M{\gamma }_{i,j,k}=1\forall k=1,\dots ,K\hfill \\ \hfill & {\gamma }_{i,j,k}\ge 0\forall i,j=1,\dots ,K,\forall k=1,\dots ,K\hfill \\ \hfill & p({\mathit{x}}^p-h(\mathit{u}))\le b\hfill \\ \hfill & p_u({\mathit{x}}^p-h(\mathit{u})-R_u)+p{R}_u\le b\hfill \\ \hfill & -p({\mathit{x}}^p-h(\mathit{u}))\le b\hfill \\ \hfill & -p_d({\mathit{x}}^p-h(\mathit{u})-R_d)+p{R}_d\le b\hfill \\ \hfill & 0\le {\mathit{u}}^c\le min\{(x_{max}^s-{\mathit{x}}^s)/({\alpha }^c\Delta t),u_{max}^c\}\hfill \\ \hfill & 0\le {\mathit{u}}^d\le min\{{\mathit{x}}^s/\Delta t,u_{max}^d\}\hfill \\ \hfill & \lambda \ge 0.\hfill \end{array}$$

(9)

Given any $\mathit{x}\in \mathcal{X}$, this optimization problem is a linear program (LP) because f and h are affine functions. Thus, this optimization problem can be efficiently solved by using existing algorithms such as interior-point methods and simplex methods (see, e.g., [31,32,33] and the references therein).

Algorithm 1 describes the design procedure for a distributionally robust storage controller. It aims to solve the Bellman equation to compute the value function $v_t$ over a discretized state space. As specified in Line 7, the value function at a particular grid point is obtained by solving the LP (9). Note that an optimal ${\mathit{u}}^{🟉}$ of this LP is assigned to be an optimal action at state $(x^{\left[i\right]},y^{\left[j\right]})$, i.e., ${\pi }_t^{🟉}(x^{\left[i\right]},y^{\left[j\right]})={\mathit{u}}^{🟉}$. An optimal control action at a nongrid point $\mathit{x}$ can also be computed by solving the LP with $\mathit{x}$. Thus, this algorithm does not require any explicit interpolation, which may introduce additional numerical errors. The input data $\{{\widehat{\xi }}_t^{\left(1\right)},\dots ,{\widehat{\xi }}_t^{\left(N\right)}\}$ can be considered as a training sample. In fact, the proposed method can be interpreted as an adversarial training of a storage controller to make it robust against distributional errors.

The distributionally robust control problem (5) is solved once using Algorithm 1 before the storage operation starts. In other words, the proposed controller design approach is an offline method unlike, for example, receding horizon control. Note also that the proposed reformulation method does not require the direct calculation of Wasserstein distances, which is #P-hard [34]. This is an advantage of Wasserstein DRO that enables us to obtain an optimal solution through the dual form (7) or (9) without explicitly computing Wasserstein distances.

Algorithm 1: Distributionally Robust Storage Controller Design via Linear Program (LP).

4. Case Studies

To demonstrate the performance and utility of the proposed distributionally robust ramp management method, we perform simulation studies using the wind power generation data in the BPA control area for the year of 2018 [35]. The storage size is chosen as $x_{max}^s=10$ MWh with power rating $u_{max}^c=u_{max}^d=10$ MW. The initial SOC of storage is chosen to be the half of its capacity, i.e., $x^s\left(0\right)=x_{max}^s/2$. The storage operation interval is $\Delta t=5$ min, and the number of time steps T is 288. Thus, the storage device manages the ramp rate for 24 h. The ramp-rate limit is chosen as $\pm 0.5$ MW/min by setting $R_u=R_d=2.5$ MW. Additionally, the following parameters are used in the simulations: $\eta =0.99$, ${\alpha }^c={\alpha }^d=0.9$, $p=0.005$, $p_u=p_d=1$, $\theta =0.1$, and $K=21$. The sample of the ramp variable ${\xi }_t$ is constructed for each month from the BPA wind generation data. Since outliers bias the trained controller, we clipped out data points lying outside $[-120,120]$ MW and replaced such outliers with $\pm 120$ MW. The state space $\mathcal{X}=[0,10]\times [-120,120]$ was discretized with $11\times 21$ grid points with grid spacing 1 for x-axis and 12 for y-axis. All the numerical experiments were performed on a Mac with 4.2 GHz Intel Core i7 and 64 GB RAM. The LP problem (9) was solved using CPLEX for MATLAB.

4.1. Comparison with Stochastic Optimal Control

We first compare the performance of the proposed distributionally robust method and that of the standard stochastic optimal control method, which is the most popular approach to energy storage operation (e.g., [9,10,11]). The stochastic optimal controller is designed by solving (3) via dynamic programming. To evaluate the performance for all four seasons, the BPA wind data of January, April, July, and October are used. For each month, we split the data into the training and test data set: the training data are chosen as the ramp data for the first 15 days, i.e., from day 1 to day 15; and the test data are selected as those for the next 15 days, i.e., from day 16 to day 30. We use three different training sample sizes, 5, 10, and 15, in this comparison study. The training data are chosen from day 15, backward in time. For example, in the case of sample size 5, the training data are selected for days 11–15.

The performance of the two controllers are evaluated as the total ramp penalty for 24 h relative to the “no storage” case. In other words, the cost is evaluated as the ratio of the total ramp penalty with energy storage to that without storage. As shown in Figure 3, the proposed distributionally robust controller outperforms the standard stochastic optimal controller for all months and for all sample sizes. Specifically, the distributionally robust method saves the ramp penalty by 4.82% on average compared to the stochastic optimal control method, as summarized in

Table 1. Average ramp penalty relative to the “no storage” case depending on the sample size.

Sample Size	5	10	15	Avg
Stochastic optimal control	0.9939	0.9902	0.9970	0.9937
Distributionally robust control	0.9612	0.9399	0.9363	0.9458
Cost saving	3.29%	5.08%	6.09%	4.82%

. This result indicates that the distributionally robust method consistently resolves the issue coming from distribution errors for every season and sample size. The stochastic optimal controller sometimes performs worse than the “no storage” case. This is because the training set distribution is different from the test set distribution, i.e., the training set does not offer useful information about the behavior of wind power ramping in the near future. The stochastic optimal controller believes such a misleading or uninformative training set distribution, while the distributionally robust controller does not. The proposed method actively takes into account potential distribution mismatches and makes control decisions robust against the distribution errors. The effect of the distributionally robust method on net power production is shown in Figure 4. The controlled storage smoothens wind power fluctuations and thus reduces the ramp penalty.

When the sample size is too small, the data may provide too little information that is useful in decision making. On the other hand, as the sample size increases, old data are used for designing controllers. This addition of old data, which may be different from the future ramping behavior, may distort the training set distribution in an undesirable way. Thus, in both control cases, the performance is improved when increasing the sample size from 5 to 10 and is almost unaffected when increasing the sample size from 10 to 15.

4.2. Effect of Ambiguity Set Size

A notable advantage of Wasserstein distributionally robust control is that it provides a nonasymptotic probabilistic guarantee on the out-of-sample performance, which is the control performance evaluated with unseen test samples drawn from the true distribution [19,20]. It is well known that the out-of-sample performance critically depends on the radius $\theta$ that controls the size of ambiguity set (4). In our simulation studies, the ramp penalty computed with test samples is the measure of out-of-sample performance. We now examine how the radius affects the ramp management performance of the distributionally robust method. Figure 5 displays the effect of $\theta$ on the total ramp penalty relative to the “no storage” case, where the data of April are used for the test. As $\theta$ increases from 0.05, the performance initially decreases and then increases for $\theta$ greater than $0.2$. When the radius is too small, the resulting controller is not sufficiently robust to take into account the deviation of the test set distribution from the training set distribution. On the other hand, in the case of a large radius, decisions made by the distributionally robust storage controller are overly conservative. Thus, it is incapable of aggressively charging or discharging energy storage to minimize the ramp rate. According to the simulation result, the proposed controller with $\theta =0.1$–$0.2$ presents the best out-of-sample performance for wind power ramp management in the setting used for these simulations.

4.3. Effect of Storage Size

To examine the impact of storage size, the total ramp penalties relative to the “no storage” case are computed for different sizes $x_{max}$ of energy storage with radius $\theta$ fixed as $0.1$, using the data of April. As shown in Figure 6, the ramp penalty decreases as the storage size increases up to 11 MWh in both standard stochastic optimal control and distributionally robust control cases. This is because a bigger storage device provides greater operational flexibility, which can be utilized to mitigate the ramp rate of wind power generation. However, the benefit of such flexibility is saturated around 11 MWh: the ramp penalty even slightly increases with storage size. This counterintuitive result is caused by the mismatch between the training and test sets. The controllers use the prior knowledge obtained from the training set about wind power ramp to fully utilize the flexibility provided by energy storage. However, such behaviors can be overly aggressive when the storage size is large. Thus, as the test set distribution deviates from the training set distribution, the aggressive storage operation produces undesirable ramp events.

5. Conclusions

We have proposed a distributionally robust storage control method for wind power ramp management using historical data that may not provide reliable information about future wind power ramp events. Our simulation studies using the BPA data indicate that the proposed storage operation strategy overcomes wind ramp distribution errors and presents a robust performance unlike the standard stochastic optimal control method. The control performance depends on the Wasserstein ambiguity set size as well as the storage size, and there exist an optimal ambiguity set size and an optimal storage size given the trade-off between robustness and efficiency. The proposed distributionally robust storage control method can be extended in several interesting ways. First, an approximate dynamic programming algorithm can be designed to solve a large-scale distributionally robust control problem, taking into account transmission constraints in the network setting. Second, a more sophisticated model for wind power dynamics or even its forecast model can be used in conjunction with the distributionally robust method. Third, other types of services including frequency regulation and energy arbitrage can be considered by extending the proposed framework.