Using Stochastic Dual Dynamic Programming to Solve the Multi-Stage Energy Management Problem in Microgrids

Abstract

In recent years, the adoption of renewable energy sources has significantly increased due to their numerous advantages, which include environmental sustainability and economic viability. However, the management of electric microgrids presents complex challenges, particularly in the orchestration of energy production and consumption under the uncertainty of fluctuating meteorological conditions. This study aims to enhance decision-making processes within energy management systems specifically designed for microgrids that are interconnected with primary grids, addressing the stochastic and dynamic nature of energy generation and consumption patterns among microgrid users. The research incorporates stochastic models for energy pricing in transactions with the main grid and probabilistic representations of energy generation and demand. This comprehensive methodology allows for an accurate depiction of the volatile dynamics prevalent in the energy markets, which are critical in influencing microgrid operational performance. The application of the Stochastic Dual Dynamic Programming (SDDP) algorithm within a multi-stage adaptive framework for microgrids is evaluated for its effectiveness compared to deterministic approaches. The SDDP algorithm is utilized to develop robust strategies for managing the energy requirements of 1, 2, and 12 prosumers over a 24 h planning horizon. A comparative analysis against the precise solutions obtained from dynamic programming via Monte Carlo simulations indicates a strong congruence between the strategies proposed by the SDDP algorithm and the optimal solutions. The results provide significant insights into the optimization of energy management systems in microgrid settings, emphasizing improvements in operational performance and cost reduction.

1. Introduction

The quest for novel and efficient energy sources, their distribution, and the transformation for use in industrial and residential sectors while mitigating environmental impact stands as a pivotal aspect of societal progression. Central to this endeavor is the electrical grid, a complex system that channels energy from generation sites to the final consumers. This grid encompasses generation plants, transmission lines, and distribution networks. Generation plants convert primary energy sources into electrical energy, which is then conveyed through transmission lines to distribution networks, reaching various load centers.

As economies grow and the well-being of citizens improves, the demand for electrical energy escalates. Traditionally, this growing demand has been met by expanding and reinforcing the electrical grid, constructing new power generation facilities, and enlarging substation capacities. However, this traditional approach, while effective, exerts considerable pressure on natural resources and the environment and demands substantial investment. Consequently, governments are exploring alternative strategies, with the integration of Distributed Energy Resources (DER) being a notable example [1].

DERs encompass unconventional energy sources such as solar and wind power, along with various energy storage systems like batteries and flywheels. An effective method to incorporate these alternative energy sources is via microgrids. A microgrid is a small-scale, decentralized electrical network that combines renewable energy sources, storage technologies, and load management to deliver power in a reliable and sustainable manner to a localized community [2]. It can operate in conjunction with the main electrical grid or independently. Microgrids include a range of components like renewable generators, energy storage devices, electric vehicles, traditional power generators, and sophisticated control systems.

The concept of “prosumers”—entities that both produce and consume energy within the microgrid—is central to microgrid operations. The operational dynamics of a microgrid involve coordinating diverse energy sources and storage systems to efficiently meet its energy needs, facilitated by energy management systems that ensure a balance between supply and demand. The overarching goal is to maximize the use of renewable energy, reduce reliance on the main power grid, and achieve cost savings and enhanced reliability [1,3]. Microgrids not only contribute to a sustainable energy matrix but also promote a more equitable energy market [4,5,6].

If appropriate control techniques are implemented, microgrids can enhance the reliability of the electrical system, increase the utilization of renewable resources, and generate greater well-being in communities [7]. Control tasks within microgrids are typically categorized into three levels [8]:

i.

Primary-level tasks: These tasks ensure the stability of frequency and voltage levels across microgrid components, with response times ranging from 1 to 10 milliseconds.

ii.

Secondary-level tasks: These tasks compensate for any deviations in voltage and power levels caused by the primary control, with response times ranging from 100 milliseconds to 1 s.

iii.

Tertiary-level tasks: These tasks manage the power flow between the microgrid and the main grid, ensuring optimal economic operation via an energy management system (EMS), with response times varying from seconds to hours.

EMS in microgrids is inherently stochastic and dynamic due to uncertainties in energy demand, renewable generation, and pricing [9]. The control system must dynamically make decisions to meet the energy needs within each stage of the designated time horizon while maintaining system stability by keeping voltages and power flows within acceptable limits [10]. This requires minimizing operational costs and achieving other objectives via complex stochastic optimization, necessitating advanced control strategies for economic and stable operation [11].

The typical planning horizon for a microgrid extends 24 h into the future, with decision stages of either 15 min or one hour [12]. These short decision stages are essential to capture the variability and uncertainty in renewable energy generation and demand effectively. The storage capacities in the microgrid range from small residential batteries (around 3 kWh) to larger community storage systems (up to 30 kWh). Typically, it takes approximately 4–8 decision periods (1–2 h) to fully charge or discharge these storage systems under average conditions. This frequent updating of decisions is crucial to managing the rapid changes in generation and demand. Effective EMS are crucial for the success of microgrids, ensuring real-time balance and system stability [13]. Poor energy management can lead to increased operational costs and potential instability in the main grid [14]. Traditional algorithms like stochastic dynamic programming (SDP) and nested benders decomposition (NBD) struggle with the ‘curse of dimensionality’, where the exponential growth in scenarios from stochastic variables challenges computational capacities [15]. As stated before, for the typical planning horizon of 24 h and stages’ length of 1 h, even for a reduced number of scenarios, the computational effort required by SDP and NBD is unaffordable. Approximate methods offer a balance between accuracy and computational feasibility and, due to their reduced computational requirements, are an attractive alternative to solve the EMS problem.

This backdrop sets the stage for the exploration of advanced approximate methods like stochastic and dynamic optimization, machine learning, and artificial intelligence in improving microgrid energy management [16]. The Stochastic Dual Dynamic Programming (SDDP) algorithm shows promise in optimizing microgrid operations by accounting for the variability and dynamism of renewable energy sources. This paper aims to critically review the literature on energy management optimization in microgrids, with a focus on stochastic and dynamic modeling techniques, and to develop a computational simulation using the SDDP algorithm to strategize control.

The primary goal of this paper is to evaluate the efficacy of the SDDP algorithm in optimizing energy management decisions within microgrids and to explore its application in simulated scenarios. This study has significant implications for the reliability and efficiency of power supplies in microgrids, aiming to reduce operational costs and improve overall energy efficiency. The paper will proceed with a comprehensive literature review on the use of SDDP in microgrid EMS, followed by a detailed description of the SDDP algorithm and its application in computational simulations. The findings of this study will be presented, culminating in an analysis and conclusion that will also suggest directions for future research in this field.

The structure of this document is delineated as follows: Section 2 provides a comprehensive review of the literature pertaining to the algorithms employed to tackle the stochastic dynamic energy management challenge. Section 3 delineates the SDDP algorithm alongside the methodology employed for its enhancement. Subsequently, in Section 4, the outcomes of the research are presented. Conclusively, Section 5 offers a concise examination alongside the conclusions drawn from this study, as well as prospective avenues for future research.

2. Literature Review

In the field of microgrid management, a diverse array of methodologies has been explored to enhance system efficiency and cost-effectiveness. These methodologies span from traditional mathematical programming to sophisticated applications of metaheuristics and artificial intelligence. Collectively, these techniques are categorized into four primary groups, as illustrated in Figure 1: artificial intelligence techniques, conventional methods, metaheuristic-based methods, and others [17].

Within the realm of artificial intelligence, notable algorithms include fuzzy logic [18], game theory [19], and various forms of reinforcement learning, such as Q-Learning [20], DQN [21], actor–critic methods [22], and TD3 [23]. These have been applied with considerable success to the challenge of energy management in microgrids. For a detailed examination of reinforcement learning applications in this context, the reader is directed to the comprehensive review in reference [24]. Additionally, metaheuristics applied to energy management systems (EMS) can be subdivided into three categories: swarm intelligent-based EMS, evolutionary algorithm-based EMS, and other related approaches [17]. Noteworthy is the work by Elsied et al. [25], which employs a genetic algorithm to concurrently optimize cost and CO2 emissions, marking a significant advancement toward sustainable microgrid management. However, this approach does not fully incorporate the stochastic variability inherent in renewable energy sources and loads, as detailed in references [17,26].

A widely recognized conventional approach is mixed-integer linear programming (MILP), known for its relative simplicity and efficacy in the domain. Kanchev et al. present a deterministic solution for an isolated microgrid comprising a solar plant, a battery system, and a gas generation unit [27]. Further studies, such as those by Tsikalakis et al. [28] and Riffoneau et al. [29], have utilized mathematical programming to focus on cost minimization in microgrids integrating renewable energy sources. These works explore sequential quadratic programming and dynamic programming, respectively, with the former considering interactions with the main grid and the latter optimizing battery charge states. Sukumar et al. [30] extend these methods to both isolated and grid-connected systems via linear programming and MILP, albeit without addressing conductor losses.

In the realm of stochastic optimization for microgrid energy management, both two-stage and multi-stage stochastic programming models play pivotal roles. Two-stage stochastic models, extensively explored in the literature [31], are structured to make initial decisions before uncertainties are revealed, and subsequent recourse actions are determined once uncertainties become apparent. For instance, a model detailed in [32] operates over two 24 h intervals, optimizing operational costs while accommodating uncertainties in a 33-node microgrid that integrates a diesel generator, a battery system, a solar plant, and a wind turbine.

Expanding beyond two-stage models, multi-stage stochastic programming introduces a more granular approach to decision-making under uncertainty [31]. These models generate a scenario tree where each node represents a possible state of the world, and edges depict transitions between these states across multiple stages [31]. This methodology is beneficial for capturing the temporal dynamics and uncertainties in microgrid operations, as exemplified by the application of NBD [33]. NBD, an advanced form of Benders decomposition tailored for multi-stage problems, efficiently tackles the complexities involved in these models by decomposing them into simpler subproblems, which are solved iteratively. This approach was utilized in [34] to manage the operations of a diesel generator, adjusting dynamically to fluctuations in load and photovoltaic energy production, thereby optimizing operational costs and enhancing microgrid frequency stability via strategic adjustments.

Turning to SDP, this method is employed for solving convex optimization problems aimed at minimizing operational costs in microgrids, as seen in [35]. However, SDP is often limited by the “curse of dimensionality”, which becomes a significant barrier when dealing with a moderate to large number of decisions or state variables, rendering the method impractical for broader applications. Addressing these challenges, Huangfu et al. [9] developed a bi-objective dynamic programming approach that integrates cost minimization with curtailment rate reduction. Further, a study in [36] combined deterministic and stochastic dynamic programming strategies to manage energy, although it assumed a deterministic load profile and energy prices, with limited stochastic inputs, which could restrict the applicability of the results in more variable real-world scenarios.

Continuing from the challenges of the “curse of dimensionality” encountered in both SDP and multi-stage stochastic models, it is essential to recognize how this phenomenon complicates the practical application of advanced stochastic models. As the number of stages in a model increases, the associated scenario tree expands exponentially, thus significantly complicating the computation. This exponential growth renders the application of NBD algorithms impractical for problems encompassing more than five stages, as noted in [37].

To address the computational hurdles imposed by the “curse of dimensionality”, several approximation methods have been developed. One of the most influential of these methods is the Stochastic Dual Dynamic Programming (SDDP) technique, introduced by Pereira and Pinto in 1991 [38]. SDDP is particularly well-suited to tackling multi-stage stochastic problems due to its innovative approach to handling the complexities associated with the scenario trees.

The core principle of SDDP hinges on the assumption of stage-wise independence, which posits that uncertainties across different stages of the decision-making process are independent. This assumption simplifies the model significantly, as it implies that all nodes within a single stage will share identical child nodes featuring the same realizations of uncertainty and their corresponding probabilities. This structure drastically reduces the complexity of the scenario tree and facilitates a more manageable approach to solving the model.

Due to the stage-wise independence, instead of requiring a distinct value function for each node within the scenario tree, SDDP utilizes a singular value function for each stage [37]. This streamlining allows for a significant reduction in computational demand. To approximate the value functions effectively, SDDP employs a set of linear functions or hyperplanes. These hyperplanes are generated via the random sampling of scenarios, which capture the diverse possible outcomes and uncertainties at each stage of the decision process.

By approximating the value functions via these hyperplanes, SDDP provides a powerful and efficient means of solving complex multi-stage stochastic problems that would otherwise be intractable due to the “curse of dimensionality”. This method not only enhances the feasibility of applying stochastic models to real-world problems but also ensures that decisions made at each stage are optimized based on the most probable scenarios and their outcomes.

In the comparison of various stochastic optimization algorithms,

Table 1. Comparison of NBD, SDP, and SDDP.

Method	Stage-Wise Independence Assumption	State and Action Discretization	Complexity in T	Complexity in State Dimension	Complexity in Number of Scenarios
DE	X	X	X	Exponential	Polynomial
NBD	X	X	X	Exponential	Polynomial
SDP	X	Yes	Yes	Linear	Polynomial
SDDP	Yes	X	X	Exponential	Polynomial

illustrates the distinctions between SDDP, NBD, and the deterministic equivalent method (DE), revealing significant differences in their operational frameworks and areas of applicability.

SDDP offers a notable reduction in computational effort per iteration by approximating a single cost function per stage rather than individual functions for each node within a stage, as required in more complex stochastic models. This simplification is achieved via the assumption of stage-wise independence, which posits that decisions and uncertainties in one stage do not depend on those in other stages. While this assumption facilitates a more streamlined computational process, it does maintain the exponential complexity in the number of decision stages—a characteristic it shares with NBD. Despite this complexity, SDDP is particularly advantageous in scenarios with continuous action spaces where the state and action spaces do not need to be discretized, unlike in traditional SDP.

NBD, on the other hand, involves decomposing a multi-stage problem into a series of sub-problems that are solved sequentially, linking the solutions via dual and primal cuts. This method is highly effective for a smaller number of stages due to its detailed and precise handling of dependencies between stages. However, as the number of stages increases, the complexity of managing and solving these interconnected sub-problems grows, making it less practical for very large-scale applications.

The deterministic equivalent method converts stochastic problems into a single large deterministic problem by considering all possible scenarios simultaneously. While this approach can theoretically guarantee an optimal solution, it becomes computationally intensive and practically infeasible as the number of scenarios and stages increases due to the combinatorial explosion in the size of the problem.

In practical applications, SDDP has often demonstrated superior performance compared to NBD and traditional SDP [37], particularly in settings involving continuous action spaces and a moderate number of decision stages. The effectiveness of SDDP in such environments can be attributed to its ability to handle large-scale problems more efficiently via the simplification of cost function approximations. However, this efficiency comes at the cost of assuming stage-wise independence, which may not be a valid assumption in all contexts and could lead to suboptimal decisions if inter-stage dependencies are significant.

One innovative application of SDDP can be observed in [39], where a hybrid strategy combining SDP and SDDP was employed for the economic dispatch of a hydroelectric power system in Norway. This study considered stochastic river inflow, along with energy and reserve prices modeled as Markov chains with discrete states, over a two-year planning horizon with weekly decision stages. This approach allowed for precise management under uncertainty, although it operated under the assumption that energy demand remained deterministic, and no energy storage was available within the network.

In a microgrid context, ref. [40] applied SDDP to manage a system with only four buses, where demand, renewable generation, and energy prices were all treated as stochastic. The results highlighted energy cost savings compared to simpler management policies; however, the limited size of the microgrid posed challenges for generalizing findings to larger, more complex systems.

Further expanding the scope, ref. [41] integrated SDDP with model predictive control (MPC) for comprehensive energy management in an industrial park featuring diverse energy sources and loads, including renewable generation and a combined heat and power (CHP) system. Despite this complexity, the study only considered deterministic energy prices and covered a small-scale network, which could hinder broader applicability.

Moreover, ref. [42] showcased the use of SDDP for economic dispatch within a transmission system over a 24 h period with 15 min stages, emphasizing the algorithm’s efficacy in handling renewable generation variability represented via a discrete Markov process. Yet, this study overlooked the dynamics of energy transactions and the stochasticity in demand or prices, which could affect the robustness of the solution in real-world scenarios.

Additionally, ref. [43] employed a robust version of SDDP for a microgrid with three diesel generators, a wind turbine, and a battery system. While this study effectively addressed the stochastic nature of renewable generation and demand, it did not consider the potential for energy transactions among prosumers, thus missing out on modeling a key aspect of microgrid dynamics.

In ref. [44], the SDDP method was again utilized for microgrids, focusing particularly on renewable generation as a stochastic element. However, this study similarly neglected the complexities of prosumer interactions and diverse pricing mechanisms, limiting its scope.

These cases reflect a broader trend in the application of SDDP, primarily for long-term planning of large-scale energy systems, as seen in [37], and short-term planning within microgrids with decision stages ranging typically from 15 min to 1 h. Despite the substantial progress in utilizing SDDP for dynamic and stochastic energy management, significant limitations persist. Many studies focus predominantly on renewable generation variability, often overlooking the stochastic nature of energy demand or price fluctuations. In addition, most of these studies consider consumers as passive agents, limiting the ability to model energy transactions among them. Additionally, the computational intensity of SDDP and the challenges in scaling the studies to more comprehensive and interactive networks pose substantial barriers. These constraints highlight the need for further research to enhance the method’s practicality and accuracy, ensuring it can more effectively represent and manage the complexities of real-world energy systems.

This study aims to bridge these gaps by applying the SDDP algorithm in the context of microgrid EMS. This way, it seeks to enhance the existing methodologies’ capabilities to handle the stochastic and dynamic aspects of microgrid operations, which are critical for efficient and cost-effective energy management. Thus, the main contributions of this work can be summarized as follows:

• Prior research has often overlooked the full integration of stochastic elements and dynamic conditions in microgrid energy management. This study addresses this gap by employing the SDDP algorithm, which excels in handling the stochastic nature of energy generation, demand, and pricing. It thus extends the existing models by incorporating a more realistic treatment of uncertainty, providing a methodological advancement over traditional deterministic and simpler stochastic methods.
• The integration of real-time data into the SDDP framework addresses another significant gap in the literature, where previous studies have not fully leveraged the potential of data for operational optimization in microgrids. This approach enhances both the economic efficiency and reliability of microgrid operations, providing practical benefits in terms of cost savings and energy utilization.
• The literature indicates a limited exploration of multi-stage stochastic optimization [42] in microgrids, particularly with a focus on multi-stage decision processes and adaptive control strategies. By applying the SDDP method, this study introduces a comprehensive framework that captures the multi-temporal nature of decision-making in microgrids, allowing for more dynamic adjustments to operational strategies as new information becomes available.
• While existing studies have often focused on specific types of microgrids, this work demonstrates the versatility of the SDDP algorithm across various configurations and scales. The inclusion of energy transactions among prosumers is another key contribution of this study, emphasizing the importance of this factor in enabling greater democratization of the energy market and contributing to the development of the electric internet [45]. This contribution addresses the need for adaptable solutions that can be customized to different microgrid setups, from rural, isolated systems to interconnected urban networks, filling a critical gap in scalable and flexible energy management solutions.
• Finally, the research lays a foundation for future studies in the field by highlighting the effectiveness of the SDDP method in managing the complexity of microgrid systems. It opens new avenues for exploring further enhancements in algorithmic approaches and their applications.

3. Stochastic Decision Processes in Microgrid Energy Management

The management of energy within microgrids epitomizes a quintessential stochastic optimization problem where decisions must be made under multiple layers of uncertainty. These uncertainties arise from volatile renewable energy outputs, fluctuating demand profiles, and dynamic pricing in energy markets. Effective EMS for microgrids must therefore not only respond to immediate energy requirements but also anticipate and strategically manage future states under uncertainty. In general, the decision process can be summarized as follows:

• Stage Setup: At each decision stage t, the algorithm updates its state based on the latest available information about uncertainties, denoted by ${\xi }_t$.
• Decision Execution: Once the uncertainty ${\xi }_t$ for the current stage is known, the decision $a_t$ is made. This decision is aimed at optimizing specific performance metrics, such as cost minimization, energy efficiency, or other operational objectives.
• Objective Function: The core objective at each stage is formulated to minimize expected costs or maximize efficiency, integrating the newly revealed uncertainties. This dynamic adaptation helps to effectively manage the variable nature of microgrids.

Figure 2 summarizes the interaction between the elements and highlights the dynamism of the decision-making process.

In the context of multi-stage decision problems, particularly within the scope of stochastic programming, the concept of a policy plays a pivotal role. A policy, as defined in the literature [37], comprises a sequence of decision rules $\{a_1,a_2,\dots ,a_T\}$, where each $a_t$ is a function of the stochastic data ${\xi }_t$ observed at each stage $t$ of the decision process. This series of decision rules effectively guides the decision-making process across all stages, tailored to the specific realizations of the stochastic variables encountered.

Each decision rule $a_t\left({\xi }_t\right)$ within the policy is crafted to respond appropriately to any given realization of the random data ${\xi }_t$ at stage $t$. This characteristic of the policy ensures that decisions are not only sequential but also adaptive, considering the evolving nature of the uncertainties inherent in the system.

The primary objective in defining and utilizing such a policy is to identify an optimal set of decisions that maximizes (or minimizes) a particular objective function while adhering to a predefined set of technical constraints. These constraints can range from physical limitations within a system to regulatory and safety standards that must be met.

3.1. Formulating the Decision-Making Process under Uncertainty

In the stochastic framework of microgrid EMS, each stage t presents a decision node where the system state $E_{i,t}$ encapsulates the current state of charge, anticipated generation based on renewable sources, and expected consumption for each prosumer $i$. Decisions made at each interval—ranging from energy trading (buying or selling) to storage management (charging or discharging batteries)—are pivotal, impacting not only current cost and energy efficiency but also the system’s future operational state E.

To model the uncertainty inherent in microgrid operations, the stochastic variables $P_{i,t,s}^{gen}$ and $P_{i,t,s}^{load}$ represent the potential generation and load for each prosumer i at stage t across possible scenarios s. These variables are drawn from probabilistically defined distributions—gamma for solar output, beta for wind generation, and log-normal for consumption profiles [46]. Similarly, the stochastic nature of market interactions is captured via beta-distributed variables ${\varphi }_{i,t,s}^{buy}$ and ${\varphi }_{i,t,s}^{sell}$, which define the fluctuating buy and sell prices at each stage [12].

All these stochastic variables are assumed to be independent of each other and exhibit stage-wise independence. This independence assumption aligns with the scenario tree methodology, as employed in [47,48]. Notably, ref. [49] has demonstrated that standard SDDP effectively converges to the optimal policy under stochastic objectives. Further advancing this approach, ref. [46] employs the open-source MSPPY package to tackle stochastic multistage decision problems via SDDP, with stochasticity in the objective function modeled by a scenario tree; this package is utilized in our current study.

It should also be noted that an alternative approach prevalent in the literature models price dynamics using a Markov chain, initially proposed in [50]. However, the assumption that prices evolve according to a homogeneous Markov chain remains a topic of debate [51], highlighting the complexity and variability in modeling financial variables within microgrid systems. This discussion underscores the necessity for flexible and robust modeling techniques to accurately reflect the dynamic and uncertain nature of energy markets and grid operations.

Mathematically, the EMS can be represented as a multi-stage stochastic problem as presented below:

$$\min \sum _{s\in S}{\pi }_s[\sum _{i\in N}\sum _{t\in H}\left({\varphi }_{i,t,s}^{buy}{p}_{i,t,s}^{buy}-{\varphi }_{i,t,s}^{sell}{p}_{i,t,s}^{sell}\right)+\sum _{i\in N}\sum _{j\in N}\sum _{t\in H}{f_{i,j,t,s}\ast y}_{i,j,t,s}]$$

(1)

which is subject to

$$\begin{array}{c}{I}_{i,t,s}=I_{i,t-1,s_{t-1}}+(-P_{i,t,s}^{load}+p_{i,t,s}^{buy}-p_{i,t,s}^{sell}+P_{i,t,s}^{gen})\ast \Delta t+{\displaystyle \sum _{j\in N|j\ne i}}{y}_{j,i,t,s}\\ -{\displaystyle \sum _{j\in N|j\ne i}}{y}_{i,j,t,s}, \forall i\in N,\forall t\in H,\forall s\in S \end{array}$$

(2)

$$I_{i,t,s}=I_{i,t-1,s_{t-1}}+\Delta t\ast \left(\dot{p_{i,t,s}^{char}}{\eta }_b-\frac{\dot{p_{i,t,s}^{dis}}}{{\eta }_b}-I_{i,t,s}{\beta }^b\right) \forall i \in N, t\in T, s\in S$$

(3)

$$I_{i,t,s}\le k_i, \forall i\in N,\forall t\in H, \forall s\in S$$

(4)

$$p_{i,t,s}^{buy}\le {\overline{P}}_{i,t}^{buy}, \forall i\in N,t\in H, \forall s\in S$$

(5)

$$p_{i,t,s}^{sell}\le {\overline{P}}_{i,t,s}^{sell}, \forall i\in N,t\in H,\forall s\in S$$

(6)

$$p_{i,t,s}^{buy},p_{i,t,s}^{sell},I_{i,t,s}\ge 0, \forall i\in N,t\in H,\forall s\in S$$

(7)

$$y_{i,j,t,s}\ge 0, \forall i\in N,\forall j\in N,\forall t\in H,\forall s\in S$$

(8)

wherein the objective Function (1) minimizes the cost of buying and selling energy by prosumers, including the transportation cost. Note that the purchase ${\varphi }_{i,t,s}^{buy}$ and sale ${\varphi }_{i,t,s}^{sell}$ prices of energy from the main grid are stochastic variables, while the power bought $p_{i,t,s}^{buy}$ and sold $p_{i,t,s}^{sell}$ to the grid are decision variables determined by the EMS.

Constraint (2) ensures the power balance of each prosumer in each stage. Note that in this constraint, the power generated by solar and wind sources $P_{i,t,s}^{gen}$, and the power demanded by home appliances of each prosumer are stochastic variables $P_{i,t,s}^{load}$, while the battery’s state of charge (SOC) $I_{i,t,s}$ and the power transactions with the main grid, $p_{i,t,s}^{buy} and p_{i,t,s}^{sell}$ are the decision variables determined by the EMS.

Constraint (3) models the battery’s SOC behaviors, including the effect of efficiency at charging and discharging modes represented by ${\eta }_b$ joining to the round-trip efficiency ${\beta }^b$ [52]. Finally, Constraints (4)–(8) represent the battery system’s capacity, the limits of power bought and sold to the main grid, and the positive nature of the decision variables, respectively.

It is important to highlight that the value of the time step. $\Delta t$ has a direct impact on the computational complexity of the algorithm. As demonstrated by [38], increasing the time step leads to greater uncertainty in predictions of variables such as prices, power demands, power generation, etc., whereas reducing the time step increases computational effort. The substantial computational effort arises from the exponential increase in computational complexity of SDDP as a function of decision epochs and the number of scenarios [37]. In this regard, a common practice, as followed in this study, is to use a time step of 1 h over a planning horizon of 24 h [7,12,53]. It is worth mentioning that these simplifications and approximations are made with the aim of making the solution process of the original problem manageable, which would otherwise be impractical with current computing resources due to the large number of scenarios generated and the complexity of the algorithms that would be feasible to apply [37].

Finally, Figure 3 illustrates the general methodology proposed in this study; solar generation data, wind generation data, electricity demand, and energy prices are processed to be used in scenario generation via the adjustment of probability distributions, as previously described. Furthermore, the original problem outlined above is modeled as a set of linear subproblems for each decision stage. Subsequently, scenario discretization is performed for each decision stage. The SDDP solver is employed to approximately solve the stochastic multi-stage decision problem. Finally, several Monte Carlo simulations are conducted to validate the solution quality. Refined details of the SDDP algorithm and Monte Carlo simulations will be expanded on in a later section of this paper.

3.2. Implementation of the Stochastic Dual Dynamic Programming (SDDP) Algorithm

The SDDP algorithm is crucial for addressing multi-stage stochastic decision problems in energy management systems. These problems involve making sequential decisions over a set of periods, t, to optimize a specified objective function while contending with uncertainties that influence both the objectives and the constraints.

In general form, the decision process for every period $t$ can be formulated as in (9).

$$\mathit{min}{C_1}^T{x}_1+\dots +{C_T}^T{x}_T$$

Subject to:

$$\begin{array}{c}{A}_{11}{x}_1 \ge b_1\\ A_{21}{x}_1+A_{22}{x}_2 \ge b_2\\ ⋮\\ A_{T,T-1}{x}_{T-1}+A_{TT}{x}_T\ge b_T\end{array}$$

(9)

wherein $x_1, x_2, \dots , x_T$ denote the decision variables associated with each respective period and $C_1,A_{11},b_1$ are known at the outset, but $C_t,A_{t,t-1},A_{tt}\mathrm{y}{b}_t$ generally involve random components that are revealed over time [37]. Consequently, the dynamic structure of the problem is often expressed in a nested stochastic formulation (10):

$$\mathit{min}{C_1}^T{x}_1+E[\mathit{min}{C_2}^T+E[\dots E[\mathit{min}{C_T}^T{x}_T]\dots ]]$$

(10)

The core principle of SDDP lies in approximating the future cost function using a series of linear functions or cuts. Initially, the problem

$$\mathit{min}{C_1}^T{x}_1+\alpha \left(x1\right)$$

(11)

subject to

$$A{x}_1\ge b_1$$

is solved, where $\alpha \left(x_1\right)$ varies based on the decision at stage one. The forward recursion phase involves using a set of trial decisions, $x_{t, t\in T}$, to estimate these cost function approximations. Given the computational impracticality of simulating every potential scenario, a Monte Carlo simulation is conducted for a sample of potential variable combinations [49]. Figure 4 shows semantically the Monte Carlo simulation for three scenarios, highlighted in green, in a multi-stage decision problem with three stages.

After the Forward pass has been performed, the following linear approximation (12) is performed for the cost function [37] and iteration $\Gamma$:

$${\tilde{\alpha }}^{\Gamma +1}\left(x_{t-1}\right)\ge \underset{\_}{{\alpha }^{\Gamma +1}\left(x_{t-1}\right)}+{{\beta }_{t,s}}^{\Gamma }\ast (x_{T-1}-x_{T-1}^{\Gamma ,s})$$

(12)

Here, ${\phi }_{T,s}^{\Gamma }(x_{T-1})$ represents the expected value of ${\tilde{\alpha }}^{\Gamma +1}\left(x_{t-1}\right)$ given all possible uncertainty realizations at stage T and scenario s. A piecewise linear approximation, known as a polyhedral approximation, is constructed as shown in (13):

$${{\Lambda }_t}^{\Gamma +1}=max({{\Lambda }_t}^{\Gamma }, {\phi }_{t,1}^{\Gamma }(x_{t-1}), {\phi }_{t,2}^{\Gamma }\left(x_{t-1}\right), ..., {\phi }_{t,\left|S\right|}^{\Gamma }(x_{t-1}))$$

(13)

The backward recursion involves computing these sub-gradients ${{\beta }_{t,s}}^{\Gamma }$ using the dual values of the linear subproblem of each stage $t$. For a more detailed explanation on Backward pass and the calculation of sub-gradients using dual values, please refer to [37,49,50].

Solving stochastic and dynamic linear problems poses challenges due to the curse of dimensionality, where the exponential growth of possible states and actions complicates solutions. Two prevalent methodologies for addressing this challenge are deterministic equivalent (DE) and Markov decision processes (MDP). DE simplifies the stochastic model for quick and efficient solutions by substituting deterministic parameters for random variables [47]. However, accuracy depends on precise parameter estimates, and oversimplification may lead to suboptimal solutions. MDP, on the other hand, models sequential decision-making with outcomes influenced by both actions and the system’s current state [52]. While powerful, MDP demands significant computation and modeling, with complexity escalating as states and actions increase. An alternative, SDDP, tackles the curse of dimensionality by approximating solutions without considering all possible scenarios.

3.3. Implementation of SDDP Algorithm for Microgrid Energy Management Problem

The SDDP algorithm is implemented to manage the complex multi-stage decision-making in microgrid energy systems. Each decision stage typically set at one-hour intervals, allows for frequent updates based on the latest available information, capturing the variability in renewable generation and demand. This is essential for effectively managing storage systems, which can typically be filled or emptied within 1–2 decision periods (1–2 h) under average conditions. This level of granularity and responsiveness is crucial for optimizing the EMS’s performance, making SDDP an ideal choice over more rigid classical multi-stage stochastic programs. This section refines the approach discussed in the Mathematical Formulation section, introducing a concrete structure for the application in microgrid energy management.

• Stage Initialization and Setup

The foundational stage, $t=0$, is critical for establishing the initial conditions of the energy system, although it does not correspond to a real-time period. This stage initializes the problem framework and sets a precedent for the dynamic decision-making process across subsequent stages. Each stage is characterized by specific scenarios, defined as subproblems, that encapsulate the following variables:

• ${\varphi }_{i,t,s}^{buy}$ and ${\varphi }_{i,t,s}^{sell}$: prices at which energy is bought and sold.
• $P_{i,t,s}^{load}$ and $P_{i,ts}^{gen}$: power load and generation for each prosumer.

The decision variables used across these subproblems include the following:

• $I_{i,t,s}$: state variables that bridge consecutive stages.
• $p_{i,t,s}^{buy},$ $p_{i,ts}^{sell},y_{i,j,t,s}$: operational variables specific to each stage and scenario.

• Problem Formulation for Each Stage:

Stage 1 (Initialization): At time $t=0$, the aim is to address the problem outlined by Equations (14)–(17). Constraints (15) and (16) are employed to initialize the SOC for each prosumer, accounting for their individual storage capacities. Consequently, if a prosumer lacks a battery bank (i.e., possesses zero capacity), its SOC is initialized equal to $k_i$. Otherwise, it is initialized to a predetermined value. The auxiliary variable $z$ is incorporated to facilitate the computational execution of this stage within the software framework.

$$\mathit{min}z$$

(14)

which is subject to

$$I_{i,0}=0 \forall i\in N| k_i=0$$

(15)

$$I_{i,0}=I_{0i} \forall i\in N| k_i>0$$

(16)

$$z free$$

(17)

Stages 2 to T: For stages from 2 to T, the subproblem aligns closely with the base model, transitioning the initial setup into actionable decision-making. The objective at these stages integrates the objective Function (1) subject to the following constraints that ensure the physical and operational feasibility of energy management (2)–(8) but disregarding the index $\mathrm{s}$ related to the set of scenarios.

As an illustration, the SDDP algorithm is applied to an electric microgrid with the following parameters: a set of prosumers, $\mathrm{P}=\left\{\mathrm{1,2}\right\}$; scenarios, $\mathrm{S}=\left\{\mathrm{1,2}\right\}$; and stages, $\mathrm{T}=\{\mathrm{0,1},2\}$. Figure 5 depicts the multi-stage adapted EMS problem, serving as a framework for implementing the SDDP, where the circles represent the uncertainty realizations of buying prices, selling prices, energy generated, and energy demanded prosumer 1 and 2, respectively, and the green circles represent the scenarios sampled in the Forward step, while blue circles represent scenarios not sampled in the current iteration.

The initial stage, $t=0$, serves as a fictitious phase, where the initial battery’s energy levels for all prosumers are initialized. Consequently, it does not entail any energy quantities or prices. The corresponding model for this first-stage scenario is illustrated in Figure 6, highlighted in orange. The subproblem for this stage is described in (14)–(17).

Stage $t=2$ to $t=T$: When examining any stage other than $t=0$, several parameters will vary depending on the specific scenario. An illustrative example is presented in the scenario highlighted in orange in Figure 7, wherein the following variables are defined:

• $P_1^{gen}$: is the power generated by prosumer 1.
• $P_2^{gen}:$ is the power generated by prosumer 2.
• ${\varphi }_1^{buy}:$ is the price for prosumer 1 to buy power from the main network.
• ${\varphi }_2^{buy}$: is the price for prosumer 2 to buy power from the main network.
• $P_1^{load}$: is the load required by prosumer 1.
• $P_2^{load}$: is the load required by prosumer 2.
• ${\varphi }_1^{sell}$: is the price for prosumer 1 to sell energy to the main network.
• ${\varphi }_2^{sell}$: is the price for prosumer 2 to sell energy to the main network.

Thus, the SDDP algorithm can capture the complexity and variability of energy prosumer scenarios across different stages, providing a robust and flexible framework for decision-making and optimization while keeping the computational resources demand at an acceptable level.

3.4. Monte Carlo Simulations

The SDDP algorithm, as detailed earlier, develops a feasible policy for the given energy management problem. This policy can be rigorously assessed across a spectrum of uncertainty scenarios inherent in the original problem formulation by employing Monte Carlo simulations. These simulations enable the construction of a $1-\alpha$ confidence interval for the average value of the policy. This interval’s upper limit effectively sets an upper bound on the value of the optimal policy, ensuring a confidence level of $1-\alpha$.

The confidence interval is calculated using [50]

$$CI=\underset{\_}{\omega }\pm t_{\frac{\alpha }{2}}\ast \left(\frac{SD}{\sqrt{n}}\right)$$

(18)

Here, $\overline{x}$ represents the average outcome of the policy values derived from the Monte Carlo simulations, $SD$ is the standard deviation of these outcomes, $n$ is the total number of simulations conducted, and $t_{\raisebox{1ex}{$\mathsf{\alpha }$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}$ is the critical value from the t-Student distribution for the specified confidence level $1-\alpha$. with $n-1$ degrees of freedom.

This approach not only quantifies the policy’s effectiveness under varying conditions but also provides statistical assurance of its performance reliability, as referenced in [50].

4. Simulation Results

This section presents the methodology and outcomes from the application of the Stochastic Dual Dynamic Programming (SDDP) algorithm for microgrid energy management. The microgrid configuration employed in this study is illustrated in Figure 8 where each prosumer is connected to a common coupling point and has the capability to engage in energy transactions with other prosumers.

The study utilizes datasets from Universidad de los Andes and PECAN STREET Dataport (https://www.pecanstreet.org/dataport/, accessed on 10 June 2023), which include wind and solar generation, energy demand, and pricing data. The data undergo cleaning and processing, fitting of probability density functions [54], and integration into the SDDP algorithm.

The SDDP algorithm is implemented using Python 3.9 on a virtual machine powered by Gurobi 10.0.0 optimization software. The setup includes an Ubuntu Linux operating system with a 12-core NV processor and 112 GB of RAM. To mitigate potential convergence issues, two stopping criteria were set: a maximum runtime of 21,600 s and a limit of 300 iterations.

The algorithm is tested via simulations, forming scenarios based on the probability distributions derived from the initial data set. Each scenario evaluates the policy’s response to different energy demands and market conditions. These are detailed in the following sub-sections.

Table 2. Comparison of number of prosumers, decision stages, state variables, and number of scenarios.

Case	Number of Stages	State Variables *	Number of Scenarios
One prosumer connected to the main grid without an energy storage system	24	1	$1\times {10}^{144}$
One prosumer connected to the main grid with an energy storage system	24	1	$1\times {10}^{144}$
Two prosumers connected to the main grid with energy storage systems	24	2	$1\times {10}^{288}$
Twelve prosumers	24	12	$1\times {10}^{1728}$

* State Variables Refer to Variables That Connected Two Consecutive Stages, as Defined by [37], Which Are Not the Total Number of the Mathematical Model.

details for each of the cases studied the number of prosumers in the microgrid, the number of decision stages, the number of state variables, and the number of scenarios.

4.1. One Prosumer Connected to the Main Grid without Energy Storage System

Figure 7 shows the behavior of the power bought to the main grid by the prosumer, as well as the net energy balance before carrying out the selling transaction. This balance corresponds to Equation (19).

$$P_{balance}=P_{generated}-P_{demand}$$

(19)

In this context, when $P_{balance}$ is greater than zero, there is an excess of energy that can be stored or sold. On the other hand, when $P_{balance}$ is less than zero, there is an energy deficit that needs to be compensated by purchasing energy from the grid. Figure 9 shows the behavior of the power sold to the grid. As seen from Figure 9 and Figure 10, without a battery bank, the system aims to satisfy the energy conservation equation by selling all the surplus when it is available and purchasing.

4.2. One Prosumer Connected to the Main Grid with Energy Storage System

In this scenario, a single prosumer with a battery bank capable of storing 3 kWh of energy was considered. Figure 11 depicts the behavior of the power bought from the grid, and Figure 12 illustrates the behavior of the sold to the grid. As can be seen in Figure 11 and Figure 12, the SDDP algorithm aims to minimize power purchases from the grid while attempting to sell energy to it when purchase prices are high. However, if the energy balance exceeds the battery capacity, it has no choice but to sell to the grid at whatever price is available (note the peak at the 20th hour).

4.3. Two Prosumers Connected to the Main Grid with Energy Storage Systems

In this case, two prosumers with 3 kWh battery banks are considered, connected to the grid and with the ability to exchange energy between them at a cost of 1 USD/kWh due to the use of the grid. Figure 13 illustrates the policy for prosumer number 1 for the energy purchased from the grid as well as from prosumer number 2. As it is cheaper to buy energy for prosumer number 2, the algorithm aims to maximize the energy bought by prosumer 2 and to minimize the energy bought to the grid.

4.4. Validation of Results

To validate the results obtained, a Monte Carlo simulation was conducted to assess the policy obtained in each of the previous cases under random realizations of uncertainty. Three thousand simulations were carried out for each case listed in Table 1. The histograms of the values obtained for the policy in the five cases listed in Table 1 can be found in Appendix A. Montecarlo simulation results. As can be seen, the confidence intervals are very narrow, meaning that the exact optimal policy is close to the policy approximated by the SDDP algorithm [50].

Table 3. Confidence interval for policy evaluated on true problems.

Number of Prosumers	Battery Bank Capacity kwh	95% Confidence Interval (USD Cents)	DE (USD Cents)
1	3	16.97–18.34	18.7040
1	12	16.48–17.82	18.6738
1	30	16.47–17.81	18.6738
2	3	29.08–30.96	-
12	5 to 10	−0.00092–0.0008	-

also presents results for certain cases of the deterministic equivalent of the problem (i.e., a deterministic linear program for the discretized original problem). This type of deterministic equivalent formulation exhibits exponential complexity [50], rendering it only feasible for relatively small-scale problems since it requires great amounts of RAM memory and computation time.

5. Conclusions

This study has detailed a solution methodology for the Stochastic Dynamic Energy Management Problem (SDEMP) within microgrids, emphasizing the incorporation of stochasticity and dynamism in demand, supply, and power pricing. By utilizing the Stochastic Dual Dynamic Programming (SDDP) algorithm, this approach facilitates the modeling of sequential decisions and the sharing of information throughout the operation of an energy management system (EMS).

The SDDP algorithm proves to be a potent tool in the domain of energy management optimization, particularly when judiciously applied to dynamic and stochastic scheduling challenges. Despite being an approximate method, SDDP enables the generation of feasible policies that closely approximate the optimal solutions, which are often unattainable due to the curse of dimensionality.

Our simulations delivered detailed energy management plans for each prosumer at every decision stage. These plans are strategically designed to minimize total energy management costs while ensuring a reliable and efficient energy supply to all participants in the microgrid. The flexibility and adaptability of the proposed method mean it can also be tailored to address additional objectives and constraints, such as reducing greenhouse gas emissions, enhancing energy efficiency, or integrating advanced energy storage solutions and their impact on electrical networks.

The findings from this study underscore the potential of the proposed methodology as a robust solution for energy management in microgrids, potentially fostering the development of more sustainable energy systems. Furthermore, recent advancements have expanded the capabilities of the SDDP algorithm to solve multi-stage mixed integer problems, thereby enhancing its suitability for managing reconfigurable microgrids.

Looking forward, the application of SDDP in scenarios that consider non-linear convex load flow models, along with associated electrical constraints, presents a promising avenue for further research. Moreover, optimal energy management in unbalanced three-phase microgrids is an area that remains largely unexplored, offering significant opportunities for future investigations, particularly as current studies have predominantly focused on two-stage models. This burgeoning field of research holds considerable promise for advancing the efficiency and sustainability of microgrid operations.