1. Introduction
As global energy demand rises and climate change poses an increasing threat, the development of sustainable, low-carbon energy solutions has become imperative. According to the International Energy Agency (IEA) [
1], global energy demand is expected to nearly double in the next two decades, while the latest assessment report of the United Nations Framework Convention on Climate Change (UNFCCC) highlights the need for significant reductions in greenhouse gas emissions in order to limit the rise in global temperature to more than 1.5 °C [
2]. Direct carbon emissions refer to the greenhouse gases emitted by power generation companies during energy consumption and production processes, which can be directly controlled and managed. In contrast, indirect carbon emissions are those resulting from the energy (such as electricity, steam, heating, and cooling) purchased by companies. Although these emissions are not produced directly by the companies, they are intrinsically linked to their operations.
The power sector is the largest consumer of energy, making it crucial to control and reduce user-side indirect carbon emissions, particularly on the distribution network side. This not only helps to decrease the overall carbon footprint but also promotes the sustainable development of the energy supply chain. Low-carbon optimization refers to the systematic approach taking the objective of minimizing carbon emissions across various operational processes while maintaining or enhancing system performance. In the context of energy systems, this involves the optimization of energy production, distribution, and consumption to reduce both direct and indirect carbon emissions. Kang et al. [
3] introduced a method for calculating indirect carbon emissions in power systems using a carbon emission flow model. This approach refines the calculation of carbon emission intensity to smaller spatial and temporal granularities, clarifying the concept of indirect carbon emission flow. Building on this, utilizing locational marginal prices and dynamic carbon intensity signals has demonstrated economic and environmental benefits within distribution networks (DN) [
4]. Moreover, a low-carbon economic dispatch model enhances energy flexibility and decreases pressure in these networks [
5]. Additionally, a tiered carbon emission model, addressing uncertainties in wind, photovoltaic generation, and loads, has been developed to minimize emissions and costs within a carbon trading framework [
6]. Finally, a production simulation method that integrates diverse energy outputs with a carbon trading mechanism has also been proposed [
7], paving the way for advances in energy system optimization.
By integrating advanced smart grid technology, renewable energy, and energy storage systems, DN can respond more flexibly to complex and variable energy demands [
8,
9]. This integration can not only promote the widespread use of renewable energy but also enhance the system’s flexibility in responding to fluctuations in energy demand, thereby providing more reliable and efficient power services [
10]. Applying various scheduling strategies and a dynamic carbon emission trading system, the efficient maximization of renewable energy for power generation and the reduction of carbon emissions have been achieved [
11]. Mitigating indirect carbon emissions on the user side is crucial. However, research on user-side SES systems has not fully considered the application of dynamic carbon emission intensity. The importance of dynamic carbon emission intensity in storage operation strategies are not well addressed. Thus, this paper aims to explore the effective integration of dynamic carbon factors and carbon emission flow theory into user-side shared energy storage–distribution network systems, addressing the research gap in this critical area.
On the distribution network side, the application of energy storage systems, particularly within user-side shared energy storage–distribution grids, has proven effective in reducing user electricity costs and decreasing indirect carbon emissions. To enhance this, a proposal suggests utilizing energy storage efficiently and safely as a flexible grid asset by employing an energy management system (EMS) and optimization techniques to deliver various electric power services to users [
12,
13]. Furthermore, ref. [
14] introduces a double-layer optimal distribution method for the cooperative optimization of distributed shared energy storage within networks, along with a corresponding operational model. Simulation analysis of numerical examples verifies the effectiveness and economic viability of this proposed configuration method.
Unlike traditional models that dedicate energy storage to individual users, “User-Side Shared Energy Storage-Distribution Grids” offer a modern approach by pooling storage assets to serve multiple users within a distribution network. This system is managed by a centralized EMS, which optimizes energy distribution based on real-time demand, pricing, and carbon emission signals [
15,
16]. This setup not only enhances system flexibility but also significantly improves the efficiency of energy storage utilization. By sharing storage costs and leveraging load complementarity among users, these systems effectively reduce the overall carbon footprint [
17,
18]. Shared energy storage (SES) plays a crucial role by assessing complementary storage capacities and proposing coordinated operation strategies to efficiently serve customers [
19]. Additionally, using Nash bargaining theory, a dynamic zonation optimization strategy for centralized SES power stations has been developed, further enhancing storage utilization and benefiting various stakeholders [
20].
Several existing studies have demonstrated that SES provides considerable advantages in terms of environmental economic benefits and practical engineering value [
21,
22] have shown that SES not only increases the flexibility and efficiency of the system, but also is a key technology to support the development of low-carbon power grids. This forward-looking mechanism not only promotes the integration of energy storage resources, but also increases the flexibility and efficiency of the system, making SES a key component to support low-carbon power grids. The development and implementation of low-carbon optimization strategies has become a necessary measure to reduce environmental impact, and SES shows great potential to address climate change by reducing carbon emissions [
23]. A double-layer carbon-aware planning method for SES stations for multi-component integrated energy systems was proposed [
24]. The conditional value-at-risk method was adopted as a risk measurement to effectively reduce the carbon emissions and system operating costs of SES stations. A power system interval optimization model based on SES and refined demand response is proposed to effectively deal with the uncertainty of source load through interval optimization and enhance the utilization rate of energy storage as well as the overall system economy [
25].
However, traditional optimization algorithms struggle with high computational complexity, dynamic environments, and uncertainty in distribution network (DN) optimization problems [
26]. With advancements in artificial intelligence, deep reinforcement learning (DRL) has emerged as a powerful tool for optimizing the operations of SES and DNs [
27]. DRL’s ability to learn and adapt through continuous interaction with complex, dynamic environments makes it particularly well-suited for DN optimization. This study specifically employs the deep deterministic policy gradient (DDPG) algorithm, a sophisticated DRL technique, to revolutionize optimization and decision-making processes within these systems. DDPG excels in handling complex optimization problems, adapting to dynamic environments, managing uncertainties, and being model-independent, thus providing SES and DN operators with highly flexible and efficient decision support tools [
28,
29]. As a model-free reinforcement learning algorithm, DDPG can independently learn and make decisions across successive actions and states [
30]. Studies have shown that DRL agents outperform stochastic optimization algorithms in extensive action and observation spaces and can effectively manage uncertainty with high accuracy [
31].
By integrating Lyapunov optimization strategies with DDPG, the challenges of real-time operation in wind-storage integrated systems (WSISs) are effectively addressed, particularly under conditions of uncertainty and fluctuating electricity prices [
32]. Case studies demonstrate the effectiveness of DDPG in enhancing power system efficiency and reliability [
33]. Additionally, a reward function based on the long-term behavior of energy systems enables optimization of online scheduling by learning the operational modes of renewable energy generation and grid dynamics [
34]. This demonstrates that DRL not only enhances SES efficiency but also supports the low-carbon transformation of DNs, offering significant economic and environmental benefits.
Traditional power grids face challenges in achieving environmentally friendly, low-carbon operations while maintaining a stable power supply. [
35].Although previous studies have utilized various advanced deep reinforcement learning algorithms to optimize power system scheduling, few have simultaneously considered dynamic carbon emission intensity, photovoltaic generation uncertainty, and shared energy storage for low-carbon economic dispatch. To achieve the dual goals of economic efficiency and environmental sustainability in energy systems, this study aims to apply the DDPG algorithm with prioritized experience replay and orthogonal regularization to achieve low-carbon optimization of shared energy storage and DN under the conditions of uncertain carbon intensity and photovoltaic output, ensuring real-time adaptability to fluctuating carbon intensities and renewable energy outputs.
The main contributions of this study are as follows:
To address the uncertainties in carbon emission intensity of the main grid and the photovoltaic generation output within the DN, a low-carbon optimization model is proposed. This model incorporates tiered carbon trading and considers the dynamic carbon emission intensity, aiming to optimize the operation of SES operators in conjunction with the DN.
Based on the aforementioned model, the DDPG algorithm with prioritized experience replay and orthogonal regularization is employed, which improves the convergence and stability of the algorithm. This approach ensures real-time adaptability to fluctuating carbon intensities and renewable energy outputs.
2. Problem Description and Framework
The framework of this study is illustrated in
Figure 1. Within the DN, various distributed energy resources, energy storage systems, and photovoltaic (PV) installations are distributed. The battery energy storage systems (BESS) in user microgrids are managed by a SES operator, ensuring efficient utilization and management of energy resources. All users within the DN participate in the carbon emission rights trading market, leveraging market mechanisms to optimize and control carbon emissions, thus promoting low-carbon development goals.
Notably, the DN is connected to the main grid, which provides time-segmented carbon emission intensity data. Combined with the internal energy structure of the DN, the carbon emission intensity within the entire DN dynamically changes. This dynamic characteristic allows the DN to flexibly adjust its operational strategies, optimize carbon emissions, and maximize the use of low-carbon energy at different times, thereby achieving more efficient carbon emission management and energy utilization.
2.1. Distribution Network Modeling
The objective function of the economic low-carbon dispatch problem is demonstrated in (1), aiming to minimize the total cost. This objective function consists of four terms, including the cost of power purchased from the higher grid by distribution grid users
, the cost of gas turbine operation by microgrid users in the system
, the profit of the SES service provider
, and the cost of carbon trading for the system segmented carbon price
, as detailed in (14).
where
is the set of all power buses in the system;
is the set of consumer gas turbines in the system;
is the set of SES;
is the time-of-use tariffs;
is the nodal consumer loads;
is the generator outputs;
,
, and
are the cost coefficients;
is the charging power of SES;
is the discharging power of SES; and
is the cost of operation of SES, taking into account the lifetime of the batteries.
The constraints are established as follows.
where
is the power flowing through the branch;
is the branch resistance;
is the square of the current flowing through the branch;
is the inflow power from the higher grid;
is the active power output from the PV inverter;
is the reactive power load at the bus;
is the reactive power flowing through the branch;
is the reactive power inflow from the higher grid;
is the reactive power output from the PV inverter;
is the reactive power output from the user’s micro-gas turbine;
is the maximum capacity of the line;
is the square of the bus voltage;
and
are the minimum and maximum values of the bus voltage;
and
are the minimum and maximum values of the branch current;
and
are the minimum and maximum values of the charging power;
and
are the minimum and maximum values of the discharging power;
and
are the minimum and maximum values of the microgrid users’ gas turbine output.
Equations (2) and (3) denote the system’s active and reactive power balance equations, respectively, ensuring that the total generated active and reactive power equals the total consumed active and reactive power within the distribution network. This balance is crucial for maintaining system stability. Equation (4) denotes the line capacity constraint, ensuring that the power flow through each line does not exceed its maximum capacity. Equation (5) denotes the bus voltage balance, ensuring that the voltage at each bus remains within acceptable limits. Equation (6) represents the bus voltage magnitude constraint, ensuring that the voltage magnitude at each bus stays within specified bounds. Equation (7) denotes the branch current magnitude constraint, ensuring that the current in each branch does not exceed its maximum limit. Equation (8) denotes the energy storage charging constraint, which ensures that the charging power of energy storage systems remains within their capacity limits. Equation (9) denotes the energy storage discharge constraint, ensuring that the discharge power of energy storage systems does not exceed their capacity. Equation (10) denotes the user microgrid gas turbine output constraint, which limits the power output of gas turbines in microgrids. Equation (11) denotes the relationship between the capacity of the energy storage during charging and discharging processes, ensuring proper coordination between the two. Finally, Equation (12) denotes the capacity constraint of the BESS, ensuring that the energy storage operates within its designated capacity.
2.2. System Carbon Intensity Calculation
Dynamic carbon emission intensity varies based on the proportion of different energy sources utilized within the grid, fluctuations in load, and the generation capacity of renewable energy sources. Unlike static carbon factors, dynamic carbon emission intensity provides real-time reflections of the environmental impact of electricity supply, thereby offering users more accurate data on carbon emissions. The carbon intensity of the system is given by Equation (13):
where
is the real-time dynamic carbon intensity of the superior grid and
is the carbon intensity of the gas turbine of the user microgrid.
2.3. Ladder-Type Carbon Emission Right Trading
Simulating a real carbon emission market trading scenario is complex; hence, the ladder-type carbon trading approach, which is widely accepted [
36,
37], is used to mimic real-world carbon trading scenarios. This method follows the market principle that the scarcer the commodity, the higher the price, while maintaining a constant total quantity. The total carbon emissions
are calculated at the end of all scheduling periods of the day, and then the growth interval
d and the growth rate of the carbon trading cost
are determined based on the historical data. The higher the carbon emissions, the higher the corresponding carbon trading price. Equation (14) denotes the ladder-type carbon emission trading cost.
3. Markov Decision Process Framework of Dynamic Dispatch
In this study, deep reinforcement learning is utilized to solve the low-carbon economic dispatch problem of a distribution grid–SES system. First, the mathematical formulation of this low carbon economy scheduling problem is transformed into the MDP framework for reinforcement learning. The design of state space, action space, and reward function is included.
A Markov decision process (MDP) [
38] is a mathematical model used to describe decision-making in environments that involve randomness and decision-making. In reinforcement learning and decision theory, MDP provides a systematic framework that allows complex decision problems to be solved through mathematical and algorithmic methods. An MDP typically consists of the following four components: State Set
: A set representing all possible states the system can be in. At any given time, the system is in a specific state. Action Set (
): A set representing all possible actions. The agent (decision-maker) can choose an action
in each state
. State Transition Probability (
): This describes the probability of transitioning from one state to another. Specifically,
represents the probability of transitioning to state
after taking action
in state
. Reward Function (
): This describes the immediate reward received after taking a specific action in a specific state. It is usually represented as
, which is the reward received after taking action
in state
. The MDP framework in our study is designed to optimize the low-carbon economic dispatch of the SES–distribution grid system. The state space includes variables such as electrical load, energy prices, carbon emissions, and the state of charge of the SES. The action space comprises the control actions for the SES and microgrid generators. The reward function is designed to minimize the total operating cost and carbon emissions, incorporating penalties for violating constraints.
The agent obtains state observations
through interaction with the actual environment, and these observations provide key data for effective training of the agent based on the perceived state information. The study utilizes a 24 h time frame, during which the system observations collected include electrical load data at a specific time period (period t), energy prices, carbon emissions trading volume, tiered carbon prices, and the state of charge of the BESS system. For this problem, the state can be expressed as
The scheduling objective is to determine the optimal SES as well as PV and wind generation outage scenarios, and the set of actions is denoted as
,
. Actions in low-carbon scheduling can be expressed as follows:
where
denotes the action of each microgrid user’s gas turbine, and
denotes the action of each storage battery for SES.
is the continuous action space that satisfies the energy and capacity constraints that satisfy the action constraints.
The state transfer is , and in any time step, state and action for a given time step are transitioned to the next state via the distribution grid–SES.
The reward received by the intelligent agent during the time period
is given by the environment in order to guide the intelligent agent to update its strategy to minimize the carbon trading price and operation cost to achieve the goal of low-carbon economy operation by designing a suitable reward function. After the agent makes an action and interacts with the environment, the environment gives it a reward value.
The reward function contains five terms: the first four terms correspond to the objective function of the modeling approach in
Section 2 as Equation (1),
is added to the reward in the last step of each episode as the settlement of daily carbon emissions trading, and the last term is the penalty term for constraint Equations (10) and (12).
4. DDPG Algorithm for Online Dispatch Problem
This study uses the DDPG algorithm enhanced with prioritized experience replay (PER) and orthogonal regularization (OR) to solve the above MDP problem, and works to learn the optimal economic low-carbon green scheduling strategy. DDPG combines the advantages of deep learning (DL) and reinforcement learning (RL), and is particularly suitable for dealing with problems in continuous action space. The algorithm mainly consists of two parts: the strategy network and its corresponding goal network ; and the value network and its corresponding goal network .
The policy network generates the action
based on the current state of the environment
, while the value network evaluates the expected payoff of this action for taking a particular action under the current policy. Thus, the strategy network is trained to maximize the action evaluation given by the value network:
where
The gradient rise is utilized to increase
, as shown in Equation (19):
Thus, the algorithm for updating the policy network
is obtained by randomly drawing one state at a time from the replay buffer, denoted as
, computing
, and updating
once with gradient ascent, as shown in Equation (20).
The value network
learns to update the parameters
by temporal difference (TD) learning errors, and each time, a quaternion
from the replay buffer computes the predicted values
and
of the target value network, and then computes the TD target as shown in Equation (21).
The loss function is defined as shown in Equation (22):
The computed gradient is used to update the parameters
of the value network as shown in Equation (23):
The parameter updates for the strategy network are calculated as shown in Equation (24):
In order to improve the stability and convergence of training, the parameters of the two networks are updated using soft update, for the target strategy network and target value network, the parameters are updated as shown in Equations (25) and (26).
where
is the soft update rate, which reduces the impact of abrupt changes in network parameters on the training process by slowly adjusting the parameters of the target network to gradually converge to those of the actual network.
Prioritized experience replay (PER) [
39] enhances the efficiency of reinforcement learning by selectively sampling experiences that are more informative. Unlike uniform sampling in traditional experience replay, PER assigns higher sampling probabilities to experiences with larger temporal difference (TD) errors. This focus on high-priority experiences accelerates learning and stabilizes the training process. In our study, PER is integrated into the DDPG algorithm, which helps the model learn more effectively from the most significant experiences. The priority of each experience is given by Equation (27):
Orthogonal regularization (OR) [
40] is employed to improve the generalization and stability of neural networks by maintaining the orthogonality of weight matrices. This method helps prevent overfitting and enhances the robustness of the learned policies. By incorporating an orthogonal regularization term into the loss function, we encourage the weight matrices to remain orthogonal, thus improving the performance of deep learning models. In our implementation, OR is added to the DDPG algorithm, ensuring that the neural network weights are regularized throughout training. The orthogonal regularization term is defined as Equation (28):
where
is the weight matrix of the
-th layer,
is the identity matrix,
is the regularization coefficient, and
denotes the Frobenius norm. This addition helps stabilize the learning process and enhances the algorithm’s ability to handle uncertainties in low-carbon energy dispatch scenarios.
When the DDPG agent completes the offline training, only the trained policy network is used to make decisions, and the Agent can make online decisions through its own observation, which is based on the state in the observation area, and online decisions at the millisecond level are made through the already trained policy network. The overall block diagram of the algorithm based on DDPG for the joint low-carbon optimization of the user-side SES–distribution network is shown in
Figure 2.
6. Conclusions
This study is dedicated to exploring optimization strategies to reduce DN costs and carbon emissions. We address the system uncertainty caused by the high percentage of renewable energy access and the variability of the dynamic carbon factor in the main grid. By incorporating real-time dynamic carbon factors, our approach enables more accurate assessments of carbon emissions associated with electricity consumption, facilitating timely adjustments in energy dispatch and consumption patterns. To this end, we propose a joint SES operator–distribution grid low-carbon optimization model that considers ladder-type carbon trading and utilizes the DDPG algorithm enhanced with PER and OR to achieve the dual goals of economic efficiency and environmental sustainability.
We first propose an optimization framework that considers SES and all-day carbon trading costs to minimize operating costs and carbon emissions. Second, by formulating the scheduling problem as a MDP, the observed states, scheduling actions and reward functions of the system are explicitly defined. Finally, the DDPG algorithm is used for low-carbon economy scheduling. The simulation results of this study validate the proposed method’s effectiveness in reducing both the operating cost of the power system and carbon emissions.
This study provides an innovative perspective to synergistically optimize SES with the distribution grid, and also offers a practical methodology for achieving low-carbon economic dispatch of the power system. Through a series of simulation experiments, the study confirms the significant effectiveness of the proposed methodology in reducing power system operating costs and reducing carbon emissions, providing an innovative perspective and a practical methodology for the low-carbon optimization of SES with distribution grids.
Future research directions include investigating the scalability of the proposed low-carbon optimization model in larger, more complex DNs and its deployment in real-world scenarios to provide valuable insights into its practical applications and limitations. Additionally, exploring multi-agent systems for the coordination and optimization of multiple shared energy storage operators and distributed energy resources could enhance the overall efficiency and robustness of the grid. Furthermore, studying the impact of consumer behavior and demand response on the effectiveness of low-carbon optimization models could lead to more accurate and user-centric strategies, including analyzing how different incentives and pricing strategies affect consumer participation in demand response programs.