Deep Reinforcement Learning-Based Distribution Network Planning Method Considering Renewable Energy

Abstract

Distribution networks are an indispensable component of modern economic societies. Against the background of building new power systems, the rapid growth of distributed renewable energy sources, such as photovoltaic and wind power, has introduced many challenges for distribution network planning (DNP), including different source-load compositions, complex network topologies, and varied application scenarios. Traditional heuristic algorithms are limited in scalability and struggle to address the increasingly complex optimization problems of DNP. The emergence of new artificial intelligence provides a new way to solve this problem. Based on the above discussion, this paper proposes a DNP method based on deep reinforcement learning (DRL). By defining state space and action space, a Markov decision process model tailored for DNP is formulated. Then, a multi-objective optimization function and a corresponding reward function including construction costs, voltage deviation, renewable energy penetration, and electricity purchase costs are designed to guide the generation of network topology schemes. Based on the proximal policy optimization algorithm, an actor-critic-based autonomous generation and adaptive adjustment model for DNP is constructed. Finally, the representative test case is selected to verify the effectiveness of the proposed method, which indicates that the proposed method can improve the efficiency of DNP and promote the digital transformation of DNP.

1. Introduction

As an important component of modern power systems, distribution networks play a crucial role in promoting economic development, improving livelihoods, and facilitating energy transitions [1,2,3]. With the development of distributed renewable energy sources (such as photovoltaics (PVs) [4] and wind turbine generators (WTGs) [5]) and the emergence of new load types (such as electric vehicle), distribution systems are characterized by diverse source-load configurations and a complex grid structure. In addition, the variability of solar irradiance and wind speed in different regions brings uncertainties, which present new challenges for the task of distribution network planning (DNP). However, traditional planning methods primarily rely on human experience, which are influenced by planners’ expertise. These methods have the drawbacks of low efficiency and suboptimal quality. Therefore, how to choose appropriate methods to achieve efficient DNP schemes generation is a worthy issue.

As an effective tool, heuristic algorithms [6,7,8] (such as the simulated annealing algorithm (SAA), genetic algorithm (GA), ant colony optimization (ACO), and particle swarm optimization (PSO)) are widely used in the field of distribution networks. By integrating the steepest descent method with the SAA, ref. [9] proposed a DNP approach in which the capital recovery, energy loss, and undelivered energy costs were considered simultaneously. By combining a GA with the k-nearest neighbors structure, a hybrid GA-based radial distribution network reconfiguration method was displayed in [10] which reduced the search space while enhancing planning performance. Ref. [11] employed fuzzy logic systems to process the objective function and further utilized ACO to determine the optimal solution, which solved the optimal configuration of switching devices in the distribution network. A two-stage heuristic structure was constructed in [12] to choose the switches, and the reconfiguration of radial distribution networks was solved. By employing heuristic algorithms to determine the service areas of substations, ref. [13] addressed the substation planning problem for large-scale distribution networks. Additionally, the unreliability of feeders and substations was studied simultaneously. Ref. [14] proposed a new hybrid optimization approach combining PSO and Tabu Search to deal with the expansion planning of large-scale electric distribution networks. Ref. [15] solved the multi-objective reconfiguration of distribution networks with distributed power sources, in which an improved binary PSO algorithm was designed to enhance global search ability and accelerate convergence. By combining binary PPO for network investment decisions and inner optimization for flexibility procurement, cost-effective and adaptable distribution network expansion planning was achieved in [16]. A GA and PSO were combined in [17] to determine the best locations and sizes of distributed generation units, which reduced power losses and enhance voltage stability while respecting safe constraints. Based on PSO, an enhanced fault-recovery reconfiguration method for DC distribution networks was proposed in [18] which ensured that power can be quickly restored after a fault occurs. However, heuristic algorithms have limited solving capabilities, and the solutions generated by heuristic algorithms exhibit low generalization, which makes it difficult to address the increasingly complex optimization problems of DNP. Nowadays, artificial intelligence technologies have been applied in the entire spectrum of power systems, including generation, transmission, substations, distribution, and consumption. Moreover, many complex tasks, such as load forecasting [19], voltage stability control [20], and emergency control [21], have been implemented by utilizing artificial intelligence technologies.

In recent years, the artificial intelligence technologies represented by deep reinforcement learning (DRL) have experienced rapid development [22]. By combining deep learning and reinforcement learning, DRL can effectively deal with complex decision problems. The main idea of DRL is that agents continuously interact with their environment to obtain optimal policies. Recently, DRL has been extensively applied in the field of power systems, such as voltage control [23], frequency control [24], and load shedding control [25]. In [23], uncertainties in operating conditions and system physical parameters were considered, and a DRL-based voltage control approach was proposed for maintaining voltage within a safe range. Based on multi-agent DRL, a data-driven frequency control method for multi-area power systems was proposed in [24] which effectively reduced the frequency fluctuations caused by load variations and renewable energy sources. By designing a novel voltage-based reward function, the deep deterministic policy gradient (DDPG) algorithm was utilized to achieve load shedding control in [25], improving the ability of anti-disturbance. Refs. [26,27,28] utilized soft actor-critic to solve issues in electric vehicle charging, dynamic network reconfiguration, and volt-VAR control, respectively. By utilizing the DDPG to distribute torque between two motors to enhance energy efficiency, a hierarchical energy management strategy structure was constructed in [29]. In [30], a multi-agent DRL approach was developed which combined discrete DQN and continuous DDPG on different time scales to optimize long-term voltage deviations. In [31], risk sensitivity and the uncertainties of renewable generation were integrated in TRPO, and a systematic way for multiple microgrids to cooperate was proposed. Proximal policy optimization (PPO) [32] was proposed by OpenAI in 2017, which is a reinforcement learning method based on the actor-critic (AC) architecture. By integrating trust region policy optimization with the advantage AC, the PPO algorithm prevents performance degradation caused by excessively large policy updates. In [33], a two-level planning model based on PPO was developed, and a grid structure optimization algorithm based on electrical coupling degree was proposed which reduced the search space and alleviated the difficulty of acquiring splitting sections. In [34], a multi-agent PPO was introduced to construct the multi-layer energy management system to deal with challenges in mechanism modeling and supply-demand uncertainty. PPO was introduced to learn the optimal policy for flexible battery energy storage systems in [35], in which the working time and the balancing performance were optimized simultaneously. Ref. [36] presented a PPO-based decentralized framework for the voltage control of microgrids. Ref. [37] took cyber contingencies into consideration, designing a graph-based DRL framework in which the graph information was embedded into the PPO algorithm to alleviate the impact of cyber contingencies. Stackelberg–Nash PPO was constructed in [38] to deal with the influence of neglecting consumers’ interests and temporal relationships between participant actions. Based on the above discussion, DRL has been applied in multiple fields such as voltage control, reactive power dispatch, load shaping, demand response, and energy storage optimization. However, there are still some issues that need to be resolved; for example, few DNP studies consider the topology structure of distribution networks, while some DRL frameworks only focus on one main objective and neglect other factors. Thus, we propose a DRL-based DNP method considering renewable energy in this paper. The contribution of this paper is as follows:

(1)

By designing the action space and state space, we construct a Markov decision process (MDP) for DNP. Through this transformation, the problem of DNP can thus be integrated into the DRL framework.

(2)

Considering line construction costs, voltage deviation, renewable energy subsidies, and electricity purchasing costs, a multi-objective optimization function is designed. Additionally, a multi-reward function is developed to guide the agent to learn the optimal policy. After sufficient training, the agent can generate optimized planning schemes.

(3)

Based on the AC architecture, a PPO-based DNP algorithm (PPODNPA) is designed. The actor network (AN) generates planning schemes upon receiving reward signals and environmental states, while the critic network (CN) further evaluates these schemes to optimize the AN, which achieves the autonomous generation and adaptive tuning of the planning scheme. Simulation results validate the superiority of the proposed algorithm.

The remaining part of the paper is as follows: In Section 2, the MDP for DNP and planning model is defined. In Section 3, the PPODNPA is designed, which achieves the autonomous generation and adaptive tuning of the DNP scheme. In Section 4, the simulation results are displayed and heuristic algorithms are introduced to demonstrate the superiority of PPODNPA. A flow chart of this paper is displayed in Figure 1.

2. MDP for DNP and Planning Model

2.1. MDP for DNP

In general, MDP can be represented by a quintuple $(S,A,P,R,\gamma )$ [39], where S denotes the set of states, A denotes the set of actions, P represents the state transition probabilities, and R is the reward. The diagram of the MDP is illustrated in Figure 2, where $S_i$ corresponds to the ith state, $P_{ij}^k$ denotes the probability of transitioning from state $s_i$ to state $s_j$ after taking action $a_k$, and $R_i^k$ indicates the immediate reward generated by taking action $a_k$ in state $s_i$.

Considering the characteristics of DNP tasks, we assign physical meanings to various parameters of the MDP model. First, we define the state space as

$$S=\left\{\begin{array}{c}{P}^{WTG},Q^{WTG},P^{PV},Q^{PV},\hfill \\ L_m,U_i,I_{ij},D^{WTG},D^{PV},S_{ij}\hfill \end{array}\right\}$$

(1)

where $P^{WTG}$ and $Q^{WTG}$ represent the active power and reactive power of WTG, $P^{PV}$ and $Q^{PV}$ represent the active power and reactive power of PV, $L_m$ represents the construction status of the lines, $U_i$ represents the node voltage, $I_{ij}$ represents the line current, $D^{WTG}$ and $D^{PV}$ represent the installation status of PV and WTG, and $S_{ij}$ represents the result of load flow calculation. Then, we define the action space:

$$A=\left\{a_l,a_{WTG},a_{PV}\right\}$$

(2)

where $a_l$ represents the construction of lines, $a_{WTG}$ represents the construction of WTG, and $a_{PV}$ represents the construction of PV.

Remark 1.

It should be noted that load fluctuations, uncertainties in solar irradiance, and regional variations in wind power are factors that need to be considered. In this study, we use normal distribution to simulate uncertainties in PV output and load, and use Weibull distribution to simulate uncertainties in WTG output.

After defining the state space and action space, we establish the transition of the distribution network framework: By applying an action to the initial network, both its topology and the installation status of renewable energy sources will be changed. Due to the inherent uncertainties associated with renewable energy, state variables such as power, line current, and node voltage will be converted into other states with certain probabilities. This definition will be utilized in the subsequent algorithm design. In the next section, the planning model will be further proposed, which includes the objective function and some necessary constraints.

2.2. Planning Model

The objective function is the economic cost of DNP considering renewable energy, which includes investment cost, electricity purchase cost, and renewable energy subsidies. It is defined as follows:

$$min:C_{\mathrm{total}}=N_{\mathrm{cost}}\frac{k{(1+k)}^y}{{(1+k)}^y-1}+{\beta }_1{E}_{\mathrm{buy}}-{\beta }_2{E}_{\mathrm{sub}}$$

(3)

where $N_{\mathrm{cost}}$ represents the initial investment cost of the grid structure, k is the discount rate, y is the service life, the overhead line is taken as 30 years, the cable is taken as 40 years, and the factor ${\displaystyle \frac{k{(1+k)}^y}{{(1+k)}^y-1}}$ can convert the initial investment into the equivalent year cost. ${\beta }_1=0.394$ CNY/kW · h represents the purchase price for purchasing electricity from the external grid, $E_{\mathrm{buy}}$ is the amount of electricity purchased from the external grid to satisfy local demand. ${\beta }_2=0.2$ CNY/kW · h represents the subsidy unit price of renewable energy, and $E_{\mathrm{sub}}$ represents the amount of renewable energy generation.

To ensure safe, reliable, and efficient operation of the DNP, the following constraints are incorporated:

(1)

Power balance constraints [40]

$$\begin{array}{cc}\hfill & U_{i^{\prime },t}^{\mathrm{ac}}=U_{i,t}^{\mathrm{ac}}-\left(r_{i{i}^{\prime }}^{\mathrm{ac}}{H}_{i{i}^{\prime },t}^{\mathrm{ac}}+x_{i{i}^{\prime }}^{\mathrm{ac}}{G}_{i{i}^{\prime },t}^{\mathrm{ac}}\right)/U_0^{\mathrm{ac}},\forall i,i^{\prime },t:{\beta }_{i{i}^{\prime },t}^{\mathrm{ac}}\hfill \\ \hfill & P_{i,t}^{\mathrm{de}}-L_{i,t}^{\mathrm{ac}}+\sum _{j_1\in \Omega \left(i_1\right)}\sum _{l\in \mathcal{L}}\left(P_{i_1{j}_1,l,t}^{\mathrm{ac}}-P_{i_1{j}_1,l,t}^{\mathrm{acdc}}\right)=\sum _{i^{\prime }\in \pi \left(i\right)}{H}_{i{i}^{\prime },t}^{\mathrm{ac}}\hfill \\ & -\sum _{i^{″}\in \delta \left(i\right)}{H}_{i^{″}i,t}^{\mathrm{ac}}-\left(P_{i,t}^{\mathrm{w}}-{\rho }_{i,t}^{\mathrm{w}}\right),\forall i,t:{\beta }_{i,t}^{\mathrm{ac}2}\hfill \\ \hfill & Q_{i,t}^{\mathrm{de}}-Q_{i,t}^{\mathrm{ac}}=\sum _{i^{\prime }\in \pi \left(i\right)}{G}_{i{i}^{\prime },t}^{\mathrm{ac}}-\sum _{i^{″}\in \delta \left(i\right)}{G}_{i^{″}i,t}^{\mathrm{ac}},\forall i,t:{\beta }_{i,t}^{\mathrm{ac}3}\hfill \end{array}$$

(4)

where $U_{i,t}^{\mathrm{ac}}$ is the voltage magnitude, $r_{i{i}^{\prime }}^{\mathrm{ac}}$ and $x_{i{i}^{\prime }}^{\mathrm{ac}}$ are the resistance and reactance, respectively. $H_{i{i}^{\prime },t}^{\mathrm{ac}}$ and $G_{i{i}^{\prime },t}^{\mathrm{ac}}$ are the active and reactive power flow, respectively. $P_{i,t}^{\mathrm{de}}$ and $L_{i,t}^{\mathrm{ac}}$ are the output of the distributed generation and active load, respectively. $P_{i,t}^{\mathrm{w}}$ and ${\rho }_{i,t}^{\mathrm{w}}$ are the predicted wind power and curtailed wind power, respectively. $Q_{i,t}^{\mathrm{de}}$ and $Q_{i,t}^{\mathrm{ac}}$ are the reactive power of diesel engines and the reactive load, respectively. Equation (4) ensures that the power flowing into each node balances the load, generation, and line losses, which reflects Kirchhoff’s current and voltage laws.

(2)

The line current constraint and voltage constraint are calculated as follows:

$$\begin{array}{cc}\hfill I_{jmin}& \le I_j\le I_{jmax}\hfill \\ \hfill U_{imin}& \le U_i\le U_{imax}\hfill \end{array}$$

(5)

where $I_{jmax}$ and $I_{jmin}$ are the maximum and minimum values of the current, respectively, which reflect the thermal limits of the overhead line and cable. $U_{imax}$ and $U_{imin}$ are the maximum and minimum values of the voltage, respectively, which ensure power quality. Considering Equation (5) can prevent the overheating of lines and maintain the node voltage within safe ranges.

(3)

The active power constraint is calculated as follows:

$$\left|P_i\right|\le P_{imax}$$

(6)

where $P_i$ is the active power of the line and $P_{imax}$ is the maximum active power. This constraint ensures that no line is influenced by real-power flows beyond its rated capability.

(4)

Line flow constraints

$$S_{ij}\le S_{ij}^{max}$$

(7)

where $S_{ij}$ is the line flow, and $S_{ij}^{max}$ is the maximum value of the line flow. Considering Equation (7) can limit the power flow to safe levels considering both real power and reactive power.

(5)

Radial grid constraint

$$n=m+1$$

(8)

where n is the number of nodes in the distribution network, and m is the sum of the original and future lines. This constraint ensures a radial topology of grid structure, which is a common structural requirement in DNP.

The investment cost, electricity purchase cost from external sources, and the subsidy income generated by renewable energy generation are comprehensively considered in the planning model. The objective is to minimize the total cost under the power balance constraints, line current and voltage constraints, active power constraints, line flow constraints, and radial grid constraints. By obtaining an equilibrium solution between economic viability and renewable energy utilization, this planning model provides a systematic framework for DNP, and the resulting solution can give a more sustainable, cost-effective, and reliable DNP scheme.

3. The Design of the PPODNPA

3.1. PPO Algorithm

PPO algorithm is an actor-critic method which includes CN and AN. As illustrated in Figure 3, AN is responsible for generating action probability distributions, and CN provides evaluated reward values. Consequently, the AN does not need to compute cumulative rewards for each episode. Instead, it learns from the reward evaluations supplied by the CN, which enables single-step updates and enhances training efficiency. The CN fits more accurate value functions through temporal difference (TD) errors to help the AN in selecting actions.

The basic principle of the PPO algorithm is to use policy gradient for updating, and the update objective and rule of policy gradient are displayed as follows:

$$L^{PG}\left(\theta \right)={\widehat{E}}_t\left[\log {\pi }_{\theta }\left(a_t\mid s_t\right){\widehat{A}}_t\right]$$

(9)

$$\widehat{g}={\widehat{E}}_t\left[{\nabla }_{\theta }\log {\pi }_{\theta }\left(a_t\mid s_t\right){\widehat{A}}_t\right]$$

(10)

where ${\pi }_{\theta }$ is a stochastic policy and ${\widehat{A}}_t$ is the estimated value of advantage function. However, the policy gradient algorithm relies on the update step size. Therefore, the trust domain method (TRPO) measures the distribution difference between new and old policies through KL divergence and constrains it. Then, the objective optimization function is modified to

$${\mathrm{maximize}}_{\theta }{\widehat{E}}_t\left[{\displaystyle \frac{{\pi }_{\theta }\left(a_t\mid s_t\right)}{{\pi }_{\theta old}\left(a_t\mid s_t\right)}}{\widehat{A}}_t\right]$$

(11)

$$\widehat{E}\left[KL\left[{\pi }_{\theta \mathrm{old}}\left(a_t\mid s_t\right),{\pi }_{\theta }\left(·\mid s_t\right)\right]\right]\le \delta$$

(12)

where ${\theta }_{\mathrm{old}}$ is the parameter of old policy. Directly solving the TRPO constraint problem will bring significant computational complexity. In order to improve computational efficiency, PPO has further simplified the TRPO objective function. By defining the probability ratio of the distribution of new and old strategies $r_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left(a_t\mid s_t\right)}{{\pi }_{\theta \mathrm{old}}\left(a_t\mid s_t\right)}}$, the object function can be written as

$$L\left(\theta \right)={\widehat{E}}_t\left[{\displaystyle \frac{{\pi }_{\theta }\left(a_t\mid s_t\right)}{{\pi }_{\theta \mathrm{old}}\left(a_t\mid s_t\right)}}{\widehat{A}}_t\right]={\widehat{E}}_t\left[r_t\left(\theta \right){\widehat{A}}_t\right].$$

(13)

In addition, PPO prunes the probability distribution ratio to limit its policy updates:

$$L^{CLIP}\left(\theta \right)={\widehat{E}}_t\left[min\left(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}\left(r_t\left(\theta \right),1-\epsilon ,1+\epsilon \right){\widehat{A}}_t\right)\right].$$

(14)

For a complete turn trajectory $T=\left\{s_0,a_0,r_0;s_1,a_1,r_1;s_2,\dots ,a_{n-1},r_{n-1},s_n\right\}$ and strategy network ${\pi }_{\theta }\left(a_t\mid s_t\right)$, use state action function $Q^{\pi }\left(s_t,a_t\right)$ as the evaluation value of state action pairs, and use state value function $V^{\pi }\left(s_t\right)$ as the evaluation baseline; then, the following advantage function is defined

$$A^{\pi }\left(s_t,a_t\right)=r+\gamma V^{\pi }\left(s_{t+1}\right)-V^{\pi }\left(s_t\right).$$

(15)

The loss function of AN is

$$L\left(\theta \right)=-E\left[min\left(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}\left(r_t\left(\theta \right),1-\epsilon ,1+\epsilon \right){\widehat{A}}_t\right)\right]$$

(16)

And the loss function of CN is

$$L\left(\omega \right)=-E\left({\left[r_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right)\right]}^2\right).$$

(17)

In order to prevent overfitting caused by the strategy network being too fast, which leads to premature convergence of the agent and inability to effectively explore environmental information, regularization entropy is added as a penalty term in the final loss function. Therefore, the loss function of the AC PPO algorithm framework can be expressed as follows:

$$\mathrm{Loss}=aL\left(\theta \right)+bL\left(\omega \right)-c\mathrm{ENT}\left({\pi }_{\theta }\right)$$

(18)

where $a=1,b=0.5,c=0.01$ are the loss proportion coefficients.

Overall, PPO has the following advantages: By modifying the objective function to penalize large deviations between new and old policies, the trust region of the policy update can be ensured, which can achieve more stable training. Unlike TRPO, the second-order derivatives and large matrix inversions are not needed, which is relatively easy to implement. The hyperparameters of PPO are relatively robust, which makes PPO perform outstandingly in various complex tasks.

3.2. PPODNPA

By integrating the MDP of DNP into the AC-based PPO algorithmic framework, we propose the PPODNPA algorithm, which is illustrated in Figure 4. In this approach, the DNP environment is first initialized with a set of states. Based on these initial states, the AN generates a series of actions, such as building lines and installing renewable energy sources in the power grid. Influenced by these actions, the environment states transition to a next set of states according to a probabilistic distribution and an immediate reward is produced. The CN receives the reward and states, evaluates the quality of the actions, and then provides the action advantage function back to AN, thereby further training the AN. After sufficient training, the actions generated by the agent (which dictate the configuration of the grid) are optimized to meet the specified planning objectives. The reward function in the PPODNPA is designed as

$$R_{\mathrm{total}}=-{\alpha }_1{C}_{\mathrm{line}}-{\alpha }_2{C}_{\mathrm{vol}}-{\alpha }_3{C}_{\mathrm{buy}}+{\alpha }_4{C}_{\mathrm{sub}}+R_5$$

(19)

where the parameters ${\alpha }_1,{\alpha }_2,{\alpha }_3,{\alpha }_4$ serve as the respective weighting factors. These factors can be adaptively adjusted to accommodate various practical requirements. Notably, the parameter $R_5$ plays a critical role: if no renewable energy resources are installed in the resulting grid configuration, $R_5$ will impose a penalty. This mechanism ensures a certain level of renewable energy penetration in the final plan.

In the PPODNPA, the distribution network planning environment (DNPE) represents the distribution network prior to the construction of lines and the integration of renewable energy sources. When agent actions are applied to the DNPE, it performs power flow calculations to calculate the node voltages, line currents, and power flows. In addition, the DNPE also calculates line construction costs, distributed renewable energy subsidies, and the costs of purchasing electricity from the up-level grid. These calculated parameters are then transmitted back to the CN, which can guide the AN updates. The procedure of the PPODNPA is displayed in Algorithm 1.

Algorithm 1 PPODNPA Algorithm Procedure

1: Algorithm Output: Actor policy network ${\pi }_{\theta }$ which can output optimized actions of line construction and renewable energy installation, Critic value network $Q_{\omega }$ which can approximate the state value function $V^{\pi }\left(s_t\right)$.   2: Randomly initialize the policy network parameters and the value network parameters.   3: for episode = 1, 2, … do   4:      (a) Data Collection:   5:      Use the old policy ${\pi }_{{\theta }_{\mathrm{old}}}$ to interact with the DNP environment.   6:      In each time step t, sample action $a_t\sim {\pi }_{{\theta }_{\mathrm{old}}}(a\mid s_t)$, update the grid/renewable configuration, run power flow calculation, and obtain next state $s_{t+1}$ and immediate reward $r_t$.   7:      Continue until reaching the episode end or a terminal condition.   8:      (b) Advantage Computation:   9:      For each transition $(s_t,a_t,r_t,s_{t+1})$, compute the TD advantage function $A_t$ using $$A_t=r_t+\gamma V^{\pi }\left(s_{t+1}\right)-V^{\pi }\left(s_t\right),$$ 10:    (c) PPO Loss Calculation: 11:    According to Equation (18), compute the loss function. 12:    (d) Update Actor: 13:    Perform gradient descent on the actor parameters $\theta$. 14:    (e) Update Critic: 15:    Perform gradient descent on the critic parameters $\omega$. 16:    After the updates, set ${\theta }_{\mathrm{old}}\leftarrow \theta$ {Update old policy} 17: end for

In this paper, the proposed PPO-based distribution network planning algorithm (PPODNPA) includes the following important processes: environment simulation, action selection, power flow computation and reward evaluation, and PPO strategy updating. First, the environment simulation builds and updates the system state space (which includes node voltages, line currents, and network topology) in response to different agent actions (which include constructing new lines and installing renewable energy). Second, the AN gives a probability distribution of possible actions. By sampling the probability distribution, the local optima phenomenon can be avoided. Third, the power flow computations and reward evaluations can measure system constraints and evaluate economic factors, and generate reward signals for AN and CN. Finally, the PPO strategy updates via advantage functions and the clipping mechanism, and the AN and CN are iteratively refined through backpropagation. The main computational burdens emerge from repeated power flow analyses, and parallel or distributed computation can help maintain computational overhead. Repeated episodes allow the PPODNPA to converge to the near-optimal solutions which balance investment cost, purchase price for purchasing electricity from the external grid, and renewable utilization subsidies within acceptable training times. Overall, environment simulation, neural network training, iterative power flow, and multi-round updates collectively illustrate the execution flow and resource requirements scale of the PPODNPA and provide guidance for actual applications.

4. Simulation Results

In this section, the simulation results will be presented. A typical DNP case is selected to verify the effectiveness of the PPODNPA, and the relevant parameters are listed in

Table 1. The parameters of the DNP case.

Node Number	Node Coordinates/km	Active Load/kW	Reactive Load/kvar
1	(1.976, 1.090)	0	0
2	(1.056, 1.026)	439.92	273.78
3	(0.480, 1.304)	421.20	262.08
4	(1.928, 1.798)	318.24	196.56
5	(0.196, 1.076)	430.56	266.76
6	(3.640, 0.474)	374.40	231.66
7	(0.524, 0.914)	402.48	250.38
8	(2.876, 1.808)	383.76	238.68
9	(0.184, 1.602)	570.96	353.34
10	(1.008, 1.586)	589.68	365.04
11	(0.664, 1.822)	421.20	262.08
12	(3.360, 0.904)	477.36	231.66
13	(0.548, 0.430)	580.32	360.36
14	(0.916, 0.182)	374.40	231.66
15	(3.424, 1.192)	458.64	285.48
16	(2.856, 0.182)	336.96	208.26
17	(2.488, 0.272)	402.48	250.38
18	(3.272, 1.738)	439.92	273.78
19	(2.876, 1.560)	402.48	250.38
20	(3.112, 1.394)	683.28	423.54
21	(2.348, 0.112)	449.28	278.46
22	(2.128, 0.334)	486.72	301.86
23	(3.300, 0.474)	318.24	196.56
24	(3.440, 1.490)	393.12	243.36
25	(2.304, 1.556)	276.12	170.82
26	(1.172, 0.354)	159.12	98.28
27	(2.388, 0.506)	313.56	194.22
28	(2.944, 1.196)	480.18	402.32
29	(3.616, 0.718)	159.12	98.28

. The first column of Table 1 displays the node number, the second column shows each node’s coordinates (in kilometers), the third and fourth columns show the active and reactive power of the nodes, respectively (in kW and KVAR). The hidden layer size of the AN and CN was selected as 512. Larger hidden layers can help the neural network learn the complex relationships between state and action. However, larger hidden layers may also lead to higher computational costs and increase the risk of overfitting with limited training data. The learning rate of AN is selected as 0.001 and the learning rate of CN is selected as 0.001. A higher learning rate may cause the policy to oscillate, and a lower rate can slow convergence. Therefore, we select the learning rates for both AN and CN as 0.001 to achieve a balance between convergence speed and stability. The discount factor is chosen as 0.9. The discount factor can balance the short-term and long-term rewards; selecting the discount factor as 0.9 can ensure that cumulative returns are sufficiently reflected, even in relatively shorter training sequences. The clipping parameter is set to 0.1. The clipping parameter is a common hyperparameter in the PPO algorithm that restricts the policy’s deviation from the old policy. Selecting the clipping parameter as 0.9 can maintain enough policy improvement while avoiding overly large policy updates. After collecting each trajectory, 20 optimization steps are performed. The number of total episodes is 20,000, and each episode includes 10 steps, which ensures that the information gathered from each sample is effectively leveraged. The DNP lines are selected as LGJ-185 and the construction cost is 1 million Chinese Yuan (CNY)/kilometer. The PV installation capacities are 600 kW, 900 kW, 400 kW, and 700 kW, while the WTG are 500 kW, 600 kW, and 400 kW. Moreover, we select PSO [41], GA [42], ACO [43], and SAA [44] for comparison, which demonstrates the superiority of the proposed algorithm.

The simulation results are presented in Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12. Figure 5 illustrates the training process curve of the PPODNPA, which includes the instantaneous rewards for each episode and the average rewards (calculated every 200 episodes). From Figure 5, we can conclude that after sufficient training, the agent’s rewards approach convergence. Figure 6 depicts the grid topology structure generated under the PPODNPA, in which the red nodes represent the node connected to the main grid, the blue nodes are load nodes, the yellow nodes are PV nodes, and the green nodes are WTG nodes. Additionally, we select PSO, GA, ACO, and SAA for comparison, and the line construction costs (LLC), renewable energy subsidies (RES), and electricity purchasing costs (EPC) from the upper-level grid as comparative indicators. The grid topology structures generated under PSO, GA, ACO, and SAA are shown in Figure 7, Figure 8, Figure 9 and Figure 10. In addition, we also use DQN as the comparative algorithm; Figure 11 illustrates the training process curve of DQN and Figure 12 depicts the grid topology structure generated under DQN. The relevant indicators are shown in

Table 2. Comparative indicators.

Method	LCC/million CNY	VD	EPC/CNY	RES/CNY
PPODNPA	10.87	2.0744	78,857	16,320
PSO	11.37	2.3352	83,298	14,880
GA	11.83	0.5897	85,184	14,400
ACO	11.79	0.5105	77,620	15,840
SAA	12.05	0.3003	81,699	13,920
DQN	11.59	2.6364	96,241	9120

. As shown in Table 2, the PPODNPA effectively reduces the costs of construction and electricity purchasing while achieving higher subsidies for renewable energy sources. It should be noted that, as the number of loads and power sources on a single branch increases, higher currents can lead to more pronounced voltage drops along that branch. This problem will be one of our future research directions.

5. Conclusions

In this paper, a PPODNPA has been proposed and the MDP for the task of DNP has been constructed. The objective function and reward function have been formulated, and the PPO algorithm has been employed to train the agent. Furthermore, PSO, GA, ASO, and SAA algorithms have been utilized to validate the effectiveness of the proposed method. In future work, we will investigate the application of DRL in high-voltage DNP, and more factors such as voltage deviation and station location corridors will be considered. Moreover, we will consider further improving DRL algorithms such as SAC and DDPG to further optimize planning performance.