A Frequency–Pressure Cooperative Control Strategy of Multi-Microgrid with an Electric–Gas System Based on MADDPG

Abstract

With the development of micro gas turbines (MT) and power-to-gas (P2G) technology, the electric–gas system plays an important role in maintaining the stable, economical, and flexible operation of the microgrid. When subjected to power load disturbance and natural gas load disturbance, the system controller needs to coordinately control the frequency of the microgrid and the gas pressure at the natural gas pipeline nodes. Additionally, the reliability and stability of a multi-microgrid system are much higher than that of a single microgrid, but its control technology is more complicated. Thus, a frequency–pressure cooperative control strategy of a multi-microgrid oriented to an electric–gas system is proposed in this paper. Firstly, based on the analysis of the operating characteristics of the natural gas network and the coupling equipment, the dynamic model of natural gas transmission is built. Secondly, a multi-microgrid load frequency control model including MT, P2G equipment, electric vehicles (EVs), distributed power sources and loads has been established. In addition, according to the three control objectives of microgrid frequency, node pressure and system coordination and stability, the structure of a Muti-Agent Deep Deterministic Policy Gradient (MADDPG) controller is designed, then the definition of space and reward functions are completed. Finally, different cases are set up in the multi-microgrid, and the simulation results are compared with PI control and fuzzy control. The simulation results show that, the proposed MADDPG controller can greatly suppress the frequency deviation caused by wind power and load disturbances and the air pressure fluctuations caused by natural gas network load fluctuations. Additionally, it can coordinate well the overall stability between the sub-microgrids of multi-microgrid.

1. Introduction

With the rapid economic development, energy shortages and environmental pollution problems have gradually become prominent [1,2]. China and most countries in the world take the construction of a green, low-carbon and clean energy Internet as a strategic goal and vigorously support the development and access of renewable energy [3]. The energy Internet is an energy peer-to-peer exchange and sharing network that can connect new power networks, oil networks, natural gas networks and other energy nodes. Power-to-Gas (P2G) technology is an important means to realize energy transmission and distribution in the energy Internet, which can realize joint operation with renewable energy power generation and change the direction of energy flow between the power system and natural gas from the traditional one-way to two-way [4]. However, compared with a single microgrid and natural gas network, the structure of the P2G coupling network is more complex, and the relationship between the systems through the equipment coupling is more diverse and flexible, which undoubtedly increases the operation and control difficulty of the P2G coupling network [5].

First of all, the structural design and control model of the natural gas network is the research foundation of the P2G interconnection system [6,7]. In the research of natural gas network structure, the dynamic characteristic of gas pipelines is modeled with energy storage capability in [8], and a cooperative dispatching strategy for P2G and pipeline storage capability is presented. In [9], a dynamic model of gas network pipelines is also used to establish the online optimal dispatching model of the integrated energy system. Additionally, regarding the research of natural gas network control methods, a comprehensive non-combination of wind turbines, gas turbines, electric-to-gas turbines, transmission lines and natural gas pipelines in the gas–electric integrated system linear programming model is established in [10], and a collaborative planning method for gas–electric systems based on non-cooperative game theory is proposed. In [11], a control algorithm using radial basis function neural network to optimize controller parameters is proposed for the complex non-linear regulator model, which can effectively improve the control accuracy, response speed and dynamic tracking performance of the voltage regulator. However, with the development of the P2G interconnection system, the operation status of the natural gas net-work side will be more complicated, so the load disturbance of the natural gas network and the microgrid can affect the frequency and pressure balance of the system together. Meanwhile, the P2G system also carries the task of multi-energy complementation with the grid, and the stability puts forward higher requirements. Thus, the above natural gas network control method for a single scenario cannot well coordinate various units in the microgrid to participate in the stable control of natural gas and realize the stable operation of the natural gas network under a variety of operating conditions.

Secondly, when the microgrid is in an islanded mode, the switch at the public connection point between the upper-level grid and the microgrid is disconnected, and power support can no longer be provided. It is necessary to control the distributed power supply in the microgrid to provide frequency support for the microgrid [12,13]. In the P2G inter-connection system, the MT and P2G equipment, as stable micro sources and controllable loads on the microgrid side, also have the ability to participate in the stability control of the microgrid. In [14], a nonlinear control model of wind power accommodation for an electricity-heat-gas integrated microgrid with P2G is proposed, which can meet the demand for electricity, heat, gas loads, and the security constraints of the electricity microgrid. In [15], a strategy to regulate and control the frequency of the microgrid through EVs and controllable natural gas loads is proposed. Moreover, the simulation is carried out in a power system including large-scale wind power and photovoltaic power generation to verify the effectiveness of this method for load frequency control. However, the above-mentioned research is only aimed at the frequency control problem of microgrid containing natural gas system, which obviously cannot satisfy the cooperative the control between the whole systems of microgrid frequency and natural gas network air pressure.

In addition, the core unit of the microgrid frequency control system is the load frequency controller, and its control effect is critical to whether the microgrid can perform effective frequency modulation and whether the P2G interconnection system can be coordinated [16]. In the current research, a variety of control methods have been applied to the design of load frequency controllers. In [17], a coordinated operation strategy for a microgrid that uses P2G technology to produce natural gas from abandoned wind and participates in frequency modulation is proposed. Furthermore, based on the actual abandonment of a wind farm and the actual command issued by an automatic power generation control (AGC) unit, the particle swarm optimization algorithm is used to perform verification calculations. In [18], a load frequency controller is designed for the simultaneous participation of a MT, electrolytic cells and EVs, and the PI control parameters are optimized based on the hybrid leapfrog algorithm of H2/H∞ control technology. However, in the P2G interconnection system, the state parameters and operating conditions of the microgrid have been changing due to the fluctuation of renewable energy and the coupling of the network. The traditional control method is difficult to ensure the stability of the microgrid under complex and changeable operating conditions, and it can only deal with the frequency modulation problem of the microgrid alone but cannot coordinate the control of microgrid and natural gas network greatly.

Meanwhile, with the proposal of the multi-regional energy Internet, the multi-microgrid system (MMS) formed by the interconnection of multiple sub-microgrids has developed rapidly. However, the strong random disturbance brought by the access of large-scale new energy sources and the power exchange between multiple microgrids bring more challenges to the multi-energy cooperative control of the MMS. Methods such as classical control [19], adaptive control [20], and robust control [21] cannot save the experience and knowledge of past tasks and need to be re-initialized in the face of various complex operating conditions so that the response speed of the control is slowed down and the performance is deteriorated.

In summary, an P2G interconnection system with MTs and P2G equipment as the coupling link is modeled, and the interactive characteristics of the microgrid and natural gas network are obtained from this. Thus, a load frequency control strategy of islanded microgrid based on DDPG is proposed in this paper, which is oriented to P2G system. The main contributions are as follows:

(1)

A frequency–pressure cooperative control structure of multi-microgrid with electric-gas system is proposed. Based on the analysis of the operating characteristics of the natural gas network and the coupling equipment, the natural gas network model and the P2G system coupling model are established. The coordinated transformation of the electrical coupling system in the two modes of the gas determined by power (GDP) and the power determined by gas (PDG) is realized, which provides a model basis for the coordinated control of frequency and natural gas pressure.

(2)

In order to coordinate power transmission between multiple microgrids and realize coordination and cooperation, the MADDPG algorithm with the cooperative control idea of “centralized training, decentralized execution” is proposed to design the frequency controller for the multi-microgrid. Moreover, on the basis of the two areas of the IEEE standard, the area control error (ACE) is used as one of the state spaces in the design process of the controller so as to obtain the optimal solution of multi-microgrid coordination quickly and accurately in the control process.

(3)

In order to coordinate the control of the frequency of the microgrid and the air pressure of the natural gas network, the structure and the reward function of the MADDPG controller is designed according to the two control objectives in order for it to meet the two control goals of the frequency of the microgrid and the air pressure of the natural gas network together.

The remainder of this paper is organized as follows. In Section 2, the natural gas network model and the P2G system coupling model are established. The load frequency control structure of multi-microgrid is introduced in Section 3. The load frequency controller based on MADDPG is introduced in Section 4. The simulation results are presented and analyzed in Section 5, and the conclusions are summarized in Section 6.

2. The Model of the Power-to-Gas System

As the connection between the power grid and the natural gas network is getting closer, the construction of the P2G interconnection model and the mutual influence be-tween the two networks have received more and more attention [22]. The microgrid model in this paper includes EVs, distributed power sources, MTs and ordinary loads, while the natural gas network model includes natural gas transmission pipelines, pressure regulating valves and shut-off valves, which together constitute the model of the P2G inter-connected terminal integrated energy system [23], as shown in Figure 1.

2.1. The Model of Natural Gas Network

When the interactive characteristics of the P2G interconnection system is being studied, the operation status of the grid side changes rapidly and the time scale is very short. Therefore, when establishing a natural gas pipeline model, it is necessary to consider the dynamic process of the natural gas pipeline operating state change and establish its dynamic model.

Additionally, because the natural gas terminal system covers a small area, the radial change of natural gas flow can be ignored. The natural gas flow change of the natural gas pipeline can be assumed to be a one-dimensional fluid, and the pressure, flow rate and density only change with time and the axial position in the pipeline, as shown in Figure 2.

According to the law of conservation of mass and Newton’s second law of motion, the dynamic natural flow can be described by the following Formulas (1) and (2). Additionally, because the pipeline length of the low-pressure natural gas network is usually several hundred meters, it is shorter than the pipelines of the medium-pressure network and the high-pressure network. Therefore, according to the law of conservation of mass, Newton’s second law of motion, and the idea of finite element approximation, the dynamic natural gas flow in a short natural gas pipeline can be described in the form of ordinary differential Equations (3) and (4) [24]:

$$\frac{S}{c^2}\frac{\partial p}{\partial t}=-\frac{\partial M}{\partial x}$$

(1)

$$\frac{\partial p}{\partial x}=-\frac{\partial (\rho \omega )}{\partial t}-\frac{\partial (\rho {\omega }^2)}{\partial t}-\frac{2f\rho {\omega }^2}{D}$$

(2)

$$\frac{d{p}_{out}}{dt}=\frac{c^2}{S}\frac{M_{in}-M_{out}}{L}$$

(3)

$$\frac{d{M}_{in}}{dt}=\frac{S}{L}(p_{in}-p_{out})-\frac{2f{c}^2{M}_{in}}{SD{p}_{out}}$$

(4)

where p_in and p_out are the air pressure at the inlet and outlet of the natural gas pipeline, M_in, M_out are the flow at the inlet and outlet of the pipeline, c is the sound propagation speed in the natural gas pipeline, D is the diameter of the pipeline, S is the cross-sectional area of the pipeline, L is the length of the pipeline, f is the friction coefficient of the pipeline, ρ is the density of natural gas, ω is the flow rate of natural gas in the pipeline, and x is the distance.

Furthermore, because the friction coefficient and pipeline diameter of the natural gas pipeline are constant, the friction coefficient is typically 0.003. Then, in Equations (1) and (2), there are only four variables, namely p_in, p_out, M_in and M_out. For a section of natural gas pipeline, when two variables are given, the values of the other two variables can be solved.

In this low-pressure natural gas network, the pressure regulating valve functions as a step-down transformer in the power grid, which reduces the air pressure and maintains the outlet air pressure to track the set reference value. The dynamic characteristics of the pressure regulating valve can be simplified as:

$$r=K_r{\displaystyle \int (p_{out\_ref}}-p_{out})$$

(5)

$$p_{out}=r{p}_{in}$$

(6)

$$M_{out}=M_{in}$$

(7)

where r is the variable ratio of the pressure regulating valve, K_r is the proportional control coefficient in the pressure regulating valve, and p_{out_ref} is the air pressure reference value at the outlet of the pressure regulating valve.

Thus, a pressure regulator model can be built to test its dynamic characteristics. When the outlet reference pressure of the pressure regulating valve changes, its outlet pressure and pressure regulation ratio can change accordingly, as shown in Figure 3. It can be seen that the pressure regulating valve model can track the target outlet pressure well. Meanwhile, when a leak is detected in a natural gas pipeline, the shut-off valves at both ends of the pipeline can be controlled to shut off, and the natural gas supply of the faulty pipeline can be cut off in time to prevent the leakage from further expanding and cause safety accidents. Its dynamic characteristics are shown in Figure 4. It can be seen that the shut-off valve can effectively cut off the flow of the natural gas network.

2.2. Coupling Relationship of Power-to-Gas System

As shown in Figure 1, in the P2G interconnection system, the microgrid and the natural gas network are connected to P2G equipment through MTs, and the coupling between the microgrid and the natural gas network is also caused by changes in the operating state of the coupling equipment.

The active power output by the MT is related to the amount of gas, and the rate of methane production by the P2G equipment is related to the active power consumed. Since the operating status of the MT and P2G equipment can be determined by the demand on the microgrid side or the demand on the natural gas network, the operation modes of the two can be divided into two modes: gas determined by power (GDP) and power determined by gas (PDG): When working in the GDP mode, the micro gas turbine is equivalent to a stable distributed power source, and the P2G device is equivalent to a controllable electric load; when working in the PDG mode, the P2G device is a stable gas source, and the micro gas turbine is a controllable gas load.

In the internal control system of the MT, the output active power is regulated by the DDPG controller. The block diagram of the control system is shown in Figure 5.

Where, the fuel system and the gas turbine adopt the first-order inertia link to be equivalent, ∆A_DDPG is the control signal sent by the agent DDPG, ∆X_MT is the state quantity that characterizes the valve position change of the fuel system, T_f and T_t are the time constants of the fuel system and the gas turbine respectively, ±δ_MT is the power climbing constraint of the micro gas turbine, ±μ_MT is the power variation constraint. The working mode of the micro gas turbine control system is adjusted by the DDPG controller.

When the control system works in the GDP mode, it can adjust the output active power of the gas turbine by adjusting the gas volume of the gas turbine according to the frequency of the microgrid so as to realize the functions of frequency modulation. The block diagram of FM control is shown in Figure 6. Additionally, when the control system works in the PDG mode, it can adjust the MT’s intake air volume according to the change of the gas pressure at the node of the natural gas network in order to adjust the air pressure at the inlet end to further maintain the node air pressure. The block diagram of natural gas pressure balance control is shown in Figure 7.

3. Load Frequency Control Model of Multi-Microgrid

Frequency determines the quality of electric energy, and air pressure determines the quality and safety of gas supply. Therefore, in the P2G interconnected terminal system, the frequency stability of the microgrid and the gas pressure stability of the natural gas network are very important for improving the quality of energy supply and the stability of the integrated energy system. For a multi-microgrid system, the controller needs to coordinate and control the unit output of each sub-microgrid, and select the control command that is most conducive to the overall stability of the system for various operating conditions. Therefore, this section analyzes the load frequency responses of MT, P2G equipment, EV stations, distributed power sources and loads in the microgrid, and constructs a multi-microgrid load frequency control model by introducing the ACE and power coupling links.

3.1. Load Frequency Control Model of P2G Equipment

The role of P2G equipment is equivalent to virtual energy storage. The P2G electrolyzer can adjust the voltage amplitude of the DC voltage source through the rectified Buck circuit and realize power adjustment by changing its electrolysis current. Since the gas flow rate supplied by P2G equipment to the natural gas network is small, and the duration of frequency adjustment is very short, its frequency control model does not need to consider the influence of natural gas users and has the advantage of fast response speed. The frequency control response model of P2G equipment is shown in Figure 8:

Where, ∆u_P2G is the load frequency control signal received by the P2G equipment, T_ele is the time constant of the electrolytic cell, ±δ_P2G is the upper and lower limits of the power change rate constraint, ±μ_P2G is the upper and lower limits of the power increment constraint, and ∆P_P2G is the power increment of the P2G device.

3.2. Load Frequency Control Structure of a Multi-Microgrid Based on MADDPG

Based on the above, the wind power, micro gas turbines [25], and electric vehicle charging stations [26] are used to establish a multi-microgrid load frequency control model in this paper, as shown in Figure 9. Among them, wind power has great uncontrollability, so it is only treated as a random load, MT is used as the main frequency modulation unit, and EV stations are used as auxiliary frequency modulation units to ensure the flexibility of microgrid regulation [27].

Where, ΔP_L is the load disturbance power, ΔP_W is the wind disturbance power, ΔP_MT is the power variation of MT, ΔP_EV is the power variation of EVs, ΔP_P2G is the power variation of P2G, ΔP_line is the power variation of tie-line, H_t is the constant of inertia of the microgrid, T_sij is the coupling link parameter, ACE is the area control error, which is a signal used to coordinate the two controllers.

4. Load Frequency Controller of Islanded Microgrid Based on MADDPG

In the multi-microgrid with P2G equipment, it is important to maintain the stability of the microgrid frequency and gas network pressure. However, due to fluctuations in the output of wind turbines and natural gas flow, it would inevitably cause the microgrid frequency and natural gas network pressure to deviate from the standard value.

Furthermore, DDPG has the advantages of online learning and experience playback capabilities, which can effectively solve the control problems in nonlinear systems [28]. Meanwhile, compared to the Q-learning algorithm, which cannot process continuous signals, DDPG learning can be performed in a continuous action space [29] without discretizing the action space. Therefore, it can better realize the precise control of each unit in the microgrid, which is suitable for the design of this paper. Therefore, this paper designs a load frequency controller of multi-microgrid based on MADDPG, which can adjust the output power of the unit in the microgrid through the frequency deviation and the change of the air pressure so that the frequency and the air pressure can be stabilized.

4.1. Theoretical Analysis of DDPG

In this model, the fluctuation of random power supply and load, the random output boundary of EV and the real-time frequency deviation are all complex nonlinear data that are difficult to quantify. The combination of deep neural network and reinforcement learning can effectively process large-scale data. It can rely on the data in the microgrid model in this paper to give real-time decisions according to the corresponding operating requirements and optimization goals and obtain the optimal control scheme and optimization strategy [30]. Therefore, this paper takes the state vector S as the input sequence through the convolutional neural network and passes through the long short-term memory network (LSTM) in the convolutional neural network. Then, the output will enter the fully connected layer and finally get the approximated Q value in the output layer; the network structure diagram is shown in Figure 10.

There are four networks in DDPG: actor target network (ATN), actor current network (ACN), critic target network (CTN) and critic current network (CCN).

At t, the parameter of ACN is θ, the parameter of ATN is θ’, the parameter of CCN is ω, and the parameter of CTN is ω’. Among them, the action at can be generated according to the current status s_t by the ACN. The action a_t+1 at the t+1 time can be generated according to the subsequent state of the environment by the ATN. The value R_t corresponding to the status s_t and action a_t can be calculated by the CCN. The value of Q’ (s_t+1, a_t+1|ω’), which is used to calculate the target value y, can be generated by the CTN based on subsequent state s_t+1 and action a_t+1, as shown in the Formula (8):

$$y=r_t+\gamma Q\prime (s_{t+1},a_{t+1},\omega \prime )$$

(8)

where, γ is a discount factor and 0 < γ < 1.

Meanwhile, the ω of CCN and the θ of ACN are updated through the gradient of the neural network, as shown in the Formulas (9) and (10):

$$L=\frac{1}{m}{{\displaystyle \sum _{j=1}^m\left(y_j-Q\left(s_j,A_j,\omega \right)\right)}}^2$$

(9)

$$\nabla J\left(\theta \right)=\frac{1}{m}{\displaystyle \sum _{j=1}^m\left[{\nabla }_{a}Q\left(s,a,\omega \right){|}_{s=s_j,a={\pi }_{\theta }\left(s\right)}{\nabla }_{\theta }{\pi }_{\theta }\left(s\right){|}_{s=s_j}\right]}$$

(10)

where, m is the number of samples, y_j is the target value of the j sample, Q(s_j,a_j,ω) is the output value of the CCN for the j sample, π_θ(⋅) is the output value of the ACN.

Further, the parameters of CCN and ATN can be updated by Equation (11):

$$\begin{array}{l}\omega \prime \leftarrow \tau \omega +(1-\tau )\omega \prime \\ \theta \prime \leftarrow \tau \theta +(1-\tau )\theta \prime \end{array}$$

(11)

where, τ is an update coefficient, which is generally small.

Moreover, the termination function E is used to determine whether the Agent enters the termination: If it does, the iterative process stops, and a new round of state sequence starts. Otherwise, the iterative process can be continued.

In summary, the sample unit {s, a, R, s’, E} is stored in the empirical playback set D. Then the m sample units of the set D are trained for T rounds using Equations (8)–(11), and the training step length of each round is T_m.

4.2. Theoretical Analysis of MADDPG

In addition, the theoretical basis of MADDPG is the Markov game, which is an extension of Markov decision process under multi-agent conditions [31]. In a multi-agent environment, the Markov game of N agents consists of the following elements:

(1)

State S, which can describe all possible configurations of all agents;

(2)

Actions A₁,…,A_N of each agent;

(3)

Observations O₁,…,O_N of each agent.

The goal of each agent is to maximize its final expected reward, and its calculation formula is as follows.

$$\max \left(R_i={\displaystyle \sum _{t=0}^T{\gamma }^t}{r}_i^t\right)$$

(12)

where, γ is the discount factor, which can affect the convergence speed and training stability of the agent; r_i is the reward value of each agent.

The structural comparison of MADDPG and DDPG is shown in Figure 11. Among the actor–critic network in each agent, the network framework adopts the mode of centralized training and decentralized execution as shown in Figure 11: Critic (Q network) can obtain global information (s₁, s₂,…,s_n) and global actions (a₁, a₂,…,a_n); Actor (policy network) can only make actions based on local observations. The learning process of MADDPG is similar to the above-mentioned DDPG algorithm, and its objective function is shown in Formula (13) [32]:

$$L\left(\theta \right)=\underset{(s,a,other\_s\_a,r,s\prime )~D}{E}[(Q(s,a,other\_s\_a|\theta )-{(r+Q(s\prime ,a\prime ,other\_s\prime \_a\prime |\theta \prime ))}^2]$$

(13)

where, other_s_a is the states and actions of other attackers and defenders. For the states and actions at the next time instant other_s′_a′, the states other_s′ are from the replay buffer, and the related actions other_a′ are obtained through the delayed policy network.

The update of the gradient is shown in Equation (14), and the objective function of the policy network is shown in Equation (15):

$$\theta =\theta -\eta \frac{\partial }{\partial \theta }L(\theta )$$

(14)

$$J(\widehat{\theta })=\underset{s,other\_s\_a~D}{E}[Q(s,P(s|\widehat{\theta }),other\_s\_a|\theta )]$$

(15)

From this, the gradient can be calculated by Equation (16), and calculate the final update of the gradient by Equation (17):

$${\nabla }_{\widehat{\theta }}J(\widehat{\theta })=\underset{s,other\_s\_a~D}{E}[{\nabla }_{\theta }P(s|\widehat{\theta }){\nabla }_{\theta }Q(s,other\_s\_a|\theta ){|}_{a=P(x|\widehat{\theta })}]$$

(16)

$$\widehat{\theta }=\widehat{\theta }+\eta \frac{\partial }{\partial \widehat{\theta }}J(\widehat{\theta })$$

(17)

4.3. Definition of State Space and Action Space

As shown in Figure 12, the controller structure is composed of the input layer and the control layer. The input layer can collect and process status signals such as microgrid frequency deviation Δf, the natural gas network pressure deviation Δp, the boundary of the output power of EVs charging stations ${\mathrm{P}}_{\mathrm{EV}}^{\pm }$, and the value of ACE, which can be updated by Equation (18):

$$ACE=\mathsf{\Delta }{P}_t-10\beta \cdot \mathsf{\Delta }f$$

(18)

where, β is the frequency response coefficient in each microgrid, the unit is MW/0.1 Hz.

Meanwhile, the control layer can coordinate the control objectives of the two microgrids to provide a combined action command that is optimal for overall stability ΔA_i (Δa_EVi, Δa_MTi, Δa_P2Gi) to each unit in the multi-microgrid so as to control the output power of MT, EV and P2G in multi-microgrid, and achieve rapid suppression of frequency fluctuations and pressure deviations.

Thus, the state set of the controller is the microgrid frequency deviation ∆F(t), the natural gas network pressure deviation ∆p(t), the boundary of output power of EVs station $\Omega P_{EV}^{\pm }\left(t\right)$, and the value of ACE(t), so the state space S can be defined as follow:

$$S=\left[\mathsf{\Delta }F\left(t\right),\mathsf{\Delta }p\left(t\right),ACE\left(t\right),\Omega P_{EV}^{+}\left(t\right),\Omega P_{EV}^{-}\left(t\right),\right]$$

(19)

The action set A of the MADDPG controller should be the real time scheduling instruction set for the output active power of MT, EVs, and P2G equipment. Therefore, the joint action set A of Agent1 can be defined as follows:

A=[∆A_MT (t), ∆A_EV (t), ∆A_P2G (t)]

(20)

Furthermore, according to [15], the discrete set of frequency deviation ΔF(t) can be set as (−∞, −0.2), [−0.2, −0.15), [−0.15, −0.10), [−0.10, −0.03), [−0.03, 0.03], (0.03, 0.10], (0.10, 0.15], (0.15, 0.2], (0.2, +∞), unit is Hz. And real-time air pressure fluctuation ∆p(t) can be set to (−∞, −0.5), [−0.5, −0.4), [−0.4, −0.3), [−0.3, −0.2), [−0.2, 0.2], (0.2, 0.3], (0.3, 0.4), (0.4, 0.5), (0.5, ∞), unit is kPa.

4.4. Design of Reward Function

The control objectives of this paper are: (1) Maintain the frequency stability of the microgrid; (2) Maintain the stability of the node air pressure in the natural gas network; (3) Coordinate the overall stability of two microgrids. Therefore, a comprehensive reward function in each sub-microgrid can be established to coordinate frequency restoration and air pressure adjustment.

$$R_i=r_f+r_p-\mu {[{\mathrm{ACE}}_i(t)]}^2$$

(21)

$$r_f=\{\begin{array}{l}0\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}|\mathsf{\Delta }f|<0.03\\ -{\mu }_1|\mathsf{\Delta }f|\hspace{1em}0.03\le |\mathsf{\Delta }f|<0.10\\ -{\mu }_2|\mathsf{\Delta }f|\hspace{1em}0.10\le |\mathsf{\Delta }f|<0.15\\ -{\mu }_3|\mathsf{\Delta }f|\hspace{1em}0.15\le |\mathsf{\Delta }f|<0.2\\ -{\mu }_4|\mathsf{\Delta }f|\hspace{1em}0.2\le |\mathsf{\Delta }f|\end{array}$$

(22)

$$r_p=\{\begin{array}{l}0\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}|\mathsf{\Delta }p|<0.2\\ -{\delta }_1|\mathsf{\Delta }p|\hspace{1em}0.2\le |\mathsf{\Delta }p|<0.3\\ -{\delta }_2|\mathsf{\Delta }p|\hspace{1em}0.3\le |\mathsf{\Delta }p|<0.4\\ -{\delta }_3|\mathsf{\Delta }p|\hspace{1em}0.4\le |\mathsf{\Delta }p|<0.5\\ -{\delta }_4|\mathsf{\Delta }p|\hspace{1em}0.5\le |\mathsf{\Delta }p|\end{array}$$

(23)

where, R_i is the global reward in MG i, r_f is the reward of microgrid frequency, r_p is the reward of air pressure, μ₁, μ₂, μ₃ and μ₄ are the parameters of each control area in the frequency reward r_f, and δ₁, δ₂, δ₃ and δ₄ and are the parameters of each control area in the air pressure reward r_p, the ACE(t) represents the instantaneous value of ACE at time t. η represents the weight of ACE, which is 0.5 in this paper. And the convergence effect and the learning speed can be affected by the size of the reward value, so it is necessary to perform simulation tests according to actual examples, and the specific process would be discussed in next section.

Thus, the overall reward function of the multi-microgrid system is shown in Formula (24):

$$R_{total}={\displaystyle \sum _{i=1}^n{R}_i}$$

(24)

5. Simulation Results

In order to test the control effect of the strategy proposed in this paper, a multi-microgrid system is constructed through simulation experiments as shown in Figure 13, and the specific settings of equipment parameters are shown in

Table 1. Parameters of equipment in microgrids.

Unit	Parameter	Meaning	Value
MT	Tf	time constant of governor	10 s
	Tt	time constant of generator	0.1 s
	Rf	speed regulation factor	0.005 Hz/p.u.
	λmtd	lower limit of active power variation	−0.015 p.u.
	λmtp	upper limit of active power variation	0.02 p.u.
EVMG1	Tev1	time constant of EV station of MG1	1 s
	±λev1	the limit of active power variation	±0.018 p.u.
	nev1	Initial number of EVs in station of MG1	36
EVMG2	Tev2	time constant of EV station of MG2	1 s
	±λev2	the limit of active power variation	±0.016 p.u.
	nev2	Initial number of EVs in station of MG2	32
P2G	Tele	time constant of P2G	0.5 s
	PCLmax	maximum controllable amount	0.01 p.u.
	PCLmin	minimum controllable amount	−0.01 p.u.
Other	Ht	Microgrid inertia parameters	7.11 s
	Tsij	Microgrid coupling link parameters	0.545 p.u.

. Therefore, according to the microgrid structure and a large number of tests, the specific values of the controller reward function, hyperparameters and training parameters can be obtained as shown in

Table 2. Convergence test results under different parameters.

Parameter	Meaning	Value
μi (i = 1, 2, 3, 4)	time constant of governor	3, 8, 20, 45
δi (i = 1, 2, 3, 4)	time constant of generator	1, 3, 8, 20
γ	The discount factor	0.95
α	The learning rate	0.001
Ne	The maximum episode number	500
ns	The number of steps in each round	300
l	The step size	0.1

. The verification of the calculation examples in this paper is carried out through simulation experiments. The computing platform is a PC with [email protected] CPU and 16 GB RAM, and the software environment is Windows 10 Professional and MATLAB R2021a.

Meanwhile, 6 groups of neural network structure parameters (h,u) are set for the convergence test, and the learning results of the agent are shown in

Table 3. Convergence test results under different parameters.

SN	Parameter Settings	Average Reward	Final Award
1	h = 3, u = 50	−34.037	−1.83622
2	h = 3, u = 200	−26.673	−1.20923
3	h = 5, u = 50	−21.096	−0.65307
4	h = 5, u = 200	−27.075	−1.35723
5	h = 10, u = 50	−34.572	−2.54778
6	h = 10, u = 200	−40.922	−3.04643

. It can be seen that when the number of layers of the neural network h = 5, and the neurons in each layer u = 50, the agent has the best convergence characteristics and optimization results.

5.1. Pre-Learning Stage

Before the DDPG controller is used in the formal simulation test, it needs to go through a pre-learning stage. In the early stage of pre-learning, the controller did not accumulate enough experience, so there is no intelligent control ability [33]. Therefore, this paper sets up the wind power disturbance function formed by the superposition of various different amplitudes and different types of functions, the node air pressure function of the natural gas network, and the electric vehicle output power boundary function that changes randomly over time, which are used to train the Agent, so as to obtain the optimal value function Q network Q_φ(s,a). The training process is shown in Figure 14.

Thus, the pre-learning process of the Agent 1 is shown in Figure 15. It can be seen that the learning process has been completed at 372 episodes. After training, the average reward and the final reward of Agent 1 is −1.93044 and −29.64933, the average reward and the final reward of Agent 2 is −2.52452 and −30.45247, which shows that the controller has good convergence characteristics to complete the simulation experiment.

5.2. Case Study

In order to verify the control effect of the MADDPG controller proposed in this paper, the PID controller and the Fuzzy controller are selected as comparisons, and the experimental tests are carried out in the following cases.

5.2.1. Case 1: The Response of Microgrid Load and Wind Power Combined Disturbance

The load power on the microgrid side may have a step disturbance, and the combined power disturbance caused by the superposition of the wind power disturbance is extremely complex. This section aims to simulate such combined disturbances to verify the robustness of the proposed load frequency controller. The power changes of the combined disturbances experienced by the multi-microgrid system are shown in Figure 16. In Figure 17 is shown the action of the load frequency controller based on the MADDPG control, fuzzy control and PI control on the load frequency fluctuation of the microgrid. Additionally, the |Δf| of MG1 can be used as the assessment object, and the threshold value of the assessment excellence rate can be set to 0.03 Hz. It is worth mentioning that, although the wind turbine has great randomness, its change speed is fast, and the time scale is small. Therefore, the power imbalance of the micro-grid caused by the wind turbine will be solved by MADDPG controller in a short time. As a result, in this section, the influence of random disturbance on the air pressure in the gas network will not be considered.

Thus, it can be seen from Figure 17 that, under the combined disturbance, the load frequency controller based on MADDPG algorithm maintains excellent control effect, the amplitude of frequency fluctuation is smaller, which can be controlled within the range of 0.03 Hz, reaching 100% excellent rate. Under PI and Fuzzy control, the maximum |Δf| reaches 0.1168 and 0.08239 Hz, the excellent rate of |Δf| is much lower than MADDPG controllers.

Meanwhile, the power variations of each equipment and tie line in the MG1 under the MADDPG controller are shown in Figure 18. The inertia of the MT is large, and the response speed is slow. The P2G equipment maintains a fast response speed. However, due to the limitation of its controllable capacity, in the case of combined disturbances, the EV stations with a large capacity are mainly responsible for the main frequency regulation task. In addition, there are random limit constraints on the output power increment of EV stations, so the power change curve of the EV stations is obviously jagged.

5.2.2. Case 2: The Response of Natural Gas Load Disturbance

The sudden change of the load flow of the natural gas network will cause the air pressure at the interface between the pipeline and the load to fluctuate, causing the pressure difference between the inlet and the outlet of the pipeline to change. When the gas pressure disturbance at the key node of natural gas is detected, the MT and P2G equipment are switched to the PDG mode to participate in the gas pressure stability control.

Thus, on the basis of case1, at the initial moment of the simulation, the natural gas network is running in a steady state. When t = 10 s, the natural gas flow of pipe 2 in MG1 drops from 25 g/s to 15 g/s, as shown in Figure 19.

Since the MT and P2G equipment are operating in the PDG mode at this time, they cannot participate in the microgrid load frequency control, and act as a disturbance source, superimposed with the wind power disturbance and load disturbance to form a combined disturbance. Take MG 1 as an example, the power changes of the MT and P2G equipment under the control of MA-DDOG are shown in Figure 20. Moreover, the frequency variation under the action of the proposed MADDPG controller, Fuzzy controller and PI controller is shown in Figure 21. Meanwhile, the pressure changes of the natural gas network under the control of PI, fuzzy and MADDPG are shown in Figure 22.

It can be seen that, under the combined disturbance, the load frequency controller based on MADDPG algorithm maintains excellent control effect, the amplitude of frequency fluctuation is smaller, the |Δf| can be controlled within the range of 0.03 Hz, reaching 100% excellent rate. Addirionally, the air pressure deviation can quickly return to 0. Therefore, the MADDPG controller can coordinate the frequency recovery and air pressure regulation of the microgrid with superior dynamic control characteristics.

6. Conclusions

In order to make the P2G system in the multi-microgrid have the ability to maintain frequency stability when affected by electrical load disturbances, the ability to maintain node air pressure stability when affected by natural gas load disturbances, and the ability to coordinate the overall stability of the multi-microgrid, a load frequency control strategy of multi-microgrid oriented to P2G system is proposed in this paper. The model of natural gas network and the coupling model of P2G system are built, and a multi-microgrid system including MT, P2G, EVs stations, WT and loads is established. Thus, a load frequency control strategy based on MADDPG is proposed. The simulation results show that:

• The model of P2G system with MT and P2G equipment as the coupling device is constructed, and the great interaction characteristics between the microgrid and the natural gas network are proved through simulation. The MADDPG algorithm with the cooperative regulation idea of “centralized training and decentralized execution” can meet the coordinated control of multi-microgrid.
• Compared with traditional controller, the MADDPG controller with the ability of online learning and experience playback can more effectively deal with the random disturbance of the multi-microgrid, the |Δf| can be limited in 0.03 Hz, and the excellent rate can reach 100%, which is significantly better than the PI controller and Fuzzy controller.
• The MADDPG controller can greatly coordinate the frequency recovery and air pressure adjustment of the multi-microgrid. When the node air pressure of the natural gas network changes, the MADDPG controller can ensure that the |Δf| of the microgrid is kept within 0.03 Hz, and quickly bring the node air pressure deviation close to 0.