A Cloud-Edge Computing Method for Integrated Electricity-Gas System Dispatch

Abstract

An integrated electric–gas system (IEGS) is the manifestation and development direction of a modern smart power system. This paper employs the cloud-edge computing method to research IEGS’s optimal dispatch to satisfy data protection requirements between power systems and natural gas systems and reduce data transmission pressure. Based on cloud-edge computing architecture, this paper constructs a cloud-edge computing method based on the Multi-agent Deep Deterministic Policy Gradient (MADDPG) algorithm to solve optimal dispatch problems. Then, this paper proposes an IEGS dispatch strategy based on cloud-edge computing, which conducts distributed computing independently at the edge of power and natural gas, and the cloud implements global dispatch based on boundary information and edge learning parameters. This method does not require the exchange of all information between the power system and natural gas system, effectively protecting data privacy. This paper takes the improved IEGS of the IEEE 9 node and Gas 8 node as an example to analyze. The equipment output of this dispatch method is within a reasonable range, and the cost is reduced by 0.21% to 1.03% compared with other methods, which verifies the effectiveness of the cloud-edge computing method in solving dispatch problems.

1. Introduction

As the contradiction between environmental protection and energy consumption intensifies, the demand for efficient development and utilization of clean energy is becoming more and more urgent. Natural gas has the outstanding advantages of safety, cleanliness, low carbon, and high calorific value, and has a good coupling effect with the power system. As a carrier of energy transportation and comprehensive utilization, IEGSs cannot be ignored [1,2,3,4].

For IEGSs, the tidal current problem, the dispatch problem, and the optimization problem have been studied extensively in previous publications. Reference [5] proposed a unified analytical approach that uses the Newton–Raphson formulation to realize the dispatch and optimization of an IEGS. In this method, the nonlinear equations representing the combined system of natural gas and electric power are obtained, linearized, and solved based on the node equilibrium of natural gas and electric power flow, respectively. Through a hierarchical Stackelberg game model, reference [6] proposed a hybrid multi-objective optimization and game theory approach to realize the coordinated operation strategy of an IEGS. The method realizes the balanced dispatch of a variety of distributed energy systems, but the calculation process is complicated and the calculation speed is slow. For the optimization problem of an IEGS with bidirectional energy flow, reference [7] reduced it to a hybrid integer nonconvex and nonlinear programming problem solved by a second-order cone programming (SOCP) method. Reference [8] used the sparse semidefinite programming (SDP) relaxation method to construct a tight and traceable SDP relaxation of the lifting variable matrix to achieve the collaboration operation. The above method realizes the day-ahead dispatch scheme of an IEGS and provides a data reference for the subsequent research. Meanwhile, considering that the coupling of an IEGS brings challenges to the system in terms of coordination and uncertainty management, reference [9] considers the storage dispatch and renewable energy uncertainty in IEGSs and designs a new second-order cone relaxation method for the non-convexity of the Weymouth equation to achieve optimal dispatch of the IEGS. Reference [10] proposed an uncertainty management framework based on physically accurate and validated models to study the importance of uncertainty in IEGSs to avoid violations of operational constraints and gas delivery to generators. The method only considers the effect of natural gas on the system, and schedules the IEGS by changing the consumption of gas turbines and natural gas storage tanks. To verify the impact on the system when a failure occurs in the operating unit of the IEGS, reference [11] proposes a new dynamic load recovery model to study the operational reliability of the IEGS by simulating failures. Reference [12] analyzed the probabilistic energy flow model of an IEGS using the cumulative volume method and investigated the effect of power-to-gas (P2G) on network security in a grid-connected wind power environment. Although this method only considers the effect of wind power, it describes the IEGS energy flow model in detail, which provides a model reference for future research. Currently, machine learning methods have been widely used, whereby scholars are trying to introduce artificial intelligence to solve power system problems. Reference [13] used deep Q networks (DQN) and deep deterministic policy gradients (DDPG) to develop autonomous voltage control (AVC) strategies. Reference [14] used the MADDPG approach to achieve multi-timescale voltage control. The method realizes the application of deep reinforcement learning in the field of AVC, and proves the superiority of machine learning in the field of power systems compared with other methods. In the research related to integrated energy systems, reference [15] used a deep reinforcement learning (DRL) approach to solve the optimal dispatch problem of an IEGS. Reference [16] proposed a real-time adaptive dynamic optimal control strategy based on deep learning to learn the optimal operation behavior of micro-energy grids to realize the operation control of electricity–thermal–gas energy grids. However, the above reference adopts a centralized calculation and centralized execution approach, and the power system and natural gas system are solved as a whole, without considering the data protection between different systems.

Previous research adopts the method of centralized computing, which requires data exchange between different energy systems. However, energy systems need to protect private data while data exchange is not always feasible for IEGSs. Meanwhile, with the increase in the amount of calculated data, the network transmission pressure increases, which sets a greater challenge for the network structure. Cloud-edge computing architectures can better coordinate multiple energy systems. The application of cloud-edge computing technology in power systems has attracted extensive attention and has great potential to help solve power system problems. Reference [17] investigates the application scenarios of cloud-edge computing based on the distributed architecture of central and edge clouds. Reference [18] constructs a physical model of power grid information for joint control based on the framework of cloud-edge collaboration technology, which achieves optimization and improvement of the traditional mathematical model of a power grid. The above references build a power system model based on cloud-edge computing architecture. Due to the superiority of cloud-edge computing architecture in privacy data protection, it provides the infrastructure to meet the requirements of power grid data protection. Reference [19] combines a multi-agent system and cloud edge collaboration architecture to establish a deep reinforcement learning model of the multi-level dynamic reconstruction of an urban distribution network, adopts the learning mode combining offline and online, and uses a multi-agent algorithm to complete the optimization. For both distribution grids and multi-microgrids, cloud-edge computing technology can realize control and prediction. Reference [20] designed a low-voltage distribution network operation and control architecture based on cloud-edge computing. Reference [21] proposed a cloud-edge collaborative forecasting model to achieve dynamic load combination forecasting for microgrids. Reference [22] proposes a cloud-edge computing method for the economic dispatch of multi-microgrid active distribution networks, which adopts multi-agent deep reinforcement learning to realize cloud-edge collaborative computing. The above references realize the dispatch of different power systems based on the cloud-edge computing method. However, the above references often set the distribution network as an edge side, and the internal distribution network is still regarded as a whole, and the data of different systems within the distribution network cannot be protected. Reference [23] proposed a cloud-edge cooperative distributed optimization dispatch strategy for an electrical IES based on the proximal Jacobian alternating direction method of multipliers, and the general optimization variable interaction iteration and parallel solution of the multi-dispatch model were established. However, in reference [23], although the cloud-edge collaborative method is adopted to deal with the IEGS dispatch problem, the independent calculation of the power system and natural gas system is not realized. At the same time, the algorithm solves complex optimization problems in real time, the decision time is long, the calculation speed is slow, the edge side still has the risk of privacy disclosure, and multiple agents have a strict calculation order, which cannot be calculated in parallel. The above research classification of power system dispatch problems is shown in

Table 1. Research classification of power system dispatch problems.

Structure	Algorithm	Advantages	Disadvantages
Centralized computing	Game theory [5]	High precision	Calculation process is complex, long computing time.
	PSO [12]	Strong versatility, fewer adjustment parameters, fast convergence speed	Due to the limitation of local search ability, the algorithm cannot guarantee the global optimal solution.
	DDPG [13]	Relatively stable, training easy convergence.	The parameter change is small, and the learning process is slow.
	SAC [15]	More exploration, faster training	The training speed is slow and the training process is unstable.
Distributed computing	ADMM [23]	High efficiency, improve computing efficiency, strong scalability	To solve complex optimization problems in real time, the decision time is long and the parameters are sensitive.

.

Power systems and gas systems are separate sectors, and each needs to protect its own data. The dispatch methods mentioned above need data exchange between different systems and cannot meet the privacy data protection needs of different departments. Therefore, in order to better realize the privacy protection of different systems, this paper proposes a cloud-edge computing method for IEGS dispatch. In this dispatch strategy based on cloud-edge computing, computing platforms are deployed on the edge side of the power system and natural gas system, respectively, to complete distributed computing independently, and the cloud computing platform is responsible for receiving, distributing, and coordinating the data on the edge side. Meanwhile, a multi-agent approach that treats different systems in the IEGS as agents separately has yet to be discovered. By combining the cloud edge collaboration method and the MADDPG algorithm, the cloud edge sets up agents, respectively, to achieve cooperation and collaboration. Power systems, natural gas systems, and coupled unit systems are modeled, which are integrated into the dispatch model. Finally, the IEGS dispatch strategy based on cloud-edge computing is proposed. The contributions of this paper can be summarized in the following three aspects:

• Protecting data privacy and reducing network transmission pressure. A cloud-edge computing approach is proposed for dealing with dispatch problems of IEGSs. In this approach, there is no need for data exchange between two systems to meet the data privacy protection needs between different systems. Meanwhile, it is not necessary to transmit all data information between the cloud and the edge to reduce the network transmission pressure.
• The cloud-edge computing dispatch method based on MADDPG is adopted. The edge server deploys a computing platform to learn the regulation strategy and compute the intelligence based on the local information. The cloud computing platform computes reinforcement learning parameters based on synergistic information to obtain the optimal regulation strategy.
• The approach not only meets the data privacy protection needs for IEGSs, but also controls the cost in a reasonable interval.

The rest of this paper is organized as follows. Section 2 presents the IEGS model. Section 3 describes the cloud-edge computing method. Section 4 investigates the IEGS dispatch strategy based on cloud-edge computing and provides a case study to illustrate the effectiveness of the proposed approach. The paper makes a conclusion in Section 5.

2. IEGS Model

The electric power system, the natural gas system, and the electric–gas coupling unit constitute the IEGS. The IEGS couples with a gas turbine (GT) and P2G. In the system, the P2G converts electrical energy into natural gas, and the GT outputs electrical energy by consuming natural gas. The IEGS model is shown in Figure 1.

2.1. Power System Model

In this paper, the power system model includes conventional generating units, wind power, photovoltaic (PV), and energy storage (ES) [12].

To ensure the normal operation of the power system, the power balance constraint, power limit constraint, and ramping constraint should be satisfied.

The power balance constraint is as follows:

$$\begin{array}{c}{P}_{G,i,t}+P_{GT,i,t}+P_{PV,i,t}+P_{W,i,t}+P_{ES,i,t}=\\ P_{L,i,t}+P_{P2G,i,t}\end{array}$$

(1)

The power limit constraints are as follows:

$$0\le P_{G,i,t}\le P_{G,i,max}$$

(2)

$$P_{ES,i,min}\le P_{ES,i,t}\le P_{ES,i,max}$$

(3)

For the ES, the state of charge is used to calculate the charging and discharging power of the ES, the ES state of charge $SO{C}_{es,i,t}$ satisfies the following formula:

$$SO{C}_{es,i,t}=\frac{E_{es,i,t}}{E_{es,i,max}}$$

(4)

$$SO{C}_{es,t+1}=SO{C}_{es,t}-\frac{{\eta }_{soc}{P}_{ES}\Delta t}{E_{es,max}}$$

(5)

The ramping constraint is as follows:

$$-SO{C}_{esdn,i}\le SO{C}_{es,i,t}-SO{C}_{es,i,t-1}\le SO{C}_{esup,i}$$

(6)

2.2. Natural Gas System Model

In this paper, the natural gas system includes natural gas sources and the storage capacity of the gas storage (GS), ignoring the impact of gas pipelines, pressure stations, and compressors on the system [12].

To ensure the normal operation of the natural gas system, the gas flow balance constraint, limit constraints, and ramping constraint should be satisfied.

The gas flow balance constraint is as follows:

$$\begin{array}{c}{f}_{G,i,t}+f_{P2G,i,t}+\\ f_{gs,out,i,t}-f_{GT,i,t}-f_{L,i,t}-f_{gs,in,i,t}=0\end{array}$$

(7)

For the GS, the storage capacity calculation formula is as follows:

$$S_{GS,i,t}=S_{GS,i,t-1}+f_{gs,in,i,t}-f_{gs,out,i,t}$$

(8)

The limit constraints are as follows:

$$S_{GS,i,min}\le S_{GS,i,t}\le S_{GS,i,max}$$

(9)

$$f_{G,i,min}\le f_{G,i,t}\le f_{G,i,max}$$

(10)

The ramping constraint stands for as follows:

$$-S_{GSdn,i}\le S_{GS,i,t}-S_{GS,i,t-1}\le S_{GSup,i}$$

(11)

2.3. Electric–Gas Coupling Unit Model

The consumption of the GT and P2G characteristic equation is as follows [24].

$$H_{g,i,t}={\alpha }_{g,i}+{\beta }_{g,i}{P}_{GT,i,t}+{\gamma }_{g,i}{P}_{GT,i,t}^2$$

(12)

$$f_{GT,i,t}=H_{g,i,t}/G_{HV}$$

(13)

$$f_{P2G,i,t}={\eta }_{P2G}{P}_{P2G,i,t}/G_{HV}$$

(14)

To ensure the normal operation of the electric–gas coupling unit, the system constraints are as follows:

$$0\le P_{P2G,i,t}\le P_{P2G,i,max}$$

(15)

$$0\le P_{GT,i,t}\le P_{GT,i,max}$$

(16)

2.4. IEGS Model Operating Cost

The total operating cost of the IEGS consists of the operating cost of the electric power system, the natural gas system, and the electric–gas coupling unit. For the IEGS in this paper, the total operating cost is minimized to achieve power system, natural gas system, and coupling unit costs.

$$minF=F_E+F_{NG}+F_{P2G}+F_{GT}$$

(17)

The power system operation costs consider the ES operation losses cost and the generation cost of conventional generating units, ignoring the operation losses cost of PV and wind power.

$$F_E={\displaystyle \sum }_{t=1}^T\left[\begin{array}{c}{\displaystyle \sum }\left(a_i{P}_{G,i,t}^2+b_i{P}_{G,i,t}+c_i\right)+\\ {\displaystyle \sum }{C}_{ES,i}{P}_{ES,i,t}\end{array}\right]$$

(18)

The natural gas system operation costs consider the GS operation losses cost and the generation cost of c natural gas sources.

$$F_{NG}={\displaystyle \sum }_{t=1}^T\left[\begin{array}{c}{\displaystyle \sum }{C}_{G,i}{f}_{G,i,t}+\\ {\displaystyle \sum }{C}_{GS,i}{S}_{GS,i,t}\end{array}\right]$$

(19)

The electric–gas coupling unit operation costs consider the operation losses cost, ignoring the consumed electricity and natural gas cost.

$$F_{P2G}={\displaystyle \sum }_{t=1}^T{C}_{P2G,i}{P}_{P2G,i,t}$$

(20)

$$F_{GT}={\displaystyle \sum }_{t=1}^T{C}_{GT,i}{P}_{GT,i,t}$$

(21)

3. Cloud-Edge Computing Architecture

3.1. Cloud-Edge Computing Architecture for IEGS

Instead of a centralized control structure where all computing tasks are performed by a central computing platform, cloud-edge computing is to realize unified dispatch and optimization by setting edge server and cloud server and taking advantage of the synergistic advantages of cloud computing and traditional edge calculation.

The IEGS architecture based on cloud-edge computing is shown in Figure 2. The IEGS deploys servers on their respective edges. Edge servers are mainly distributed between the cloud platform and physical devices, especially near the devices to perform device protocol conversion, data collection, storage analysis, online simulation, real-time control, and other functions, and at the same time to efficiently communicate and cooperate with the cloud platform. The cloud server collects required data, performs storage analysis, and implements IEGS optimization and dispatch.

Combined with Figure 2, the IEGS dispatch process under cloud-edge computing architecture is as follows:

• The IEGS edge computing platform collects relevant equipment data, carries out distributed training calculation and assignment of dispatching tasks, and then performs task unloading assignment to complete the upload of the response required by the cloud computing platform.
• After global dispatch, the cloud computing platform sends dispatch instructions to each edge server, and the edge computing platform controls the device to complete dispatch.

3.2. Decomposition of the IEGS

In order to meet the privacy protection requirements, no private data are exchanged between the power system and the natural gas system, and the electric–gas coupling unit needs to be decoupled. As shown in Figure 1, the electric power system, the natural gas system, and the electric–gas coupling unit constitute the IEGS. As shown in Figure 2, the IEGS is divided into a power system and natural gas system, and the edge intelligence perform distributed computing to achieve local optimization goals. Each edge performs distributed computing to make dispatch decisions with the goal of maximizing its own response gain. We set up an IEGS dispatch model including action space, state space, and reward functions. After the separation of IEGS, the consumption characteristic equations of GT and P2G, including operating constraints and operating costs, are transferred to the natural gas system and the power system, respectively. The IEGS can only control the local units separately, and cannot control the output of coupling units. Therefore, we choose to introduce dummy variables in the IEGS to replace the output variables of coupling units and move the coupling unit into the system where its input variables reside. Since the tear is from the same variable, the input variable and output variable of the coupling unit should be equivalent: that is, the following constraints can be satisfied:

$$P_{pre,i,t}=P_{GT,i,t}$$

(22)

$$f_{pre,i,t}=f_{P2G,i,t}$$

(23)

4. Cloud-Edge Computing Method

4.1. Cloud-Edge Computing Algorithm

The MADDPG algorithm is a natural extension of the DDPG algorithm in multi-agent systems. A stochastic strategy is adopted for actors and a deterministic strategy for critics. The behavior value function and deterministic strategy are approximated by deep neural networks. The algorithm does not need to know the dynamic model of the environment and special communication requirements, and can give the optimal action by using only the local information through learning the optimal strategy.

At each time step, for each agent $i$, select and execute the action $a_i$, obtain the reward $r_i$ and the new environment state $s_i^{\prime }$, and store the respective experience pool ${\mathcal{D}}_i$ [22].

$$a_i=\mu \left(s_i|{\theta }^{\mu }\right)+{\xi }_t$$

(24)

To train the main network, the critic network is updated by randomly sampling a sufficient number of $B$ multi-digit groups $\left(s_i,a_i,r_i,s_i^{\prime }\right)$ from the experience pool ${\mathcal{D}}_i$.

Minimize Loss function:

$$Loss=\frac{1}{B}{\displaystyle \sum }_i{\left(\begin{array}{c}{r}_i+ϵQ_i^{\prime }\left(s_i^{\prime },{\mu }^{\prime }\left(s_i^{\prime }|{\theta }^{{\mu }^{\prime }}\right)|{\theta }^{Q^{\prime }}\right)\\ -Q\left(s_i,a_i|{\theta }^Q\right)\end{array}\right)}^2$$

(25)

Update the Critic network, sampling gradient update strategy network:

$$\begin{array}{c}{\nabla }_{{\theta }^{\mu }}J\left({\theta }^{\mu }\right)=\\ \frac{1}{B}\left[{\displaystyle {\displaystyle \sum }_i}{\nabla }_{a}Q\left(s_i,a_i|{\theta }^Q\right)\cdot {\nabla }_{{\theta }^{\mu }}\mu \left(s_i|{\theta }^{\mu }\right) \right]{\left.\right|}_{a_i=\mu \left(s_i\right)}\end{array}$$

(26)

Then, the target parameters of strategy and value networks with a joint soft update:

$${\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+\left(1-\tau \right){\theta }^{{\mu }^{\prime }}$$

(27)

$${\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+\left(1-\tau \right){\theta }^{Q^{\prime }}$$

(28)

4.2. IEGS Dispatch Strategy Based on Cloud-Edge Computing

The IEGS dispatch strategy based on cloud-edge computing is shown in Figure 3. Firstly, the edge computing platform carries out distributed training and set agents. After completing the local optimization calculation, coupling unit data information and reinforcement learning parameters are transmitted to the cloud. The cloud server realizes the optimization dispatch, and then sends instructions to the respective areas. It is difficult for the centralized system to collect detailed information on distributed resources because of the privacy of the owner. In this architecture, the edge agents only need to transfer a small amount of information after calculating, according to the locally collected information, to realize the cooperative solution of the problem, which avoids the transmission of a lot of private data in the dispatch process. This method provides data privacy protection for the IEGS while reducing transmission pressure.

4.3. Power System Edge Dispatch Strategy

In this paper, the power of the generator, the power of P2G, and the power of energy storage (ES) constitute the action spaces.

$$A_E=\left\{P_G,P_{P2G},P_{ES}\right\}$$

(29)

We use the power system load, wind power output, and PV output as the state spaces. In addition, the action of the previous moment is selected as part of the state spaces.

$$S_E=\left\{P_L,P_w,P_{pv},A_E\right\}$$

(30)

To reduce the fitting difficulty and improve the accuracy and convergence stability, this paper chooses to use a linear penalty form in the edge dispatch strategy. The uniform expression form of the penalty term is as follows:

$${\phi }_i={\beta }_i\ast \left[\begin{array}{c}\left(v-v_{max}\right)f\left(v-v_{max}\right)+\\ \left(v_{min}-v\right)f\left(v_{min}-v\right)\end{array}\right]$$

(31)

$$f\left(x\right)=\left\{\begin{array}{c}0,\hspace{1em}x\ge 0\\ 1,\hspace{1em}x<0\end{array}\right.$$

(32)

The corresponding constant coefficients are set according to the different transgression penalties.

The reward function of the power system should include the power system operation cost, the P2G operation costs, and the operation unit out-of-bounds penalty.

$$F_E{}^{\prime }=-\left(F_E+F_{P2G}+{\phi }_G+{\phi }_{ES}+{\phi }_{P2G}+{\phi }_{\Delta ES}\right)$$

(33)

4.4. Natural Gas System Edge Dispatch Strategy

In this paper, the action spaces include the GT power, the flow of gas source action, and GS.

$$A_L=\left\{P_{GT},f_G,S_{GS}\right\}$$

(34)

We use the natural gas loads as the state space. Just like the power system, the action of the previous moment is selected as part of the state space.

$$S_L=\left\{f_L,A_L\right\}$$

(35)

The reward function of the natural gas system should include the natural gas system operation cost, the GT operation costs, and the operation unit out-of-bounds penalty.

$$F_L{}^{\prime }=-\left(F_L+F_{GT}+{\phi }_{GT}+{\phi }_{S_{GS}}+{\phi }_{f_G}+{\phi }_{\Delta S_{GS}}\right)$$

(36)

4.5. Cloud Dispatch Strategy

The cloud agent obtains the boundary information and reinforcement learning parameters of the edge agents. Cloud dispatch strategy needs to meet global operation constraints and formulate global reward functions

Formulas (26) and (27) constitute global power system and natural gas system operation constraints, respectively. Since the local optimal dispatch model adopts dummy variables to calculate the tidal balance, the edge does not need to send the information of all operating unit data of the system to the cloud. The cloud dispatch strategy penalty term adopts the form of a step penalty.

For any constraint shown as:

$$g\left(x_i\right)\ge 0$$

(37)

Set the penalty term as:

$${\psi }_i=\left\{\begin{array}{c}0,\hspace{1em}g\left(x_i\right)=0\\ {\alpha }_i,\hspace{1em}g\left(x_i\right)>0\end{array}\right.$$

(38)

Since the edge reward function takes a negative value, the global penalty term value also takes a negative value.

For the power system agent, the cloud reward function includes the power system edge reward function, and the global power system operation constraint penalty:

$$r_E=\left(F_E{}^{\prime }+{\psi }_E\right)$$

(39)

For the natural gas system smart body, the reward function includes the natural gas edge reward function, and the global natural gas system operation constraint penalty:

$$r_L=\left(F_L{}^{\prime }+{\psi }_L\right)$$

(40)

In this paper, the multiple intelligences are cooperative and synergistic, so the environment feeds the same global reward to each intelligence per time period.

$$r=\left(r_E+r_L\right)$$

(41)

The above IEGS dispatch strategy based on cloud-edge computing is trained by the algorithm shown in

Table 2. Pseudo-code of cloud-edge computing algorithm based on MADDPG.

Cloud-Edge Computing Algorithm Based on MADDPG

for episode = 1 to M do Initialize IEGS state $\mathit{s}$ and random process $\mathit{\xi }$ for $\mathit{t}=\mathbf{1}$ to max-episode-length do    for each agent $\mathit{i}$, select action ${\mathit{a}}_{\mathit{i}}={\mathit{\mu }}_{{\mathit{\theta }}_{\mathit{i}}}\left({\mathit{o}}_{\mathit{i}}\right)+{\mathit{\xi }}_{\mathit{t}}$ w.r.t. the current policy and exploration    Execute action ${\mathit{a}}_{\mathit{i}}$ and observe reward ${\mathit{r}}_{\mathit{i}}$ and new state ${\mathit{s}}_{\mathit{i}}^{\prime }$    Observe global reward $\mathit{r}=\left({\mathit{r}}_{\mathbf{1}}+{\mathit{r}}_{\mathbf{2}}\right)$    Store $\left({\mathit{s}}_{\mathit{i}},{\mathit{a}}_{\mathit{i}},\mathit{r},{\mathit{s}}_{\mathit{i}}^{\prime }\right)$ in replay buffer ${\mathcal{D}}_{\mathit{i}}$    $\mathit{s}\leftarrow {\mathit{s}}^{\prime }$    for each agent $\mathit{i}$ do      Sample a random minibatch of $\mathit{S}$ samples $\left({\mathit{s}}_{\mathit{i}}^{\mathit{j}},{\mathit{a}}_{\mathit{i}}^{\mathit{j}},{\mathit{r}}^{\mathit{j}},{{\mathit{s}}^{\prime }}_{\mathit{i}}^{\mathit{j}}\right)$ from ${\mathcal{D}}_i$ Set                    ${\mathit{y}}_{\mathit{k}}^{\mathit{i}}={\mathit{r}}_{\mathit{k}}^{\mathit{i}}+\mathit{ϵ}{\mathit{Q}}_{\mathit{k}}^{\prime }\left({\mathit{s}}_{\mathit{k}}^{\mathit{i}+\mathbf{1}},{\mathit{a}}^{\mathit{i}+\mathbf{1}}|{\mathit{\theta }}_{\mathit{k}}^{{\mathit{Q}}^{\prime }}\right){\left.\right|}_{{\mathit{a}}^{\mathit{i}+\mathbf{1}}={\mathit{\mu }}_{\mathit{k}}^{\prime }\left({\mathit{o}}_{\mathit{k}}^{\mathit{i}+\mathbf{1}}\right)}$      Update critic of each agent $\mathit{i}$ by minimizing the loss by (25)      Update actor of each agent $\mathit{i}$ by (26)    end for    Update target network parameters for each agent $\mathit{i}$ by (27) and (28) end for end for

, and the flow chart of the training algorithm based on cloud-edge computing is shown in Figure 4.

4.6. Case Study

4.6.1. Parameter Setting

To test cloud-edge computing’s effectiveness for IEGS dispatch strategy, an IEGS consisting of an IEEE 9-node power system and an 8-node natural gas system is proposed, as shown in Figure 5 in this paper. The IEEE 9 node model consists of nine nodes, including three generator nodes, three load nodes, three transformer nodes, and one transmission line node. The IEEE 8-node natural gas model consists of 8 nodes, including 2 gas source nodes, 2 pressure station nodes, and 4 gas pipelines. The power system includes three generator units and the natural gas system includes two gas sources. The power system includes wind power, PV, and ES, and the P2G connects to the natural gas system with a GS. The system parameter values are shown in

Table 3. System parameter values.

System Parameter	Numerical Value
${\mathit{\eta }}_{\mathit{s}\mathit{o}\mathit{c}}$	0.95
${\mathit{\eta }}_{\mathit{P}\mathbf{2}\mathit{G}}$	0.6
$\mathit{S}\mathit{O}{\mathit{C}}_{\mathit{e}\mathit{s},\mathit{u}\mathit{p}}$	0.9
$\mathit{S}\mathit{O}{\mathit{C}}_{\mathit{e}\mathit{s},\mathit{d}\mathit{n}}$	0.1
${\mathit{E}}_{\mathit{e}\mathit{s},\mathit{m}\mathit{a}\mathit{x}}$	50 $\mathrm{MWh}$
${\mathit{P}}_{\mathit{P}\mathbf{2}\mathit{G},\mathit{m}\mathit{a}\mathit{x}}$	50 $\mathrm{MW}$
${\mathit{P}}_{\mathit{G}\mathit{T},\mathit{m}\mathit{a}\mathit{x}}$	50 $\mathrm{MW}$
${\mathit{S}}_{\mathit{s},\mathit{m}\mathit{a}\mathit{x}}$	0.5
${\mathit{G}}_{\mathit{H}\mathit{V}}$	39 $\mathrm{MJ}/{\mathrm{m}}^3$
${\mathit{\alpha }}_{\mathit{g},\mathit{i}}$	0
${\mathit{\beta }}_{\mathit{g},\mathit{i}}$	5.5 ${\mathrm{m}}^3/\mathrm{s}\left(100 \mathrm{MW}\right)$
${\mathit{\gamma }}_{\mathit{g},\mathit{i}}$	0.16 ${\mathrm{m}}^3/\mathrm{s}{\left(100 \mathrm{MW}\right)}^2$

[24]. The conventional generating unit parameter values are shown in

Table 4. Conventional generating unit parameter values.

$\mathit{i}$	${\mathit{a}}_{\mathit{i}}$	${\mathit{b}}_{\mathit{i}}$	${\mathit{c}}_{\mathit{i}}$	${\mathit{P}}_{\mathit{G},\mathit{i},\mathit{m}\mathit{a}\mathit{x}}$	${\mathit{P}}_{\mathit{G},\mathit{i},\mathit{m}\mathit{i}\mathit{n}}$
1	0.021	40.14	85.10	100 $\mathrm{MW}$	60 $\mathrm{MW}$
2	0.020	41.62	84.10	105 $\mathrm{MW}$	65 $\mathrm{MW}$
3	0.024	39.10	70.90	120 $\mathrm{MW}$	55 $\mathrm{MW}$

[24]. The length of the system dispatch period is 24 h, and the interval between 2 adjacent periods is 1 h.

4.6.2. Experimental Result and Analysis

The power grid system and the natural gas system constitute the edge agent, independently. G1, G2, G3, P2G, and ES constitute the power system action spaces. The power system load, wind power, and PV are the power system state spaces. In addition, the action of the previous moment is selected as part of the power system state spaces. The gas system action spaces include GT, W1, W2, and GS. We use the natural gas loads as the state space. Just like the power system, the action of the previous moment is selected as part of the state space. The cloud agent obtains the boundary information and reinforcement learning parameters of the edge agents. The cloud agent needs to meet global operation constraints and formulate global reward functions.

The network consists of the input layer, hidden layer, and output layer. The number of hidden layers in the strategy network is 3, and the number of neurons is 500, 128, and 32 in order. The value function network contains 3 hidden layers with 200, 100, and 50 neurons. The activation function of the output layer of the Actor network is set as a hyperbolic tangent function (tanh) to restrict the action output to [−1, 1] in order to prevent the gradient from disappearing and reduce the learning efficiency of the neural network. The number of iterations was set to 5000. The corresponding convergence curve is shown in Figure 6.

From Figure 6, we can see that the cloud-edge computing algorithm can achieve convergence and converge around the 2000th iteration.

To test the cloud-edge computing’s performance, the MADDPG algorithm is selected for comparison. The comparison result is shown in Figure 7.

From Figure 7, compared with MADDPG, cloud-edge computing has improved the execution time, and the reward curve fluctuates less and convergence is more stable.

The objective of this paper is to optimize the dispatch cost of an IEGS. In order to verify the optimization of cloud-edge computing methods for IEGS dispatch problems, dispatch methods based on MADDPG, DDPG, and PSO algorithms are selected for the comparison of dispatch objectives.

Table 5. Cost of dispatch cost of different algorithms.

Dispatch Algorithms	Dispatch Cost
cloud-edge	4.81 × ${10}^6$ RMB
MADDPG	4.86 × ${10}^6$ RMB
DDPG	4.82 × ${10}^6$ RMB
PSO	4.85 × ${10}^6$ RMB

shows the dispatch cost of the IEGS of different algorithms.

From Table 5, we can prove that the cloud-edge computing method has feasibility for the IEGS and optimizes the IEGS dispatch cost; the cost is reduced by 0.21% to 1.03% compared with other methods.

The total power system load, wind, and PV power are shown in Figure 8A, and the total natural gas system load is shown in Figure 8B.

The IEGS dispatch results based on cloud-edge computing of the power system and natural gas system in one day are shown in Figure 9A,B, respectively.

From Figure 9, at the beginning of the period, due to the low level of electric load, the output of generating units is also at a low level, and the GTs hardly produce any power, the excess wind power produces natural gas through the P2G, and part of natural gas is directly used and the remainder is stored. In the period of 20:00–23:00, as the electric load decreases, the wind power goes up, the GT power goes down, the P2G device is restarted, and the surplus energy is stored.

4.7. Discussion

The cloud-edge method can solve the IEGS dispatch problem, which provides ideas and guidance for solving the complex power system dispatch problem. However, there are some shortcomings in this paper. In real power systems, the server deployment to meet cloud-edge computing often has high deployment requirements, such as better edge device performance and larger cloud-edge communication bandwidth. The cloud-edge computing method requires normal communication. In the interrupted mode of cloud edge communication, the cloud edge cannot transmit information and realize real-time dispatch.

In addition, the scale of the IEGS affects the results of response dispatch. As the scale continues to expand, the time delay generated by the cloud server response dispatch increases, and the different response rates and response speeds of different systems also affect the dispatch effect. Therefore, considering the economy and dynamic dispatch of multi-node complex integrated energy systems will be the focus of future research.

Meanwhile, in order to better meet the needs of actual operation, considering a capacity configuration scheme for high renewable energy penetration scenarios in reference [25] and the power system challenged by emergency mentioned in reference [26], and considering the ability of the cloud-edge computing method to adapt to uncertainty and emergency will be the next research direction.

5. Conclusions

Facing the dispatch problem of IEGSs, we propose a dispatch method based on cloud-edge computing, and design the action space, state space, and reward mechanism of agents in power systems and natural gas systems in detail. The edge computing platform completes the data collection and preliminary calculation, and the cloud computing platform realizes the collaborative calculation of the integrated system. The proposed cloud-edge computing method is used to test and analyze the results in the IEGS model, which verifies the effectiveness of the proposed method. As can be seen in Table 5 and Figure 8, we can generate reasonable IEGS dispatch policies with reasonable dispatch cost, and the cost is reduced by 0.21% to 1.03% compared with other methods. As can be seen in Figure 6, compared with MADDPG, the cloud-edge computing convergence speed is faster, the reward curve fluctuation is less, and the convergence is more stable. Moreover, compared with the overall centralized calculation method, this method not only meets the data privacy protection requirements between different systems, but also reduces the data transmission volume and avoids excessive transmission pressure caused by data communication.