Optimal Operation of Virtual Power Plants Based on Stackelberg Game Theory

Abstract

As the scale of units within virtual power plants (VPPs) continues to expand, establishing an effective operational game model for these internal units has become a pressing issue for enhancing management and operations. This paper integrates photovoltaic generation, wind power, energy storage, and constant-temperature responsive loads, and it also considers micro gas turbines as auxiliary units, collectively forming a typical VPP case study. An operational optimization model was developed for the VPP control center and the micro gas turbines, and the game relationship between them was analyzed. A Stackelberg game model between the VPP control center and the micro gas turbines was proposed. Lastly, an improved D3QN (Dueling Double Deep Q-network) algorithm was employed to compute the VPP’s optimal operational strategy based on Stackelberg game theory. The results demonstrate that the proposed model can balance the energy complementarity between the VPP control center and the micro gas turbines, thereby enhancing the overall economic efficiency of operations.

1. Introduction

A virtual power plant (VPP) is an energy management system realized through advanced information and communication technologies. Reference [1] proposes integrating distributed generation, controllable loads, and storage devices for energy management and scheduling, thereby reducing the impact of the uncertainties associated with distributed generation on the stable operation of the power system. As the scale of distributed generation continues to expand, VPPs offer a broader space for improving energy utilization efficiency in the electricity market.

The control problem of peak shaving demand response (DR) participation in a virtual power plant involves the effective integration of the downward adjustment potential of a large number of distributed energy resources (DERs) to dynamically track predefined target load profiles. These target load profiles can be determined based on settlement results from electricity ancillary service markets [2], customized operational strategies [3], and discrepancies between DR baselines and actual response capacities [4]. This control problem can be classified into two approaches: open-loop and closed-loop methods. Conventional open-loop control methods are suited for centrally dispatching energy storage systems, gas turbines, and other DERs with strong control capabilities within VPP operations [5]. DERs with weaker control authority, such as air conditioning units, are more suitable for participating in peak shaving DR using adaptive closed-loop control methods that require rapid adaptation to changes. It should be noted that the effectiveness of closed-loop control methods may be adversely affected by uncertainties such as control delays and model-measurement errors [6].

Reference [7] proposes that many scholars both domestically and internationally have conducted extensive research on demand-side management inspired by game theory. For instance, reference [8] considers the degree of demand response while setting electricity prices at the VPP control center, ultimately establishing a Stackelberg game optimization model for the internal units of the VPP. Reference [9] proposes the concepts of “price-based quantity” and “quantity-based price” in the context of electric vehicle charging management, modeling the pursuit of maximum benefits by users and electricity retailers as a Stackelberg game. Reference [10] presents a real-time optimal control model for demand-side response based on the Stackelberg game, with electricity retailers as leaders and various loads as followers, obtaining optimal strategies for all participants. Reference [11] establishes a two-tier Stackelberg game model for market transactions involving electric vehicle operators and distributed generators. Reference [12] constructs a two-stage Stackelberg model considering the operational hierarchy of the electricity market, involving the upper-level market, electricity retailers, and users. However, these studies mainly focus on demand-side management without game modeling for energy-side units, which is crucial for reducing power wastage on the energy side.

Deep reinforcement learning (DRL) provides powerful algorithms for solving problems in power system research. Reference [13] proposes an optimization method for the operation of a photovoltaic-storage combined power plant’s energy storage system in the spot market environment based on DQN (Deep Q-network), considering deviation penalty costs, energy revenue, and frequency regulation ancillary service revenue, to maximize the economic benefits of the PV-storage combined power plant. Reference [14] introduces the DDPG (Deep Deterministic Policy Gradient) algorithm, utilizing the experience replay mechanism and proportionally regulating perception learning to effectively enhance algorithm stability, successfully applied to the dynamic energy scheduling of power systems. References [15,16] propose the application of both the DDQN (Double Deep Q-network) algorithm and the Dueling-DQN (Dueling Deep Q-network) algorithm to the optimal scheduling of power systems. Additionally, reference [17] uses the D3QN (Dueling Double Deep Q-network) algorithm for real-time energy management, but this algorithm may encounter strategy oscillation issues in continuous and complex decision-making environments, primarily because it relies on extensive exploration to maintain performance, potentially leading to instability in rapidly changing environments.

This paper designs a VPP containing energy storage, micro gas turbines, wind farms, photovoltaics, and temperature-controlled loads. Under the framework of setting electricity sales and purchase price strategies at the VPP control center with energy storage, considering the flexibility of micro gas turbines and the dispatchability of demand response resources, a VPP operation optimization model based on the Stackelberg game is established. An improved D3QN algorithm is proposed to adapt to the interactive supply and demand within the VPP, and the economic efficiency of the proposed model is verified through case studies.

The remaining sections of this work are organized as follows: Section 2 presents the modeling of sources, loads, and storage within the virtual power plant. Section 3 introduces the Stackelberg game model, focusing on the game-theoretic modeling between the virtual power plant and gas turbines. Section 4 employs deep reinforcement learning to solve the game model, specifically attempting to integrate the D3QN algorithm with the electricity market to address strategic interactions in the virtual power plant context. In Section 5, a case study is used to analyze and compare the algorithm, thereby validating its practical effectiveness. Finally, Section 6 summarizes the main conclusions of this paper and outlines potential future research challenges.

2. Framework of Virtual Power Plant with Energy Storage

The number of loads within a virtual power plant jurisdiction can be quite large, making it impractical for the dispatch center to manage each load individually. A more feasible approach to categorize the loads and model different types of loads separately is proposed in Reference [18]. To streamline optimization, the load user groups within the VPP are equivalently considered as non-dispatchable loads and temperature-responsive loads, with the latter divided into air conditioning loads and water heater loads. In addition to distributed generation and loads, the VPP also includes energy storage and gas turbines, as shown in Figure 1.

The operation of each entity within the VPP is as follows: The VPP control center predicts the output of distributed generation within the virtual power plant for the next day based on historical data and sets reasonable electricity purchase and sale prices in advance. It then selects the set temperatures for air conditioning and water heater loads based on weather information and real-time electricity prices. Finally, the micro gas turbine optimizes its output throughout the day based on the electricity prices provided by the VPP to maximize its own revenue.

This paper investigates the energy optimization and scheduling problem of a VPP. The VPP is connected to the main grid, allowing it to purchase electricity from the main grid in case of a power shortage and sell electricity to the main grid in case of a power surplus. This effectively addresses the supply–demand imbalance issues during the operation of the VPP system.

2.1. Distributed Generation Units

Distributed generation components are the core elements of a VPP. As the penetration rate of renewable energy generation within the VPP continues to increase, the uncertainty and variability of its power output present challenges for the economic operation of the VPP. The distributed generation data used in this paper are processed from real-time power generation data of typical photovoltaic and wind power plants in the East China region.

2.2. Micro Gas Turbine Power Generation

On the power supply side, there is a micro gas turbine primarily fueled by natural gas to provide electrical energy to the user side. The relationship between its fuel cost and the output electrical power of the unit in the $t-th$ time period within a day can be expressed as

$$E_t^{MT}={\displaystyle \frac{q_{gas}}{Q_{gas}{\eta }^{MT}}}{P}_t^{MT}$$

(1)

$$P_{MT}^{\min }\le P_t^{MT}\le P_{MT}^{\max }$$

(2)

$$P_{MT}^{down}\le P_t^{MT}-P_{t-1}^{MT}\le P_{MT}^{up}$$

(3)

In the equations, $E_t^{MT}$ represents the generation cost of the micro gas turbine during time period $t$. $P_t^{MT}$ represents the electricity generated by the micro gas turbine during time period $t$. $q_{gas}$ represents the unit price of natural gas. $Q_{gas}$ represents the lower heating value of natural gas. ${\eta }^{MT}$ represents the efficiency of the gas turbine. $P_{MT}^{\min }$ represents the minimum active power of the generator. $P_{MT}^{\max }$ represents the minimum active power of the generator. $P_t^{MT}$ and $P_{t-1}^{MT}$ represent the power output of the gas turbine at the current moment and the previous moment, respectively. $P_{MT}^{down}$ and $P_{MT}^{up}$ represent the upper and lower limits of the ramping power of the gas turbine.

2.3. Constant-Temperature Control Load

2.3.1. Air Conditioning Load Model

The power variation of the air conditioning system is generally related to the temperature difference between indoors and outdoors. To model the air conditioning system, including the influence of outdoor temperature on its performance, a variation model dependent on outdoor temperature can be introduced. This type of model more accurately reflects the operational efficiency of the air conditioning system under different environmental temperatures. A common approach is to represent $\lambda$ as a function of outdoor temperature:

$$Q_{load}=k\cdot (T_{in}-T_{out})$$

(4)

$$P_t^{AC}(T_{out})={\displaystyle \frac{Q_{load}}{\lambda (T_{out})}}$$

(5)

$$\lambda (T_{out})=a+b\cdot T_{out}+c\cdot T_{out}^2$$

(6)

In these equations, $Q_{load}$ represents the heat or cold load caused by the temperature difference between indoors and outdoors. The calculation method may vary depending on the cooling or heating requirements. $k$ is the heat transfer coefficient, representing the insulation performance of the building, with units of kW/°C. $T_{in}$ is the indoor temperature. $P_t^{AC}(T_{out})$ represents the power consumption of the air conditioning load at time $t$. $\lambda (T_{out})$ is the performance coefficient of the air conditioning system, which varies with outdoor temperature $T_{out}$. $a$, $b$, and $c$ are model parameters obtained through fitting with actual data. $T_{out}$ represents the outdoor temperature.

The air conditioning switch is set to

$${S_t}^{AC}=\left\{\begin{array}{cc}0& T_{in}>T_{set}^{+}\\ 1& T_{in}<T_{set}^{-}\\ S_{t-1}^{AC}& otherwise\end{array}\right.$$

(7)

In this equation, $S_t$ and $S_{t-1}$ represent the states of the air conditioning system at the current moment and the previous moment, respectively. $T_{set}^{+}$ represents the upper limit of the set temperature, while $T_{set}^{-}$ represents the lower limit of the set temperature. The power of the air conditioning can be obtained as

$$P_t^{AC}=S_t{P}_t^{AC}(T_{out})$$

(8)

An air conditioning energy consumption model has been obtained that takes into account the influence of outdoor temperature. This model can calculate the power consumption of the air conditioning system based on different outdoor temperatures, thereby assisting the virtual power plant in optimizing its dispatching strategy to adapt to various environmental conditions and operational efficiencies.

2.3.2. Water Heater Load Model

Electric water heaters are primarily used for bathing and kitchen water in daily life, and their internal load must accurately reflect the water usage habits at different times. Throughout the day, peak periods of hot water demand occur during morning and evening bathing peaks and meal preparation periods, during which the electricity consumption of the water heater noticeably increases.

Probability density functions for water heater usage are established for different time periods, and the overall probability density is calculated by integrating the probability density functions across different time periods. This probability density can describe the variation in the probability of water heater usage at different times throughout the day, as shown in Equation (9):

$$\begin{array}{cc}{f}_i(x)={\displaystyle \frac{1}{\sqrt{2\pi }{\sigma }_i}}\exp (-{\displaystyle \frac{{(x-{\mu }_i)}^2}{2{{\sigma }_i}^2}})& i=1,2,3,4,5\end{array}$$

(9)

$$\left\{\begin{array}{l}{f}_t(x)={\displaystyle \sum _{i=1}^5{\rho }_i\cdot f_i(x)}\hfill \\ {\rho }_1+{\rho }_2+{\rho }_3+{\rho }_4+{\rho }_5=1\hfill \end{array}\right.$$

(10)

In these equations, $f_i(x)$ $i=1,2,3,4,5$ represents the probability density of water usage during different time periods, including morning bathing, breakfast, lunch, dinner, and evening bathing.

Similar to air conditioning switch settings, electric water heater switches can be configured as follows:

$${S_t}^{WH}=\left\{\begin{array}{cc}0& T_{in}>T_{set}^{+}\\ 1& T_{in}<T_{set}^{-}\\ S_{T-1}^{WH}& otherwise\end{array}\right.$$

(11)

In this equation, $S_t$ and $S_{t-1}$ represent the states of the water heater system at the current moment and the previous moment, respectively. $T_{set}^{+}$ represents the upper limit of the set temperature, while $T_{set}^{-}$ represents the lower limit of the set temperature.

2.4. Energy Storage Battery Model

The VPP control center directly controls the charging and discharging actions of the energy storage battery component. The charging and discharging constraints of the energy storage battery are as follows:

$$S_t=S_{t-1}+{\eta }^{+}{e}_t^{+}-e_t^{-}/{\eta }^{-},\forall t$$

(12)

$$\left\{\begin{array}{cc}0\le e_t^{+}\le u_t{R}_m^{+}& \forall t\\ 0\le e_t^{-}\le (1-u_t)R_m^{-}& \forall t\\ 0\le S_t\le S_{\max }& \forall t\\ S_1=S_T& \end{array}\right.$$

(13)

In these equations, $S_t$ represents the state of charge of the energy storage battery at time $t$. $e_t^{+}$ and $e_t^{-}$ represent the charging and discharging amounts of the energy storage device at time $t$. $u_t$ is a Boolean variable indicating the state of the energy storage device at time $t$. $R_m^{+}$ and $R_m^{-}$ represent the charging power limits. ${\eta }^{+}$ and ${\eta }^{-}$ represent the efficiency of the energy storage charging and discharging equipment. $S_{\max }$ represents the maximum capacity of the energy storage. $S_1$ represents the state of charge of the battery at the beginning of the day. $S_T$ represents the state of charge of the battery at the end of the day.

3. Real-Time Interactive Stackelberg Game Model

3.1. Stackelberg Game Framework

To study the game optimization operation of internal units within a VPP, this paper establishes a Stackelberg game model featuring the VPP control center and a micro gas turbine as the main entities, as shown in Figure 2.

The upper-level leader is the VPP control center, while the lower-level follower is the micro gas turbine. Both the VPP control center and the micro gas turbine aim to maximize their respective profits. The decision variable for the VPP control center is the electricity purchase price from the micro gas turbine, and the decision variable for the micro gas turbine is its own power generation output.

The upper-level leader, the VPP control center, is constrained by the status of temperature-responsive loads, the operation status of energy storage, and the electricity purchase price from loads. The lower-level follower, the micro gas turbine, is constrained by its maximum and minimum operational power, as well as its maximum ramp rate.

3.2. Objective Function Design

Economic dispatch aims to maximize expected returns. To achieve the economic optimal operation control of the VPP control center, this paper takes the maximum operating profit of the VPP control center as the objective function for the economic optimization scheduling of the VPP. The operational costs for the VPP control center mainly include the cost of purchasing electricity from the micro gas turbine, the cost of distributed generation, and the cost of purchasing electricity from the main grid. The revenue of the VPP primarily comes from the profits gained from meeting various power load demands and selling electricity to the main grid. Therefore, the objective function of the VPP control center is set as follows:

$$\begin{array}{c}\max {\displaystyle \sum _{t=0}^T{F}_{vpp,t}}=p_{load}{L}_{load}+p_{tcl}{\displaystyle \sum _{i=0}^{N_{tcl}}{L}_{tcl}^{i,t}}+p_t^s{E}_t^s+\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{p}_{soc}(e_t^{+}+e_t^{-})-p_{\cos t}{G}_t-p_t^b{E}_t^b\end{array}$$

(14)

In this equation, $F_{vpp,t}$ represents the profit of the VPP control center at time $t$. $p_{load}$ is the electricity price for price-responsive loads. $L_{load}$ represents the output status of the daily load. $p_{tcl}$ is the electricity price for constant-temperature control loads. $N_{tcl}$ is the quantity of constant-temperature control loads. $p_t^s$ is the selling price of electricity to the main grid. $p_t^b$ is the purchasing price of electricity from the main grid. $E_t^s$ is the amount of electricity sold to the main grid. $p_{soc}$ is the unit price for charging and discharging of the energy storage battery. $E_t^b$ is the amount of electricity purchased from the main grid. $p_{\cos t}$ is the generation cost of distributed generation components. $e_t^{+}$ and $e_t^{-}$ are the charging and discharging amounts of the energy storage device at time $t$. $G_t$ is the power generation of distributed generation components.

The objective function of the micro gas turbine is as follows:

$$\max F_t={\displaystyle \sum _{t=0}^T(p_t^{MT}{P}_t^{MT}}-E_t^{MT})$$

(15)

In this equation, $F_t$ represents the profit of the micro gas turbine during time period $t$. $E_t^{MT}$ represents the generation cost of the micro gas turbine during time period $t$. $P_t^{MT}$ represents the electricity generation of the micro gas turbine during time period $t$. $p_t^{MT}$ represents the purchasing price offered by the virtual power plant to the micro gas turbine during time period $t$.

4. Model Solution Based on Deep Reinforcement Learning

4.1. Problem Description

4.1.1. State Space

The state space $s_t$ contained in this paper includes the following: information regarding the relative state of charge of constant-temperature control load components; information about the power generation of distributed generation components; information about the power generation of the gas turbine unit; information about the overall power consumption of price-responsive load components; information about the state of charge of energy storage components; information about the charging and discharging prices of energy storage components; information about the prices of price-responsive load components; time information; the electricity price for purchasing from the main grid; the electricity price for selling to the main grid.

4.1.2. Action Space

The action space $a_t$ in this paper consists of four parts: In Section 2.2, action space $a_c$ for gas turbine generation can be obtained; in Section 2.3, action space $a_{tcl}$ for constant-temperature control load can be obtained; action space $a_e$ for system power shortage and action space $a_d$ for system power surplus can be obtained by interacting with the main grid. This paper defines the action space for temperature-controlled loads to include eight possible actions, which specify the adjustment of temperatures for air conditioning and water heater loads $a_t={(a_{tcl},a_c,a_d,a_e)}_t$. The gas turbine has four actions: base load mode, peak shaving mode, standby mode, and start/stop operations. The VPP control center has two actions for power shortage and two for power surplus, which specify the priority between battery charging/discharging and purchasing/selling electricity from/to the main grid. The action space in the virtual power plant comprises 128 possible potential actions.

4.1.3. Reward Function

Reinforcement learning aims to maximize expected returns. To achieve the economic optimization control of the VPP control center, this paper sets the highest operating profit of the VPP control center as the objective function for VPP economic optimization scheduling. It is considered that the operating costs of the VPP mainly include electricity trading transmission costs, wind power generation costs, and purchasing costs from the main grid, and the revenue of the VPP mainly includes profits from meeting various electricity load demands and profits from selling electricity to the main grid. Therefore, the reward function of the VPP control center in this paper is derived from Equation (14) as follows:

$$\begin{array}{c}{r}_t=p_{load}{L}_{load}+p_{tcl}{\displaystyle \sum _{i=0}^{N_{tcl}}{L}_{tcl}^{i,t}}+p_t^s{E}_t^s+\\ p_{soc}(e_t^{+}+e_t^{-})-p_{\cos t}{G}_t-p_t^b{E}_t^b\end{array}$$

(16)

The reward function for the micro gas turbine is derived from Equation (15) as follows:

$$r_t=p_t^{MT}{P}_t^{MT}-E_t^{MT}$$

(17)

4.2. Designing the Improved D3QN Algorithm

On the basis of the game model of the virtual power plant with energy storage mentioned above, this paper proposes an improved D3QN game algorithm to assist agents in finding pricing or quantification strategies that maximize their own rewards. During the learning process, the intelligent agent interacts with the environment in the improved D3QN game algorithm, selecting the best actions to achieve the optimization control objectives of the VPP system. The Q-value function can be defined as follows [17]:

$$\begin{array}{l}Q(s_t,a_t;\theta ,{\theta }_v,{\theta }_A)=V(s_t;\theta ,{\theta }_v)\\ +(A(s_t,a_t;\theta ,{\theta }_A)-{\displaystyle \frac{1}{\left|A\right|}}A(s_t,a_{t+1};\theta ,{\theta }_A))\end{array}$$

(18)

In this equation, $Q(•)$ is the output function of the Q-network algorithm. $V(•)$ is the state value function that characterizes the goodness or badness of the state. $A(•)$ is the state-action advantage function. $\theta$ represents the shared network parameters. ${\theta }_v$ and ${\theta }_A$ are the neural network parameters unique to the state value function and the action advantage function, respectively. $a_{t+1}$ is the action at time $t+1$. $1/\left|A\right|$ is the average value of the action advantage function [17].

$$\begin{array}{l}{y}_t=R_{t+1}+\gamma Q(s_{t+1},\\ \arg {\max }_{a+1}Q(s_{t+1},a_{t+1};\theta ,{\theta }_v,{\theta }_A);{\theta }^{\prime },{{\theta }^{\prime }}_v,{{\theta }^{\prime }}_A)\end{array}$$

(19)

In this equation, $r_{t+1}$ is the reward obtained by the agent at time $t+1$. $s_{t+1}$ is the state set at time $t+1$. $\gamma$ is the discount factor. $\theta$, ${\theta }_v$, and ${\theta }_A$ are parameters of the evaluation network, and the parameters $\theta$, ${\theta }_v$, and ${\theta }_A$ are continuously updated. ${\theta }^{\prime }$, ${{\theta }_v}^{\prime }$, and ${\theta }_A$ are parameters of the target network, and they are replaced with the latest evaluation network parameters every N time steps.

To enhance the exploration ability of the improved D3QN reinforcement learning agent and prevent the algorithm from converging to local optima, the concept of a noise network is proposed in Reference [19]. Noise is added to the fully connected layers of the neural network, allowing parameters to be learned in a noisy environment, thus effectively enhancing network generalization and improving algorithm robustness. Considering the forward calculation formula of the next fully connected layer as $y=\omega \bullet x+b$, the network parameters $\omega$ in $u^{\omega }+{\delta }^{\omega }\bullet {\xi }^{\omega }$ and the network parameters $b$ in $u^b+{\delta }^b\bullet {\xi }^b$ are transformed, where the network parameters $u$ and $\delta$ represent the mean and standard deviation, respectively. $\xi$ represents random noise, and $\xi \sim (0,1)$. By adding noise to the neural network parameters, the exploration capability of the reinforcement learning algorithm is effectively enhanced.

In the traditional D3QN algorithm, the optimal action is selected through strategy $\epsilon -greedy$. The agent selects the action with the highest Q-value with probability $\epsilon$ and a random action with probability $1-\epsilon$. As the number of episodes increases, the value does not increase significantly. Considering that the electricity selling price to the grid $p_t^s$ is relatively low, this paper combines strategy $\epsilon -greedy$ with the power surplus action $a_d$ to balance action exploration. When $a_d$ is true, the value of $\epsilon$ is reduced to increase the chances of favorable exploration.

Table 1. Improved D3QN game algorithm.

input: episodes, $\gamma$, $\theta$, ${\theta }_v$, ${\theta }_A$, ${\theta }^{\prime }$, ${{\theta }_v}^{\prime }$, ${{\theta }_A}^{\prime }$.

exportation: action $E_t^{+}$, $E_t^{-}$, $e_t^{+}$, $e_t^{-}$, ${\lambda }_{it}$, $q_{it}$

1.  initialize $Q(•)$, $y_t$

2.  for every episode in episodes

3.    for step $t=1$ to $T$ do

4.       The leader takes action $a_t^{vpp}$ based on $s_t$;

5.       The follower takes action $a_t$ according to $s_t$ and $a_t^{vpp}$;

6.       Observe $s_{t+1}$;

7.       Calculate $R_t$ by (15), (16);

8.       Renewal Q-value by (17), (18);

9.       $s_t\leftarrow s_{t+1}$

10.     $t=t+1$

11.  end for

12.  Every N time steps $\theta$, ${\theta }_v$, ${\theta }_A$$\to$${\theta }^{\prime }$, ${{\theta }_v}^{\prime }$, ${{\theta }_A}^{\prime }$

13. return action $a_t$

14. end for

presents the pseudocode for the improved D3QN algorithm. This table outlines the key steps and modifications implemented in the algorithm Figure 3 shows the structure of the D3QN algorithm with a noisy network.

Based on the above network structure and real-time electricity price incentive mechanism, the training rules for the VPP energy management method using the improved D3QN algorithm are as follows.

5. Example Analysis

5.1. Scenario Description

This paper employs the Python language and utilizes the TensorFlow tool to design a virtual power plant as depicted in Figure 1, to validate the feasibility and effectiveness of the proposed virtual power plant internal unit game method based on the improved D3QN algorithm.

The distributed generation data and meteorological information required for the experiments were processed from real-time power generation data of typical photovoltaic and wind power plants in the East China region. The experimental sample data were collected once per hour, covering a year of characteristic information for the components within the virtual power plant. To maintain generality, the power generation data of distributed energy components were averaged over the year, as shown in Figure 4. Figure 5 shows the price curve of the virtual power plant for selling electricity to residents.

The loads include non-adjustable loads and temperature-controlled loads, with the latter divided into air conditioning loads and water heater loads. Figure 6 shows the power load of air conditioning for one day at different set temperatures. The outdoor temperature was processed using real-time summer temperatures from a specific location, with experimental sample data collected every 5 min.

Table 2. Probability density function parameter.

Parameter	Value	Parameter	Value	Parameter	Value
${\mu }_1$	7	${\sigma }_1$	0.5	${\rho }_1$	0.3
${\mu }_2$	7.5	${\sigma }_2$	0.5	${\rho }_2$	0.1
${\mu }_3$	12.5	${\sigma }_3$	0.8	${\rho }_3$	0.15
${\mu }_4$	18	${\sigma }_4$	0.9	${\rho }_4$	0.15
${\mu }_5$	20	${\sigma }_5$	1.2	${\rho }_5$	0.3

presents the probability density parameters for water heater usage.

Table 3. Environmental parameters of virtual power plant system.

Parameter	Value
Coefficient of charging efficiency ${\eta }_{\mathrm{c}}$	0.9
Discharge efficiency coefficient ${\eta }_{\mathrm{d}}$	0.9
Maximum storage capacity of battery $B_{\max }$	500 kW
Maximum charging power of a battery $c_{\max }$	500 kW
Maximum discharge power of a battery $D_{\max }$	800 kW
The number of air conditioning loads $N_{loads}$	2000
Water heater load quantity $N_{tcls}$	2000
Distributed energy generation cost value $p_{\cos t}$	69.8 CNY/MWh
Cost of purchasing electricity from the grid $p_t^b$	650 CNY/MWh

lists other parameters in the virtual power plant system environment.

5.2. Algorithm Performance Analysis

The algorithm was run on a computer with an Intel(R) Core(TM) i5-7300HQ CPU @ 2.50 GHz and 8 GB RAM. It was implemented using Python 3.10 and TensorFlow 2.11.0 as the deep learning framework.

In the algorithm, rewards were calculated every hour and accumulated to obtain the total daily reward. The training set for the VPP control center consisted of 2000 episodes, and 10 experiments were conducted under different algorithms to obtain the average reward value. The convergence of the VPP control center during iterations is shown in Figure 7. The algorithm exhibited significant fluctuations in the first 30 episodes, primarily due to the randomness in strategy selection during this stage. Subsequently, while gradually optimizing its strategy, the VPP control center and the gas turbine engaged in a non-cooperative Stackelberg game, which dominated the reward fluctuations during this period.

As observed from the figure, the convergence range of the Dueling DQN algorithm is between 510 and 530, the DDQN algorithm converges between 530 and 550, and the improved D3QN algorithm converges between 530 and 590. The convergence curves indicate that the improved D3QN algorithm exhibits better network optimization effects and has larger fluctuation amplitudes, making it less likely to fall into local optima.

To further verify the impact of real-time electricity price incentives on energy scheduling within a virtual power plant, Figure 8 presents a comparison of the VPP control center’s profits over seven days under three different algorithms.

Under the DDQN algorithm, the total profit of the virtual power plant control over seven days is CNY 350,453.7. When using the Dueling DQN algorithm, the total profit is CNY 384,555.2. With the improved D3QN algorithm, the total profit over seven days for the VPP control center is CNY 391,069.6, representing an increase of 11.6% compared to the DDQN algorithm and 1.7% compared to the Dueling DQN algorithm.

5.3. Analyzing the Results of Reinforcement Learning Optimization Scheduling

The improved D3QN algorithm was used for training the VPP control center and gas turbine agents. Using historical data, the agents were trained offline, with Figure 9 showing the price changes over 5000 iterations. The colormap used ranges from blue to yellow, where blue represents lower price values and yellow represents higher price values. This visual aid allows for an easier comparison of how prices fluctuate over time and across different simulations. Figure 10 presents the scheduling results obtained after convergence.

From Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10, it can be seen that in the first half of the day, the wind farm generates a significant amount of electricity. During the 00:00–03:00 time period, when the electricity selling price to the main grid is low, the energy storage battery prioritizes charging. During the 12:00–18:00 period, when the electricity selling price is higher, the energy storage battery discharges in a timely manner. Additionally, the VPP purchases any shortfall from the main grid to meet the internal power load demands of the virtual power plant.

During periods when distributed generation in the virtual power plant management system is high, sufficient electricity is allocated to temperature-controlled loads to maintain a comfortable temperature for the rest of the day without needing additional power. Using the improved D3QN algorithm for virtual power plant energy scheduling can avoid purchasing large amounts of electricity during high-demand and low-generation periods. This approach achieves peak shaving and valley filling, enhancing the overall daytime revenue of the VPP control center. Consequently, it maximizes the economic dispatch benefits of the internal game units within the virtual power plant.

6. Conclusions

This paper successfully proposes an improved D3QN algorithm for optimizing the economic dispatch within a virtual power plant. First, a comprehensive VPP economic dispatch game model was established, encompassing various energy resources such as energy storage, micro gas turbines, wind power, and photovoltaics. By integrating a greedy selection mechanism and a noisy network into the D3QN algorithm, this study not only enhances the model’s capability to handle complex and dynamic market environments but also improves the real-time responsiveness and robustness of the decision-making process.

Simulation experiments based on real data demonstrated that the improved D3QN algorithm significantly outperforms traditional DDQN and Dueling DQN algorithms. It shows better adaptability in handling the challenges of continuous decision-making environments, effectively reducing operational costs while maximizing the satisfaction rate of power load demands and the profitability of power transactions with the main grid. Compared to the DDQN and Dueling DQN algorithms, the improved D3QN algorithm exhibited profit increases ranging from 11.6% to 1.7% across multiple cycles of experiments, thereby confirming its economic advantages.

Furthermore, the primary objective of this study is to investigate the optimal scheduling problem within a virtual power plant and to construct a model for the game between the virtual power plant and its internal units. Against this backdrop, the research contribution lies in the development of an internal game model for the virtual power plant and the application of an enhanced D3QN algorithm integrated with real-time electricity prices to solve the model. Ultimately, the case study demonstrates the economic feasibility of the model and the effectiveness of the algorithm.

The study also reveals the potential of demand-side management achieved through intelligent scheduling strategies, further validating the application value of deep reinforcement learning technologies in the field of energy system optimization. Future work will explore the integration of more real-time datasets and algorithm adjustments to further enhance the operational efficiency and responsiveness of the VPP.