Multiple DC Modulation Coordination Strategies Based on Transient Energy Function and Deep Deterministic Policy Gradient Algorithm

Abstract

Ultra-low-frequency oscillation (ULFO) is a new problem of frequency stability in asynchronous network back-end power networks. The negative damping provided by hydropower units is the direct cause of ultra-low-frequency oscillation, and DC modulation is an effective means of suppressing system frequency oscillation. Based on the transient energy function and the limitation of DC modulation, this paper analyzes the conditions of suppressing system oscillation by DC modulation from the perspective of energy. The weight of unit energy participation is defined, and the calculation formula of transient oscillation energy in a certain oscillation mode is derived. DC modulation sensitivity is then used to create a DC modulation sequence table based on the Prony identification approach. Ultimately, the DC involved in the modulation is identified using the DC modulation sequence table and the transient oscillation energy, and the Deep Deterministic Policy Gradient (DDPG) algorithm is used to optimize the chosen DC modulation parameters.

1. Introduction

After asynchronous networking, the problem of low frequency oscillation caused by weak damping of the AC system in weakly interconnected large systems has been improved to a certain extent, but ultra-low-frequency oscillation occurs in non-synchronous hydropower high-proportion transmission systems. It has been shown that the ultra-low frequency oscillation of asynchronous network transmitter systems is mainly due to the water hammer effect of hydropower units. As an additional control of traditional direct current, DC modulation has been used to suppress ultra-low frequency oscillation. At present, there have been many studies on the use of DC additional control to suppress low-frequency oscillations. Based on the analysis of the action mechanism of governor and FLC, a decoupling coordination design method of governor and FLC was proposed in the literature [1]. The literature [2] proposed and used FLC fast modulation power to jointly improve the damping of oscillation mode of HVDC island system to suppress ultra-low-frequency oscillation and suggested that the FLC dead zone should be smaller than the governor dead zone to enhance the FLC effect.

However, with the introduction of voltage source converter based high voltage direct current transmission (VSC-HVDC) technology based on a modular multilevel converter (MMC), a line-commutated converter (LCC) combined with MMC forms a new direct current transmission topology [3,4,5,6]. With the increasing scale of power grid interconnection, inter-regional low-frequency oscillation caused by system interconnection has gradually become one of the bottlenecks that trouble the safe and stable operation of power grid and limit the transmission capacity of power grid [7,8,9,10]. By controlling the active power output of HVDC system, the oscillation can be quickly suppressed so as to improve the dynamic stability of the AC system and increase the safe transmission capacity. The literature [11,12] used emergency power support and coordinated power control to suppress low-frequency oscillation of AC power grid for a conventional HVDC system. Compared with the conventional HVDC system, VSC-HVDC technology has technical advantages such as self-phase change capability, active power and reactive power can be decoupled and controlled, and no phase change failure will occur. In recent years, it has been widely developed and applied in the world, and China is also vigorously building VSC-HVDC projects. Because VSC-HVDC technology has large degrees of control freedom and switching between operation modes of the converter station, it can further suppress the AC low-frequency oscillation in theory. For an AC–DC hybrid power grid with VSC-HVDC system, an additional controller is designed to adjust the active and reactive power of the flexible direct converter, which can effectively suppress interregional low-frequency oscillation and improve the dynamic performance of the system [13,14,15,16,17]. Based on a wide-area measurement system, the literature [13] determined the optimal value of active and reactive power of DC system through model predictive control, and it adopted fixed frequency prediction and adjustment power instructions to suppress low-frequency oscillation. In the literature [14], a design method of a centralized coordination controller suitable for multiple HVDC lines was proposed by adopting a centralized control scheme, which could improve the damping of small power systems considerably. In order to avoid the risk of losing centralized control communication, literature [15] applied the homologous ethics theory to the controller design of VSC-HVDC converter, and it obtains a robust distributed controller. In literature [16], the nonlinear system is transformed into a linearized system through input–output accurate feedback linearization. Then, the coordinated control structure of multiple HVDC lines is designed through the design method of a linear system controller to improve the stability of the system. The literature [17] proposed a robust fixed-order frequency domain controller design method to maximize the tracking performance with specified stability margin in multiple operating points. In order to suppress ultra-low frequency oscillation, the most effective strategy is governor parameter optimization to improve the system damping within the ultra-low-frequency band. The literature [18] put forward the design principle of the damping control index of ULFO. These studies can provide references for the suppression of ultra-low frequency oscillation. For low-frequency oscillation suppression, the DC modulation power required is small, and it is generally not necessary to have multiple DC simultaneous modulation. However, for ultra-low-frequency oscillation, because of the different oscillation energy, it may require multiple DC modulation at the same time. Methods of using two kinds of DC to suppress the ultra-low-frequency oscillation of the system have become a new problem, but there is some research investigating this.

In this paper, a multi-DC modulation coordination strategy for suppressing ultra-low frequency oscillations is proposed. Firstly, the mechanism of suppressing ultra-low frequency oscillation in hybrid DC modulation system is analyzed from the perspective of energy, and the calculation method of system energy in practical engineering is given. Finally, a deep deterministic policy gradient (DDPG) algorithm is used to optimize the selected DC modulation parameters.

2. Ultra-Low-Frequency Oscillation Suppression Based on Transient Energy Function

2.1. Mechanism of Hybrid DC Modulation to Suppress Ultra-Low-Frequency Oscillation Based on Transient Energy Function

There are n units in the whole system. Each unit uses the second-order classical model and takes damping into account, which is represented by the center of inertia (COI). COI is defined as follows [19]:

$$\{\begin{array}{l}{T}_{\mathrm{JT}}={\displaystyle {\displaystyle \sum _{i=1}^n}}{T}_{\mathrm{J}i}\\ {\delta }_{\mathrm{COI}}=\frac{1}{T_{\mathrm{JT}}}{\displaystyle \sum _{i=1}^n}{T}_{\mathrm{J}i}{\delta }_i\\ {\omega }_{\mathrm{COI}}=\frac{1}{T_{JT}}{\displaystyle \sum _{i=1}^n}{T}_{\mathrm{J}i}{\omega }_i\\ P_{\mathrm{COI}}={\displaystyle \sum _{i=1}^n}(P_{\mathrm{m}i}-P_{\mathrm{e}i})\end{array},$$

(1)

where $T_{\mathrm{J}i}$, ${\delta }_i$, ${\omega }_i$, $P_{\mathrm{m}i}$, and $P_{\mathrm{e}i}$ are the inertia time constant, power angle, angular velocity, mechanical power, and electromagnetic power of the rotor of unit $i$ in synchronous coordinates, respectively. $T_{JT}$ is the total inertia time constant of rotors of all units. $P_{\mathrm{COI}}$ is the acceleration power of COI. ${\delta }_{\mathrm{COI}}$ and ${\omega }_{\mathrm{COI}}$ are the equivalent rotor angle and equivalent rotor angular velocity of COI, respectively.

When DC modulation is not considered, the motion equation of each unit in COI coordinates is:

$$\{\begin{array}{c}d{\theta }_i/dt={\tilde{\omega }}_i\\ T_{\mathrm{ji}}\frac{d{\tilde{\omega }}_i}{dt}=P_{\mathrm{m}i}-P_{\mathrm{e}i}-\frac{T_{Ji}}{T_{JT}}{P}_{\mathrm{COI}}-D_i({\tilde{\omega }}_i+{\omega }_{\mathrm{COI}})+\\ \hspace{1em}\hspace{1em}\hspace{1em}\frac{T_{Ji}}{T_{JT}}{\displaystyle \sum _{j=1}^n}{D}_j({\tilde{\omega }}_j+{\omega }_{\mathrm{COI}})\end{array}$$

(2)

where ${\theta }_i={\delta }_i-{\delta }_{\mathrm{COI}}$ and ${\tilde{\omega }}_i={\omega }_i-{\omega }_{\mathrm{COI}}$ are the rotor angle and rotor angular velocity of unit $i$ in COI coordinates, respectively.

The oscillating kinetic energy $V_{Ki}$, oscillating potential energy $V_{Pi}$ and damping energy $V_{Di}$ of unit i are defined as follows:

$$\{\begin{array}{l}{V}_i(t)=V_{Ki}(t)+V_{Pi}(t)+V_{Di}(t)\\ V_{Ki}(t)=T_{Ji}{\widehat{\omega }}_i^2(t)/2\\ V_{Pi}(t)={\displaystyle {\int }_{{\theta }_i^s}^{{\theta }_i(t)}{P}_{mi}-P_{ei}-\frac{T_{Ji}}{T_{JT}}{p}_{COI}\mathrm{d}{\theta }_i}\\ V_{Di}(t)={\displaystyle {\int }_{{\theta }_i^s}^{{\theta }_i(t)}-D_i{\omega }_i+\frac{T_{Ji}}{T_{JT}}{\displaystyle \sum _{j=1}^n}{D}_j({\tilde{\omega }}_j+{\omega }_{COI})\mathrm{d}{\theta }_i}\end{array},$$

(3)

where ${\theta }_i(t)$ and ${\theta }_i^s$ are rotor angle and equilibrium rotor angle under COI coordinates of the system at time t respectively.

After the system is disturbed, its transient energy is:

$$V(T)={\displaystyle {\displaystyle \sum _{i=1}^n}{V}_i}(t)={\displaystyle {\displaystyle \sum _{i=1}^n}[V_{\mathrm{K}i}(t)+V_{\mathrm{P}i}(t)+V_{\mathrm{D}i}(t)]}.$$

(4)

Now assume that there is only one dominant oscillation mode ${\lambda }_k={\sigma }_k+j{\chi }_k$ in the system, After the system is disturbed, according to the difference of ${\sigma }_k$, the system may appear in the first oscillation period [$t_s$,$t_{s+k}$] (where $\tilde{\omega }(t_s)=\tilde{\omega }(t_{s+k})=0$, $t_{s+k}=t_s+2\pi /{\chi }_k$) in the following three oscillation cases:

• ${\sigma }_k$ < 0; because of the damping energy, the transient energy of the system decreases with time $\mathsf{\Delta }{V}_{t_k}=V(t_{s+k})-V(t_s)<0$ ($\mathsf{\Delta }{V}_{t_k}$ is the change in the system energy during the first oscillation period), and the system is stable in the first oscillation period.
• ${\sigma }_k$ = 0; because the damping energy of the system $V_D$ ≈ 0, in the process of oscillation and oscillation, the kinetic energy and potential energy components increase and decrease, and the transient energy of the system basically remains unchanged, and the system is stable. And $V(t)$ ≈ C (C is constant), so $\mathsf{\Delta }{V}_{t_k}=V(t_{s+k})-V(t_s)=0$.
• ${\sigma }_k$ > 0; because of the damping energy $V_D$, the transient energy of the system increases with time, and the system becomes unstable. At the end of the first oscillation period, the transient energy change of the system is:

  $$\begin{array}{l}\mathsf{\Delta }{V}_{t_k}=V(t_{s+k})-V(t_s)={\displaystyle {\displaystyle \sum _{i=1}^n}[V_{\mathrm{P}i}(t_{s+k})+V_{\mathrm{D}i}(t_{s+k})]}-\\ {\displaystyle {\displaystyle \sum _{i=1}^n}[V_{\mathrm{P}i}(t_{s+k})+V_{\mathrm{D}i}(t_{s+k})]}={\displaystyle {\displaystyle \sum _{i=1}^n}{\displaystyle {\int }_{{\theta }_i^s}^{{\theta }_i(t)}(P_{mi}-P_{ei}-}}\\ \frac{T_{Ji}}{T_{JT}}{p}_{COI})-D_j({\tilde{\omega }}_j+{\omega }_{COI})+\frac{T_{Ji}}{T_{JT}}{\displaystyle \sum _{j=1}^n}{D}_j({\tilde{\omega }}_j+{\omega }_{COI})\mathrm{d}{\theta }_i\end{array}.$$

  (5)

In order to realize the suppression effect of DC modulation on oscillation, when ${\sigma }_k$ > 0, the first oscillation period is divided into four stages. Stage [$t_s$,$t_1$), $\tilde{\omega }(t)$ decreases from 0 to ${\tilde{\omega }}_{\min }$; stage [$t_1$,$t_2$), $\tilde{\omega }(t)$ increases from ${\tilde{\omega }}_{\min }$ to 0; stage [$t_2$,$t_3$), $\tilde{\omega }(t)$ increases from 0 to ${\tilde{\omega }}_{\max }$; stage [$t_3$,$t_{s+k}$], decreases from ${\tilde{\omega }}_{\max }$ to 0. It is assumed that l DC in the system can control the ultra-low-frequency oscillation mode, and when p (0 < p ≤ l) DC participates in DC modulation, the system oscillation can be suppressed. The maximum percentage of each DC that can participate in power modulation is $\epsilon$, where the maximum controllable DC power of article $\tau$ is $\mathrm{\Delta }{P}_{dc\tau }=\epsilon P_{dc{\tau }^{*}}$, and where $P_{dc{\tau }^{*}}$ is the rated active power of article t. Within [$t_s$,$t_2$), the DC of the system can reduce the transmission power so that the transient energy change of the system can be reduced.

In [$t_2$,$t_{s+k}$], the DC of the system can reduce the transient energy change of the system by increasing the transmission power. Therefore, the system transient energy reduction in the first oscillation period due to DC modulation is as follows:

$$\mathsf{\Delta }{V}_{\mathrm{dc}{t}_k}={\displaystyle \sum _{\tau =1}^p}({\displaystyle {\int }_{{\theta }_i(t_s)}^{{\theta }_i(t_2)}\mathsf{\Delta }{P}_{\mathrm{dc}\tau }\mathrm{d}{\theta }_i}+{\displaystyle {\int }_{{\theta }_i(t_2)}^{{\theta }_i(t_{s+k})}\mathsf{\Delta }{P}_{\mathrm{dc}\tau }\mathrm{d}{\theta }_i}).$$

(6)

When ${\sigma }_k$ ≤ 0, the system can still maintain stability after disturbance, and DC modulation can make the system oscillation decay as soon as possible and shorten the time for the system to recover stability. When ${\sigma }_k$ > 0, based on the principle of transient energy decline, in the first oscillation period, the energy change $\mathsf{\Delta }{V}_{\mathrm{dc}{t}_k}$ of the system p direct current used to suppress the oscillation must be at least greater than the change of the system transient energy—that is, $\mathsf{\Delta }{V}_{\mathrm{dc}{t}_k}>\mathsf{\Delta }{V}_{t_k}$.

2.2. Calculation of System Transient Energy

In a system with n units, there are theoretically n − 1 oscillation modes, and each unit may participate in multiple oscillation modes at the same time. Now, the left eigenvector $\psi$ and the right eigenvector $\varphi$ of the system are introduced. According to the definition of the left eigenvector of the system, ${\psi }_{{\lambda }_k}$ can be used to measure the influence weight of each unit on each unit of oscillation mode ${\lambda }_k$.

$$\begin{array}{c}\psi =[{\psi }_1^{\mathrm{T}}\cdots {\psi }_{{\lambda }_k}^{\mathrm{T}}\cdots {\psi }_n^{\mathrm{T}}]\\ \varphi =[{\varphi }_1\cdots {\varphi }_{{\lambda }_k}\cdots {\varphi }_n]\end{array},$$

(7)

The rotor motion equation is linearized near the stable operating point,

$$\{\begin{array}{l}{T}_{\mathrm{J}i}\mathsf{\Delta }{\dot{\omega }}_i=-\mathsf{\Delta }P\\ \mathsf{\Delta }{P}_i=\mathsf{\Delta }{P}_{\mathrm{m}i}-\mathsf{\Delta }{P}_{\mathrm{e}i}-D_i\mathsf{\Delta }{\omega }_i\end{array},$$

(8)

where $\mathsf{\Delta }{P}_i$ is the acceleration power deviation.

For oscillation mode ${\lambda }_k$, the acceleration power deviation of unit i can be expressed in a generalized phasor form, namely:

$$\mathsf{\Delta }{\overrightarrow{P}}_{i,{\lambda }_k}=-T_{\mathrm{J}i}({\sigma }_k+\mathrm{j}{\chi }_k)\mathsf{\Delta }{\overrightarrow{\omega }}_{i,{\lambda }_k}.$$

(9)

According to literature [20], under oscillation mode, the ratio of acceleration power deviation of any two units a and b is as follows:

$$\frac{\mathsf{\Delta }{\overrightarrow{P}}_{\mathrm{a},{\lambda }_k}}{\mathsf{\Delta }{\overrightarrow{P}}_{\mathrm{b},{\lambda }_k}}=\frac{T_{\mathrm{Ja}}\mathsf{\Delta }{\overrightarrow{\omega }}_{\mathrm{a},{\lambda }_k}}{T_{\mathrm{Jb}}\mathsf{\Delta }{\overrightarrow{\omega }}_{\mathrm{b},{\lambda }_k}}.$$

(10)

According to the free motion analysis theory of dynamic systems with zero input, for a certain mode, the ratio of corresponding terms of a and b units in the right characteristic phasor is equal to the ratio of corresponding terms of state variables, that is:

$$\frac{{\varphi }_{\mathrm{a},{\lambda }_k}}{{\varphi }_{\mathrm{b},{\lambda }_k}}=\frac{\mathsf{\Delta }{\overrightarrow{\omega }}_{\mathrm{a},{\lambda }_k}}{\mathsf{\Delta }{\overrightarrow{\omega }}_{\mathrm{b},{\lambda }_k}}=\frac{\mathsf{\Delta }{\overrightarrow{P}}_{\mathrm{a},{\lambda }_k}/T_{\mathrm{ja}}}{\mathsf{\Delta }{\overrightarrow{P}}_{\mathrm{b},{\lambda }_k}/T_{\mathrm{jb}}}.$$

(11)

In the actual system, the left characteristic phasor and the right characteristic phasor have the following approximate relationship [21].

$$\begin{array}{c}{\psi }_{{\lambda }_k}\approx [diag(T_J/{\omega }_s)]{\varphi }_{{\lambda }_k}\\ diag(T_{\mathrm{j}}/{\omega }_s)=\frac{1}{{\omega }_s}\left[\begin{array}{cccc}{T}_{\mathrm{Jl}}& 0& \cdots & 0\\ 0& T_{\mathrm{J}}& \cdots & 0\\ ⋮& ⋮& & ⋮\\ 0& 0& \cdots & T_{\mathrm{Jn}}\end{array}\right]\end{array},$$

(12)

where ${\omega }_s$ is the steady-state value of the generator speed.

Then the ratio of corresponding terms of units a and b in the left characteristic phasor is:

$$\frac{{\varphi }_{\mathrm{a},{\lambda }_k}}{{\varphi }_{\mathrm{b},{\lambda }_k}}=\frac{\mathsf{\Delta }{\overrightarrow{\omega }}_{\mathrm{a},{\lambda }_k}}{\mathsf{\Delta }{\overrightarrow{\omega }}_{\mathrm{b},{\lambda }_k}}=\frac{\mathsf{\Delta }{\overrightarrow{P}}_{\mathrm{a},{\lambda }_k}/T_{\mathrm{ja}}}{\mathsf{\Delta }{\overrightarrow{P}}_{\mathrm{b},{\lambda }_k}/T_{\mathrm{jb}}}.$$

(13)

Based on the law of conservation of energy, this paper defines the integration of power to time as an energy calculation formula, and it obtains the acceleration power $p_i$ of each unit by time domain simulation. The maximum accelerated power deviation of the unit under the leading oscillation mode is selected as the reference value, and the ratio of the accelerated power deviation amplitude of the unit i to the reference value is obtained. Then, the ratio is defined as the unit energy participation weight ${\eta }_{{\lambda }_{k}i}$—that is:

$${\eta }_{{\lambda }_{k}i}=\frac{\left|\mathsf{\Delta }{P}_{i,{\lambda }_k}\right|}{\max \left\{\left|\mathsf{\Delta }{P}_{i,{\lambda }_k}\right|,(i=1,2,\cdots n)\right\}}.$$

(14)

By weighting the energy provided by each unit participating in the oscillation mode, the transient energy change of the system under the dominant oscillation mode ${\lambda }_k$ is:

$$\mathrm{\Delta }{V}_{t_k}={\displaystyle {\displaystyle \sum _{i=1}^n}\mathrm{\Delta }{V}_{t_{k}i}={\displaystyle {\displaystyle \sum _{i=1}^n}{\eta }_{{\lambda }_{k}i}{\displaystyle {\int }_{t_s}^{t_{s+k}}{p}_{i}dt}}},$$

(15)

$$\mathrm{\Delta }{V}_{dc}{t}_k={\displaystyle {\displaystyle \sum _{m=1}^p}\left({\displaystyle {\int }_{t_s}^{t_2}\mathrm{\Delta }{P}_{dcm}dt}+{\displaystyle {\int }_{t_2}^{t_{s+k}}\mathrm{\Delta }{P}_{dcm}dt}\right)}.$$

(16)

When the system frequency oscillation deviation is within the dead range, the DC additional control will not operate, and the DC modulation automatically adjusts with the frequency deviation and does not always stay at the maximum. Therefore, the amount of energy change used by DC to suppress oscillations needs to retain a certain margin—that is:

$$\mu \mathsf{\Delta }{V}_{\mathrm{dc}{t}_k}\ge \mathsf{\Delta }{V}_{t_k}, 0<\mu <1.$$

(17)

3. Prony Algorithm Identification and DC Sensitive Point Sorting

3.1. Prony Algorithm Identification

The Prony method can be used for both on-line and off-line monitoring of ULF oscillation modes. The Prony algorithm utilizes real-time data from the phase measurement unit (PMU) for online monitoring and historical data for offline monitoring. The data measured by the PMU can be categorized into time-varying, ambient, and sounding data. The Prony algorithm employs the time-varying data, which is the more densely populated data, for the estimation of various modes. Time-varying data $y(i)$ can be expressed as:

$$y(j)={\displaystyle {\displaystyle \sum _{i=1}^{n_{\lambda }}}{c}_i{\mathrm{e}}^{j\lambda (i)\mathsf{\Delta }t}}\hspace{1em}j=0,1,\cdots ,N-1,$$

(18)

where, $n_{\lambda }$ is the total characteristic root number; $c_i$ is the amplitude of the $i$ mode; $\lambda (i)$ is the $i$th eigenvalue of the system; $\mathrm{\Delta }t$ is the sampling interval; $N$ is the total data points.

The steps of Prony algorithm for pattern recognition of power system are as follows [7]:

Step 1. Linear prediction model (LPM) is established. The nth-order LPM is composed of the predicted value $y(k)$ and the first $n$ exact value ($y(k-1),y(k-2),\cdots ,y(k-n)$) and has: $y(k)=a_{1}y(k-1)+a_{2}y(k-2)+\cdots +a_{n}y(k-n)$, where $a_i$ is the coefficient of the characteristic equation.

Let $k=n,n+1,\cdots ,N-1$. There is:

$$\begin{array}{c}\left[\begin{array}{c}y(n)\\ y(n+1)\\ ⋮\\ y(N-1)\end{array}\right]=\\ \left[\begin{array}{cccc}y(n-1)& y(n-2)& \cdots & y(0)\\ y(n)& y(n-1)& \cdots & y(1)\\ ⋮& ⋮& ⋮& ⋮\\ y(N-2)& y(N-3)& \cdots & y(N-n-1)\end{array}\right]\\ \left[\begin{array}{c}a\left(1\right)\\ a\left(2\right)\\ ⋮\\ a\left(n\right)\end{array}\right]+\left[\begin{array}{c}\epsilon \left(n+1\right)\\ \epsilon \left(n+2\right)\\ ⋮\\ \epsilon \left(N\right)\end{array}\right]\end{array},$$

(19)

where $a(i)$ is the coefficient of the characteristic equation.

The coefficient $a_1$, $a_2$, …, $a_n$ of the characteristic equation is obtained by using the least square method.

Step 2. Discrete mode calculation. Variable $z$ is introduced as the variable of the characteristic polynomial, and combined with the coefficient $a$ of the characteristic equation obtained in step 1, the $n$-degree characteristic polynomial of variable $z$ is constructed as follows:

$$z^n-(a_1{z}^{n-1}+a_2{z}^{n-2}+\cdots +a_n{z}^0)=0.$$

(20)

Step 3. Residual term calculation in discrete mode. Residual terms of the discrete mode ($c_1$, $c_2$, …, $c_n$) can represent the amplitude and phase angle, and its calculation formula is as follows:

$$\left[\begin{array}{cccc}{z}_1^0& z_2^0& \cdots & z_n^0\\ z_1^1& z_2^1& \cdots & z_n^1\\ ⋮& ⋮& ⋮& ⋮\\ z_1^N& z_2^N& \cdots & z_n^N\end{array}\right]\cdot \left[\begin{array}{c}{c}_1\\ c_2\\ ⋮\\ c_n\end{array}\right]=\left[\begin{array}{c}y(1)\\ y(2)\\ ⋮\\ y(N)\end{array}\right].$$

(21)

Step 4. Discrete mode parameter calculation. The formula for calculating amplitude ($A_i$), phase angle ($P_i$), frequency ($f_i$), and oscillation factor (D) is as follows:

$$\{\begin{array}{l}{A}_i=\left|c_i\right|\\ P_i=\arctan \left|\frac{Im(c_i)}{\mathrm{Re}(c_i)}\right|\\ f_i=\frac{1}{2\pi \mathsf{\Delta }t}\arctan \left|\frac{Im(z_i)}{\mathrm{Re}(z_i)}\right|\\ {\sigma }_i=In\left|z_i\right|/\mathsf{\Delta }t\end{array}.$$

(22)

Step 5. Distinguish the real number mode and conjugate complex number mode, and calculate the energy of each mode.

For real numbers, there are:

$$E_i={\displaystyle {\displaystyle \sum _{j=1}^N}{[real(c_i{z}_i^j)]}^2}.$$

(23)

For conjugate complex modes, there are:

$$E_i={\displaystyle {\displaystyle \sum _{j=1}^N}{\left[\begin{array}{c}2real(c_i{z}_i^j)\end{array}\right]}^2}.$$

(24)

Different modes have different energy levels. By calculating the energy of each mode and classifying it, the dominant mode of the power system can be identified. Mode energy is defined as $E_i={\displaystyle {\displaystyle \sum _{j=1}^N}{[real(c_i{z}_i^j)]}^2}$ when $c_i$ or $z_i$ mode value is large, $E_i$ is high, and this mode is the dominant mode.

3.2. DC Sensitive Points Sort

In order to sort the control sensitive points of l DC, the same disturbance is applied to each DC at the same time to stimulate the oscillation of the system, and the power deviation of l DC is taken as the input signal and the system frequency is taken as the output signal. The transient energy variation of the system’s dominant oscillation mode is obtained by the Prony identification method.

For a certain oscillation mode ${\lambda }_k$, the mode value of each element in the mode matrix represents the controllability of a certain input to the oscillation mode ${\lambda }_k$, so $E_i$ is defined as DC modulation sensitivity $\phi$, where the DC modulation sensitivity corresponding to article m is ${\phi }_m$ ($m=1,2,\cdots ,l$). Set the maximum value of $\phi$ to ${\phi }_{\max }$, which corresponds to the DC number 1, and the minimum value of $\phi$ to ${\phi }_{\min }$, which corresponds to the DC number l, and sort the DC from the largest to the smallest. A DC modulation sequence table is formed, as shown in

Table 1. DC modulation sequence table.

DC Modulation Sequence	1	…	m	…	l
name	1	…	m	…	l
DC modulation sensitivity	${\phi }_{\max }$	…	${\phi }_m$	…	${\phi }_{\min }$

, where the larger the $\phi$, the more sensitive it is to the oscillation mode ${\lambda }_k$ control.

4. Deep Deterministic Policy Gradient

Frequency limit controller (FLC) is a common DC modulation method in practical engineering [22,23], and its control block diagram is shown in Figure 1. When the system fails and the system frequency oscillation deviation exceeds the upper or lower limit of the FLC dead zone, the FLC is automatically activated and the DC power is automatically adjusted by a closed-loop PI controller.

A deep deterministic policy gradient algorithm (DDPG, Figure 2) is proposed to rectify the PI parameters of FLC. The Deep Deterministic Policy Gradient method uses the Actor–Critic algorithm as its basic framework, approximates the policy and action-value functions using a deep learning network, and uses the stochastic gradient method to train the parameters in both the policy network and value network models. A dual neural network model of real-time network and target network is used for both policy and value functions. At the same time, the algorithm borrows the experience playback mechanism from the DQN algorithm, in which the experience data generated by the interaction between the Actor and the environment will be stored in the experience pool. Then, a batch of data samples will be extracted for training so that the algorithm can be converged more easily.

At each time period t, the agent observes the current state $s_t$, performs an action $a_t$, receives a reward value $r_t$ from the environment E, and then transitions to the next state $s_{t+1}$ according to the state transition probability function.

For FLC, which adjusts the system frequency, the integral of the absolute value of the system frequency deviation and the product of time is selected as the reward function of DDPG optimization algorithm—that is, the optimization objective function is as follows:

$$\min J={\displaystyle {\int }_0^{T}t\left|\mathrm{\Delta }f\right|}dt.$$

(25)

The state space $s_t$ is:

$$s_t=(t,K_P,K_I,\mathrm{\Delta }f,\mathrm{\Delta }P),$$

(26)

the state space includes the current period t, PI parameters $K_P$, and $K_I$ of DC FLC. $\mathrm{\Delta }f$ is the frequency deviation, and $\mathrm{\Delta }P$ is the active power deviation.

Action space $a_t$ is:

$$a_t=(\mathrm{\Delta }{K}_P,\mathrm{\Delta }{K}_I).$$

(27)

The reward function $r_t$ is:

$$r_t=-{\displaystyle {\int }_0^{T}t\left|\mathrm{\Delta }f\right|}dt.$$

(28)

The first part of the reward function is chosen as the inverse of the integral of the absolute value of the system frequency deviation and the product of time. The greater the reward the agent receives, the smaller the integral of the absolute value of the system frequency deviation and the time product.

The loss function is calculated according to the following formula, where $y_t$ is the target $Q$ value.

$$\{\begin{array}{l}{y}_t=r_t+\gamma Q^{\prime }(s_{t+1},{\pi }^{\prime }(s_{t+1}|{\theta }^{{\pi }^{\prime }})|{\theta }^{Q^{\prime }})\\ L({\theta }^Q)=E{({\gamma }_t-Q(s_t,a_t|{\theta }^Q))}^2\end{array}.$$

(29)

According to the gradient update rule, the current value network is updated by calculating the gradient of the loss function [24]:

$$\{\begin{array}{c}{\theta }_k^Q={\theta }_{k-1}^Q-{\mu }_Q{𝛻}_{{\theta }^Q}L({\theta }_{k-1}^Q)\\ {𝛻}_{{\theta }^Q}L({\theta }_{k-1}^Q)=E(2(y_t-Q(s,a|{\theta }_{k-1}^Q){|}_{s=s_t,a=a_t})\\ \hspace{1em}\hspace{1em}\hspace{1em} 𝛻Q_{{\theta }^Q}(s,a|{\theta }_{k-1}^Q){|}_{s=s_t,a=a_t})\end{array}$$

(30)

The policy network uses a function of the value network output as a loss function. By finding the strategy gradient of the function, the updated formula is obtained [25]:

$$\{\begin{array}{l}{\theta }_k^{\pi }={\theta }_{k-1}^{\pi }-{\mu }_{\pi }{𝛻}_{{\theta }^x}L({\theta }_{k-1}^{\pi })\\ {𝛻}_{{\theta }_{k-1}^{\pi }}\pi ={𝛻}_{a}Q(s,a\mid {\theta }_{k-1}^{\pi }){\mid }_{s=s_t,a_t=\pi (s_t)}{𝛻}_{{\theta }^x}\pi (s\mid {\theta }_{k-1}^{\pi }){\mid }_{s=s_t})\end{array}.$$

(31)

The target network adopts the soft update mode as shown in the following formula.

$$\{\begin{array}{c}{\theta }_k^{Q^{\prime }}=\tau {\theta }_{k-1}^Q+(1-\tau ){\theta }_{k-1}^{Q^{\prime }}\\ {\theta }_k^{{\pi }^{\prime }}=\tau {\theta }_{k-1}^{\pi }+(1-\tau ){\theta }_{k-1}^{{\pi }^{\prime }}\end{array}.$$

(32)

When learning the strategy, m sample data points are randomly sampled from the experience pool, and the algorithm is optimized by constantly updating the gradient value of the network. The optimization objective function constructed in this paper aims to minimize the integral of the absolute value of the system frequency deviation and the product of time.

5. Multiple DC Modulation Coordination Strategies

The coordinated optimization strategy of multi-DC modulation is shown in Figure 3, and its specific steps are as follows:

• For the leading oscillation mode ${\lambda }_k$, the maximum acceleration power deviation of each unit is taken as the reference value, and the energy participation weight ${\eta }_{{\lambda }_k}$ of each unit in oscillation mode ${\lambda }_k$ is calculated;
• Calculate the transient energy change $\mathrm{\Delta }{V}_{t_k}$ of oscillation mode ${\lambda }_k$;
• Apply the same power disturbance to the DC of the system l to excite the oscillation of the system;
• Prony algorithm is used to identify the system;
• Determine the modulation sensitivity ${\phi }_m$ of each DC, and sort l DC to form a DC modulation sequence table;
• Calculate $\mathrm{\Delta }{V}_{dc{t}_k}$ and determine $p$ DC that participates in DC modulation;
• Using the DDPG algorithm, the FLC parameters of $p$ DC were optimized and the optimization results were output.

6. Example Analysis

The effectiveness of multi-DC modulation coordination optimization strategy is verified by taking the power grid of a region after back-to-back DC projects are put into operation.

After back-to-back DC projects are put into operation in a certain region, all the connections between the power grid and the external power grid will be realized through the DC network, as shown in Figure 4. The dashed line box in the figure represents the synchronous power grid. A hybrid AC–DC network containing a high voltage direct current transmission (HVDC) system is taken as the main research object. Regardless of the influence of frequency change on the DC transmission power, three ultra-high-voltage direct current (UHVDC) networks are selected to develop a coordinated optimization scheme. Constant current control is adopted on the rectifier side, constant voltage control is adopted on the inverter side, and FLC control is added.

When the system’s critical AC communication channel AC2 experiences an N−1 fault, the primary oscillation modes of the power grid under operating modes are as shown in

Table 2. Critical oscillation modes of the power grid in summer peak and summer base cases.

Mode of Operation	Real Part	Imaginary Part	Frequency/Hz	Amplitude
summer peak	0.00504	0.44965	0.073	0.269
summer base	0.00093	0.38400	0.060	0.443

. Assume that the maximum percentage ε of each DC that can participate in power modulation is 5%. When the system frequency changes, the DC modulation amount is automatically adjusted with the frequency deviation, FLC has an adjustment dead zone (FLC dead zone is set at 0.05Hz in this simulation example), a certain DC modulation capability is retained, and μ is set at 0.68. The multi-DC modulation coordination optimization strategy under two operating modes is simulated and verified, respectively.

6.1. Example 1—Summer Peak

In summer peak operation mode, in 0.073 Hz oscillation mode, the energy of some units of the power grid participates in weight ${\eta }_{{\lambda }_{k}i}$, and the transient energy change $\mathrm{\Delta }{V}_{t_{k}i}$ of the units after being converted according to ${\eta }_{{\lambda }_{k}i}$ is shown in

Table 3. ${\eta }_{{\lambda }_{k}i}$ and $\mathrm{\Delta }{V}_{t_{k}i}$ of some units in critical oscillation mode (summer peak case).

Unit i	${\mathit{\eta }}_{{\mathit{\lambda }}_{\mathit{k}}\mathit{i}}$	$\mathit{\Delta }{\mathit{V}}_{{\mathit{t}}_{\mathit{k}}\mathit{i}}$/(kW·h)	Unit i	${\mathit{\eta }}_{{\mathit{\lambda }}_{\mathit{k}}\mathit{i}}$	$\mathit{\Delta }{\mathit{V}}_{{\mathit{t}}_{\mathit{k}}\mathit{i}}$/(kW·h)
1	1	67.7	9	0.212	13.8
2	0.412	28.1	10	0.212	13.8
3	0.383	24.5	11	0.215	13.6
4	0.364	24.4	12	0.206	13.5
5	0.355	24.3	13	0.185	12.2
6	0.313	14.8	14	0.183	11.5
7	0.315	19.7	15	0.166	10.6
8	0.213	14.7

. It is calculated that the transient energy change of the system under this oscillation mode is $\mathrm{\Delta }{V}_{t_k}=523 \mathrm{kW}\cdot \mathrm{h}$.

A 2% power disturbance is applied to three DCS at the same time, and the DC modulation sensitivity φ_m is obtained by Prony algorithm identification and calculation. The DC modulation sequence is obtained by sequencing, as shown in

Table 4. DC modulation sequence table (summer peak case).

DC Modulation Sequence	1	2	3
Name	a	b	c
DC modulation sensitivity	0.649	0.632	0.317

. According to the calculation in Table 4, when p > 1, the energy change $\mathrm{\Delta }{V}_{\mathrm{dc}{t}_k}$ used by DC to suppress oscillation is 1255 kW·h, and it is determined that DC a participates in modulation.

Taking the power disturbance of a DC as an input, and the optimization model was built in MATLAB combined with DDPG algorithm. The learning rate $\alpha$ = 0.00001, discount factor $\gamma$ = 0.97, and the PI parameters of a DC FLC were iteratively obtained as K₁ = 1.5213, T₁ = 0.4205. When AC channel AC2 fails N − 1, the system frequency change curve is shown in Figure 5.

As can be seen from Figure 5a, compared with the system without DC modulation, when DC a participates in modulation, the system frequency oscillation is significantly suppressed, which verifies the effectiveness of the proposed strategy.

In order to further verify the correctness of analyzing the oscillation mechanism of DC modulation suppression system from the perspective of energy and the ordering of DC modulation sensitivity, the frequency variation curves of the system with three DCS participating in modulation were compared. As can be seen from Figure 5b, since the energy variation $\mathrm{\Delta }{V}_{\mathrm{dc}{t}_k}$ of the three DCS that can be used to suppress the oscillation is greater than the transient energy variation $\mathrm{\Delta }{V}_{t_k}$ of the system, the oscillation of the system can be suppressed when the three DCS are modulated separately. At the same time, compared with c DC modulation, the amplitude of system frequency oscillation is smaller when a DC participates in modulation. Compared with b DC, it takes the shortest time for the system frequency to stabilize when a DC participates in modulation. Therefore, choosing a DC as the DC participating in modulation is the most effective for suppressing system oscillation.

6.2. Example 2—Summer Base

In summer base operation mode, in 0.062 Hz oscillation mode, the energy of some units of the power grid participates in weight ${\eta }_{{\lambda }_{k}i}$, and the transient energy change $\mathrm{\Delta }{V}_{t_{k}i}$ of the units after being converted according to ${\eta }_{{\lambda }_{k}i}$ is shown in

Table 5. ${\eta }_{{\lambda }_{k}i}$ and $\mathrm{\Delta }{V}_{t_{k}i}$ of some units in critical oscillation mode (summer base case).

Unit i	${\mathit{\eta }}_{{\mathit{\lambda }}_{\mathit{k}}\mathit{i}}$	$\mathit{\Delta }{\mathit{V}}_{{\mathit{t}}_{\mathit{k}}\mathit{i}}$/(kW·h)	Unit i	${\mathit{\eta }}_{{\mathit{\lambda }}_{\mathit{k}}\mathit{i}}$	$\mathit{\Delta }{\mathit{V}}_{{\mathit{t}}_{\mathit{k}}\mathit{i}}$/(kW·h)
1	1	85.3	9	0.674	55.5
2	0.985	85.4	10	0.672	55.6
3	0.848	73.5	11	0.672	56.6
4	0.796	62.2	12	0.671	55.7
5	0.785	66.3	13	0.653	56.4
6	0.774	62.5	14	0.584	47.3
7	0.753	60.3	15	0.545	33.3
8	0.673	55.3

. It is calculated that the transient energy change of the system under this oscillation mode is $\mathrm{\Delta }{V}_{t_k}=1614 \mathrm{kW}\cdot \mathrm{h}$.

A 2% power disturbance is applied to three DCS at the same time, and the DC modulation sensitivity φ_m is obtained by Prony algorithm identification and calculation, and the DC modulation sequence is obtained by sequencing, as shown in

Table 6. DC modulation sequence table (summer base case).

DC Modulation Sequence	1	2	3
Name	a	c	b
DC modulation sensitivity	0.604	0.124	0.044

. According to the calculation in Table 6, when p = 2, the energy change $\mathrm{\Delta }{V}_{\mathrm{dc}{t}_k}$ used by DC to suppress oscillation is 2876 kW·h, and it is determined that DC a and DC c participate in modulation.

Taking the power disturbance of a and c DC as input, the PI parameters of a DC FLC can be iteratively obtained as K₁ = 1.0096, T₁ = 1.0584, and PI parameters of c DC FLC as K₁ = 1.1063, T₁ = 0.5437 by using the same steps as example 1. When the N − 1 fault occurs in AC2, the system frequency change curve is shown in Figure 6.

It can be seen from Figure 6a that compared to the system without DC modulation, when DC a and DC c participate in the modulation at the same time, the system frequency oscillation is significantly suppressed, which verifies the effectiveness of the proposed strategy.

In order to further verify the correctness of analyzing the oscillation mechanism of the DC modulation suppression system from the perspective of energy and the ordering of DC modulation sensitivity, three kinds of a DC participate in modulation alone, a and b DC participate in modulation simultaneously, and a and c DC participate in modulation simultaneously are simulated and compared. The frequency change curve of the system is shown in Figure 6b. As can be seen from the figure, when DC a participates in modulation alone, the system oscillation amplitude decreases but the system frequency still oscillates. As a and b DC participate in modulation at the same time, and a and c DC participate in modulation at the same time, the system frequency can be effectively suppressed, but the system frequency oscillation is most effective when a and c DC participate in modulation at the same time.

7. Conclusions

With the rapid development and wide application of HVDC transmission technology, the number of DC drops is gradually increasing in China, and the intensity and transmission scale are rare in the world. Based on the system transient energy and DC modulation sensitivity ranking, this paper proposes a multi-DC modulation coordination optimization strategy, which is verified by numerical simulation. The following conclusions are obtained:

(1) Using the transient energy function, the mechanism of DC modulation to suppress the frequency oscillation of the system can be analyzed from the point of view of energy, and the oscillation energy of the system can be calculated so as to determine the amount of modulation required to be involved by DC.

(2) For a multi-DC system, analyzing the dc modulation sensitivity based on the state-space description of the system, the multiple dc’s involved in the modulation can be ranked so that the dc with the optimal modulation effect can be selected.